Vehicle-to-infrastructure system and method with long wave infrared capabilities - Patents.com

JP2024502133A5Active Publication Date: 2024-11-21ADASKY LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023541083
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-30
Filing Date
2021-11-24
Publication Date
2024-11-21
Estimated Expiration
2041-11-24

AI Technical Summary

Technical Problem

Existing vehicle-to-infrastructure (V2I) systems face limitations in weather conditions and night visibility due to the performance issues of sensors like video cameras, radar, and LIDAR, which are affected by glare, snow reflections, and adverse weather, leading to reduced detection accuracy and image quality.

Method used

Incorporation of infrared (IR) cameras with non-uniformity correction (NUC) and shutter mechanisms, along with image processing algorithms, to enhance object detection and maintain field of view clarity under various weather conditions, using IR cameras that operate in the far-infrared spectrum and perform non-uniformity correction to improve image quality and reliability.

Benefits of technology

The IR camera system provides reliable object detection up to 250 meters without active illumination, maintaining image quality and accuracy during day and night, and in all weather conditions, enhancing safety and efficiency in vehicle communication systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A system and method for operating a vehicle-to-infrastructure (V2I) system, the method including receiving an image from an infrared (IR) camera, determining whether non-uniformity noise is present in the received image, performing a calibration if non-uniformity noise is present, performing a non-uniformity correction (NUC) on the IR if after performing the calibration, it is determined that there is residual non-uniformity noise and the IR camera does not detect a moving object approaching the IR camera, determining whether a field of view (FOV) of the IR camera is obstructed after performing the NUC, and cleaning the IR camera if it is determined that the FOV of the IR camera is obstructed.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Nonprovisional Application No. 17 / 138,054, filed December 30, 2020. The contents of the above related applications are incorporated herein by reference.

[0002] The present disclosure relates generally to vehicle communication systems, and more particularly to vehicle-to-infrastructure systems and methods that include infrared cameras. [Background technology]

[0003] Vehicle-to-Infrastructure (V2I) is a communications model that allows vehicles to share information with the infrastructure and components that make up the road system. The components may include traffic signals, lane markers, street lights, road signs, parking meters, building structures, etc. Data from sensors mounted on these components can be collected in real time and transmitted over a network to vehicles to deliver warnings about road conditions, traffic jams, potential obstacles, and parking possibilities. Other systems may also use both infrastructure and vehicle data to optimize traffic flow and improve vehicle fuel efficiency while promoting safety for both drivers and pedestrians on the roads.

[0004] Typically, sensors used in V2I systems include video cameras, radio detection and ranging (RADAR), ultrasonic sensors, and light detection and ranging (LIDAR), as well as other stationary sensors mounted at fixed locations with good visibility of traffic objects outside the moving vehicle. V2I systems may also add information that cannot be adequately captured by sensors mounted on the vehicle. For example, a V2I system may include an RGB camera mounted on a pole at an intersection that is adapted to capture and analyze video of vehicles and pedestrians approaching the intersection and transmit information to vehicles approaching from around the corner.

[0005] Because V2I systems are stationary and not located on the vehicle, and the costs of the systems can be shared by members of the public who use these systems rather than by individual vehicle owners, V2I systems can augment the analytical capabilities of sensors mounted on vehicles in V2V (Vehicle-to-Vehicle) systems by having sensors mounted at viewing angles and positions with increased sensor sensitivity, improved resolution, increased detection range, and wider field of view, without being limited to the size, position location, and cost constraints that apply to V2V systems installed on individual vehicles.

[0006] However, the various sensing modalities have limited fields of view and cannot perform in all weather conditions. The performance of these devices is particularly poor in bad weather or at night. For example, even under bright daylight and clear sky conditions, video cameras can be rendered useless by glare from sunlight and snow reflections. Also, at night, video cameras require additional active lighting to identify distant objects, which consumes a lot of electrical energy. Furthermore, video cameras may be useless under heavy rain or fog.

[0007] Radar can identify the relative speed of an object, but cannot identify the details of the object. Furthermore, ultrasonic sensors can only sense nearby objects with low resolution. Additionally, LIDAR may be able to create a three-dimensional image of the object and provide depth perception, but the range is still limited and detection may be obscured in rainy or foggy conditions.

[0008] Overall, the various systems described do not perform well in adverse weather conditions or at night.

[0009] Furthermore, when imagers of different modalities are fixed on an infrastructure for extended periods of time, such as in V2I applications, noise can accumulate in the images produced, reducing the quality of the images and making it difficult for on-board processors to distinguish between different objects detected over time. Summary of the Invention [Problem to be solved by the invention]

[0010] In view of the above discussion, there is a need for a V2I system that uses different modalities that overcomes the above mentioned shortcomings. [Means for solving the problem]

[0011] A summary of some exemplary embodiments of the present disclosure is provided below. This summary is provided for the convenience of the reader to provide a basic understanding of such embodiments, but does not completely define the breadth of the present disclosure. This summary is not a comprehensive overview of all contemplated embodiments, nor does it identify key or critical elements of all embodiments, nor delineate the scope of any or all aspects. Its sole purpose is to present some concepts of one or more embodiments in a simplified form as a prelude to the more detailed description that is presented later. For convenience, the terms "some embodiments" or "specific embodiments" may be used herein to refer to a single embodiment or multiple embodiments of the present disclosure.

[0012] Certain embodiments disclosed herein include a method for operating a vehicle-to-infrastructure (V2I) system, the method including receiving an image from an infrared (IR) camera, determining whether non-uniformity noise is present in the received image, performing a calibration if non-uniformity noise is present, performing a non-uniformity correction (NUC) on the IR if it is determined after performing the calibration that there is residual non-uniformity noise and the IR camera does not detect a moving object approaching the IR camera, determining whether a field of view (FOV) of the IR camera is obstructed after performing the NUC, and cleaning the IR camera if it is determined that the FOV of the IR camera is obstructed.

[0013] Certain embodiments disclosed herein also include a non-transitory computer readable medium having stored thereon instructions for causing a processing circuit to perform the following processes, the processes including receiving an image from an infrared (IR) camera, determining whether non-uniformity noise is present in the received image, and if it is determined that non-uniformity noise is present, performing a calibration, and if it is determined after performing the calibration that there is residual non-uniformity noise and that the IR camera has not detected a moving object approaching the IR camera, performing a non-uniformity correction (NUC) for the IR, and after performing the NUC, determining whether the field of view (FOV) of the IR camera is obstructed, and if it is determined that the FOV of the IR camera is obstructed, cleaning the IR camera.

[0014] Certain embodiments disclosed herein also include a vehicle-to-infrastructure system that includes a processing circuit and a memory that includes instructions that, when executed by the processing circuit, configure the system to receive an image from an infrared (IR) camera, determine whether non-uniformity noise is present in the received image, perform a calibration if non-uniformity noise is present, determine that there is residual non-uniformity noise after performing the calibration, perform a non-uniformity correction (NUC) on the IR if the IR camera does not detect a moving object approaching the IR camera, determine whether a field of view (FOV) of the IR camera is obstructed after performing the NUC, and clean the IR camera if the FOV of the IR camera is determined to be obstructed. [Brief description of the drawings]

[0015] The subject matter disclosed herein is particularly pointed out and distinctly claimed in the claims appended hereto. The foregoing and other objects, features, and advantages of the disclosed embodiments will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0016] [Figure 1] FIG. 1 is a schematic diagram of a vehicle-to-infrastructure (V2I) communication system using an infrared camera, according to an embodiment of the present specification.

[0017] [Diagram 2] FIG. 2 is an exploded view of an infrared camera used in a V2I communication system, according to an embodiment of the present specification.

[0018] [Diagram 3] FIG. 3 is a detailed system diagram of an infrared camera system used in a V2I communication system according to one embodiment of the present specification.

[0019] [Figure 4]FIG. 4 is a detailed system diagram of a (processing circuit / server) used in a V2I communication system according to one embodiment of the present specification.

[0020] [Diagram 5] FIG. 5 illustrates a pipeline of processing tasks performed by the IC of FIG. 2 according to one embodiment.

[0021] [Figure 6] FIG. 6 is a flow chart of a method for operating V2I communication according to an embodiment of the present disclosure.

[0022] [Figure 7] FIG. 7 is a schematic diagram of a V2I communication system according to an embodiment of the present specification. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] It is important to note that the embodiments disclosed herein are merely examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of this application do not necessarily limit any of the various claimed embodiments. Moreover, some statements may apply to some inventive features but not to others. In general, unless otherwise stated, singular elements may be in the plural and vice versa without loss of generality. In the drawings, like numbers refer to like parts throughout the several views.

[0024] Various disclosed embodiments include methods and systems for operating vehicle-to-infrastructure (V2I) systems using long-wave infrared cameras. The use of infrared cameras allows detection of moving objects such as vehicles and pedestrians from long distances (up to about 250 m) without the need for active lighting, which uses up large amounts of energy and creates optical noise for the sensors. Images produced by long-wave infrared cameras are also invariant to other light sources, haze, nighttime light, rain, snow, fog, or other factors that may reduce visibility of other types of imaging systems, such as light detection and ranging (LIDAR). The V2I methods and systems also include image processing that improves both object detection, tracking, and distance determination, which is optimized for cameras that are stationary and fixed on a pole. Additionally, a shutter mechanism can be used to close the camera's field of view and perform important camera maintenance when the infrared camera is not in use. Thus, algorithms can be developed to determine when the shutter can be closed.

[0025] 1 is a schematic diagram of a vehicle-to-infrastructure (V2I) communication system 100 using a far-infrared (FIR) device 110, according to an embodiment of the present disclosure. The V2I communication system 100 includes the FIR device 110, an infrastructure 120, a terminal 130, a network 140, and a display 152 located in the vehicle.

[0026] The FIR device 110 may be mounted on the infrastructure 120 or may be independently mounted on its own mount at a fixed location. The FIR device 110 is adapted to detect the far infrared spectrum of electromagnetic radiation at wavelengths between about 15 micrometers and 1 mm, which also corresponds to the range of 20 THz to 300 GHz.

[0027] The infrastructure 120 may be, but is not limited to, traffic cameras, road signs, street lights, buildings, bridges, toll booths, parking lots, charging stations, road markings, and other transportation equipment.

[0028] Network 140 may be, but is not limited to, a wireless, cellular or wired network, a local area network (LAN), a wide area network (WAN), a metro area network (MAN), the Internet, the World Wide Web (WWW), a similar network, and any combination thereof. Network 140 may also include a cellular network, e.g., LTE or 5G. Also connected to network 140 is server 142. Display 152 may be a screen on the vehicle.

[0029] During operation, FIR device 110 mounted on infrastructure 120 continuously captures images of pedestrians crossing the intersection and vehicles 140 approaching the intersection. In addition to images, whether by itself or in combination with other sensors, FIR device 110 may also capture other information related to both vehicles and pedestrians. FIR device 110 then forwards the image information over network 130 to server 142 for processing.

[0030] 2 is an example exploded view of an FIR device 110 used in a V2I communication system 100, according to embodiments herein. The FIR device 110 includes a central housing 200 that encloses a shutter mechanism (shutter) 210, or non-uniformity correction (NUC) mechanism, an FIR camera 220, and a mount 250. The mount 250 is fixed on a back cover 260 on a proximal end of the FIR device 110, and a front cover 270 encloses a distal end of the FIR device 110 opposite the proximal end.

[0031] The shutter 210 is for protecting the FIR camera 220 and includes a window 213 and a motor 215. The motor 215 moves the window 213 to cover the field of view (FOV) of the FIR camera 220 when performing camera maintenance or resource-intensive image processing. The window 213 may also cover the FIR camera 220 in a standby state if the FIR sensor 220 has not detected any moving objects, such as vehicles or pedestrians, for a certain period of time.

[0032] The shutter 210 may also be automatically activated to perform a NUC operation upon power up, or may be activated if image quality is too degraded to be corrected when the shutter 210 does not close and cover the FOV of the FIR camera 220. For example, if an image captured by the FIR camera 220 is detected to contain noise, the shutter 210 may be activated to perform a NUC operation to perform calibration.

[0033] That is, during NUC operation when calibration is being performed, the shutter 210 can be moved by the motor 215 to a "closed" position, where the FIR sensor 220 is blocked by the shutter 210, so that the NUC operation can be performed to take corrective action without external disturbances.

[0034] Alternatively, the shutter 210 may be manually operated in conjunction with the NUC operation.

[0035] The front cover 270 includes a lens 280. A de-icing window heater (heater) 217, which may include a wire or coil, is also present between the lens 280 and the window 213. The heater 217 prevents icing on the lens 280 and protects the components of the FIR camera 220. Also, in one embodiment, the lens 280 may be made from a low conductivity material that is transparent to electromagnetic waves in the IR range, so that a current may be applied directly to heat the lens without the need for a separate heater 217. For example, a material such as germanium may be used to form the lens 280. In some embodiments, the lens 280 may also have a diamond-like carbon (DLC) coating. Additionally, lenses using germanium and other semiconductors with similar electrical and thermal conductive properties may be doped to control the electrical conductivity of the material (e.g., N-type or P-type semiconductors).

[0036] The heater 217 is powered and controlled by the developer kit 134 via a heater relay 138 of the controller 132. The heater 217 may be powered by the heater relay 138 based on temperature sensing of the window 213. For example, the heater 217 may be triggered when the sensed temperature falls below about 5 C (41f), and once triggered, the heater 217 may stop heating when the temperature rises above about 14 C (57f).

[0037] 3 is a side view of an example block diagram of an FIR camera 220 implemented according to one embodiment. The FIR camera 220 includes an optical unit 310 and an FIR sensor 320 coupled to an integrated circuit (IC) 330. The output of the FIR camera 220 is a video stream of thermal images captured by the FIR sensor 320 and processed by the IC 330.

[0038] In one embodiment, two thermal video streams are output by the FIR camera 220. The first stream 304 may include a thermal image captured by the FIR camera 220 and processed by the IC 330 for image correction and enhancement. In an exemplary embodiment, the first thermal video stream 304 may be an 8-bit grayscale video stream. The first thermal video stream 304 is provided to the display 152. In a further embodiment, multiple thermal video streams are output by the FIR camera 220. The output streams may include results from images received from a non-FIR sensor (e.g., visible wavelength, CCD, or CMOS sensor).

[0039] The second stream 305 includes thermal images captured by the FIR sensor 320 and processed by the IC 330 for NUC correction. In an exemplary embodiment, the second thermal video stream 305 is a 14-bit data stream. The second stream 305 is provided to the server 142 for computer vision processing. In particular, the server 142 may be configured to execute various computer vision algorithms related to AVS as well as other autonomous systems and applications. The algorithms may include, but are not limited to, detection of objects, obstacles, or hazards on the road. For example, the processed thermal video stream output by the FIR camera 220 may be utilized to detect vehicles, pedestrians, animals, motorbikes, black-ice spots, trash, debris, potholes, gully covers, curbs, and the like. In one embodiment, the server 142 may be an on-board computer or a dedicated computer installed in the vehicle (not shown). In some exemplary embodiments, the computer may be a virtual or physical computing machine capable of operating within a cloud computing platform (not shown).

[0040] As discussed below, according to some embodiments, the processing performed by IC 330 is configured to improve the quality of the captured thermal image to enable accurate and rapid detection of objects, obstacles, and hazards on the road. The processing by IC 330 ensures passive detection of objects during daytime and nighttime lighting conditions, as well as in all weather conditions. To this end, IC 330 is configured to perform one or more image processing tasks. Such tasks include, but are not limited to, NUC, sunburn protection, bad pixel replacement, near-field correction, temporal noise removal, spatial noise removal, edge enhancement, automatic gain control, local contrast, enhancement, and brightness and polarity adjustment. In one embodiment, these tasks are performed in a pipelined manner where some or all tasks may be utilized to process an FIR image (frame), thus ensuring low latency processing.

[0041] Image processing that may be performed by IC 330 is shutterless correction of FIR images captured by FIR sensor 320. In one embodiment, FIR camera 220 may not include a shutter (or any moving parts that can be viewed as a shutter). Shutterless correction performed by IC 330 allows flat-field correction to be performed without a shutter. That is, shutterless correction allows for a uniform FIR image. However, in cases where shutterless image processing cannot provide correction of FIR images (e.g., in the presence of rapid ambient changes) and given a static placement of FIR camera 220, a shutter may be used as a backup to allow the system to perform more resource-intensive corrections. In this case, the shutterless correction method may lengthen the time between shutter calls by processing simple corrections while keeping the shutter open and allowing FIR camera 220 to continue recording images.

[0042] It should be appreciated that the shutterless FIR camera 220 is mechanically more reliable and can better meet the safety standards required for AVS systems. Thus, the time required to keep the shutter open during imaging and no need to close the shutter may provide optimal reliability and safety in a V2I setting if the captured image is less noisy and unobstructed, requiring minimal processing (or keeping the shutter closed may provide optimal reliability and safety in a V2I setting if the captured image is less noisy and unobstructed). However, shutterless correction methods may be more computationally demanding and require more powerful processors / accelerators, whereas shutter-based corrections are typically very basic, which can be performed on the fly by less sophisticated optical circuits, on-chip, FPGA, small controllers that may even be part of the detector circuitry.

[0043] In one embodiment, optical unit 310 includes one or more lens elements, each having a predetermined field of view (FOV). For example, the FOV of the lens elements may range from about 30 to 60 degrees. In one embodiment, the lens elements are chalcogenides. The lenses may be protected by a protective window (not shown in FIG. 3).

[0044] In one embodiment, the FIR camera 220 may include multiple optical units 310. Each optical unit 110 may be used independently or in conjunction with one or more other optical units 310. For example, a single FIR sensor 320 may be connected to two or more optical units 310. Additionally, in one embodiment, multiple FIR sensors 320 may be connected to a single IC 330.

[0045] In an exemplary embodiment, multiple optical units 310 and / or multiple FIR sensors 320 can be used in tandem to capture and create a resultant image with a higher final resolution than a single optical unit 310 or FIR sensor 320 could otherwise provide. As an example, two VGA (640 pixels by 480 pixels) sized FIR sensors 320 can be used to capture a single 1280 pixel by 480 pixel FIR image. The IC 330 may be configured to automatically stitch two adjacent images together and may apply all of the corrections discussed herein to the final composite image. Alternatively, each individual image may be corrected first and then stitched together. In further embodiments, the images are not stitched but are concatenated horizontally or vertically when output to a display or computer. In further embodiments, multiple optical units 310 or FIR sensors 320 may be used for stereo vision applications or to provide multiple viewing angles of a scene.

[0046] In yet another embodiment, multiple image streams are received and processed by the FIR camera 220. Such an embodiment may include creating an output stream that is a fusion of multiple video streams including different imaging modalities, e.g., fusion of FIR images with visible spectrum images, fusion of FIR images with different focal lengths, etc. Any number of input streams may be received and processed to generate a fused output stream.

[0047] The FIR sensor 320 can be an uncooled FIR sensor. That is, the sensor 320 operates in the FIR spectrum from 7.0 to 14.0 μm wavelengths. In one example, the FIR sensor 220 is coupled to the IC 230 via a first bus 301 and inputs the captured FIR image and metadata. In one embodiment, a junction temperature sensor (temperature sensor) 340 is included within the housing of the FIR sensor 320 and measures the junction temperature therein.

[0048] A control bus 302 is also connected between the FIR sensor 320 and the IC 330. On the bus 302, signals related to the status of the sensor 320, clocks, synchronization, and other digital control signals are transferred. In one embodiment, the bus 302 may carry an analog signal indicating, for example, the current ambient temperature at the FIR sensor 320. Alternatively, the analog signal may not be part of the control bus 302.

[0049] IC 330 includes memory, processing circuitry, and various circuits, engines, and engines that enable the execution of the tasks described herein. A detailed block diagram of IC 330 is shown in Figure 4. IC 330 may be realized as a chipset, a system on a chip (SoC), a field programmable gate array (FPGA), a programmable logic device (PLD), an application specific integrated circuit (ASIC), or any other type of digital and / or analog hardware component.

[0050] The IC 330, and therefore the FIR camera 220, can operate in four different modes: calibration, power-on test, function, and standby. The calibration mode is performed in the lab, where the IC 330 performs a calibration process based on the previous calibration point. During the process, the FIR camera 220 is stabilized at a given temperature, and both the FPGA and ambient temperatures are periodically read from the FIR sensor 320 to determine temperature stability.

[0051] In the power-on test mode, various circuits of the IC 330 are checked for proper operation during the FIR camera 220 startup (or power-up). This includes, for example, performing internal self-tests for memory, logic monitors, digital and analog components. In the functional mode, the IC 330 performs image processing tasks and communicates with external systems and components (e.g., sensors and computers). In the standby mode, no image processing takes place, only communication with external systems and components takes place. The standby mode can be activated when the vehicle is not moving.

[0052] The FIR sensor 320 and IC 330 are encapsulated in a thermal core 325. The thermal core 325 is utilized to ensure a uniform temperature for the FIR camera 220. Temperature calibration of the thermal core is performed during a calibration mode. In another embodiment, a thermal core can also be disposed in the central housing 200, the front cover 270, and the back cover 260 to protect the components of the FIR device 110 from heat, cold, and can be made to waterproof the components.

[0053] 4 illustrates an exemplary block diagram of an IC 330 designed according to one embodiment. The IC 330 includes a processing circuit 410, a residual non-uniformity noise correction processor (RNUNCP) 420, and an image enhancement processor (IEP) 430 configured to perform various image processing tasks described herein. The IC 330 further includes a memory 440 and a temperature sensor 460.

[0054] The IC 330 interfaces with external elements such as the server 142 and the display 152 via a multimedia link 402. In an exemplary embodiment, the media link is a gigabit multimedia serial link (GMSL). As mentioned above, in one embodiment, the IC 230 can be configured to output two thermal video streams of FIR images, where a first stream (304) is processed by the RNUNCP 420 and the IEP 430 and input to the display 152, and a second stream (305) is processed by the RNUNCP 420 and input to the display 152. Both thermal video streams are output via the multimedia link 402. In a further embodiment, a single thermal video stream is output to the server 142. For example, a particular autonomous vehicle system may not include a display and therefore only require an input from the thermal video stream to the server 142.

[0055] In some optional embodiments, the IC 330 does not include an IEP 430. In such embodiments, the correction is performed by the RNUNCP 420 and the output is provided to the server 142. A configuration of the FIR camera 220 without an IEP 430 may be utilized in an autonomous vehicle system where a display may not be required.

[0056] In one embodiment, the memory 440 is configured to store a calibration table. The calibration table includes at least various calibration values ​​for each picture element (or pixel) calculated in a lab during camera manufacturing. The calibration values ​​may include gain and offset calculated from two temperature points (T1, T2) to overcome irregularities in the FIR sensor and unify the pixel's response to IR radiation for a wide range of ambient temperatures. The calibration table also includes drift values ​​determined for each pixel at each temperature point during the calibration process. In one embodiment, the table also stores values ​​of various parameters for configuring the FIR camera 220.

[0057] The memory 440 may further store computer readable instructions executed by the processing circuitry 410, the RNUNCP 420, and the IEP 430. Computer readable instructions should be broadly construed to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or the like. The instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable code format). The instructions, when executed by the processing circuitry 410, the RNUNCP 420, and / or the IEP 430, cause those processors to perform the various embodiments and tasks described herein. The memory 440 may be volatile (e.g., RAM, etc.), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof.

[0058] In an optional configuration, the camera IC 330 includes a motion detection unit (MDU) 450. The MDU 450 includes one or more accelerometers that are utilized to determine if the FIR camera 220 (and thus the vehicle) is moving. When the MDU 450 is not utilized, motion detection is based on captured images. Motion indications are provided to the RNUNCP 420 and the IEP 430. A temperature sensor 460 measures the ambient temperature within the housing of the FIR camera 220.

[0059] The processing circuit 410 is configured to control various elements of the IC 230. The circuit 410 is further configured to power up the FIR sensor 320, upload initial calibration tables, and configure the FIR sensor 320 as well as the RNUNCP 420 and IEP 430 based on the values ​​of parameters in such tables. In one embodiment, the circuit 410 periodically samples the temperature measured by the temperature sensor 460 and uploads a set of calibration tables from the memory 440 that contain calibration values ​​for the measured temperature range. To allow for fast execution, some tables may be uploaded in the background (i.e., while the FIR image is being processed). The processing circuit 410 further includes a shutter motor driver (driver) 412 and a window heater relay (relay) 414, both of which control components of the FIR device 110. For example, the motor 215 of the shutter 210 may be powered and controlled by the motor driver 412 of the processing circuit 410.

[0060] In one embodiment, the RNUNCP 420 is configured to perform pixel-based correction, which corrects for unmodeled offset, gain, drift, noise, or errors for each pixel. In one embodiment, the offset and gain of the pixel are corrected based on respective values ​​in a calibration table that correspond to respective temperatures measured by the temperature sensor 460. This correction compensates for sensor irregularities and unifies the pixel response to IR radiation for a wide range of ambient temperatures.

[0061] The drift correction, which extends the period between shutter calls when a shutter is used during imaging, includes adding a drift correction to each of the pixels. Specifically, the correction for each pixel is the result of subtracting a predetermined drift value provided from a calibration table loaded separately for each pixel and multiplying it by a gamma value calculated from a frame generated by the FIR sensor. The predetermined drift value is selected from the table based on the current measured temperature at the FIR sensor 320.

[0062] According to some embodiments, drift correction is based on a temperature drift correction for each pixel in the image output to (or from) the FIR sensor 320. This may be necessary if the temperature values ​​of the entire FIR camera 220 and the individual FIR sensor 320 and IC 330 shift during operation.

[0063] The drift correction may be based on estimating the drift factor γ using drift patterns recorded during a calibration phase of the FIR camera 220. In one embodiment, the estimation of the drift factor γ is performed iteratively with small incremental updates based on each input image, so that any residual or new noise, e.g. due to ambient temperature shifts, is detected and removed in each subsequent image.

[0064] The ambient drift of the input image is corrected using an estimated drift coefficient γ. The drift-corrected image (Iout) can be expressed as: TIFF2024502133000002.tif10169 where I_in is the input image (produced by the FIR sensor 320), γ is the drift coefficient, which is the same for all pixels, “D” is a drift pattern unique to each pixel, “O” is an offset gain pattern, and “G” is a scene gain pattern value unique to each pixel, and the D, O, and G values ​​are determined during an initial calibration of the FIR sensor 320.

[0065] In some configurations, the gain and offset are corrected before drift correction, in which case such values ​​are not needed to estimate the drift coefficient γ. Thus, a more generalized version of the drift correction calculation uses the two-point corrected input image (I2pt), the drift coefficient, and the drift pattern from the calibration, and can be expressed as the following equation: TIFF2024502133000003.tif9169In equation (2), D^ is the value of the drift pattern (D) divided by the gain pattern (G).

[0066] The drift factor γ used for the correction is derived from the drift pattern from the initial calibration and the resulting pattern of drift in the input image after the high-pass filter is applied.

[0067] The generalized formula for the drift coefficient is the quotient of the product of the drift pattern and the input image pixel value after applying the high-pass filter and the product of the drift pattern and the drift pattern after applying the high-pass filter, and is given by: TIFF2024502133000004.tif15169 where H_1 and H_2 are pre-processing operators designed to minimize correlation between the clean image of the scene and the drift pattern. Without loss of generality, these operators may include high-pass filtering and masking. Without loss of generality, examples of such operations may be H_1 = identity map and H_2 = high-pass filter.

[0068] In Eq. 3, γ is a scalar value calculated as a ratio of two dot products. Therefore, it is one value for all pixels at this stage. In one embodiment, the scalar value may be converted to a (slowly varying) value for each pixel by using Eq. 3 on different segments of the image and interpolating the result. An example of such an embodiment is described below. By applying γ in drift correction, the frequency of shuttering the FIR camera for image processing can be reduced while keeping the FIR camera for a longer period, thereby improving the reliability and safety performance of the FIR camera in a V2I setting.

[0069] In one embodiment, for V2I applications where the position of the FIR camera 110 is fixed, only still images may be used for noise correction. This may be implemented by using a deep learning model that allows separating the actual scene image from the fixed pattern noise pattern by training the model using multiple clean images. Here, the associated shutter call made during noise correction may be made "safe" or "smart" by preventing the camera shutter while critical tasks that require a clear view of the FIR camera 110 are performed. For example, when tracking a fast approaching vehicle, the shutter should remain open and the fixed pattern noise correction may be skipped or delayed until the vehicle passes the field of view of the FIR camera 110.

[0070] If it is necessary to provide an actual temperature measurement of objects in the scene, the FIR camera 110 may be adapted to convert the gray level pixel values ​​to temperature. To do this, the FIR camera 110 may undergo a calibration procedure during manufacture or at the scene. Alternatively, or as a complement to the external calibration, one or more black bodies, or reference objects, may be used as a more accurate temperature reference. The black bodies may be placed at a fixed position in the scene, or may be attached to the detector window in front of the lens, or may be attached between the lens and the detector, so that black body radiation is received by the detector pixels. These pixels may be in the visible image area or may be hidden (i.e., redundant pixels exposed to the scene, or black body radiation (but not shown in the output image)).

[0071] As described in more detail below, the RNCP 420 implements multiple engines to perform correction and other image enhancement tasks. In one embodiment, scene-based correction also includes correcting pixels that suffer from the "sunburn effect." Such an effect is caused by exposing an FIR camera to direct sunlight, which causes the exposed pixels to go into deep saturation due to overexposure. The result of this effect is characterized by a ghost footprint of the sun that resembles fixed pattern noise that can persist for minutes or weeks depending on the exposure time. According to the disclosed embodiment, the sunburn correction also removes the ghost footprint from the image in a few seconds (e.g., 2-5 seconds).

[0072] At a minimum, the RNUNCP 420 is implemented to perform pixel-based correction, column noise suppression, and near field correction. In one embodiment, the RNUNCP 420 may be configured to perform additional image processing tasks to provide a high quality thermal video stream that enables accurate detection of objects. The image processing tasks are performed in a pipelined manner by various engines, as shown in FIG. 5.

[0073] The bad pixel correction (BPC) engine 501 is configured to mask pixels designed in a bad pixel map. Such pixels do not meet basic quality requirements required by the manufacture of the FIR sensor. Bad pixels in the map may be identified by the manufacture or during tests performed on the FIR sensor in the lab. The bad pixel map is stored in the memory 440.

[0074] The pixel-based correction (PBC) engine 503 is configured to perform offset, gain, and drift value correction for each pixel as described above. The column noise suppression (CNS) engine 504 suppresses fixed pattern column noise in FIR imaging with a FIR sensor. The bad pixel correction (BPC) engine 505 attempts to correct bad pixels by approximating each bad pixel using optimal interpolation based on a predefined number (e.g., 8) of its nearest neighbors.

[0075] The Near Field Correction (NFC) engine 506 is configured to correct unwanted patterns superimposed on the scene image due to various physical effects such as camera case and lens reflections. Such patterns are modeled during the calibration mode. During the operational mode, the NFC engine 306 identifies the best match between the pattern and the input image and subtracts the pattern from the image accordingly. The NFC engine 506 treats two such patterns independently or as a linear combination. The required pattern images and parameters are provided by the processing circuit 410 on power-up and updated as necessary.

[0076] The h-flip engine 508 is configured to perform a horizontal flip of the received sensor image. In one embodiment, the engine 508 is further configured to clip pixel values ​​between a minimum and maximum value, which are preset.

[0077] The time denoise engine 509 is configured to perform a time denoising process of the infrared video. The video acquired from the FIR sensor 320 contains various types of temporal noise such as white noise, salt and pepper noise (occasionally flickering) and row / column noise. In one embodiment, the engine 509 is implemented by an IIR filter with adaptive threshold and an anti-flicker mechanism. The time denoise engine 309 compares the change in pixel values ​​from frame to frame with the estimated noise variance to determine whether such values ​​are caused by noise or some real change in the scene. Based on the comparison, the signal to noise ratio is determined.

[0078] The spatial denoising engine 510 is configured to denoise spatial noise. Such noise is due to internal electronic noise, pixel gain and offset errors, and drift caused by temperature variations. In one embodiment, the engine 510 can further replace each noisy pixel with a weighted average of its neighboring pixels. In one embodiment, only pixels having gray levels closer to a predefined gray level threshold are replaced.

[0079] It should be noted that in a particular configuration, only some of the engines described are required as part of the processing. The various parameters, maps, and calibration values ​​are not necessary to operate the various engines and processes stored in memory 440 and are not shown herein solely for the sake of simplicity.

[0080] In the exemplary embodiment, the output of the spatial de-noising engine 510 is a 14-bit grayscale video stream that is provided to the server 142 (FIG. 1) and the IEP 430 .

[0081] Returning to Figure 4, the IEP 430 processes the 14-bit stream to generate a human-viewable enhanced thermal video stream. In one embodiment, the IEP 430 converts the 14-bit stream to a high-quality 8-bit stream that can be efficiently displayed on a screen.

[0082] In one embodiment, the IEP 430 includes engines shown in FIG. 5 configured to perform image enhancement processing. The first engine is a bilateral filter 511 that is used for edge preservation and noise reduction filtering. An automatic gain control (AGC) engine 512 is configured to reduce the influence of empty bins from the gray level range. That is, the engine 512 is configured to minimize the distance between the lowest and highest gray levels in the image without essentially losing information. The reduction of the dynamic range allows the image histogram to be stretched as much as possible.

[0083] The local contrast enhancement (LCE) engine 513 is configured to perform a process for contrast-limited adaptive histogram equalization (CLAHE). In image processing, contrast-limited adaptive histogram equalization is a technique used to improve the local contrast of an image. Adaptive histogram equalization is performed to find a mapping for each pixel based on its local (neighborhood) grayscale distribution.

[0084] In one embodiment, the LCE engine 513 is configured to divide the image into overlapping blocks to further reduce the blocking effect. In some cases, when the grayscale distribution is highly localized, it may not be desirable to convert a very low-contrast image by full histogram equalization. In such cases, the mapping curve may contain segments with high slopes, which means that two very close grayscales may be mapped to significantly different grayscales. This is resolved by limiting the contrast allowed by histogram equalization.

[0085] The final image processing enhancement engine is the Polarity-and-Brightness (PB) engine 514. In one embodiment, engine 514 is configured to change between white-hot pixels and black-hot pixels and adjust the brightness of pixels of the image by a predefined offset.

[0086] It is noted that the processing of the various engines shown in Figure 5 is performed on the infrared image input by the FIR sensor 320. The resulting image is still an infrared image (referred to herein as an "FIR image"). Thus, the processing does not change the nature of the image (e.g., from infrared to a visible spectrum image), but rather improves the quality of the infrared image.

[0087] 4, in one embodiment, the processing circuitry 410 may be a single-core or multi-core CPU. Each of the RNUNCP 420 and the IEP 430 may be implemented as one or more hardware logic components and circuits. For example, and without limitation, exemplary types of hardware logic components that may be used include FPGAs, ASICs, ASSPs, SOCs, general purpose microprocessors, microcontrollers, DSPs, GPUs, etc., or any other hardware logic components that may perform calculations or other manipulations of information.

[0088] 6 is an exemplary flowchart 600 of a method of operating a V2I communication system, according to an embodiment of the present disclosure. In one embodiment, the method may be performed by a processing circuit 410 in an IC 330 programmed to execute specific executable code or software.

[0089] In S610, with the shutter 210 open, it is detected whether there is non-uniformity noise in the image captured by the FIR camera 220. Non-uniformity noise is noise caused by lack of sensor equalization in the FIR sensor 320. Non-uniformity noise may be present in the V2I system due to the fixed FIR device 110 fixed on a pole, and therefore cannot benefit from motion-based noise suppression algorithms. For example, minute position changes due to temperature changes, dust, and wind may introduce noise. If noise is detected in S620, the method may proceed to S630 and perform a standard calibration on the FIR camera 220.

[0090] Here, standard calibration in S630 may include first receiving a video image, an intrinsic calibration, which may include calibration internally in IC 330, correction using additional image modalities such as FIR correction or LIDAR, or external calibration performed over network 140. The calibration engine (not shown) may report multiple states including an idle state, a processing state, a completed state, a processing error state, or a processing timeout state. The calibration engine may also select frames to process according to processing execution time or any heuristics on the image data.

[0091] Additionally, calibration may include geometry calibration based on scene distance measurements performed during placement, detection, and spacing (or distance / range) estimation to verify that the FIR is aligned and the FOV is correct.

[0092] Then, in S640, it is determined whether noise remains. If the noise is removed, the process proceeds to S680. Otherwise, the process proceeds to S650, where it is determined whether there is a moving object detected by the FIR camera 220. If so, the shutter 110 remains open to allow FIR imaging to continue, thus preventing a dangerous case where objects such as vehicles or pedestrians move towards each other when the shutter is closed during additional correction. Otherwise, if a moving object approaching the FIR camera 220 is not detected by the FIR camera 220, the process proceeds to S670.

[0093] In S670, the shutter is closed or placed in the field of view of the FIR camera 220 and a NUC operation can be performed. Here, the shutter 110 presents a uniform temperature source to all detector elements in the FIR so that the FIR can perform offset correction. In this way, a more uniform image is obtained compared to standard calibration. In another embodiment, NUC may also be performed at the start of FIR imaging.

[0094] Once the close shutter NUC is complete, processing proceeds to S680 to reopen the shutter. Here, it is determined whether the field of view of the FIR camera 220 is obstructed by dirt or the like. If so, additional cleaning may be performed to clear the field of view, or a report may be generated over the network in S590 to alert the user that the FIR 220 camera is not functioning properly, so that the FIR may be manually serviced. After S690, processing ends.

[0095] 7 is an exemplary schematic diagram of a server 142 according to one embodiment. The system 142 includes a processing circuit 710 coupled to a memory 620, a storage device 730, and a network interface 740. In one embodiment, the components of the server 142 may be communicatively coupled via a bus 750.

[0096] The processing circuitry 710 may be implemented as one or more hardware logic components and circuits. For example, and without limitation, exemplary types of hardware logic components that may be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), general purpose microprocessors, microcontrollers, digital signal processors (DSPs), or the like, or any other hardware logic components capable of performing calculations or other manipulations of information.

[0097] The memory 720 may be volatile (e.g., RAM, etc.), non-volatile (e.g., ROM, flash memory, etc.), or a combination thereof. In one configuration, computer readable instructions for implementing one or more embodiments disclosed herein may be stored in the storage device 730.

[0098] In another embodiment, memory 720 is configured to store software. Software should be broadly construed to mean any type of instructions, whether referred to as software, firmware, middleware, microcode, hardware description language, or the like. The instructions may include code (e.g., in source code format, binary code format, executable code format, or any other suitable code format). The instructions, when executed by processing circuitry 410, cause processing circuitry 410 to perform various processes described herein.

[0099] The storage device 730 may be a magnetic storage device, an optical storage device, etc., and may be implemented as, for example, flash memory or other memory technology, a CD-ROM, a digital versatile disk (DVD), or any other medium that can be used to store the desired information.

[0100] The network interface 740 allows the server 142 to communicate with the IC 230, for example, to receive data, to transmit data, etc. Additionally, the network interface 740 allows the server 142 to communicate with the display 152 for purposes of collecting vehicle data.

[0101] It should be understood that the embodiments described herein are not limited to the particular architecture shown in FIG. 7, and that other architectures may equally be used without departing from the scope of the disclosed embodiments.

[0102] Various embodiments disclosed herein may be implemented as hardware, firmware, software, or any combination thereof. Furthermore, software is preferably implemented as an application program tangibly embodied on a program storage unit or computer-readable medium consisting of parts or specific devices and / or combinations of devices. The application program may be uploaded to and executed by a machine having any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units ("CPU"), memory, and input / output interfaces. The computer platform may also include an operating system and microinstruction code. Various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, that may be executed by the CPU, whether or not such a computer or processor is explicitly shown. In addition, various other peripheral units may be connected to the computer platform, such as additional data storage units and a printing unit. Furthermore, a non-transitory computer-readable medium is any computer-readable medium, except for a transitory propagating signal.

[0103] All examples and conditional language recited herein are for educational purposes to aid the reader in understanding the principles of the disclosed embodiments and concepts contributed by the inventors to advance the art, and should not be construed as being limited to such specifically recited examples and conditions. Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosed embodiments, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Moreover, such equivalents are intended to include both currently known equivalents and equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0104] It should be understood that any reference to an element herein using a designation such as "first," "second," etc. does not generally limit the quantity or order of those elements. Rather, these designations are generally used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, a reference to a first and a second element does not imply that only two elements may be used therein or that the first element must precede the second element in any way. Also, unless otherwise specified, a set of elements includes one or more elements.

[0105] As used herein, the phrase "at least one of" can be followed by a list of items meaning that any of the listed items can be utilized individually, or any combination of two or more of the listed items can be utilized. For example, if a system is described as including "at least one of A, B, and C," the system can include A alone, B alone, C alone, 2A, 2B, 2C, 3A, a combination of A and B, a combination of B and C, a combination of A, B, and C, a combination of 2A and C, a combination of A, 3B, and 2C, and the like.

Claims

1. 1. A method of operating a vehicle-to-infrastructure (V2I) system, comprising: receiving an image from an infrared (IR) camera; determining whether non-uniformity noise is present in the received image; performing a calibration upon determining that the non-uniformity noise is present; performing a non-uniformity correction (NUC) on the IR after performing the calibration and upon determining that there is residual non-uniformity noise and that the IR camera does not detect a moving object approaching the IR camera; determining whether the IR camera's field of view (FOV) is obstructed after performing the NUC; and upon determining that the FOV of the infrared camera is obstructed, cleaning the IR camera.

2. the IR camera includes a shutter; The method of claim 1 , wherein the shutter is closed on the infrared (IR) camera while the non-uniformity correction (NUC) is performed.

3. The method of claim 2 , further comprising the step of keeping the shutter open during the determination of whether non-uniformity noise is present.

4. The method of claim 2 , further comprising the step of keeping the shutter open while the calibration is performed.

5. The method of claim 1 , wherein the calibration comprises one of a geometric calibration or a correction with another image.

6. The method of claim 5 further comprising the step of performing a geometric calibration based on a scene distance measurement performed during separation estimation between the IR camera and the vehicle.

7. 2. The method of claim 1, further comprising forming a pixel-based correction for each pixel in the received image to correct for one of unmodeled offset, gain, or drift noise.

8. The method of claim 7 , further comprising correcting the drift noise based on a temperature drift correction for each pixel in the received image.

9. 8. The method of claim 7, further comprising the steps of: calculating a gamma drift factor; and correcting the drift noise by subtracting a product of a predetermined drift value and the calculated gamma drift factor for each pixel.

10. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuit to perform a process, the process comprising: receiving an image from an infrared (IR) camera; determining whether non-uniformity noise is present in the received image; performing a calibration upon determining that the non-uniformity noise is present; performing a non-uniformity correction (NUC) on the IR after performing the calibration and upon determining that there is residual non-uniformity noise and that the IR camera does not detect a moving object approaching the IR camera; determining whether the IR camera's field of view (FOV) is obstructed after performing the NUC; and upon determining that the FOV of the infrared camera is obstructed, cleaning the IR camera.

11. 1. A vehicle-to-infrastructure (V2I) system for detecting objects, comprising: A processing circuit; a memory that, when executed by the processing circuitry, receiving an image from an infrared (IR) camera; determining whether non-uniformity noise is present in the received image; Upon determining that the non-uniformity noise is present, performing a calibration; performing a non-uniformity correction (NUC) on the IR camera upon determining that there is residual non-uniformity noise after performing the calibration and determining that the IR camera does not detect a moving object approaching the IR camera; determining whether the IR camera's field of view (FOV) is obstructed after performing the NUC; and the memory containing instructions that configure the system to clean the IR camera upon determining that the FOV of the IR camera is obstructed.

12. the IR camera includes a shutter; The system of claim 11 , wherein the shutter is closed on the infrared (IR) camera while non-uniformity correction (NUC) is being performed.

13. The system of claim 12 , wherein the shutter remains open during the determination of whether the non-uniformity noise is present.

14. The system of claim 12 , wherein the shutter remains open while the calibration is performed.

15. The system of claim 11 , wherein the calibration comprises one of correction with another image or geometric calibration.

16. The system of claim 15 , wherein the geometric calibration is determined based on scene distance measurements performed during distance estimation between the IR camera and the vehicle.

17. 12. The system of claim 11, wherein the system is further configured to form a pixel-based correction for each pixel in the received image to correct for one of unmodeled offset, gain, or drift noise.

18. The system of claim 17 , wherein the drift noise is corrected based on a temperature drift correction for each pixel in the received image.

19. 20. The system of claim 17, wherein the drift noise is corrected by calculating a gamma drift factor and subtracting a product of a predetermined drift value and the calculated gamma drift factor for each pixel.

20. The method of claim 1 , wherein the non-uniformity noise present in the image from the IR camera is caused by at least one of temperature changes and position changes due to wind affecting the IR camera.