Glare removal using dual cameras

Abstract

Glare can be removed from an RGB image transformed into a YUV image by replacing a glare region in a luminance component of the YUV image with a pixel value in a corresponding region of an IR image captured simultaneously with the RGB image using a dual camera capable of capturing both the RGB and IR images of a scene. In addition, color information in the glare region can be adjusted by averaging or extrapolating from color information in surrounding regions.

Description

Background Technology

[0001] When a camera is used to capture video or take photographs, the presence of any active light source or high-intensity specular reflections from objects in the scene can cause overexposed areas known as glare, resulting in pixel distortion and loss of detail in one or more images. One approach to this problem involves capturing images of the scene from two or more different perspectives and post-processing the images to create a composite image from which glare has been removed, taking advantage of the fact that glare caused by a given light source or specular reflection will occur at different locations in the images. Multiple cameras can be used to capture different images simultaneously. Alternatively, if images of a static scene or object are to be captured, the same camera can be moved to different positions and / or orientations relative to the scene to capture images sequentially. In variations of this approach, in some applications, it is also possible to capture multiple images using the same camera perspective, but with the scene illuminated from different angles. Solutions relying on sequentially captured images are generally not suitable for removing glare from videos of moving objects. Even when using multiple cameras to image the scene simultaneously from multiple angles, subsequent image processing is computationally expensive, severely limiting its feasibility in real-time glare removal applications. Attached Figure Description

[0002] Various systems and methods for removing glare from images are described herein with reference to the accompanying figures, wherein:

[0003] Figure 1 This is a flowchart illustrating a method for generating glare-free images based on red-green-blue (RGB) and infrared (IR) image pairs according to various embodiments;

[0004] Figure 2 This is based on the use of various embodiments as in Figure 1 A schematic diagram of the image processing method used in the method to remove glare from RGB images using IR images;

[0005] Figure 3 It is a schematic diagram illustrating a system architecture for generating glare-free images according to various embodiments; and

[0006] Figure 4 This is a block diagram of an exemplary machine that can perform glare removal according to various embodiments. Detailed Implementation

[0007] This document describes a system and method for automatically removing glare from images simultaneously captured using RGB cameras, based on images acquired using IR cameras. The disclosed method leverages the fact that IR cameras do not capture light in the visible spectrum, and therefore, IR images do not contain glare. Systems for glare removal according to various embodiments include dual RGB and IR cameras overlapping in their respective fields of view, and are capable of simultaneously capturing a pair of images of a scene using associated electronic circuitry that processes raw image data read from the image sensors of the RGB and IR cameras to generate output images with the same field of view and resolution. Furthermore, the system includes computing hardware and / or software that, after converting the RGB image to YUV format, processes the Y-plane of the YUV image to detect any glare present in the image and identify its contours, defining a “region of interest in glare (glare ROI)” or simply a “glare region”; replaces the glare ROI in the Y-plane with the pixel values ​​of the corresponding region in the IR image to remove glare artifacts; and (optionally) adjusts the color information of the pixels in the glare ROI based on the UV plane of the original YUV image. For example, color information in pixels surrounding a glare area can be averaged or extrapolated to determine the color information within the glare area. Alternatively, in some cases, the original UV plane values ​​can be used as is.

[0008] The proposed scheme is suitable for computationally efficient implementation, facilitating real-time glare removal, for example, within 30 ms or less. Therefore, the glare removal described therein can be applied to video streaming to, for example, improve the user experience during video calls or recordings under various lighting conditions, or improve video provided by dashboard and other vehicle camera systems, and so on.

[0009] Figure 1 This is a flowchart illustrating a method 100 for generating a glare-free image based on a pair of RGB and IR images, according to various embodiments. Two digital cameras simultaneously capture a pair of images of a scene; that is, RGB camera 102 captures light in the visible wavelength range, and IR camera 104 captures light in the infrared wavelength range. The pair of images can be, for example, synchronized frames within a video stream generated by cameras 102, 104. RGB camera 102 may include an image sensor covered with a color filter array (CFA); in this case, the image processing pipeline involves “de-mosaicing” the sensor output (action 106), that is, digitally processing it based on the CFA to construct a full-color image or RGB image 108. RGB image 108 is further converted (in action 110) into a YUV image 112, which represents the image according to the luminance (physical luminance) component Y (114) and two chromaticity components U (blue projection) and V (red projection) (115 and 116).

[0010] Glare removal operates on the Y component of the YUV image 112, or on the Y plane 114. As shown, method 100 involves determining whether glare exists in the Y plane 114 (action 118), and if so (as determined at 120), identifying the contours of the glare region (action 122). Glare detection can typically be performed using conventional explicit image processing algorithms or machine learning models. Conventional algorithms often rely on statistically defined thresholds and speckle sizes to classify image regions as glare or non-glare. These algorithms often identify glare regions with specific predefined shapes (e.g., circles, squares, rectangles), which limits detection accuracy and can lead to false negatives (missed glare regions) and false positives (non-glare regions incorrectly classified as glare). With machine learning models, thresholds and speckle sizes are not statistically predefined but are automatically adjusted based on data. Furthermore, even after deployment, specific parameters of the machine learning model can be fine-tuned based on additional data. In an exemplary embodiment, a convolutional neural network (CNN) is employed to determine which pixels within the image are within the glare region. Note that although glare detection and glare area identification are in Figure 1 The process is conceptually described as two separate steps 118, 122, but in some embodiments, it can constitute a single operation for identifying any (one or more) glare areas, thereby inherently determining whether glare is present in the image.

[0011] Once the glare ROI 124 has been identified within the Y-plane 114 of the YUV image 112, the corresponding region of pixel 126 is identified (action 128) in the IR image 130. To simplify calculations, the determination of corresponding pixels is typically performed on pairs of RGB / YUV images 108 / 112 and IR image 130, which have been preprocessed, for example, using spatial transformation and cropping, to share a common field of view and the same resolution. Then, the pixel values ​​in the glare region 124 of the Y-plane are replaced with the pixel values ​​of the corresponding region 126 within the IR image 130 (action 132), resulting in a glare-free Y-plane 134. Furthermore, the corrected glare-free Y-plane 134 is recombined with the chromaticity components (i.e., the UV planes 115, 116). In some embodiments, taking advantage of the fact that colors tend to be highly correlated (where adjacent pixels typically have the same color value), the color information within the glare ROI is adjusted based on the color values ​​surrounding the glare ROI. For example, for glare regions comprising a relatively small number of neighboring pixels, the UV pixel values ​​within the glare region of the YUV image can be replaced by the average of the UV pixel values ​​of the pixels surrounding the glare region (e.g., a weighted average, where pixels from surrounding pixels closer to the glare region have greater weight). For larger glare regions, the UV plane can be fitted to the region surrounding the glare region and extrapolated to the glare region. A glare-free image, including luminance information of the glare-free Y plane 134 and chromaticity information of (e.g., color-adjusted) UV planes 115, 116, can be sent to a display device 136 (e.g., a computer screen, telephone screen, or video display in a vehicle) and / or stored in memory 138. If no glare is detected in the Y plane (as determined at 120), the Y plane 114 and UV planes 115, 116 are used as is in the stored or displayed image.

[0012] Figure 2 This is based on the use of various embodiments as in Figure 1 This is a schematic diagram of the image processing method 200 for removing glare from RGB images using IR images, as employed in Method 10. The Y-plane 114 from the RGB camera is input to the glare detection algorithm 202, which classifies each pixel as either a glare pixel or a normal pixel, creating a binary bitmap (B) 204, where each glare pixel has a value of one, and each normal pixel has a value of zero. As will be appreciated, the bitmap 204 reflects the presence of glare in the image and, if present, inherently defines the glare region. Therefore, the glare detection algorithm 202 combines glare detection and the identification of glare regions and their contours in one step.

[0013] Glare detection algorithm 202 can be explicitly programmed or implemented by a machine learning model. In the latter case, the machine learning model can be, for example, a neural network model, such as a CNN or a fully connected network. CNNs, which are generally well-suited for image processing tasks, are particularly beneficial for implementing glare detection algorithm 202 because they can provide high accuracy and performance. The neural network model for glare detection can be trained, for example, in a supervised manner, based on training data comprising many pairs of Y-planes and their corresponding binary bitmaps, wherein the bitmaps are, for example, manually generated by a user to identify the contours of glare regions in an image. As is known in the art, training the neural network involves iteratively adjusting the network weights to optimize a cost function that measures the difference between the model's output (in this case, the binary bitmap) and the ground-based output provided with the training data (e.g., the manually generated binary bitmap); a widely used algorithm for this optimization is backpropagation of the error.

[0014] Once a binary bitmap 204, inherently identifying glare regions in the Y-plane of the YUV image, has been generated, the corresponding region in the IR image (I) 130 is identified at 206 by multiplying the IR image with the binary bitmap 204. This operation generates an intermediate image (I') 208, in which all pixel values ​​except those in the glare regions are zero. In the glare regions, the intermediate image 208 has the same pixel values ​​as the IR image 130 calculated therefrom. Furthermore, in action 210, the binary bitmap 204 is inverted by forming the two's complement (B') 212 of the binary bitmap 204, which is zero for all pixels within the glare regions and one for all pixels outside the glare regions. At 214, the Y-plane 114 of the YUV image is multiplied with the two's complement 212 of the binary bitmap 204. This operation generates another intermediate image (Y') 216, in which the pixel values ​​corresponding to the glare regions become zero. At 218, the two intermediate images 208 and 216 are added to generate a single modified Y-plane (Y”) 220 without glare pixels, wherein the glare pixels of the original Y-plane 114 have been replaced by the corresponding pixel values ​​of the IR image 130.

[0015] In an efficient implementation, method 200 is computationally fast, allowing glare removal from an RGB image stream in real time (i.e., within timeframes imperceptible to human consumers of the video stream). For example, video captured at a frame rate of 30 frames per second (or more) (corresponding to a new video frame every 33 milliseconds) is perceived as continuous video by humans. In various embodiments, the glare removal process is completed in less than 33 milliseconds, enabling the generation of glare-free video frames at the rate of acquiring the original video frames.

[0016] Figure 3This is a schematic diagram illustrating a system architecture for generating glare-free images according to various embodiments. System 300 can be implemented by a computing machine, for example, as in... Figure 4 The schematic depiction includes one or more hardware processors (e.g., one or more central processing units (CPUs) and / or graphics processing units (GPUs)), memory storing program instructions (including operating systems and one or more software applications) for execution by the processor(s), and various peripheral hardware devices, including, for example, user input / output devices (such as keyboards or touchpads, display devices or touchscreens) and cameras. The computing machine can be, for example, but not limited to, a personal desktop or laptop computer, a tablet, or a smartphone.

[0017] At hardware level 302, system 300 includes an RGB camera 102 and an IR camera 104. Cameras 102 and 104 typically have associated digital signal processors (DSPs), more specifically image signal processors (ISPs), which preprocess the image sensor signals, for example, to perform de-mosaicing, noise reduction, etc., on the RGB images. In a dual-camera system, preprocessing may also involve image transformation and cropping operations to create a pair of images with the same resolution and field of view. Cameras 102 and 104 have their respective associated camera device drivers 304 and 306, which typically run in the kernel space 308 of the operating system. At application level 310, system 300 runs software application 312, which provides video capabilities via access to the RGB camera 102; hereinafter, such an application is also referred to as camera application 312. For example, camera application 312 could be, for instance, an online collaboration platform supporting video calls, such as... Teams. As another example, camera app 312 could be a dedicated smartphone app for taking pictures and recording videos. Camera app 312 communicates with user space driver 314 in user space 316 of the operating system. User space driver 314 then communicates with camera device drivers 304, 306.

[0018] In the depicted example, glare removal is implemented in user space 316. Specifically, user space driver 314 includes processing components that operate on preprocessed YUV and IR images received from RGB and IR camera drivers 304, 306 to detect and locate the outline of any (one or more) glare regions in the Y-plane of the YUV image (driver component 318), locate the corresponding pixel in the IR image (driver component 320), replace the glare pixel value in the Y-plane with the corresponding IR pixel value (driver component 322), and adjust the color information of the replaced pixel based on the chromaticity components of the YUV image in the glare region (driver component 324). The glare-free, color-adjusted YUV image 326 can then be stored in memory 328 (e.g., double data rate (DDR) random access memory) allocated to camera application 312, and / or output by camera application 312 to hardware display device 330.

[0019] As will be readily apparent to those skilled in the art, glare removal functionality is typically implemented in user space 316 or alternatively in kernel space 308. For example, in Figure 3 It is advantageous to implement the kernel space 308 as part of the user space driver 314 as illustrated, because it isolates the kernel space 308 from any errors that may occur during glare removal, and thus avoids the need to reboot the system or reload the camera device drivers 304, 306 in case of an error.

[0020] In alternative embodiments, glare removal can also be provided at hardware level 302 in the ISP, for example, as part of a system-on-a-chip (SoC) for a dual-camera system. For example, various existing dual-camera systems already perform image alignment between two images in the ISP (to obtain images with a shared field of view). Snapdragon TM The SoC features an on-chip ISP, the Spectra 580, that supports dual-camera systems. of The SoC also supports dual-camera systems. According to various embodiments, these and similar systems can be enhanced by further processing the image in the ISP to remove glare from the RGB image when used with RGB and IR cameras.

[0021] In yet another embodiment, as described herein (e.g., refer to...), Figure 2Glare removal can be implemented in application-level 310 software. This approach allows the use of two individual cameras (one RGB camera and one IR camera) instead of a combined dual-camera system. The image processing software can receive the raw YUV and IR data from both cameras and process them as needed to achieve the same resolution and a common field of view. For example, the image processing software can utilize image registration techniques to identify corresponding pixels in the two images. The image processing method 200 for glare removal can then be applied to the properly aligned image.

[0022] The software implementation also supports remote, cloud-based glare removal in principle. For example, images from RGB and IR cameras of devices connected to the internet via appropriate wired or wireless communication channels can be sent to a remote server computer for glare removal, and the server can then return the processed, glare-free image. Such remote glare removal can be used in post-processing of images without strict time constraints. When processing video streams in real time, remote processing is feasible if the communication rate is sufficient to transmit images back and forth between the camera-equipped device and the server computer in real time.

[0023] Figure 4 This is a block diagram of an exemplary machine 400 capable of performing glare removal according to various embodiments. The machine may implement method 100, including the exemplary image processing method 200, in whole or in part, and may form all or part of system 300. In alternative embodiments, machine 400 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, machine 400 may operate as a server machine, a client machine, or both in a server-client network environment. In the example, machine 400 may act as a peer-to-peer (P2P) (or other distributed) network environment. Machine 400 may be a personal computer (PC), tablet PC, set-top box (STB), personal digital assistant (PDA), mobile phone, smartphone, web device, network router, switch or bridge, server computer, database, conference room equipment, or any machine capable of (sequentially or otherwise) executing instructions specifying actions to be taken by that machine. Furthermore, although only a single machine is illustrated, the term "machine" should also be understood to include any collection of machines that individually or jointly execute a set (or more sets) of instructions to perform any one or more of the methods discussed herein, such as cloud computing, Software as a Service (SaaS), and other computer cluster configurations.

[0024] Machine (e.g., computer system) 400 may include a hardware processor 402 (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), main memory 404, and static memory 406, some or all of which may communicate with each other via interconnect (e.g., bus) 408. Machine 400 may also include a display unit 410, an alphanumeric input device 412 (e.g., a keyboard), and a user interface (UI) navigation device 414 (e.g., a mouse). In this example, the display unit 410, input device 412, and UI navigation device 414 may be a touchscreen display. Machine 400 may additionally include a storage device (e.g., a drive unit) 416, a signal generation device 418 (e.g., a speaker), and a network interface device 420. Furthermore, machine 400 includes two or more cameras 421 to acquire still images or video. Machine 400 may include an output controller 428, such as a serial (e.g., Universal Serial Bus (USB)), parallel, or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection, to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0025] Storage device 416 may include machine-readable medium 422 on which one or more sets of data structures or instructions 424 (e.g., software) are stored, said data structures or instructions embodying or used by any one or more of the technologies or functions described herein. During execution of instructions 424 by machine 400, instructions 424 may also reside wholly or at least partially within main memory 404, static memory 406, or hardware processor 402. In this example, one or any combination of hardware processor 402, main memory 404, static memory 406, or storage device 416 may constitute the machine-readable medium.

[0026] Although machine-readable medium 422 is illustrated as a single medium, the term "machine-readable medium" can include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) configured to store one or more instructions 424.

[0027] The term "machine-readable medium" can include any medium capable of storing, encoding, or carrying instructions executable by machine 400 and causing machine 400 to perform any one or more of the technologies disclosed herein, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Examples of non-limiting machine-readable media can include solid-state memory as well as optical and magnetic media. Specific examples of machine-readable media can include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; disks, such as internal hard disks and removable disks; magneto-optical disks; random access memory (RAM); solid-state drives (SSDs); and CD-ROM and DVD-ROM disks. In some examples, the machine-readable medium is a non-transitory machine-readable medium. In particular, in some examples, the machine-readable medium is a medium other than one that temporarily propagates signals.

[0028] Instruction 424 can also be sent or received over communication network 426 via a transmission medium through network interface device 420. Machine 400 can communicate with one or more other machines using any of a number of transport protocols (e.g., Frame Relay, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), etc.). Exemplary communication networks may include local area networks (LANs), wide area networks (WANs), packet data networks (e.g., the Internet), mobile phone networks (e.g., cellular networks), common old-style telephone (POTS) networks, and wireless data networks (e.g., referred to as wireless networks). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 standard series, known as The standards include the IEEE 802.16 series, the IEEE 802.15.4 series, the Long Term Evolution (LTE) series, the Universal Mobile Telecommunications System (UMTS) series, and peer-to-peer (P2P) networks. In the example, network interface device 420 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to communication network 426. In the example, network interface device 420 may include multiple antennas to perform wireless communication using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technologies. In some examples, network interface device 420 may use multi-user MIMO technology for wireless communication.

[0029] Examples as described herein may include or operate logic or multiple components, modules, or mechanisms (collectively, “modules”). A module is a tangible entity (e.g., hardware) capable of performing a specified operation and may be configured or arranged in a certain manner. In the examples, circuitry may be arranged as a module in a specified manner (e.g., internal or external to other circuitry). In the examples, all or part of one or more computer systems (e.g., standalone, client, or server computer systems) or one or more hardware processors may be configured by firmware or software (e.g., instructions, application portions, or applications) to operate as a module to perform the specified operation. In the examples, the software may reside on a machine-readable medium. In the examples, when executed by the underlying hardware of the module, the software causes the hardware to perform the specified operation.

[0030] Therefore, the term "module" is understood to include tangible entities, that is, entities that are physically constructed, specifically configured (e.g., hardwired), or temporarily (e.g., provisionally) configured (e.g., programmed) to operate or perform any of the operations described herein in a particular manner. Considering the example of temporarily configured modules, each module does not need to be instantiated at any given time. For example, in the case where modules include general-purpose hardware processors configured using software, the general-purpose hardware processors can be configured as different modules at different times. Software can implicitly configure the hardware processor, for example, to constitute a particular module at one time instance and different modules at different time instances.

[0031] The examples numbered below are illustrative embodiments.

[0032] Example 1: A method for generating a glare-free image, comprising: acquiring an RGB image of a scene in the visible wavelength range; acquiring an IR image of the scene in the infrared (IR) wavelength range; and processing the RGB image in conjunction with the IR image. As part of the processing, the RGB image is converted to a YUV image. Glare regions in the YUV image are identified based on the luminance components of the YUV image; and then, corresponding regions for the glare regions in the YUV image are identified in the IR image. In the luminance components of the YUV image, pixel values ​​in the glare regions are replaced with pixel values ​​from the corresponding regions in the IR image. The result is a glare-free image. Advantageously, using an IR image to remove glare from an RGB image in this manner is computationally low-cost, saving hardware resources and supporting fast processing times, including real-time processing in some embodiments.

[0033] The methods of Example 2 and Example 1 can be implemented by adjusting the pixel values ​​in the chromaticity component of the YUV image based on the pixel values ​​in the region surrounding the glare region.

[0034] Example 3: In the method of Example 1 or Example 2, identifying the glare region based on the luminance component of the YUV image may include: generating a binary bitmap that indicates whether each pixel in the YUV image is a glare pixel.

[0035] Example 4: In the method of Example 3, the bitmap can be generated using a machine learning model. Advantageously, with a machine learning model, the pixel value thresholds and spot sizes of the glare region are not predefined but can be automatically adjusted based on data, and even fine-tuned based on additional data after deployment. This flexibility and data-driven optimization of the model contributes to achieving highly accurate glare / glare-free classification.

[0036] Example 5: In the method of Example 4, the machine learning model may include a convolutional neural network (CNN). CNNs are well-suited for image processing applications and can result in particularly high performance.

[0037] Example 6: In any of the methods in Examples 3-5, a corresponding region in the IR image for the glare region in the YUV image can be identified by multiplying the IR image with the binary bitmap to create a first intermediate image.

[0038] Example 7: In the method of Example 6, the pixel values ​​in the glare region of the luminance component of the YUV image are replaced by the pixel values ​​of the corresponding region in the IR image to create the glare-free image by multiplying the luminance component of the YUV image by the two's complement of the binary bitmap to create a second intermediate image, and adding the first intermediate image to the second intermediate image.

[0039] Example 8: In any of the methods described in the preceding examples, the RGB image and the IR image may be synchronized frames of a first video stream and a second video stream captured by an RGB camera and an IR camera, respectively.

[0040] Example 9: In the method of Example 8, a glare-free image is generated in real time based on additional synchronization frames of the first video stream and the second video stream.

[0041] Example 10: In any of the methods described in the preceding examples, the glare-free image may be displayed on a display device.

[0042] Example 11. A system for generating glare-free images, comprising: one or more hardware processors for processing an RGB image of the scene in the visible wavelength range in conjunction with an IR image of the scene in the infrared (IR) wavelength range. The processing operations include: converting the RGB image to a YUV image; identifying glare regions in the YUV image based on luminance components of the YUV image; identifying corresponding regions in the IR image for the glare regions in the YUV image; and replacing pixel values ​​in the glare regions with pixel values ​​of the corresponding regions in the IR image, based on the luminance components of the YUV image. The system outputs a glare-free image and facilitates real-time glare removal.

[0043] Example 12: The system in Example 11 may further include: an RGB camera configured to acquire the IR image; and an IR camera configured to acquire the IR image.

[0044] Example 13: In the system described in Example 11 or Example 12, the processing operation may further include: adjusting the pixel values ​​in the glare region of the YUV image based on the pixel values ​​in the region surrounding the glare region.

[0045] Example 14: In the system of any of the examples 11-13, the one or more hardware processors may include one or more general-purpose processors configured by instructions stored in memory to perform the processing operations.

[0046] Example 15: In the system of Example 14, the instructions may be part of a user-space driver associated with the RGB and IR cameras and executed within the user space of the operating system.

[0047] Example 16: In the system of any of the examples 11-13, the one or more hardware processors may include a hardware-implemented image signal processor associated with the RGB camera and the IR camera.

[0048] Example 17. One or more machine-readable media storing processor-executable instructions for processing synchronously acquired YUV and IR image pairs to generate a glare-free image. When executed by a hardware processor, the instructions cause the hardware processor to perform the following operations: identifying glare regions in the YUV image based on a luminance component of the YUV image; identifying a corresponding region in the IR image for the glare regions in the YUV image; and replacing pixel values ​​in the glare regions with pixel values ​​from the corresponding region in the IR image, based on the luminance component of the YUV image.

[0049] Example 18. In one or more machine-readable media of Example 17, the operation may further include: adjusting pixel values ​​in the glare region based on pixel values ​​in the region surrounding the glare region in the chromaticity component of the YUV image.

[0050] Example 19. In one or more machine-readable media as claimed in claim 17 or 18, the glare region can be identified based on the luminance component of the YUV image by generating a binary bitmap indicating whether each pixel in the YUV image is a glare pixel; the corresponding region in the IR image can be identified by multiplying the IR image with the binary bitmap to create a first intermediate image; a second intermediate image can be generated by multiplying the luminance component of the YUV image with the complement of the binary bitmap; the first intermediate image and the second intermediate image can be added to generate the glare-free image; and the pixel values ​​of the glare region in the luminance component of the YUV image can be replaced with the pixel values ​​of the corresponding region in the IR image.

[0051] Example 20: In one or more machine-readable media of Example 19, bitmaps can be generated using machine learning models, including, for example, convolutional neural networks (CNNs).

[0052] Although embodiments have been described with reference to specific exemplary embodiments, it will be apparent that various modifications and changes can be made to these embodiments without departing from the broader scope of the invention. Therefore, the specification and drawings should be considered illustrative rather than restrictive. The drawings, which form part of this invention, illustrate specific embodiments in which the subject matter can be practiced by way of illustration rather than limitation. The illustrated embodiments have been described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be used and derived therefrom, allowing structural and logical substitutions and changes to be made without departing from the scope of this disclosure. Therefore, this specification should not be construed as restrictive, and the scope of the various embodiments is defined only by the appended claims and the full scope of their equivalents.

Claims

1. A method for generating a glare-free image, the method comprising: Acquire RGB images of scenes within the visible wavelength range; Acquire an IR image of the scene within the infrared (IR) wavelength range; Convert the RGB image to a YUV image; Based on the luminance component of the YUV image, identify the glare region in the YUV image; Identify the corresponding region in the IR image for the glare region in the YUV image; as well as In the luminance component of the YUV image, the pixel values ​​in the glare region are replaced by the pixel values ​​of the corresponding region in the IR image.

2. The method according to claim 1, further comprising: The pixel values ​​in the glare region are adjusted in the chromaticity component of the YUV image based on the pixel values ​​in the region surrounding the glare region.

3. The method according to claim 1, wherein, Identifying the glare region based on the luminance component of the YUV image includes: generating a binary bitmap that indicates whether each pixel in the YUV image is a glare pixel.

4. The method according to claim 3, wherein, The bitmap was generated using a convolutional neural network (CNN) model.

5. The method according to claim 3, wherein, Identifying the corresponding region in the IR image includes multiplying the IR image by the binary bitmap to create a first intermediate image.

6. The method according to claim 5, wherein, Replacing the pixel values ​​in the glare region of the luminance component of the YUV image with the pixel values ​​of the corresponding region in the IR image includes: multiplying the luminance component of the YUV image by the complement of the binary bitmap to create a second intermediate image, and adding the first intermediate image and the second intermediate image to create the glare-free image.

7. The method according to claim 1, wherein, The RGB image is a frame of a first video stream captured using an RGB camera, and the IR image is a synchronized frame of a second video stream captured using an IR camera.

8. The method according to claim 7, further comprising: Glare-free images are generated in real time based on additional synchronization frames from the first and second video streams.

9. The method of claim 1, further comprising displaying the glare-free image on a display device.

10. A system for generating glare-free images, comprising: One or more hardware processors are configured to process an RGB image of the scene in the visible wavelength range in conjunction with an IR image of the scene in the infrared (IR) wavelength range by performing operations including: Convert the RGB image to a YUV image; Based on the luminance component of the YUV image, identify the glare region in the YUV image; Identify the corresponding region in the IR image for the glare region in the YUV image; as well as In the luminance component of the YUV image, the pixel values ​​in the glare region are replaced by the pixel values ​​of the corresponding region in the IR image.

11. The system of claim 10, further comprising: An RGB camera, configured to acquire the RGB image; as well as An IR camera is configured to acquire the IR image.

12. The system according to claim 11, wherein, The one or more hardware processors include one or more general-purpose processors configured by instructions stored in memory to perform the operations.

13. The system according to claim 12, wherein, The instructions are part of the user-space driver associated with the RGB camera and the IR camera, and are executed within the user space of the operating system.

14. The system according to claim 10, wherein, The one or more hardware processors include a hardware-implemented image signal processor associated with an RGB camera and an IR camera configured to acquire the RGB image and the IR image.

15. A machine-readable medium storing one or more processor-executable instructions for processing synchronously acquired YUV and IR image pairs to generate glare-free images, the instructions, when executed by a hardware processor, causing the hardware processor to perform operations including: Based on the luminance component of the YUV image, identify the glare region in the YUV image; Identify the corresponding region in the IR image for the glare region in the YUV image; as well as In the luminance component of the YUV image, the pixel values ​​in the glare region are replaced by the pixel values ​​of the corresponding region in the IR image.

Images