Image Processing for Pixel Correction

Abstract

A method for correcting image data includes receiving image data associated with an image captured by an optical sensor array and determining positions of target pixels in the optical sensor array associated with defects of the imaging device. A first operating pixel is identified in a first region of the optical sensor array that overlaps with the position of the target pixel, and a second operating pixel is identified in a second region of the optical sensor array that overlaps with the position of the target pixel and a portion of the first region. A pixel value simulated for the target pixel is determined according to a weighted average of the first operating pixel and the second operating pixel. The simulated pixel value is applied to the target pixel in the image data.

Description

Technical Field

[0001] The present disclosure generally relates to image processing techniques for pixel correction, and more specifically, to image processing techniques for correcting pixels associated with defective regions of an imaging device.

Summary of the Invention

Means for Solving the Problems

[0002] According to one aspect of the present disclosure, a method for correcting pixels of a display includes receiving, by a processor, image data related to an image captured by an imaging device. The method further includes determining, by the processor, a position of a target pixel in the display associated with a defect of the imaging device. The method further includes identifying operating pixels adjacent to the target pixel. The operating pixels have a color common to the target pixel. The method further includes applying a kernel to a first region of the image data that overlaps the position of the target pixel. The method further includes calculating a sum of at least one type of value for each of the operating pixels in the first region, and storing, in a memory, the sum and the number of the operating pixels in the first region. The method further includes applying the kernel to a second region of the image data that overlaps the position of the target pixel. The method further includes determining a weighted average of the operating pixels based on the sum and the number of the operating pixels in the first region, and the values of the operating pixels in the second region. The method further includes applying a correction to the target pixel based on the weighted average.

[0003] According to another aspect of the present disclosure, a system for correcting image data includes an imaging device configured to capture the image data. A display is configured to present the image data. An image processor communicates with the display and the imaging device. The image processor is configured to receive the image data, determine the position of target pixels in the display associated with defects of the imaging device, and identify operating pixels adjacent to the target pixels. The operating pixels have a color common to the target pixels. The image processor applies a kernel to a first region of the image data that overlaps the position of the target pixels, calculates the sum of at least one type of value for each of the operating pixels in the first region, and is further configured to store in a memory the sum and number of the operating pixels in the first region. The image processor is further configured to apply a kernel to a second region of the image data that overlaps the position of the target pixels. The image processor is further configured to determine a weighted average of the operating pixels based on the sum and number of the operating pixels in the first region and the values of the operating pixels in the second region, and apply a correction to the target pixels based on the weighted average.

[0004] These and other configurations, advantages, and objects of the invention of the present disclosure will be further understood and recognized by those skilled in the art with reference to the following specification, claims, and accompanying drawings.

[0005] The drawings are as follows.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

DETAILED DESCRIPTION OF THE INVENTION

[0007] For the purposes of the description herein, terms such as "upper", "lower", "right", "left", "rear", "front", "vertical", "horizontal", and derivatives thereof are related to the present invention in the orientation shown in FIG. 2. Unless otherwise specified, the term "front" shall refer to the surface of the component on the side closer to the intended observer of the display mirror, and the term "rear" shall refer to the surface of the component on the side farther from the intended observer of the display mirror. However, it will be understood that the present invention may take various alternative orientations, unless otherwise explicitly stated. It will also be understood that the specific devices and processes illustrated in the accompanying drawings and described in the following specification are merely exemplary embodiments of the inventive concepts defined in the appended claims. Therefore, specific dimensions and other physical characteristics related to the embodiments disclosed in the present disclosure are not to be considered limiting, unless otherwise explicitly stated in the claims.

[0008] The terms "including", "comprises", "comprising" or other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes the recited elements does not include only those elements, but may include other elements not expressly recited or inherent to such process, method, article, or apparatus. The use of "including" after an element does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes that element, without further limitation.

[0009] The present disclosure generally provides an image processing apparatus that operates on image data captured by an imaging device. During operation, the image processing apparatus may generate a corrected image to mask or hide defective or dead portions (e.g., pixels or segments) of an image sensor of the imaging device from a display on a display screen. The process employed by the image processing apparatus provides a reduction in memory requirements while also being able to collect sufficient data to appropriately adjust pixels associated with defective portions of the image sensor. Generally, the image processing apparatus may perform pixel correction across a given region of an image (e.g., a 5×5 pixel region) using two line buffers in a single scan. Thus, implementation of the present disclosure may reduce the line buffers required to process and correct image data within a given region imaged due to a malfunction of the image sensor. For example, the present disclosure may cover a 5×5 pixel region including "dead" pixels using two line buffers instead of four line buffers. In this way, the image processing apparatus of the present disclosure may provide a more robust and cost-effective technique for generating accurate images from inaccurate or defective image sensors.

[0010] Referring now to FIG. 1, the image processing system 10 is implemented as a simplified schematic structure representing a vehicle 12. Although described with reference to the vehicle 12 (e.g., an automobile, an aircraft, a bus, a train, etc.), the system 10 may be implemented using various cameras or imaging systems that may be applied to portable devices as well as appliances to be adapted for various applications. In the particular example shown, the system 10 may be configured to process images inside 14 or outside 16 of the vehicle 12. Thus, the present disclosure may provide a flexible image processing system 10 suitable for various applications.

[0011] System 10 may include an imaging device 18 that includes an imaging device 20 and, in some cases, one or more illumination sources 22 for irradiating a spatial region within the field of view of the imaging device 20. During operation, the imaging device 20 may be configured to capture image data and supply the data to one or more displays. In some cases, the imaging device 20 may suffer from one or more impairments due to manufacturing problems or wear and tear due to use. For example, due to changes in the operating environment, vibration, etc., the imaging device 20 and corresponding components may be susceptible to damage and defects involving one or more dead pixels or defective pixels of an optical sensor or a photodiode array (e.g., CMOS, charge-coupled, etc.). During operation, such "dead state" or defective photosensor pixels may be identified by the controller 36 based on the corresponding pixel data read from the pixel array 48. In various implementations, the present disclosure may simulate pixel data associated with one or more defective pixels and display corrected image data that corrects and removes visible defects associated with the one or more defective pixels, thereby providing for the detection and correction of such defects. In this way, the system 10 may extend the operation of the imaging device 20 by enhancing the image data, extending the corresponding service life, and reducing repair costs.

[0012] The imaging device 20 may be configured to include an image sensor including a plurality of pixels such as a CMOS photodiode array configured to generate characteristic electrical signals corresponding to the wavelengths of light received by the image sensor. The imaging device 20 may be configured to communicate with an image processor 24 that processes images within a field programmable gate array (FPGA) 26. The FPGA 26 may include a block random access memory (BRAM) 28 having a configuration that may include individual blocks 30 for storing embedded memory in the FPGA 26. Each block 30 may have a configuration having a predetermined storage space (for example, 4, 8, 16, 32 kilobits, etc.). Generally, values related to image data (for example, characteristic electrical signals) may be configured to be stored by the BRAM 28 in the block 30 for each frame. Further, in some embodiments, the BRAM 28 may be configured to store a plurality of previous lines and the current frame of image data to enable manipulation of the image data based on the previously captured lines.

[0013] Referring further to FIG. 1, the interface 32, such as a human machine interface (HMI), a touch screen, a head-up display, etc., may be configured to communicate with the image processor 24 and may be configured to output image data to the display 34 of the interface 32. For example, as described with reference to FIG. 2, the interface 32 may be configured to stream video data or image data (the image data and / or video data shows the interior 14 or the exterior 16 of the vehicle 12) captured by the imaging device 20 on the display 34 of the rearview mirror assembly. The display device 34 may include a light emitting diode, a liquid crystal display (LCD), or another type of display device that presents colors at various light intensities using pixels to form an image on the display device 34. For example, the display 34 may be an RGB display 34 (e.g., a display having red, green, and blue pixels arranged in an array) corresponding to the image data captured by the light sensor of the imaging device 20. In some embodiments, the pixels correlate or generally correspond to a square grid of light sensors of the imaging device 20. In particular, the pixels may be configured to correspond to a group of light sensors aligned with a Bayer filter that arranges RGB color filters on the light sensors. In this way, each light sensor of the imaging device 20 may be configured to apply to a given wavelength or color of visible light, and its luminance or light intensity may be configured to be processed by the image processor 24. The image processor 24 may then be configured to communicate the image data associated with the light sensor to a specific pixel mapped to the light sensor. As further described, the image data may be configured to be processed by the image processor 24 before determining the pixel data of the individual pixels.

[0014] Referring further to FIG. 1, the image processing system 10 may further include a controller 36 that communicates with the interface 32 and the imaging device 18. The controller 36 may be configured to include a memory and a processor. The memory may be configured to store instructions that, when executed by the processor, cause the processor to control the illumination source 22 and / or the imaging device 20. For example, the controller 36 may control the shutter of the imaging device 20 and / or the flashing of the illumination source 22 or the brightness of the illumination source 22 to improve the quality of the image captured by the imaging device 20. The controller 36 may be configured to communicate with other devices (not explicitly illustrated) that may correspond to other aspects of the imaging device 18, such as an ambient light sensor, an infrared sensor, an infrared light source, etc. Thus, the controller 36 may employ other functions related to image capture and / or illumination not specifically described in the present disclosure.

[0015] Referring now to FIG. 2, the image processing system 10 may be incorporated into a rearview mirror assembly 38 configured to display an image of the exterior 16 of the vehicle 12, such as a rear portion of the exterior 16 of the vehicle 12. As illustrated, the image processing system 10 may be configured to receive image data related to the image captured by the imaging device 20. The image data may be processed in individual segments or columns 40, such as the vertical column 40 shown in FIG. 2. Although illustrated as a column 40 having a width of five units, the size of the column 40 may be different in other embodiments of the present disclosure. Generally, the scan of the pixel / image data may be completed in one or both directions 42, 44 of the display 34, such as the horizontal direction 42 and the vertical direction 44.

[0016] As described above, the system 10 may be configured to detect and correct one or more defective pixels or target pixels 60 of the imaging device 20, or may be configured to simulate image data associated with the target pixels 60 and present it for viewing on the display 34. During operation, the controller 36 may be configured to detect a failure of one or more photosensor pixels of the pixel array 48 in response to a read signal indicating one or more faults. Examples of photosensor pixel failures may include dead pixels, stacked pixels, or other permanent or temporary faults resulting from the detection of image data representing a scene imaged by the imaging device 20. Such failures may occur, for example, from a photodiode that has stopped receiving power, a pixel that cannot change color or brightness, and the like. The controller 36 may associate such failures with specific addresses within the pixel array 48 that generally cannot respond to changes in the read signal over time, or that cannot read dynamic or varying values in the same manner as a group of adjacent or neighboring pixels over time.

[0017] The detection of one or more defective pixels or detection in the pixel array may be configured to be mainly detected by the controller 36 by monitoring the readout values over time. For example, the defective operation of one or more photosensor pixels may be compared with one or more pixels in the same region or the same block, and detected according to the fact that it does not change dynamically in response to the change in the readout value within the field of view. The color or luminance of one or more of the pixels may not change in response to the condition that it changes dynamically as specified by one or more adjacent pixels over time. By monitoring the output read values associated with the operation of each of the photosensor pixels in the pixel array 48, the controller 36 identifies one or more pixels that do not change over time or change differently from neighboring pixels, and attributes such abnormal behavior to the fact that the pixels in the array are malfunctioning or otherwise blocked due to contamination or debris attached to the camera device. Such a failure state can be monitored at a desired period. In response to the detection of the failure state, the controller 36 may be configured to identify the address of the corresponding pixel as that of the failure state and identify the pixel as the target pixel 60 for pixel correction.

[0018] Referring further to FIG. 2, an example of correcting or simulating the image data associated with the target pixel 60 will be described in more detail. As illustratively shown, the kernel 46 is applied to the image data successively row by row from the top of the image to the bottom of the image to process the image data vertically (44). The kernel 46 may be configured to scan pixel data and store the pixel data in the block 30 of the BRAM 28. For example, the kernel 46 may be a convolution matrix or mask that images a specific region (e.g., a 3×5 unit region of pixels) of the pixel array 48 that includes at least one line buffer 49 of the image data. Based on the pixel data stored in the BRAM 28, the target pixel 60 (e.g., a damaged or dust-covered photosensor) associated with the defect of the imaging device 18 may be updated to reflect a possible representation of the object in the captured image. The illustrated and described target pixel 60 is associated with a defective, dead, or malfunctioning portion of the imaging device 20 and may be referred to as a "dead" or "defective" pixel.

[0019] In the illustrated example, the area scanned by the kernel 46 is a 5×5 pixel array 48 of column 40 having a first area 50 and a second area 52 overlapping the first area 50 in the overlapping area 54. The overlapping area 54 may be the bottom line of a 3×5 matrix (e.g., the kernel 46) that the kernel 46 scans or passes through in two scans 56, 58. In the first scan 56 by the kernel 46, the pixels in the range of (0,0) to (3,5) are scanned, and the corresponding image data is stored in the BRAM 28. For example, the photometric value associated with the red pixel at (0,0) in the pixel array 48 may be stored in a 32-bit word together with the luminance or photometric values of some or all of the other pixels in the 3×5 kernel 46. In addition to the individual pixel values, the average or weighted average of the pixels having a similar type (e.g., color) for the kernel 46 area may be stored in the BRAM 28 in a 32-bit word or other word. In this way, for the pixel associated with the defective photosensor (e.g., the target pixel 60 at (3,3)), the pixel value associated with the target pixel 60 may be adjusted or corrected using the average of the surrounding red pixels. For example, the pixels at positions (0,0), (0,2), (0,4), (2,0), and (2,4) may be averaged, or otherwise statistically correlated with the target value of the target pixel 60 after the first scan 56. Then, the image processor 24 may be configured to capture the target value, process the image data, and output pixel data different from the values captured by the imaging device 20.

[0020] Specifically referring to the second pass 58 of the kernel 46 on the pixel array 48, the second region 52 includes pixels in the range of, for example, (2, 0) to (4, 4), and overlaps with the first region 50 by the common overlapping region 54. Similar to the process following the first scan 56, the image processor 24 may capture the pixels around the second region 52 having the same type of pixel data as the target pixel 60, and adjust the target pixel 60 to the target value. In this way, the target value may be updated based on the pixel data collected in the second pass 58. For example, the pixel data related to the pixels at positions (2, 0), (2, 4), (4, 0), (4, 2), and (4, 4) may be incorporated into the algorithm used by the processor to adjust the target value to correct the target pixel 60.

[0021] When the pixel data of the pixels of the same type (e.g., color) as the type of the target pixel 60 is determined from both the first scan 56 and the second scan 58, the weighted average of this value (e.g., luminance, light intensity, and amount / quantity) may be used as the target value. By overlapping the first scan 56 and the second scan 58 along the common overlapping region 54 where the target pixel data is captured twice, the "good" pixels in the pixel array 48 may be used to correct the "defective" target pixel 60. Further, the amount of image data stored in the BRAM 28 may be less than the amount of image data that needs to be stored in the BRAM 28 when the kernel 46 is large (e.g., a 5×5 region). In this way, the image processing system 10 of the present disclosure can provide a more efficient and robust image correction process.

[0022] In some embodiments, the algorithm employed may be a hybrid algorithm that stores, in BRAM28, the total value of the surrounding similar type pixel data in a 32-bit word, along with information indicating the number of scanned pixels (e.g., the number of scanned red pixels), between the first pass 56 and the second pass 58. During the second scan 58 (after passing through the two line buffers 49 in the second scan 56), the kernel 46 may be configured to image the pixel values associated with additional similar type pixels in the second region 52 of the second scan 58, and may be configured to add the values of these similar type pixels to a 32-bit word along with the number of additional scanned pixels, thereby forming the sum of the two regions 50, 52. Next, the image processor 24 may be configured to calculate (e.g., by averaging) the average value of the 5×5 pixel array 48 scanned twice by the kernel 46. In this way, the target value can be updated to incorporate the weighted average of the two scans and image a wider range of good pixels than may be possible with a 3×5 kernel 46 that does not include the overlapping region 54.

[0023] Although not shown by way of example, sensors corresponding to pixels of a similar type surrounding the "defective" target pixel 60 may also be defective optical sensors. Therefore, the surrounding pixels may not necessarily be reliable for the calculation of the target value. Image processing techniques can account for such defects by excluding the values of these dead pixel values by averaging or other calculations to determine the target value. Further, by employing a two-scan approach, a sufficient amount of pixel data of a similar type can be collected to properly correct the "defective" target pixel 60. Stated another way, the advantage of employing a larger kernel size can be verified without using as much storage / memory as would be required for a 5×5 kernel. In fact, the two-scan approach and corresponding method described in the present disclosure can limit the memory requirements to six blocks 30 of the BRAM 28 for a display 34 having a width or height of 1920 pixel size. For example, three pixel lines (e.g., two line buffers 49 and an overlapping line 54) each consuming two blocks 30 of the BRAM 28 can be stored for each scan of a column of pixels along either the horizontal direction 42 or the vertical direction 44 for a width or height of 1920 pixels. In contrast, a 5×5 kernel may consume ten blocks 30 of the BRAM 28 for a similar scan of a display of a similar size since five lines of pixels are imaged per scan.

[0024] Referring now to FIG. 3, a method for correcting pixels of a display includes a process 302 for identifying "dead" pixels (e.g., target pixel 60). In process 302, method 300 checks for the presence of dead pixels in step 304, and if detected, method 300 proceeds to step 306 of determining the location of one or more dead pixels based on the image data. The dead pixel identification routine may employ sensor pattern noise detection and / or other image processing techniques that can associate a consistent pixel value that does not correspond to the surrounding pixel values with the dead pixel. Next, image processor 24 may be configured to determine the location of the defective photosensor, thereby determining the location of dead pixel

[0025] In step 308, it may be configured that the operating pixels adjacent to the target / dead pixel 60 are identified by the image processor 24, and the operating pixels may have a color common to the target pixel 60. In the embodiment shown in FIG. 2, the red pixels surrounding the target pixel 60 may be operating pixels because they have a color common to the color of the target pixel 60. In step 310, it may be configured that the kernel 46 is applied to the first region 50 of the image data that overlaps the location of the target pixel 60. In step 312, it may be configured that the operating pixels within the first region 50 in a defective or dead state are also excluded from the calculations for determining the value of the target pixel 60 via steps 314 and 316. When the dead operating pixels of the first region 50 are removed from the calculation of the target value of the target pixel 60, method 300 proceeds to step 318 of storing the average value of the operating pixels of the first region 50 in memory. For example, it may be configured to store the average value in a 32-bit word within the BRAM28 block

[0026] In step 320, it may be configured to apply the kernel 46 to the second region 52 of the image data that overlaps with the position of the target pixel 60. For example, it may be configured to scan the overlapping region 54 shown in FIG. 2 twice with the kernel 46. In step 322, it may be configured to exclude the operating pixels in the defective or dead second region 52 from the calculation for determining the value of the target pixel 60 via steps 314 and 316. In step 324, it may be configured to calculate the weighted average of the operating pixels based on the average value and the values of the operating pixels in the second region 52. In step 326, it may be configured to apply a correction, update, or offset to the target pixel 60 based on the weighted average to generate a simulated pixel value in the image data. With the simulated pixel value, it can be ensured that the position of the target pixel represented on the display 34 is similar to the surrounding pixels and dynamically changes as the operating pixels in the vicinity of the position of the target pixel 60 change over time.

[0027] The correction may be configured to include applying the weighted average to the target pixel 60. In this way, it may be configured to apply the target value of the pixel based on the weighted average to correct the target pixel 60. The target value may be a luminance level, luminous intensity, or other things corresponding to a specific color corresponding to the pixel. Although it is described in relation to the red pixel in the embodiment illustrated in FIG. 2, the image correction technique can be applied to green, blue, or other color pixels. Also, in other embodiments, it may be considered to be configured to incorporate the luminance levels related to the surrounding pixels of a non-similar type.

[0028] Generally, the size (e.g., resolution) of the image captured by the imaging device 20 and / or the capacity of the image processor 24 can affect the amount of memory that can be freed by using the algorithms of the present disclosure. For example, a particular display 34 used may have a resolution in the range of 16 to 15,360 pixels by 16 to 15,360 pixels, or any combination thereof. Other resolutions beyond such ranges can also be utilized in the image processing techniques of the present disclosure. Further, the FPGA 26 may be configured with any amount of BRAM 28 capable of storing image data within such resolution ranges, and the amount of memory freed may be proportional or scalable to a particular resolution. In one embodiment where each row 40 is a maximum of 1920 pixels long, a total of four blocks 30 of BRAM 28 can be freed for other image processing techniques. For example, compared to the implementation of a 5×5 non-overlapping kernel 46, two blocks 30 of BRAM 28 per row 40 can be saved. For images having a resolution greater than 1920 pixels (e.g., 4K resolution) and / or an image processor 24 of a different size (e.g., larger), according to the algorithms of the present disclosure, more blocks 30 of BRAM 28 can be saved. Similarly, the present algorithm can also free blocks 30 of BRAM 28, although fewer, for an image processor 24 of a resolution and / or smaller size than a system for processing and presenting images at a resolution lower than 1920 pixels. Thus, the algorithms employed by the present disclosure may be scaled to account for different imaging device resolutions. It is further contemplated that the image processor 24 may be configured to employ additional kernels 46 for other image processing on the image prior to presentation on the display 34. Generally, by requiring less BRAM 28 for image processing, a smaller and / or less complex FPGA 26 may be employed, thereby resulting in a more efficient image processing apparatus.

[0029] According to some aspects of the present disclosure, a method for correcting image data captured by an optical sensor array includes receiving, by a processor, image data related to an image captured by the optical sensor array, and determining, by the processor, positions of target pixels in the optical sensor array associated with defects of the imaging device. Next, the method includes identifying a first operating pixel in a first region of the optical sensor array that overlaps with the position of the target pixel, and a second operating pixel in a second region of the optical sensor array that overlaps with the position of the target pixel and a portion of the first region. A simulated pixel value is determined for the target pixel according to a weighted average of the first operating pixel and the second operating pixel. Next, the simulated pixel value is applied to the target pixel in the image data.

[0030] According to various aspects, the present disclosure may provide one or more of the following configurations or settings in various combinations.

[0031] - The method is applied by a kernel operated by the processor. - Determining the simulated pixel value by calculating a first sum of a plurality of first pixel values for the operating pixels in the first region. - Further determining the simulated pixel value by storing, in a memory, the first sum and the first number of the operating pixels in the first region. - Determining the simulated pixel value by identifying a plurality of second pixel values for the operating pixels in the second region. - Determining the simulated pixel value as a weighted average based on the sum and the first number of the operating pixels in the first region, and the values of the operating pixels in the second region. - applying a correction to a target pixel value of the target pixel based on the weighted average, where the applying of the simulated pixel values is included; - the weighted average being a luminance level; - calculating a second sum of second pixel values for the active pixels within the second region; - determining a second number of active pixels within the second region, wherein the determination of the weighted average is further based on the second sum and the second number of the active pixels within the second region; - identifying the position of the target pixel in response to detecting at least one pixel failure within an array of photosensor pixels based on the image data; - detecting the at least one pixel failure in response to a first read value of one of the positions of the active pixels not conforming to a plurality of second read values associated with other active pixels spatially adjacent to one of the active pixels; and / or - detecting the at least one pixel failure in response to a pixel read value of one of the positions of the active pixels not changing over time.

[0032] According to another aspect of the present invention, a system for correcting image data includes an imaging device configured to capture the image data, a display configured to present the image data, and an image processor that communicates with the display and the imaging device. The image processor is configured to receive the image data and determine the position of target pixels in the display associated with the defects of the imaging device. Next, the processor may be configured to identify a first operating pixel in a first region of the photosensor array that overlaps the position of the target pixel, and a second operating pixel in a second region of the photosensor array that overlaps the position of the target pixel and a portion of the first region. Next, a simulated pixel value for the target pixel is determined according to the weighted average of the first operating pixel and the second operating pixel. Next, the imaging device can display the simulated pixel value on the target pixel in the image data on the display.

[0033] According to various aspects, the present disclosure may provide one or more of the following configurations or settings in various combinations.

[0034] - Determining the simulated pixel value by calculating a first sum of a plurality of first pixel values for the operating pixels in the first region; - Saving the first sum and the first number of the operating pixels in the first region; - Determining the simulated pixel value by calculating a second sum of a plurality of second pixel values for the operating pixels in the second region; - Saving the second sum and the second number of the operating pixels in the second region; - Determining the simulated pixel value as a weighted average based on the first sum, the first number, the second sum of the operating pixels in the first region, and the second sum and the second number of the operating pixels in the second region; - applying the simulated pixel value to the target pixel in the image data includes applying a correction to the target pixel value of the target pixel based on the weighted average, and / or - the determination of the position of the target pixel is specified in response to detecting at least one pixel failure in an array of photosensor pixels based on the image data.

[0035] According to yet another aspect of the present invention, a system for correcting image data received from an imaging device includes a display configured to present the image data and a processor in communication with the display and the imaging device. The processor receives the image data from a plurality of operating pixels forming the image data and determines the position of a target pixel in the display associated with a defect of the imaging device. Next, the processor calculates a first sum of a first number of first pixel values for the operating pixels in a first region that overlaps the position and stores the first sum and the first number of the operating pixels in the first region. A second sum is calculated for a second number of second pixel values for the operating pixels in a second region that overlaps the position and a portion of the first region. The processor then calculates a weighted average based on the first sum, the first number, the second sum, and the second number. Next, a simulated pixel value is generated for the target pixel based on the weighted average. The simulated pixel value is displayed at the position of the image data on the display.

[0036] It will be understood that any of the processes described or any of the described steps within the described processes that form a structure in combination with other disclosed processes or steps are also within the scope of the present invention. The exemplary structures and processes disclosed in this disclosure are for illustrative purposes and are not to be construed as limiting.

[0037] Moreover, modifications and changes can be made to the above-described structure and method without departing from the concept of the present invention. Further, it should be understood that such a concept is intended to be encompassed by the following claims, unless otherwise explicitly stated in the claims in words.

[0038] The above description is considered to be only for explaining the illustrated embodiments. Those skilled in the art or those who manufacture or implement the present invention will come up with variations of the present invention. Therefore, the embodiments shown in the drawings and the above description are for illustrative purposes only and are not intended to limit the scope of the present invention. It should be understood that the scope of the present invention is defined by the following claims, which are interpreted in accordance with the principles of patent law including the doctrine of equivalents.

Claims

1. A method for correcting image data captured by an optical sensor array, comprising: Receiving, by a processor, image data related to an image captured by the optical sensor array; Determining, by the processor, positions of target pixels in the optical sensor array associated with defects of the imaging device; Identifying first operating pixels in a first region of the optical sensor array that overlap with the positions of the target pixels; Identifying second operating pixels in a second region of the optical sensor array that overlap with the positions of the target pixels and a portion of the first region; Determining a simulated pixel value of the target pixels according to a weighted average of the first operating pixels and the second operating pixels; Applying the simulated pixel value to the target pixels in the image data.

2. The method according to claim 1, applied by a kernel operated by the processor.

3. The method according to claim 1 or 2, wherein the simulated pixel value is determined by calculating a first sum of a plurality of first pixel values for the operating pixels in the first region.

4. The method according to claim 3, wherein the simulated pixel value is further determined by storing, in a memory, the first sum and the first number of the operating pixels in the first region.

5. The method according to claim 4, wherein the simulated pixel value is determined by identifying a plurality of second pixel values for the operating pixels in the second region.

6. The method according to claim 5, further comprising determining the simulated pixel value as a weighted average based on the sum, the first number of the operating pixels in the first region, and values of the operating pixels in the second region.

7. The method according to claim 6, wherein the applying of the simulated pixel value comprises applying a correction to a target pixel value of the target pixels based on the weighted average.

8. The method according to claim 6, wherein the weighted average is a luminance level.

9. The method according to claim 6, further comprising calculating a second sum of second pixel values for the operating pixels in the second region.

10. Determining a second number of operating pixels in the second region, wherein determining the weighted average is further based on the second sum and the second number of the operating pixels in the second region. The method according to claim 4, further comprising determining.

11. The method according to any one of claims 1 to 10, wherein the position of the target pixel is identified in response to detecting at least one pixel failure in an array of photosensor pixels based on the image data.

12. The method according to claim 11, wherein the at least one pixel failure is detected in response to a first read value at the position of one of the operating pixels not matching a plurality of second read values associated with other operating pixels spatially proximate to one of the operating pixels.

13. A system for correcting image data, An imaging device configured to capture image data, A display configured to present the image data, An image processor in communication with the display and the imaging device, Receiving the image data, Determining the position of a target pixel in the display associated with a defect of the imaging device, Identifying first operating pixels in a first region of the photosensor array that overlap the position of the target pixel, Identifying second operating pixels in a second region of the photosensor array that overlap the position of the target pixel and a portion of the first region, Determining a simulated pixel value of the target pixel according to a weighted average of the first operating pixel and the second operating pixel, An image processor configured to display the simulated pixel value at the target pixel in the image data on the display. A system comprising.

14. The system according to claim 13, wherein the simulated pixel value is determined by calculating a first sum of a plurality of first pixel values for the operating pixels in the first region and storing the first sum and the first number of the operating pixels in the first region.

15. The system according to claim 13 or 14, wherein the simulated pixel value is determined by calculating a second sum of a plurality of second pixel values for the active pixels in the second region and storing the second sum and the second number of the active pixels in the second region.

Images