Image processing method

Abstract

An image processing method is operable in an image capture system comprising a camera arranged to capture successive images having a field of view, FOV, of a portion of a vehicle's surroundings, the FOV intersecting a window of the vehicle, the window comprising one or more heater elements visible within the FOV of the camera. The method comprises: obtaining an image; determining whether one or more sequences of pixels within the image correspond to an image of a respective heater element, each pixel within the or each sequence of pixels having a colour and intensity within respective thresholds; correcting the or each sequence of pixels within the image by replacing pixel values of the or each sequence of pixels corresponding to a respective heater element with pixel values derived from pixels not corresponding to a heater element; and displaying the corrected image.

Description

Technical Field

[0001] This invention relates to an image processing method. Background Technology

[0002] Currently, vehicles are equipped with driver assistance systems that rely on image acquisition systems. Such image acquisition systems may include one or more cameras coupled to a processor, which can display the acquired images on a driver's display. The processor can also use such images to identify road markings or three-dimensional objects in a portion of the vehicle's surrounding environment and assist in the autonomous or semi-autonomous control of the vehicle.

[0003] A camera can be located inside the vehicle to capture images of a portion of the vehicle's surroundings. Heating elements can be placed where the windshield (or window) intersects with the camera's field of view (FOV) to reduce fogging or remove ice from the windshield. While this can significantly improve visibility of that portion of the windshield, these heating elements are captured by the camera, and they appear as corresponding lines in the acquired image. When displaying or processing the image, these lines may undesirably reduce image sharpness.

[0004] US 2016 / 0091714 provides a solution in which the heater element is an induction heater element of ferromagnetic material located outside the camera's field of view (FOV). This heater element is configured to direct heat to the portion where the window intersects with the camera's FOV. This approach requires modification to the type of heater element and a specific arrangement between the heater element and the camera, which impacts the overall cost and complexity of the vehicle image acquisition system. Summary of the Invention

[0005] According to the first aspect, this application provides a method operable in an image acquisition system to provide a corrected image.

[0006] In a second aspect, an image acquisition system configured to perform the above-described methods is provided.

[0007] In another aspect, a computer program product is provided, which, when executed on a computing device, is configured to perform the methods described above. Attached Figure Description

[0008] Embodiments of the invention will now be described by way of example with reference to the accompanying drawings, wherein:

[0009] Figure 1 A vehicle including an image acquisition system for improving image quality is shown according to an embodiment of the present invention;

[0010] Figure 2 It shows Figure 1An image of a portion of the vehicle's surroundings; and

[0011] Figure 3 This is a block diagram illustrating a method for improving image quality according to an embodiment of the present invention. Detailed Implementation

[0012] Figure 1 A vehicle 102 is illustrated with an image acquisition system 100 for improving image quality, according to an embodiment of the present invention. The image acquisition system 100 includes a camera 110 and a processor 120 for processing images captured by the camera 110. The camera 110 can be any type of image capture device. For example, the camera can be of a type commonly referred to as a digital camera, such as a complementary metal-oxide-semiconductor (CMOS) camera, a charge-coupled device (CCD) camera, etc. The camera 110 can capture images in the visible and / or invisible spectrum (e.g., the infrared spectrum) and provide these images to the processor 120. The processor 120 processes the images using any combination of color or intensity image plane information provided by the camera 110.

[0013] In this embodiment, camera 110 is a forward-facing camera mounted on the forward-facing surface of a rearview mirror (not shown) of vehicle 102. Camera 110 is configured to capture a downward-looking ground plan view facing forward of vehicle 102 and through the windshield 104 of vehicle 102. The FOV 112 of camera 110 intersects a portion of the windshield 104 of vehicle 102, including heater elements 130-1…130-n. It should be understood that camera 110 can be mounted at any desired location within the vehicle to capture images viewed from vehicle 102 and through the windshield 104 of vehicle 102 in a generally forward direction. Thus, for example, camera 110 can also be located within an overhead console unit (not shown) inside the passenger compartment of vehicle 102.

[0014] Alternatively, the camera can be mounted in any desired location within the vehicle 102 to capture a downward view of the ground plan facing the rear of the vehicle 102, or to capture an image viewed from the vehicle 102 through the rear windshield 106 in a generally rearward direction. For example, the camera can also be mounted inside the passenger compartment of the vehicle 102, and preferably on the rear windshield 106. Thus, the camera's field of view (FOV) intersects with a portion of the rear windshield 106 of the vehicle 102, which includes a heater element similar to element 130 on the windshield 104.

[0015] In this embodiment, the portion where the windshield 104 intersects with the FOV 112 of the camera 110 includes a heater element 130 embedded in the glass of the windshield 104. The heater element 130 may alternatively be fixed or formed on the surface of the windshield to provide the desired function. The heater element may be of inductive and / or resistive type.

[0016] Heater element 130 includes one or more linear heater elements 130-1, 130-2, 130-n that are visible within the spectrum of camera 110. In this embodiment, the heater elements are visible, and camera 110 captures images within the visible spectrum. Alternatively, or additionally, the heater elements may be visible only in the invisible spectrum, for example, only when they are heated, and the camera may be configured to capture infrared images.

[0017] In this embodiment, heater elements 130-1, 130-2, and 130-n are solid lines extending along a line parallel to the longitudinal axis of the windshield 104 and along its substantially entire length of the windshield 104. In an alternative embodiment, as described below, heater elements 130-1, 130-2, and 130-n may have different characteristics. For example, heater elements 130-1, 130-2, and 130-n may extend wholly or partially along the axis of the windshield and may have any orientation relative to that axis. Heater elements 130-1, 130-2, and 130-n may have any predetermined shape, such as zigzag lines, curves, straight lines, or any combination thereof; color; color intensity; and pattern, such as varying thickness, or any combination thereof.

[0018] like Figure 2 As shown, the heater elements 130-1, 130-2, and 130-n intersecting with the FOV 112 of camera 110 are visible within the imaging spectrum of camera 110 and are therefore captured by camera 110 on corresponding lines of images 230-1, 230-2, and 230-n. Specifically, heater elements 130-1, 130-2, and 130-n are projected onto the image plane according to the orientation and angular position of camera 110 relative to windshield 104. In this embodiment, it is assumed that camera 110 is aligned with windshield 104 such that the corresponding imaging heater lines 230-1, 230-2, and 230-n extend along corresponding lines of the image parallel to the longitudinal X-axis of image 200 and along the entire length of image 200.

[0019] In another alternative, the rear windshield 106 may include heater elements as one or more heater lines, which may be embedded, formed, or fixed to the glass of the rear windshield 106 to reduce fogging or frost formation on that portion of the rear windshield 106. In this case, the camera captures an image while viewing the vehicle 102 in a generally rearward direction through that portion of the rear windshield 106. Similarly, heater elements visible within the camera's spectrum and intersecting with the camera's field of view (FOV) appear as corresponding imaged heater lines in the camera image.

[0020] In a further variation, the vehicle window through which the camera acquires images can be either a side window or a dedicated window. The window can be made of transparent, translucent, or a combination thereof, allowing images of the surrounding environment to be captured through the window.

[0021] In any case, it should be understood that when displayed on the driver's display (not shown), the image lines corresponding to the heater element may reduce the quality of the image.

[0022] Individually, such image artifacts in the acquired images can cause problems for driver assistance systems based on this image analysis.

[0023] Figure 2 Image 200 is part of the environment surrounding vehicle 102. Parts of the environment surrounding vehicle 102 include lane markings, such as edge lines 220 defining the road surface on which vehicle 102 can navigate, and stop lines 240. Image lines 230-1, 230-2, 230-n may be processed as potential features of image 200. For example, the system may consider one or more pixels of image lines 230-1, 230-2, 230-n as stop lines 240.

[0024] In this image acquisition system 100, the processor 120 is tasked with detecting and removing the imaged heater lines 230-1, 230-2, and 230-n before displaying the corrected image on the driver's display. Furthermore, the processor 120 can perform parking assistance functions and / or autonomous or semi-autonomous driving, such as active cruise control or virtually any other vehicle function. Alternatively, the processor 120 may include a dedicated processor for detecting and removing the imaged heater lines 230-1, 230-2, and 230-n; or in fact, the processor 120 may include any processor with multiple processor cores within the vehicle control system, and this processor may be used as needed to improve the acquired image.

[0025] In any case, the processor 120 is programmed to process digital image data representing an image captured by the camera 110 to detect defective pixels in the image corresponding to the imager lines 230-1, 230-2, 230-n, and to correct defective pixels in the image.

[0026] Now for reference Figure 3 The diagram illustrates a block diagram of an image processing method for improving acquired images according to an embodiment of the present invention.

[0027] In step 1, the image 200 from camera 110 is fed to processor 120 of image acquisition system 100. Image 200 includes one or more imaging heater lines 230-1, 230-2, 230-n corresponding to the respective heater elements 130-1, 130-2, 130-n. (Note that, in practice, image 200 corresponds to a region of interest (ROI) selected from the FOV 112 of camera 110, and this ROI has been transformed into a flat rectangular image of the scene.)

[0028] In step 2, the processor 120 scans the image to detect pixels within image 200 corresponding to heater elements 130-1, 130-2, 130-n, to detect sequences of any adjacent pixels in the horizontal lines (rows) of the image that have characteristics corresponding to predetermined characteristics of the imaged heater lines. In this embodiment, the processor 120 determines whether each pixel within the pixel sequence has a color within a given threshold and an intensity below a given threshold. In this embodiment, the color threshold corresponds to a predetermined chromaticity of the imaged heater lines, such as black (the lowest chromaticity value), and low intensity. Nevertheless, it should be understood that other heater lines may have strong non-black chromaticity components, such as orange hues, and appropriate thresholds can be set to determine whether a pixel has that color.

[0029] Therefore, it should be understood that it may be beneficial to provide image information to processor 120 in an intensity / chroma format (e.g., YUV or LAB space) so that intensity can be checked against scalar values ​​first, and then chroma can be checked only if the pixels meet the intensity threshold.

[0030] Therefore, if a given pixel (pixel [x]) and consecutive pixels (where n≥1 pixels [x+n]) in an image row have a given color and are dark, then the first pixel and the consecutive pixels satisfy the criterion. In response, the counter is incremented by one unit. This process is repeated for multiple consecutive pixels in the image row until a minimum threshold is reached. The minimum threshold is a predetermined threshold that is less than the total number of pixels in the image row of image 200. For example, in this embodiment, the image width is 1024 pixels, and the maximum value of the minimum threshold is 1023 (image width – 1). When the counter reaches the minimum pixel threshold that satisfies the criterion, it is determined that the pixel sequence corresponds to the image heater line 230. The process continues along the image line until the last adjacent pixel that satisfies the color / intensity criterion is detected. Then, all pixels along the image heater line are marked accordingly for subsequent correction.

[0031] For example, in Figure 2 In the image line 230-1, the processor 120 determines that the first pixel is dark and its intensity is within a predetermined intensity threshold. Therefore, the first pixel is classified as a potential pixel corresponding to the image line 230-1 of the heater element 130-1. The processor selects consecutive pixels on the image line and determines that these consecutive pixels are dark and their intensity is within a predetermined intensity threshold. Therefore, these consecutive pixels are classified as another potential pixel of the image line 230-1, and the processor increments a counter value. Once the counter reaches a minimum threshold, it is determined that the pixel sequence (and any further adjacent heater line pixels on that line) corresponds to the heater line 230-1 representing the imaging of the corresponding heater element 130-1.

[0032] Conversely, if the first pixel classified as a potential pixel for image line 230-1 is a pixel for parking line 240, then at least one subsequent pixel in that line will not have the color and intensity within the corresponding threshold until the counter reaches its minimum threshold. Therefore, the pixel sequence will not be classified as a heater line representing imaging. Thus, the process is repeated starting from subsequent pixels as long as the number of remaining pixels in a row is greater than the pixel threshold required for imaging a heater line.

[0033] Similarly, if the first pixel classified as a potential pixel for image line 230-1 is the rear bumper 252 of truck 250, then at least one consecutive pixel will not have the color and intensity within the corresponding threshold before the counter reaches the minimum threshold. Therefore, the pixel sequence will not be classified as representing an imaging heater line. Likewise, the process is repeated starting with subsequent pixels as long as the number of remaining pixels in a row is greater than the pixel threshold required for an imaging heater line. (Thus, it will be seen that if the threshold is high, only a small fraction of pixels within any given image will need to be tested to determine whether they belong to an imaging heater line.)

[0034] In step 3, one or more pixel sequences within image 200 identified in step 2 as corresponding to the imaging heater lines 230-1, 230-2, 230-n are corrected. Specifically, in this embodiment, processor 120 uses information from pixels adjacent to the pixels identified as corresponding to the imaging heater lines 230-1, 230-2, 230-n to correct the pixels corresponding to the imaging heater lines 230-1, 230-2, 230-n.

[0035] In a simple implementation, this can be done by locating the nearest pixel that does not belong to the corresponding imaging heater line for each pixel of each imaging heater line 230-1, 230-2, 230-n, and assigning the value of the nearest pixel to the pixel value of the imaging heater line.

[0036] Alternatively, correction can be performed using average, linear, or polynomial extrapolation of multiple neighboring pixels, or by interpolation of pixels surrounding pixels belonging to the heater line of the image.

[0037] Therefore, when a given pixel from an imaging heater line is adjacent to other pixels from an imaging heater line, it may be desirable to extrapolate image information from non-heater line pixels located to one side of the given pixel.

[0038] On the other hand, if the image heater line extends only along one line or a minimum number of lines of a given image, it may be more appropriate to interpolate or average image information from the pixels above and below the image lines corresponding to the image heater line.

[0039] Therefore, even if the imaged heater line occupies more than one neighboring row of the image, it is still possible to interpolate image information from either side of the imaged heater line by weighting the pixel information based on its distance from a given heater line pixel.

[0040] Each of these implementations includes a general linear type of correction for a given image; however, it should be understood that, using a properly trained classifier, pixels from within a given image can be used to correct defective pixels in a non-linear manner, thereby allowing for potentially more intelligent image correction.

[0041] Once completed, the image corrected according to step 3 can be provided for display on the driver's display and may also be provided to other applications or processing modules for driver assistance systems.

[0042] This process can then be repeated for consecutive images in the image stream as needed.

[0043] It should be understood that further variations of the above embodiments are possible.

[0044] For example, the above embodiments do not assume the location of the imaging heater line in any acquired image, but only assume that the neighboring pixels of the imaging heater line are expected to be on the same line of the image. This allows the location of the imaging heater line to change in response to changes in environmental conditions.

[0045] In an alternative embodiment, when installed in a vehicle, the system can be calibrated once by first capturing a blank image, such as a blank image of a white card. Then, any pixels in the acquired calibrated image not identified as white can be marked as including the heater line in the image. This avoids the need to scan each acquired image as described above regarding step 2.

[0046] Then, the processor 120 will only need to correct the pixels in subsequent images acquired during vehicle operation based on the position of such pixels that were identified during calibration as corresponding to the heater line of the image.

[0047] A more sophisticated method for identifying these pixels could be designed to feed continuously acquired images to a classifier that identifies rows of pixels within the acquired images whose chromaticity and intensity correspond substantially in time to the imaging heater lines. Such a classifier could include any suitable classifier, including those based on neural networks.

[0048] In practice, this classifier can provide a non-binary mapping for pixels corresponding to the imaged heater line, where some pixels are more likely to be images of the heater line than others. Therefore, signaling may indicate that different degrees of correction can be applied to different pixels of the imaged heater line.

[0049] It should also be understood that although the above embodiments have been described from the perspective of correcting images based on information from the images themselves, image information can also be extracted from one or more directly previously acquired images, or even from subsequently acquired images (if available before displaying the given previously acquired images), and image information can be determined from these images using vehicle odometry, which are occluded by the heater element 130 in the given image but not in other acquired images, to correct pixels in the given image.

[0050] Finally, the above embodiments are described under the assumption that the heater element is linear and horizontal within each acquired image. On the other hand, if such an element has a more complex shape, then identifying the scan path along the imaging heater line to determine neighboring pixels may be more complex than simply proceeding along a line of the image. The shape, pattern, and / or orientation of such imaging heater line can be stored in memory accessible to the processor 120, enabling the processor to determine where the next expected pixel of the imaging heater line should be located once it has found one or more neighboring pixels of the imaging heater line.

[0051] It should also be understood that in some implementations, the heater elements do not need to be continuous, and similarly, the processor 120 may also take this into consideration.

Claims

1. An image processing method operable in an image acquisition system (100) of a vehicle (102), the image acquisition system (100) including a camera (110) arranged to capture continuous images (200) of a field of view (FOV) (112) having a portion of the environment surrounding the vehicle (102), the FOV (112) intersecting with a window of the vehicle (102), the window including one or more heating elements (130-1, 130-2, 130-n) visible within the FOV (112) of the camera (110), the method comprising: Obtain (1) image (200); Determine (2) whether one or more pixel sequences within the image (200) correspond to the image of the corresponding heater element, each pixel in the one or more pixel sequences having a color and intensity within a corresponding threshold; Determining whether one or more pixel sequences within the image (200) correspond to the image of the corresponding heater element includes: Determine the shape of the one or more heater elements; and A scan is performed on the one or more pixel sequences within the image (200) along a scan path corresponding to the shape; Furthermore, determining the shape of the corresponding heater element includes: The image acquisition system (100) is calibrated using the following steps: (i) Capture scene images with constant colors, (ii) Marking pixels not identified as having the constant color as pixels of the one or more heater elements, (iii) Determine the shape corresponding to the pixel distribution of the one or more heater elements, and (iv) Store the shape in memory; and / or Retrieve the shape of the corresponding heater element from the memory; Within the image (200), the one or more pixel sequences are corrected (3) by replacing the pixel values ​​corresponding to the respective heater elements with pixel values ​​derived from pixels that do not correspond to the heater elements; and The corrected image is displayed.

2. The method of claim 1, further comprising providing the driver assistance system with one or both of the image and the corrected image.

3. The method according to claim 1, wherein, The one or more heater elements have a linear shape.

4. The method according to claim 3, wherein, The images of the one or more heater elements extend along corresponding lines of the image parallel to the longitudinal X-axis of the image (200) and along the entire length of the image (200).

5. The method according to claim 1, wherein, Performing a scan of the one or more pixel sequences within the image (200) along a scan path suitable for the shape includes: a) In response to determining a given pixel [i] and consecutive pixels [i+n] along the one or more pixel sequences of the scan path, where n≥1, having color and intensity within their respective thresholds, increment the counter by one unit; b) Repeat step a) and , for multiple consecutive pixels along the scan path. c) When the minimum threshold of the counter is reached, determine that the one or more pixel sequences in the image (200) correspond to the image of the corresponding heater element.

6. The method according to claim 5, further comprising: d) Repeat step b) until the last pixel in the one or more pixel sequences along the scan path has a color and intensity within the corresponding threshold; and e) Mark each pixel in the one or more pixel sequences for subsequent correction.

7. The method according to claim 1, wherein, The image (200) corresponds to a region of interest (ROI), which is selected from the field of view (FOV) (112) of the camera (110) and has been transformed into a flat rectangular image of the scene.

8. The method according to claim 1, wherein, A pixel that does not correspond to a heater element is a pixel adjacent to a pixel in the image (200) that corresponds to the corresponding heater element.

9. The method of claim 1, wherein replacing the pixel values ​​of the one or more pixel sequences with pixel values ​​derived from pixels that do not correspond to heater elements comprises: Interpolate pixel values ​​from pixels located on either side of the image of the respective heater element, and wherein the interpolation includes pixel values ​​weighted according to the distance from a given pixel of the respective heater element; or The average pixel value from pixels located on either side of the image corresponding to the heater element; or Extrapolate pixel values ​​from pixels located on one side of the image corresponding to the heater element; or Defective pixels are corrected in a non-linear manner by selecting pixels from the image (200) using an appropriately trained classifier.

10. An image acquisition system configured to perform the method according to claim 1.

11. The image acquisition system according to claim 10, wherein, The camera (110) captures images (200) in the visible and / or invisible spectrum.

12. A vehicle comprising an image acquisition system as defined in claim 10 and a window, the window comprising one or more heater elements (130-1, 130-2, 130-n) visible within the FOV (112) of the camera (110).

13. A non-transitory computer program product, when executed on a computing device, is arranged to perform the method according to claim 1.

Images