An image demosaicing method for autonomous driving

By detecting and correcting common color differences, adaptive filtering and color value correction are performed, the artifact problems caused by image de-mosaic algorithm in autonomous driving systems are solved, and the accuracy of the image and edge clarity are improved.

CN116630187BActive Publication Date: 2024-11-29北京辉羲智能科技有限公司
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202310576477.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-22
Publication Date
2024-11-29
Estimated Expiration
2043-05-22

AI Technical Summary

Technical Problem

Image demosaic algorithms in existing autonomous driving systems often lead to reconstruction errors, resulting in artifacts such as zipper effect, pseudo-color and aliasing, affecting the accuracy of obstacle detection and lane line recognition.

Method used

By calculating the color difference images △GB and △GR, detecting and correcting the color difference common points, performing adaptive filtering, and using the filtered color difference image to correct the color value to generate the final full-color image to reduce artifacts and color noise.

Benefits of technology

Effectively suppress edge pseudo-color, reduce color noise, improve edge clarity, and improve the accuracy and reliability of demosaic images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116630187B_ABST
    Figure CN116630187B_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of autonomous driving, and discloses an image demosaicing method for autonomous driving, which is characterized in that: an initial full-color image RGB is generated by a demosaicing interpolation method, color difference images △GB and △GR are calculated, recorded as color difference image Δ, and then the color difference abnormal points in the color difference image are detected and corrected, and then the corrected color difference image is adaptively filtered, and the filtered color difference image is recorded as Finally, the filtered color difference image is color-value corrected to obtain a final full-color image. The demosaicing method of the present invention can effectively remove edge pseudo-color caused by demosaicing interpolation, and improve color noise and edge blur caused by interpolation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of autonomous driving, and in particular to an image demosaicing method for autonomous driving. Background Art

[0002] The vehicle autonomous driving system needs to obtain image information around the vehicle through image sensors in order to perform tasks such as road condition judgment, obstacle detection, and lane line recognition. Common image sensors are covered with color filter arrays, and each pixel only collects part of the spectral information, such as red R, green G, and blue B light components. The demosaicing algorithm reconstructs the complete color image by supplementing the missing color information of each pixel. Since the missing color components need to be restored based on the single color of each pixel, reconstruction errors are inevitable, and common artifacts such as zipper effect, false color, and aliasing will be caused. In autonomous driving systems, these artifacts may lead to misjudgment of tasks such as obstacle detection and lane line recognition, so it is particularly important to improve the accuracy and reliability of the demosaicing algorithm. Summary of the invention

[0003] The present invention provides an image demosaicing method for autonomous driving, which solves the technical problems of inevitable reconstruction errors in existing algorithms, which commonly cause artifacts such as zipper effect, false color, aliasing, etc.

[0004] The present invention can be achieved through the following technical solutions:

[0005] An image demosaicing method for autonomous driving uses a demosaicing interpolation method to generate an initial full-color image RGB, calculates color difference images △GB and △GR, recorded as color difference image Δ, detects color difference abnormal points in the color difference image and corrects them, and then adaptively filters the corrected color difference image. The filtered color difference image is recorded as Finally, the color value of the filtered color difference image is corrected to obtain the final full-color image.

[0006] Further, color difference abnormal point detection

[0007] Calculate the gradient value gradis, is = H, V, 45 and 135 of the current color point in each color difference image along the horizontal, vertical, 45 degree and 135 degree oblique edge directions, as well as the difference diffis, is = H, V, 45 and 135 between the color difference value of the current color point and the color difference value of the adjacent color point along the horizontal, vertical, 45 degree and 135 degree oblique edge directions. Note that the gradient thresholds along the horizontal, vertical, 45 degree and 135 degree oblique edge directions are th1is, is = H, V, 45 and 135, the color difference difference thresholds are th2is, is = H, V, 45 and 135, and the brightness thresholds are th3is, is = H, V, 45 and 135.

[0008] If the three conditions of gradis>th1is, diffis>th2is, and maximum brightness lumamax>th3is in any direction are met at the same time, then the corresponding current pixel point is a color difference abnormal point;

[0009] Color difference correction

[0010] Select the direction with the largest gradient among all directions that meet the outlier detection conditions for correction. The correction method is to take the median along the direction:

[0011] △′=mediam{△ i-n ,…△ i ,…△ i+n}

[0012] Among them, △ i Indicates the color difference of the abnormal point, △ i-n ~△ i+n Represents the adjacent color difference along the direction, n is 1 or 2.

[0013] Furthermore, the color difference image is filtered with the initial full color image as a guide to remove the pseudo color while maintaining good edge clarity. The filtering formula is:

[0014]

[0015] Among them, Δ i,j Represents the color difference value of the pixel (i, j) in the local 5x5 or 7x7 window.

[0016] Indicates the establishment of a filter kernel function based on anisotropic Gaussian radial basis function, d i,j =[x i -x,y i -y] T Indicates the distance from pixel (i, j) to the current pixel.

[0017] represents the kernel covariance matrix, obtained through the local gradient structure tensor, Ix and I y Represents the local horizontal and vertical gradient values, and can be obtained Eigenvector v 1 Represents the local edge direction, with a larger eigenvalue λ associated with it 1 is the edge strength, the eigenvector v 2 represents the direction orthogonal to the local edge, and is associated with a smaller eigenvalue λ 2 .

[0018] Furthermore, the filtered color difference image is used as the optimized color difference image, and the demosaic interpolation method is reused to generate a new RGB interpolation to obtain the final full-color image.

[0019] Further, calculate the color difference correction value: Correct the color interpolation based on the color difference correction value, keep the original RAW input value unchanged, and only correct the interpolation results of other colors.

[0020] At the G pixel position, the corrected R and B channel values ​​are:

[0021]

[0022]

[0023] At the R pixel position, the corrected G and B channel values ​​are:

[0024]

[0025]

[0026] At the B pixel position, the corrected G and R channel values ​​are:

[0027]

[0028]

[0029] A computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the method described above.

[0030] The beneficial technical effects of the present invention are:

[0031] 1) By correcting the abnormal chromatic aberration in the edge area, the edge false color can be effectively suppressed.

[0032] 2) By filtering the color difference image in different regions with the color image as a guide, the color noise of the high-noise input image can be reduced.

[0033] 3) By filtering the color difference image along the direction guided by the color image, the edge color noise can be reduced and the edge clarity can be improved.

[0034] 4) The present invention corrects the pseudo-color based on the demosaiced image result, and a better image effect can be obtained by adding a small number of processing steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the overall process of the present invention.

[0036] Figure 2 It is a schematic diagram of the process of updating the demosaic output based on the correction value of the present invention.

[0037] Figure 3 It is a schematic diagram of the process of direct correction of color values ​​of the present invention. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings and preferred embodiments.

[0039] like Figure 1 As shown, the present invention provides an image demosaicing method for autonomous driving, which generates an initial full-color image RGB by using a demosaicing interpolation method, calculates color difference images △GB and △GR, detects color difference anomalies in the color difference image and corrects them, then adaptively filters the corrected color difference image, and finally corrects the color value of the filtered color difference image to obtain a final full-color image. In this way, the full-color image of each color channel generated by the demosaicing interpolation method in the prior art is used to calculate the color difference image, and then the color difference image is corrected to obtain a correction value, and the correction value is applied to the interpolation result, which can effectively remove the edge pseudo color caused by the demosaicing interpolation, and the color difference image filtering is guided by the full-color image, and the filtering direction and intensity are adaptively changed, which can improve the color noise and edge blur caused by the interpolation.

[0040] The details are as follows:

[0041] Step 1: De-mosaicing interpolation method to generate full-color image

[0042] In the existing image demosaicing algorithm, the G channel with the largest number of pixels is usually interpolated first, and then the R and B images are interpolated using the G image's better image resolution information. In order to make full use of each color channel information and reduce edge jaggedness caused by interpolation, the present invention uses a method combining color difference weighting and linear interpolation to calculate the G channel value:

[0043]

[0044] Among them, α is a weighting factor, which is determined according to the edge strength. The stronger the edge, the smaller the α value.

[0045]

[0046] Where C is the R or B pixel value, calculate the color difference ΔGC between the pixels around the current pixel, and weight the current pixel color difference, weight w i,j It is inversely proportional to the gradient from the neighboring pixel to the current pixel.

[0047]

[0048] The R and B channel interpolation is calculated according to the following formula:

[0049]

[0050]

[0051] Among them, w i,j The weight is calculated based on the gradient from pixel point (i, j) to the current pixel point. Pixel point (i, j) refers to the set of points in a window within a certain range centered on the current point.

[0052] Step 2: Chromatic Aberration Image Correction

[0053] Step 2.1: Calculate the color difference image

[0054] The color difference images used here include GB color difference image △GB and GR color difference image △GR, that is, color difference image △, which are obtained by subtracting the corresponding pixel values ​​of the G channel image and the R and B channel images of the full-size image corresponding to the full-color image after demosaic interpolation, that is,

[0055] ΔGB=GB

[0056] ΔGR=GR

[0057] Step 2.2: Detection and correction of abnormal points in chromatic aberration images

[0058] Areas where false colors appear after demosaic interpolation are usually edges with drastic changes in brightness. Since the R, G, and B colors change at different rates at the edges, one color value is greater than the other color channel values, which is displayed as an abnormal color point. It has a large difference in color difference with the surrounding color difference on the color difference image. Therefore, by identifying these abnormal points and correcting them with the surrounding color difference, the effect of removing abnormal color points can be achieved.

[0059] 1) Color difference abnormal point detection

[0060] Calculate the gradient value gradis of the current color point in each color difference image along the horizontal, vertical, 45 degree and 135 degree oblique direction, is = H, V, 45 and 135, and the difference between the color difference value of the current color point and the color difference value of the adjacent color point along the horizontal, vertical, 45 degree and 135 degree oblique direction, is = H, V, 45 and 135. The gradient can be calculated on the full color image after demosaic interpolation or on the original Bayer image, which is the first or second order derivative of the current pixel point and the adjacent pixel points along each direction. Usually, the color difference abnormal point appears in the direction with a larger gradient value, and its color difference is larger than that of other pixel points along the direction.

[0061] The gradient thresholds along the horizontal, vertical, 45 degree and 135 degree oblique edges are th1is, is=H, V, 45 and 135 respectively, the color difference thresholds are th2is, is=H, V, 45 and 135 respectively, and the brightness thresholds are th3is, is=H, V, 45 and 135 respectively.

[0062] In each direction, i.e. the above four directions, it is judged whether the conditions are met at the same time. The conditions are set as gradis>th1is, diffis>th2is, and the maximum brightness lumamax>th3is. If any direction meets the conditions, the corresponding current pixel point is a color difference abnormal point;

[0063] 2) Color difference correction

[0064] If it is judged as an outlier, the color difference is corrected along the direction in which the outlier is detected. When the judgment of multiple directions meets the conditions, the direction with the largest gradient among all directions that meet the outlier detection conditions is selected for correction. It can also be corrected along each direction that meets the conditions and then the results are averaged. The correction method is to take the median along the direction:

[0065] △′=mediam{△ i-n ,…△ i ,…△ i+n}

[0066] Among them, △ i Indicates the color difference of the abnormal point, △ i-n ~△ i+n Indicates the adjacent color difference along the direction, n is 1 or 2. If the subscript n is 1, it is 3 points, i-1 to i+1 centered on the current point. If n is 2, it is i-2 to i+2, which is 5 points.

[0067] Step 2.3: Chromatic Aberration Image Filtering

[0068] When the input image has large noise, color interpolation will introduce noise into other channels, which will appear as pseudo-color spots or patches on the image. Based on the color difference image corrected in step 2.2, the color difference image is filtered with the initial full-color image as a guide to filter out the pseudo-color while maintaining good edge clarity. The filtering formula is:

[0069]

[0070] Among them, Δ i,j Represents the color difference value of the pixel (i, j) in the local 5x5 or 7x7 window. In order to remove low-frequency color patches, a larger filter window can be used. Indicates the establishment of a filter kernel function based on anisotropic Gaussian radial basis function, d i,j =[x i -x,y i -y] T Indicates the distance from pixel (i, j) to the current pixel. represents the kernel covariance matrix, obtained through the local gradient structure tensor, I x and I y Represents the local horizontal and vertical gradient values, and can be obtained Eigenvector v 1 Represents the local edge direction, with a larger eigenvalue λ associated with it 1 is the edge strength, the eigenvector v 2 represents the direction orthogonal to the local edge, and is associated with a smaller eigenvalue λ 2 .

[0071] Step 3: Color value correction

[0072] The color difference image after filtering according to step 2.3 There are two methods to correct the color. One method is to use the color difference value after color correction as the optimized color difference, return to step 1 to recalculate the new RGB interpolation, the process is as follows Figure 2 , and obtain the final full-color image.

[0073] Another method is Figure 3 , according to the color difference image Calculate the color difference correction value: Correct the color interpolation based on the color difference correction value, keep the original RAW input value unchanged, and only correct the interpolation results of other colors.

[0074] At the G pixel position, the corrected R and B channel values ​​are:

[0075]

[0076]

[0077] At the R pixel position, the corrected G and B channel values ​​are:

[0078]

[0079]

[0080] At the B pixel position, the corrected G and R channel values ​​are:

[0081]

[0082]

[0083] According to an embodiment of the present invention, a computer-readable storage medium is also provided.

[0084] The computer-readable storage medium stores an image demosaicing program, and when the image demosaicing program is executed by a processor, the steps of the image demosaicing method described in any one of the embodiments described above are implemented.

[0085] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0086] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0087] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0089] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0090] Although specific embodiments of the present invention are described above, those skilled in the art should understand that these are merely examples and that various changes or modifications may be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is limited by the appended claims.

Claims

1. An image demosaicing method for autonomous driving, characterized in that: The demosaic interpolation method is used to generate the initial full-color image RGB, and the color difference images △GB and △GR are calculated, which are recorded as the color difference image Δ. Then, the color difference abnormal points in the color difference image are detected and corrected, and then the corrected color difference image is adaptively filtered. The filtered color difference image is recorded as Finally, the color value of the filtered color difference image is corrected to obtain the final full-color image; Color difference abnormal point detection Calculate the gradient value gradis, is = H, V, 45 and 135 of the current color point in each color difference image along the horizontal, vertical, 45 degree and 135 degree oblique edge directions, as well as the difference diffis, is = H, V, 45 and 135 between the color difference value of the current color point and the color difference value of the adjacent color point along the horizontal, vertical, 45 degree and 135 degree oblique edge directions. Note that the gradient thresholds along the horizontal, vertical, 45 degree and 135 degree oblique edge directions are th1is, is = H, V, 45 and 135, the color difference difference thresholds are th2is, is = H, V, 45 and 135, and the brightness thresholds are th3is, is = H, V, 45 and 135. If the three conditions of gradis>th1is, diffis>th2is, and maximum brightness lumamax>th3is in any direction are met at the same time, then the corresponding current pixel point is a color difference abnormal point; Color difference correction Select the direction with the largest gradient among all directions that meet the outlier detection conditions for correction. The correction method is to take the median along the direction: △′=middle{△ i-n ,…△ i ,…△ i+n } Among them, △ i Indicates the color difference of the abnormal point, △ i-n ~△ i+n represents the adjacent color difference along the direction, n is 1 or 2; The chromatic aberration image is filtered with the initial full-color image as a guide to remove the pseudo-color while maintaining good edge clarity. The filtering formula is: Among them, Δ i,j Represents the color difference value of the pixel (i, j) in the local 5x5 or 7x7 window. Indicates the establishment of a filter kernel function based on anisotropic Gaussian radial basis function, d i,j =[x i -x,y i -y] T Indicates the distance from pixel (i, j) to the current pixel. represents the kernel covariance matrix, obtained through the local gradient structure tensor, I x and I y Represents the local horizontal and vertical gradient values, and can be obtained The eigenvector v1 represents the local edge direction, and the larger eigenvalue λ1 associated with it is the edge strength. The eigenvector v2 represents the direction orthogonal to the local edge, and the smaller eigenvalue λ2 is associated with it.

2. The image demosaicing method for autonomous driving according to claim 1, characterized in that: The filtered color difference image is used as the optimized color difference image, and the demosaic interpolation method is reused to generate a new RGB interpolation to obtain the final full-color image.

3. The image demosaicing method for autonomous driving according to claim 1, characterized in that: Calculate the color difference correction value: Correct the color interpolation based on the color difference correction value, keep the original RAW input value unchanged, and only correct the interpolation results of other colors. At the G pixel position, the corrected R and B channel values ​​are: At the R pixel position, the corrected G and B channel values ​​are: At the B pixel position, the corrected G and R channel values ​​are: 。 4. A computer-readable storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1 to 3.

Citation Information

Patent Citations

  • Method for demosaicing color filtering array image based on non-local low rank

    CN103595981A

  • Method for suppressing image pseudo-color and device thereof

    CN106507066A

  • Anti-aliasing FIR filter-based demosaicking

    KR101780726B1

  • Demosaicking with adaptive reference range selection

    KR102055411B1

  • Method of demosaicing a digital mosaiced image

    US20080253652A1