A color denoising method for autonomous driving images

Abstract

The present invention relates to the technical field of image processing, and discloses a color denoising method for autonomous driving images, which is characterized by: being applied to road images during vehicle driving, including preprocessing the images with a point-by-point sliding window filtering method, suppressing outlier color points and color clusters, and obtaining a preprocessed image; performing multi-channel step-by-step decomposition of the preprocessed image with the aid of a multi-channel filter to obtain a multi-scale sub-band image; and performing joint filtering and reconstruction on the multi-scale sub-band image in succession to obtain a final denoised image. With the aid of the image color denoising method of the present invention, low-frequency color noise can be effectively reduced, which is more conducive to the recognition of traffic signal signs by the autonomous driving perception module, thereby greatly improving the subsequent decision-making stage.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a color denoising method for autonomous driving images. Background Art

[0002] The image acquisition and reading process is a complex random process. Random noise will be introduced during the acquisition process. The image processing process after ISP will also cause noise changes, so noise reduction processing is required. Image noise is usually divided into brightness noise and color noise. Brightness noise is manifested as high-mid frequency noise, and color noise is usually mainly medium-low frequency noise. However, due to bad pixels, Bell array interpolation (CFAI) and random noise introduced by image acquisition and reading, there are fine-grained outliers in the color channel, which are reflected in the image as small particles and clusters of color noise.

[0003] Conventional color noise reduction technology can handle some high-frequency and medium-frequency color noise, but there is no good solution for outlier color clusters introduced by bad pixels and interpolation, and there is no good solution for extremely low-frequency color noise. The existence of color noise has a great impact on the recognition of objects in autonomous driving, especially the recognition of traffic lights and signal signs that are mostly red, green and yellow, and even affects the subsequent perception-based decision-making stage. Summary of the invention

[0004] The present invention provides a color denoising method for autonomous driving images, which performs multi-channel step-by-step decomposition on the preprocessed image to obtain a multi-scale sub-band image; then the multi-scale sub-band image is successively jointly filtered and reconstructed to obtain a final denoised image, which filters the color noise on different frequency components, and performs joint filtering on a large-scale frequency band to effectively remove low-frequency color noise.

[0005] The present invention can be achieved through the following technical solutions:

[0006] A color denoising method for autonomous driving images, applied to road images during vehicle driving, including

[0007] The image is preprocessed using a point-by-point sliding window filtering method to suppress outlier color points and color clusters to obtain a preprocessed image.

[0008] By means of a multi-channel filter, the pre-processed image is subjected to multi-channel step-by-step decomposition to obtain a multi-scale sub-band image;

[0009] The multi-scale sub-band images are successively subjected to joint filtering and reconstruction to obtain a final denoised image.

[0010] Further, the point-by-point sliding window filtering method is used to preprocess the U-channel and V-channel data in the image respectively. Given the Sxy window, Zmin is the minimum pixel value in the Sxy window, Zmax is the maximum pixel value in the Sxy window, Zmed is the median value in the Sxy window, Zxy is the pixel value corresponding to the center point position of the Sxy window, and Smax is the maximum filtering size. The preprocessing is divided into the following two levels:

[0011] Level A: If Zmin + 1 < Zmed < Zmax - 1, jump to Level B;

[0012] Otherwise, increase the size of the window Sxy;

[0013] If Sxy < Smax, repeat Level A, otherwise output Zmed;

[0014] Level B: If Zmin + 1 < Zxy < Zmax - 1, output Zxy, otherwise output Zmed.

[0015] Further, two different multi-channel filters are used to perform convolution operations on the U-channel data and V-channel data in the preprocessed image respectively, and decompose them into multiple sub-bands of multiple frequency bands step by step to obtain multi-scale sub-band images.

[0016] Further, denoting the parameters of the two multi-channel filters as h and g respectively, the following formula is used to perform convolution operations on the input image and decompose it into multiple sub-bands of multiple frequency bands step by step. Among them, the input image for the first frequency band decomposition is the U-channel data or V-channel data in the preprocessed image, the input image for the second frequency band decomposition is the 0th sub-band of the first frequency band, the input image for the third frequency band decomposition is the 0th sub-band of the second frequency band, and so on, to obtain multi-scale sub-band images.

[0017] I 0 = *I* T

[0018] I 1 = *I*g T

[0019] I 2 = g*I* T

[0020] I 3 = g*I*g T

[0021] In the formula, I is the input image, I 0 is the 0th sub-band, I 1 is the 1st sub-band, I 2 is the 2nd sub-band, I 3For the third subband, h = [-0.0106, 0.0329, 0.0308, -0.1870, -0.0280, 0.6309, 0.7148, 0.2304], g = [-0.2304, 0.7148, -0.6309, -0.0280, 0.1870, 0.0308, -0.0329, -0.0106].

[0022] Furthermore, after the U channel data and V channel data in the preprocessed image are subjected to multi-channel three-level decomposition, the subband of the U channel image is obtained as The subband of the V channel image is obtained as

[0023] Use the following equations to perform joint filtering on each pixel point (x, y) of the 0th subband of each frequency band in the U channel image and the V channel image,

[0024]

[0025]

[0026] Where j = 1, 2, 3, w UV =w U *w V represents the joint filter weight of the corresponding filter point, w U 、w V All calculated by the following formula

[0027]

[0028] Wherein, n is the order of the inclined Butterworth filter w(z), d is the spatial cutoff threshold of the inclined Butterworth filter w(z), k is the tilt slope of the inclined Butterworth filter w(z), and z is the similarity calculated by block matching the 0th subband of each frequency band in the U channel image and the V channel image using the block matching method;

[0029] After joint filtering, the sub-bands of the U channel image are obtained as follows:

[0030] The subbands of the V channel image are

[0031] Then the reconstruction filter is used to reconstruct the U channel image and the V channel image respectively. The coefficient of the reconstruction filter is

[0032] h *=[0.2304 0.7148 0.6309 -0.0280 -0.1870 0.0308 0.0329 -0.0106]g * =[-0.0106 -0.0329 0.0308 0.1870 -0.0280 -0.6309 0.7148 -0.2304]

[0033] The reconstruction process is:

[0034]

[0035] in, represents the t-th subband of the k-th frequency band, t=0, 1, 2, 3, k=1, 2, 3;

[0036] For the U channel image after joint filtering, each sub-band of the third frequency band Reconstruction can be obtained So we get the new second frequency band Reconstruct the new second frequency band to obtain So we get the new first frequency band The new first frequency band is reconstructed to obtain the U channel image denoise_U after color denoising, and the above process is repeated to reconstruct the V channel image after joint filtering to obtain the V channel image denoise_V after color denoising, so as to obtain the final denoised image.

[0037] A computer-readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement the method described above.

[0038] The beneficial technical effects of the present invention are:

[0039] (1) The point-by-point sliding channel filtering method can effectively filter outlier color points and color clusters introduced by bad pixels and Bell interpolation CFAI, while avoiding the loss of true color details;

[0040] (2) The present invention adopts a multi-channel sub-band filtering structure, by dividing the image into multiple sub-bands, filtering the color noise at different frequency components, and performing joint filtering on a large-scale frequency band to effectively remove low-frequency color noise;

[0041] (3) The present invention improves the classic Butterworth low-pass filter and proposes a tilted Butterworth filter, which can well maintain the edge while performing noise reduction and can effectively prevent color edge leakage;

[0042] In summary, by means of the image color noise reduction method of the present invention, low-frequency color noise can be effectively reduced, which is more conducive to the recognition of traffic signal signs by the automatic driving perception module, thus improving the subsequent decision-making stage. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0044] The following describes in detail the specific embodiments of the present invention with reference to the accompanying drawings and preferred embodiments.

[0045] As Figure 1 shown, the present invention provides a color noise reduction method for automatic driving images. The method preprocesses the image by using a point-by-point sliding window filtering method to suppress outlier color points and color clusters, and obtains a preprocessed image; with the aid of a multi-channel filter, the preprocessed image is decomposed step by step in multiple channels to obtain a multi-scale sub-band image; the multi-scale sub-band image is jointly filtered and reconstructed successively to obtain a final noise reduction image. By using the image color noise reduction method of the present invention, the color noise in the image collected by the camera in the automatic driving perception stage can be effectively suppressed, the detection of color noise for traffic signal lights and other signal signs / objects can be reduced, and thus the subsequent decision-making stage can be assisted and the reliability of the decision-making stage can be improved.

[0046] Specifically as follows:

[0047] (1) Color noise reduction preprocessing:

[0048] The point-by-point sliding window filtering method can be carried out separately in the U / V channels. The following takes the U channel as an example to illustrate the algorithm. Given the Sxy window, Zmin is the minimum pixel value in the Sxy window, Zmax is the maximum pixel value in the Sxy window, Zmed is the median value in the Sxy window, that is, the pixel value arranged from small to large, and the pixel value in the middle, Zxy is the pixel value corresponding to the center point position of the Sxy window, Smax is the maximum filtering size, and the color noise reduction preprocessing is divided into the following two levels:

[0049] Layer A: If Zmin + 1 < Zmed < Zmax - 1, jump to Layer B;

[0050] Otherwise, increase the size of the window Sxy;

[0051] If Sxy < Smax, repeat Layer A, otherwise output Zmed;

[0052] Layer B: If Zmin + 1 < Zxy < Zmax - 1, output Zxy, otherwise output Zmed

[0053] For the V channel, the same method as above is used to perform point-by-point sliding window filtering. After preprocessing, isolated color points / clusters introduced by bad points and interpolation methods in the obtained preprocessed image are suppressed.

[0054] (2) Multi-channel decomposition

[0055] Usually, color denoising is performed on the YUV domain. We also choose to process on the UV image. The U / V image is decomposed into multiple sub-bands through multiple sub-bands.

[0056] Humans are usually very sensitive to color noise, and color noise exists in all high / medium / low frequency bands. This requires that noise suppression can be designed for each frequency band. We choose the multi-channel filter decomposition structure of the signal to decompose the image signal into different frequency bands, so that color noise in different frequency bands can be processed. Here, we choose the multi-channel filter groups h and g, and their coefficients are:

[0057] h=[-0.0106, 0.0329, 0.0308, -0.1870, -0.0280, 0.6309, 0.7148, 0.2304]

[0058] g=[-0.2304, 0.7148, -0.6309, -0.0280, 0.1870, 0.0308, -0.0329, -0.0106]

[0059] For the input image I, a convolution operation is performed using multi-channel filters h and g according to the following formula to decompose it step by step into multiple sub-bands of multiple frequency bands, where the input image decomposed by the first frequency band is the U channel data or V channel data in the preprocessed image, the input image decomposed by the second frequency band is the 0th sub-band of the first frequency band, the input image decomposed by the third frequency band is the 0th sub-band of the second frequency band, and so on, to obtain a multi-scale sub-band image

[0060] I 0 =*I* T

[0061] I 1 =*I*g T

[0062] I 2 =g*I* T

[0063] I 3 =g*I*g T

[0064] Where I is the input image, I 0 is the 0th subband, I1 is the first subband, I 2 For the second subband, I 3 For the third sub-band,

[0065] After the above process, the preprocessed image I is decomposed into four sub-bands on the first frequency band: If further decomposition is required, subband 0 of the first frequency band As input, it is decomposed into four sub-bands of the second frequency band The third decomposition is the 0th subband of the second frequency band Perform multi-channel decomposition to obtain four sub-bands on the third frequency band By analogy, sub-band components at more scales can be obtained, thereby being able to process lower frequency color noise.

[0066] According to the application needs and the frequency distribution of noise, the number of decompositions required is selected. It will not be described here. The color denoising method of the present invention performs three channel decompositions on the pixel values ​​of the U and V channel images in the original preprocessed image I, respectively, to obtain the subband of the U channel as The subband of the V channel is obtained as

[0067] (3) Joint filtering

[0068] After multi-channel sub-band decomposition, we get color components of different frequencies, so that we can process color noise at different scales. In order to make full use of the information between color channels, we choose the U / V channel joint filtering method. Here we use the low-pass sub-band Subband and Take this as an example to illustrate the joint filtering method. The filtering methods on other low-pass sub-bands are similar.

[0069] For sub-band and For the pixel point at the spatial coordinate (x, y), we use the SAD block matching method to perform block matching. For example, a 5x5 block is selected, and the search radius is 7. The specific search radius can be determined according to the actual situation. The similarity SAD (summation of difference) between each block is calculated, and then the weights between the corresponding blocks are calculated. The weight calculation formula is:

[0070]

[0071] Wherein, n is the order of the tilted Butterworth filter w(z), d is the spatial cutoff threshold of the tilted Butterworth filter w(z), k is the tilt slope of the tilted Butterworth filter w(z), and z is the similarity calculated by the block matching method, that is, the sum of the absolute values ​​of the corresponding point differences between the two blocks.

[0072] For the same pixel, we can get and The weight w U and w V , then for the spatial coordinates (x, y), the joint filtering weight is w UV =w U *w V , so we get the pixel value of the spatial coordinate (x, y) point of the joint filtering, and the calculation formula is

[0073]

[0074]

[0075] Among them, the two summation symbols of m / n represent the joint filter w UV The filter radius is 7, and the filter range is the pixels in the range of [x-7,x+7]×[y-7,y+7]. UV Represents the joint filtering weights of the corresponding filtering points.

[0076] for and Each pixel of is filtered by the above-mentioned joint filtering method, and the color noise of the frequency corresponding to the third frequency band is suppressed. Similarly, the same method can also be used to suppress the color noise of the frequencies corresponding to the second frequency band and the first frequency band, so as to obtain a denoised image in which the noise of each frequency band is suppressed. The sub-bands of the U channel after filtering are: The sub-bands of the V channel after filtering are:

[0077] After the joint filtering is completed, the reconstruction filter is used to reconstruct the U / V channels respectively. The coefficients of the reconstruction filter are:

[0078] h * =[0.2304 0.7148 0.6309 -0.0280 -0.1870 0.0308 0.0329 -0.0106]

[0079] g * =[-0.0106 -0.0329 0.0308 0.1870 -0.0280 -0.6309 0.7148 -0.2304]

[0080] The reconstruction process is:

[0081]

[0082] in, represents the t-th subband of the k-th frequency band, t=0, 1, 2, 3, k=1, 2, 3;

[0083] For the U channel image after joint filtering, for each sub-band of the third frequency band Reconstruction can be obtained So we get the new second frequency band Reconstruct the new second frequency band to obtain So we get the new first frequency band Reconstruct the new first frequency band to obtain denoise_U, thereby obtaining a U channel image after color denoising;

[0084] By reconstructing the V channel image data in the same way as the U channel, we can obtain the V channel image denoise_U after color denoising, thereby completing the color denoising of the image.

[0085] According to an embodiment of the present invention, a computer-readable storage medium is also provided.

[0086] The computer-readable storage medium stores an image color denoising program for autonomous driving, and when the image color denoising program for autonomous driving is executed by a processor, the steps of the color denoising method for autonomous driving images as described in any one of the above embodiments are implemented.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] It will be appreciated by those skilled in the art that the embodiments of the present invention may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0089] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0090] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0091] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0092] Although specific embodiments of the present invention are described above, those skilled in the art should understand that these are merely examples and that various changes or modifications may be made to these embodiments without departing from the principles and essence of the present invention. Therefore, the scope of protection of the present invention is limited by the appended claims.

Claims

1. A color denoising method for autonomous driving images, characterized in that: The road images applied during the vehicle driving process include Preprocessing the image by using the point-by-point sliding window filtering method to suppress outlier color points and color clusters, and obtaining a preprocessed image; With the aid of a multi-channel filter, performing multi-channel hierarchical decomposition on the preprocessed image to obtain multi-scale sub-band images; Performing joint filtering and reconstruction on the multi-scale sub-band images successively to obtain a final noise-reduced image; Using two different multi-channel filters to perform convolution operations on the U-channel data and V-channel data in the preprocessed image respectively, and decomposing them hierarchically to multiple sub-bands of multiple frequency bands to obtain multi-scale sub-band images; Denote the parameters of the two multi-channel filters as h and g respectively, and perform convolution operations on the input image by using the following formula, and decompose it hierarchically to multiple sub-bands of multiple frequency bands. Among them, the input image for the first frequency band decomposition is the U-channel data or V-channel data in the preprocessed image, the input image for the second frequency band decomposition is the 0th sub-band of the first frequency band, the input image for the third frequency band decomposition is the 0th sub-band of the second frequency band, and so on, to obtain multi-scale sub-band images. I 0 =h*I*h T I 1 =h*I*g T I 2 =g*I*h T I 3 =g*I*g T Where I is the input image, I 0 is the 0th subband, I 1 is the first subband, I 2 For the second subband, I 3 For the third subband, h = [-0.0106, 0.0329, 0.0308, -0.1870, -0.0280, 0.6309, 0.7148, 0.2304], g = [-0.2304, 0.7148, -0.6309, -0.0280, 0.1870, 0.0308, -0.0329, -0.0106]; After the U channel data and V channel data in the preprocessed image are subjected to multi-channel three-level decomposition, the subband of the U channel image is obtained as follows: The subband of the V channel image is obtained as Using the following equation to perform joint filtering on each pixel point (x, y) of the 0th sub-band of each frequency band in the U-channel image and V-channel image respectively. Where, j = 1, 2, 3, w UV =w U *w V represents the joint filter weight of the corresponding filter point, w U 、w V All calculated by the following formula Wherein, n is the order of the tilted Butterworth filter w(z), d is the spatial cut-off threshold of the tilted Butterworth filter w(z), k is the tilt slope of the tilted Butterworth filter w(z), and z is the similarity calculated when performing block matching on the 0th sub-band of each frequency band from the U-channel image and V-channel image by using the block matching method; After joint filtering, the sub-bands of the U channel image are obtained as follows: The subbands of the V channel image are Then use a reconstruction filter to reconstruct the U-channel image and V-channel image respectively, and the coefficients of the reconstruction filter are h * ＝[0.2304 0.7148 0.6309 -0.0280 -0.1870 0.0308 0.0329 -0.0106] g * ＝[-0.0106 -0.0329 0.0308 0.1870 -0.0280 -0.6309 0.7148 -0.2304] The reconstruction process is as follows: in, represents the t-th subband of the k-th frequency band, t=0,1,2,3, k=1,2,3; For the U channel image after joint filtering, each sub-band of the third frequency band Reconstruction can be obtained So we get the new second frequency band Reconstruct the new second frequency band to obtain So we get the new first frequency band The new first frequency band is reconstructed to obtain the U channel image denoise_U after color denoising, and the above process is repeated to reconstruct the V channel image after joint filtering to obtain the V channel image denoise_V after color denoising, so as to obtain the final denoised image.

2. The color denoising method for autonomous driving images according to claim 1, characterized in that: Preprocess the U-channel and V-channel data in the image respectively by using the point-by-point sliding window filtering method. Given the Sxy window, Zmin is the minimum pixel value in the Sxy window, Zmax is the maximum pixel value in the Sxy window, Zmed is the median value in the Sxy window, Zxy is the pixel value corresponding to the center point position of the Sxy window, and Smax is the maximum filtering size. The preprocessing is divided into the following two levels: Level A: If Zmin + 1 < Zmed < Zmax - 1, jump to Level B; Otherwise, increase the size of the window Sxy; If Sxy < Smax, repeat Level A, otherwise output Zmed; Level B: If Zmin + 1 < Zxy < Zmax - 1, output Zxy, otherwise output Zmed.

3. A computer-readable storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1-2.

Images