Unified framework-based automatic driving road target detection method in extreme weather

Abstract

The invention discloses an automatic driving road target detection method in extreme weather based on a unified framework. The method comprises the following steps: acquiring automatic driving road videos or images in various severe weather for processing; using a BLS network structure to classify the types of the severe weather; secondly, carrying out defogging processing on the foggy day image by using dark channel defogging, and carrying out rain streak and rain imprint removing processing on the rainy day image by using guide filtering; then uniformly putting the uniformly processed images of various extreme weathers into the real-ESRGAN for further de-noising and feature enhancement of the road target; and finally, uniformly putting the processed images into the RT-DETR for detection. Performing data classification processing by using a BLS network; a real-ESRGAN network is used for denoising and feature enhancement processing of a road target, and finally an RT-DETR network is used for final unified detection, so that the detection capability of a sensor in automatic driving is improved, and the detection precision and the detection efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a method for detecting road targets in autonomous driving under extreme weather conditions based on a unified framework. Background Art

[0002] In bad weather, factors such as dense fog, heavy snow, and heavy rain have a significant impact on the efficiency and accuracy of detection. Therefore, it is necessary to enhance the detection capability of sensors in bad weather for the development of autonomous driving. The main research directions are mainly divided into three types, namely, target detection based on 3D radar-related stereoscopic detection facilities, target detection based on 2D camera-related planar sensors, and a combination of the two. However, improving the detection capability of 2D sensors in bad weather is of great significance to the stability and safety of autonomous driving vehicles in various real-world scenarios. Most of the existing 2D detection methods propose corresponding solutions for denoising and restoring deteriorated images for a type of bad weather or a situation under bad weather. Most of the methods are implemented in a single weather or condition. Under the influence of different conditions of multiple bad weather, due to excessive noise, the detection capability of existing autonomous driving sensors is reduced, resulting in low detection accuracy and efficiency; and the effect may be different from the expected application in real scenarios.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] In view of the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide an autonomous driving road target detection method under extreme weather conditions based on a unified framework, aiming to solve the problem that the existing 2D detection method in the prior art has excessive noise under the influence of various severe weather conditions, resulting in a decrease in the detection capability of the existing autonomous driving sensor, resulting in low detection accuracy and efficiency.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for detecting road objects in autonomous driving under extreme weather conditions based on a unified framework, comprising:

[0007] Step 1: Obtain the initial image for autonomous driving under extreme weather conditions;

[0008] Step 2: Use the BLS network structure to detect the initial image obtained in step 1, and classify the extreme weather types of the image through the BLS network structure, and divide it into three types of severe weather images: foggy, rainy, and snowy;

[0009] Step 3: Process the images differently according to the classified bad weather types. Use the dark channel method to defog foggy images, and use guided filtering with itself as the guide map to reduce the impact of rain streaks and raindrops on autonomous driving images. For snowy images, we treat some snow fog images as foggy images and some snow trace images as raindrop-like rain traces. It has been verified that other images can achieve good enhancement effects without further redundant processing, so no redundant processing is added in this step.

[0010] Step 4: Use the real-ESRGAN network to further denoise and enhance the detail features of the processed three types of severe weather images to obtain high-resolution images;

[0011] Step 5: Put the processed high-resolution image into RT-DETR for detection to generate road target detection results.

[0012] Furthermore, in step 2, the classification of extreme weather types of the image through the BLS network structure includes:

[0013] Step 2-1: Get the initial image of autonomous driving under extreme weather conditions obtained in step 1 as input image data X, perform feature mapping on X in the BLS network, and obtain all mapping features z s ;

[0014] Step 2-2: According to Z s , a randomly generated matrix W tj and β tj , nonlinear function ξ j Calculate the corresponding enhanced node H l ;

[0015] Step 2-3: Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and into the BLS network to calculate the output Y.

[0016] Furthermore, the expression of output Y in step 2-3 is:

[0017] Y=[Z s |H l ]W=PW;

[0018] Where P is the mapping feature Z s and enhanced node H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, expressed as:

[0019]

[0020] Where γ→0, I is the unit matrix, and lim γ→0 (γ I +PTP) -1 P T Compute the pseudoinverse of the matrix P.

[0021] Furthermore, in step 3, the guided filtering of the rainy day image using itself as the guide image to reduce the influence of rain streaks and raindrops on the autonomous driving image includes:

[0022] Step 3-1: Obtain a guide image according to the pixel value of each pixel point of the rainy day image under severe weather, and obtain the average pixel value of the R channel, the average pixel value of the G channel, and the average pixel value of the B channel of the guide image;

[0023] Step 3-2: Convert the input rainy day image into a grayscale image, and determine the grayscale image pixel average value of the corresponding flow according to the pixel value of each pixel point of the grayscale image;

[0024] Step 3-3: Determine the variance of each channel of the guide image according to the average pixel values ​​of the three channels R, G, and B of the guide image, and determine the parameters of the guide filter function of the corresponding flow according to the variance of each channel of the guide image, the average pixel values ​​of the three channels R, G, and B, and the average pixel values ​​of the grayscale image;

[0025] Step 3-4: Determine the pixel value of the output image pixel of the corresponding stream according to each parameter and the pixel value of the guide image pixel to obtain a processed rainy day autonomous driving image.

[0026] Furthermore, in step 3-3, the parameters of the guided filter function of the corresponding stream are determined according to the variance of each channel of the guided image, the average pixel value of the three channels of R, G, and B, and the average pixel value of the grayscale image, including the following formula:

[0027]

[0028]

[0029] Wherein, j represents R channel, G channel or B channel, the guided filter function includes the first parameter and the second parameter, a kj represents the first parameter of the j-channel guided filter function, b kj represents the second parameter of the j-channel guided filter function, w k represents the kth kernel function window, |w| represents the number of pixels in the kth kernel function window, I ij represents the pixel value of the j-channel of the ith pixel point in the guidance image, p ij Represents the pixel value of the jth channel of the i-th pixel of the input image, μ kjrepresents the average pixel value of the j channel of the kth kernel function window of the guidance image, Represents the pixel average of the grayscale image of the kth kernel function window of the input image, represents the variance of the jth channel of the kth kernel function window of the guidance image, and ε represents the smoothing factor.

[0030] Furthermore, the dark channel method in step 3 includes:

[0031] Step 3-5: Get the foggy image x obtained by classification in the previous step, perform dark channel processing on the foggy image x, and obtain its dark channel image J d (x);

[0032] Step 3-6: According to the transmittance and the estimated atmospheric light component, a defogging image model of the original input image is established to obtain the defogging image J(x).

[0033] Furthermore, in step 3-5, the dark channel image J of the input image d (x) is obtained from the following formula:

[0034]

[0035] Where Ω(x) represents the pixel block area centered on x, J c Represents a color channel of the dehazed image J.

[0036] Furthermore, in steps 3-6, the defogging model of the fogged image is obtained by the following formula:

[0037]

[0038] In the formula, I(x) represents the original input image, J(x) represents the image after defogging, A is the brightness value of the sky, is the refractive index distribution function of the fogged image.

[0039] Furthermore, the real-ESRGAN denoising and detail enhancement method in step 4 is:

[0040] Step 4-1: Place the autonomous driving images under various severe weather conditions processed in the previous step into the real-ESRGAN network structure for 4x resolution super-enhancement.

[0041] Step 4-2: The image size is first enlarged by compressing the image channels to reduce the spatial size, and then the information is rearranged to the channel dimension to obtain a denoised and detail enhanced image.

[0042] Furthermore, in step 5, RT-DETR includes a hybrid encoder and a Transformer decoder with an auxiliary detection head, and RT-DETR is trained using the image training set obtained from the processing of steps 1 to 4, including:

[0043] Step 5-1: Input the training set into the MobileOne structure and output features {S3, S4, S5} of three different scales;

[0044] Step 5-2: Take {S3, S4, S5} as the input of the hybrid encoder, perform cross-scale fusion and intra-scale interactive conversion through the hybrid encoder, and obtain a series of image features;

[0045] Step 5-3: Select a fixed number of image features from the series of image features output by the hybrid encoder through the Transformer decoder, and then use the activation function to iteratively optimize the fixed number of image features to generate a generated box and a confidence score.

[0046] The technical solution adopted by the present invention has the following beneficial effects:

[0047] In the present invention, in the classification part, by combining with the BLS network, its characteristics of powerful classification ability, fast classification speed, simple and fast training and high scalability are applied; in the image enhancement part, thanks to the powerful classification ability of BLS, two simple and effective methods of dark channel defogging and guided filtering deraining are first used to preliminarily process different types of severe weather autonomous driving images, and then real-ESRGAN is used to further enhance the previous image, which greatly increases the resolution of the image, effectively enhances the detailed features of the object, and makes the image more suitable for our next detection task; finally, in the detection part, the RT-DETR network is used for final unified detection to improve the detection ability of sensors in autonomous driving, improve the detection accuracy and efficiency; through the integrated framework of the present invention, the framework can be used regardless of the type of severe weather, the integration of the framework ensures the convenience of detection, reduces the utilization of network resources and complicated input and output operations, and uses one framework to solve the complex task of autonomous driving car detection in severe weather. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is an image of the road for autonomous driving in foggy weather;

[0049] Figure 2 Road images for autonomous driving in rainy weather;

[0050] Figure 3 Road images for autonomous driving in snowy weather;

[0051] Figure 4 A comparison diagram of the detection results of the present invention and the original image;

[0052] Figure 5 is a flow chart of the present invention;

[0053] Figure 6 This is the network model diagram of BLS;

[0054] Figure 7 This is the network model diagram of real-ESRGAN. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0056] The present invention is a method for detecting road targets in autonomous driving under complex and changing extreme weather conditions based on a unified framework. The BLS classification network is introduced to classify different types of severe weather images. Guided filtering is used to remove rain from rainy images and dark channel defogging is used to remove fog from foggy images. The images are then put into the real-ESRGAN super-resolution network for further denoising and detail feature enhancement, and finally put into RT-DETR for detection. The whole process is as follows: Figure 5 As shown, the scene in this embodiment is described with a road in bad weather as the background.

[0057] A method for detecting road targets in autonomous driving under complex and changing extreme weather conditions based on a unified framework. The main steps are as follows:

[0058] Step 1: Obtain images of autonomous driving under various extreme weather conditions;

[0059] Obtain road video or image data of self-driving cars in various bad weather conditions and cut the video into images; the data content should be relatively large, such as country lanes, highways, intersections, etc. Figure 1 , Figure 2 , Figure 3 As shown, the figure shows road images of an autonomous driving car in various severe weather conditions including fog, rain, and snow.

[0060] Step 2: Use the BLS (Breadth Learning System) network structure to detect the image obtained in step 1. The BLS network is as follows: Figure 6 As shown in the figure, the BLS network structure classifies the extreme weather types of the image, and processes the image differently according to the classified severe weather types, dividing it into three severe weather types: foggy day, rainy day, and snowy day;

[0061] The classification method of width learning system is as follows:

[0062] Step 2-1: Get the pedestrian image result obtained in step 2 as the input graphic data X, where X is an R*E matrix, R represents the number of samples of the input data, and E represents the dimension of each sample. Perform feature mapping on X in the width learning system and obtain all mapping features Z s ;

[0063] Use the following formula to perform feature mapping Z on the input data X;

[0064]

[0065] in and These are all matrices we randomly generated. is a linear function, which is not unique here. Different linear functions can be selected according to different input data X or different feature mapping Z combinations. We will get s different Z combinations to get s groups of feature nodes, Z s =[Z1, Z2, Z3, ..., Z s ], which represents all the mapping features we extract from the input X through network mapping.

[0066] Step 2-2: Then according to Z s , a randomly generated matrix W tj and β tj , nonlinear function ξ j Calculate the corresponding enhanced node H l ;

[0067] The enhanced node H is calculated using the following formula;

[0068]

[0069] and It is still our randomly generated matrix, ξ j represents a nonlinear function. Here, we can also choose different ξ. From the above formula, we get H j Also in matrix form, by combining different l H together we can get the final l groups of enhanced nodes H l =[H1, H2, H3, ..., H l ], which represents all the enhanced nodes we get from mapping features.

[0070] Step 2-3: Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and to the width learning system to calculate the output Y;

[0071] Y is an R*P matrix, where P represents the number of categories of samples.

[0072] Map feature Z s and enhanced node H l The final output Y can be obtained by multiplying the combined matrix with the weights of the network connection;

[0073] Y=[Z s |H l ]W=PW (14)

[0074] Where P is Z s and H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, which can be expressed as:

[0075] W=lim γ→0 (γI+P T P) -1 P T Y (15)

[0076] In this case, we usually need to let γ→0, I be the identity matrix, and lim γ→0 (γI+P T P) -1 P T Calculate the pseudo-inverse of the matrix P, and multiply the pseudo-inverse of P by the output matrix Y to obtain the connection weight W of the network. We substitute the calculated W into (14) to obtain the final output Y of the network.

[0077] Step 3: Defogging the classified foggy images using the dark channel method, and performing guided filtering on rainy images with themselves as the guide map to reduce the impact of rain streaks and raindrops on the autonomous driving images. For snowy images, we treat some of the snow fog images as foggy images and some of the snow trace images as raindrop-like rain traces. It has been verified that other images can achieve good enhancement effects without further redundant processing, so no redundant processing is added in this step:

[0078] The guided filtering method is as follows:

[0079] Step 3-1: Obtain the average pixel value of the R channel, the average pixel value of the G channel, and the average pixel value of the B channel of the guide image according to the pixel value of each pixel point of the rainy day image under severe weather;

[0080] Step 3-2: Convert the input image into a grayscale image, and determine the grayscale image pixel average of the corresponding flow according to the pixel value of each pixel point of the grayscale image;

[0081] Step 3-3: Determine the variance of each channel of the guide image according to the average pixel values ​​of the three channels R, G, and B of the guide image, and determine the parameters of the guide filter function of the corresponding flow according to the variance of each channel of the guide image, the average pixel values ​​of the three channels R, G, and B, and the average pixel values ​​of the grayscale image;

[0082] In step 3-3, the parameters of the guide filter function of the corresponding stream are determined according to the variance of each channel of the guide image, the average pixel value of the three channels of R, G, and B, and the average pixel value of the grayscale image, including the following formula:

[0083]

[0084]

[0085] Wherein, j represents R channel, G channel or B channel, the guided filter function includes the first parameter and the second parameter, a kj represents the first parameter of the j-channel guided filter function, b kj represents the second parameter of the j-channel guided filter function, w k represents the kth kernel function window, |w| represents the number of pixels in the kth kernel function window, I ij represents the pixel value of the j-channel of the ith pixel point in the guidance image, p ij Represents the pixel value of the jth channel of the i-th pixel of the input image, μ kj represents the average pixel value of the j channel of the kth kernel function window of the guidance image, Represents the pixel average of the grayscale image of the kth kernel function window of the input image, represents the variance of the jth channel of the kth kernel function window of the guidance image, and ε represents the smoothing factor.

[0086] Step 3-4: Determine the pixel value of the output image pixel of the corresponding stream according to each parameter and the pixel value of the guide image pixel to obtain a processed rainy day autonomous driving image.

[0087] The dark channel defogging method is as follows:

[0088] Step 3-5: Get the foggy autonomous driving road image x classified in the previous step and obtain its dark channel image J d (x);

[0089] In step 3-5, the dark channel image J of the input image d (x) is obtained from the following formula:

[0090]

[0091] Where Q(x) represents the pixel block area centered on x, J cRepresents a color channel of the dehazed image J.

[0092] Step 3-6: According to the transmittance and the estimated atmospheric light component, a defogging image model of the original input image is established to obtain the defogging image J(x);

[0093] In steps 3-6, the dehazing model of the haze image is obtained by the following formula:

[0094]

[0095] In the formula, I(x) represents the original input image, J(x) represents the image after defogging, A is the brightness value of the sky, is the refractive index distribution function of the fogged image.

[0096] Step 4: Use the three bad weather images previously processed by the real-ESRGAN network to further denoise and enhance detail features. The real-ESRGAN network is as follows: Figure 7 As shown;

[0097] The steps of real-ESRGAN network to denoise and enhance detail features of images are as follows:

[0098] Step 4-1: Place the autonomous driving images under various severe weather conditions processed in the previous step into the real-ESRGAN network structure for 4x resolution super-enhancement.

[0099] Step 4-2: The image size is first enlarged by compressing the image channels to reduce the spatial size, and then the information is rearranged to the channel dimension to obtain a denoised and detail enhanced image.

[0100] Step 5: Put the processed image into RT-DETR for detection.

[0101] The steps of RT-DETR network to detect images are:

[0102] It is characterized in that the RT-DETR includes a hybrid encoder and a Transformer decoder with an auxiliary detection head, and the image training set obtained by the above processing process is used to train the RT-DETR, including:

[0103] Step 5-1: Input the training set into the MobileOne structure and output features {S3, S4, S5} of three different scales;

[0104] Step 5-2: taking {S3, S4, S5} as the input of the hybrid encoder, performing cross-scale fusion and intra-scale interactive conversion through the hybrid encoder to obtain a series of image features;

[0105] Step 5-3: Select a fixed number of image features from the series of image features output by the hybrid encoder through the Transformer decoder, and then use the activation function to iteratively optimize the fixed number of image features to generate a generated box and a confidence score.

[0106] Experimental Results

[0107] The present invention conducted the following ablation experiment: (1) a certain part of the frame was removed for testing. The experimental results are shown in Table 1 below:

[0108] Table 1. Ablation experiment results

[0109]

[0110]

[0111] As can be seen from Table 1, the present invention improves the average accuracy (mAP) of all categories by integrating different technologies. At the same time, compared with other detection methods, the technical solution adopted by the present invention has the best effect. The experimental results show that the present invention has a good effect on the road detection method of the autonomous driving car in bad weather.

[0112] In the present invention, reference is made to Figure 4 , Figure 4 The original bad weather autonomous driving image is directly detected and input, and the detection results after enhanced denoising obtained by the present invention are compared. Figure 4 It can be seen that compared with direct detection, the detector can detect more objects through the framework of the present invention, including vehicles or objects that are severely affected by weather conditions in the original image and a large number of small target objects that were originally undetectable. This proves that the present invention actually and effectively improves the problem of false detection and misdetection of autonomous driving vehicles on the road in bad weather.

[0113] In the present invention, in the classification part, by combining with the BLS network, its characteristics of powerful classification ability, fast classification speed, simple and fast training and high scalability are applied; in the image enhancement part, thanks to the powerful classification ability of BLS, two simple and effective methods of dark channel defogging and guided filtering deraining are first used to preliminarily process different types of severe weather autonomous driving images, and then real-ESRGAN is used to further enhance the previous image, which greatly increases the resolution of the image and effectively enhances the detailed features of the object, making the image more suitable for our next detection task; finally, in the detection part, the RT-DETR network is used for final unified detection to improve the detection ability of sensors in autonomous driving, and improve the detection accuracy and efficiency.

[0114] The integrated framework of the present invention can be used regardless of the type of severe weather conditions. The integration of the framework ensures the convenience of detection, reduces network resource utilization and complex input and output operations, and uses one framework to solve the complex task of detecting autonomous vehicles in severe weather conditions, effectively solving the problem of false detection and misdetection of autonomous vehicles on the road in the ever-changing severe weather conditions in real life.

[0115] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the schemes disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the claims.

Claims

1. A method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework, characterized in that: include: Step 1: Obtain the initial image for autonomous driving under extreme weather conditions; Step 2: Use the BLS network structure to detect the initial image obtained in step 1, and classify the extreme weather types of the image through the BLS network structure, and divide it into three types of severe weather images: foggy, rainy, and snowy; Step 3: Process the images differently according to the classified bad weather types. For foggy images, use the dark channel method to defog them. For rainy images, use guided filtering with itself as the guide map to reduce the impact of rain streaks and raindrops on the autonomous driving images. For snowy images, treat some snow and fog images as foggy images and some snow trace images as raindrop-like rain traces. Step 4: Use the real-ESRGAN network to further denoise and enhance the detail features of the processed three types of severe weather images to obtain high-resolution images; Step 5: Put the processed high-resolution image into RT-DETR for detection to generate road target detection results.

2. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 1, characterized in that: In step 2, the extreme weather types of the image are classified through the BLS network structure, including: Step 2-1: Get the initial image of autonomous driving under extreme weather conditions obtained in step 1 as input graphic data X, perform feature mapping on X in the BLS network, and obtain all mapping features Z s ; Step 2-2: According to Z s , a randomly generated matrix W tj and β tj , nonlinear function ξ j Calculate the corresponding enhanced node H l ; Step 2-3: Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and into the BLS network to calculate the output Y.

3. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 2, characterized in that: The expression for output Y in step 2-3 is: Y=[Z s |H l ]W=PW; Where P is the mapping feature z s and enhanced node H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, expressed as: Where γ→0, I is the unit matrix, and lim γ→0 (γI+P T P) -1 P T Compute the pseudoinverse of the matrix P.

4. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 1, characterized in that: In step 3, guided filtering is performed on the rainy day image using itself as the guide image to reduce the influence of rain streaks and raindrops on the autonomous driving image, including: Step 3-1: Obtain a guide image according to the pixel value of each pixel point of the rainy day image under severe weather, and obtain the average pixel value of the R channel, the average pixel value of the G channel, and the average pixel value of the B channel of the guide image; Step 3-2: Convert the input rainy day image into a grayscale image, and determine the grayscale image pixel average value of the corresponding flow according to the pixel value of each pixel point of the grayscale image; Step 3-3: Determine the variance of each channel of the guide image according to the average pixel values ​​of the three channels R, G, and B of the guide image, and determine the parameters of the guide filter function of the corresponding flow according to the variance of each channel of the guide image, the average pixel values ​​of the three channels R, G, and B, and the average pixel values ​​of the grayscale image; Step 3-4: Determine the pixel value of the output image pixel of the corresponding stream according to each parameter and the pixel value of the guide image pixel to obtain a processed rainy day autonomous driving image.

5. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 4, characterized in that: In step 3-3, the parameters of the guide filter function of the corresponding stream are determined according to the variance of each channel of the guide image, the average pixel value of the three channels of R, G, and B, and the average pixel value of the grayscale image, including the following formula: Wherein, j represents R channel, G channel or B channel, the guided filter function includes the first parameter and the second parameter, a kj represents the first parameter of the j-channel guided filter function, b kj represents the second parameter of the j-channel guided filter function, w k represents the kth kernel function window, |w| represents the number of pixels in the kth kernel function window, I ij represents the pixel value of the j-channel of the ith pixel point in the guidance image, p ij Represents the pixel value of the jth channel of the i-th pixel of the input image, μ kj represents the average pixel value of the j channel of the kth kernel function window of the guidance image, Represents the pixel average of the grayscale image of the kth kernel function window of the input image, represents the variance of the jth channel of the kth kernel function window of the guidance image, and ε represents the smoothing factor.

6. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 1, characterized in that: The dark channel method in step 3 includes: Step 3-5: Get the foggy image x obtained by classification in the previous step, perform dark channel processing on the foggy image x, and obtain its dark channel image J d (x); Step 3-6: According to the transmittance and the estimated atmospheric light component, a defogging image model of the original input image is established to obtain the defogging image J(x).

7. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 6, characterized in that: In step 3-5, the dark channel image J of the input image d (x) is obtained from the following formula: Where Ω(x) represents the pixel block area centered on x, J c Represents a color channel of the dehazed image J.

8. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 6, characterized in that: In steps 3-6, the dehazing model of the haze image is obtained by the following formula: In the formula, I(x) represents the original input image, J(x) represents the image after defogging, A is the brightness value of the sky, is the refractive index distribution function of the fogged image.

9. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 1, characterized in that: The real-ESRGAN denoising and detail enhancement method in step 4 is: Step 4-1: Put the autonomous driving images under various severe weather conditions processed in the previous step into the real-ESRGAN network structure for 4x resolution super-enhancement; Step 4-2: The image size is first enlarged by compressing the image channels to reduce the spatial size, and then the information is rearranged to the channel dimension to obtain a denoised and detail enhanced image.

10. The method for detecting road targets for autonomous driving in extreme weather conditions based on a unified framework according to claim 1, characterized in that: In step 5, RT-DETR includes a hybrid encoder and a Transformer decoder with an auxiliary detection head. The image training set obtained from the processing of steps 1 to 4 is used to train RT-DETR, including: Step 5-1: Input the training set into the MobileOne structure and output features {S3, S4, S5} of three different scales; Step 5-2: Take {S3, S4, S5} as the input of the hybrid encoder, perform cross-scale fusion and intra-scale interactive conversion through the hybrid encoder, and obtain a series of image features; Step 5-3: Select a fixed number of image features from the series of image features output by the hybrid encoder through the Transformer decoder, and then use the activation function to iteratively optimize the fixed number of image features to generate a generated box and a confidence score.

Images