An enhanced refinement network device and method for target detection in adverse weather

Abstract

The application discloses an enhanced refinement network device and method for target detection in severe weather, and relates to the field of traffic image processing. The enhanced refinement network comprises an adaptive image enhancement module, a feature refinement module and a target detection module which are sequentially connected. The adaptive image enhancement module comprises a limited contrast adaptive histogram equalization filter, an adaptive illumination enhancement filter and a Laplacian pyramid sharpening structure which are sequentially connected. The feature refinement module comprises a down-sampling layer, a feature refinement layer and an up-sampling layer which are sequentially connected through a residual block. The application uses a digital image enhancement technology to perform image enhancement processing first. In order to retain more detailed features of the image during image enhancement, the feature refinement module refines the blurred detailed features based on the feature refinement module of Unet.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of traffic target detection, and more particularly to an enhanced refinement network device and method for target detection in adverse weather. BACKGROUND

[0002] Traffic target detection is one of the important tasks in intelligent transportation systems. Through thousands of cameras on the road, traffic target detection can better obtain road traffic related information to support downstream tasks in intelligent transportation systems. At present, the traffic target detection method based on deep learning has made certain progress, but the real traffic scene is complex and changeable, such as rainy, foggy, overcast and other scenes. In the face of complex real traffic scenes, the general target detection model trained in the environment with sufficient lighting and no interference often performs poorly. This is because in adverse weather, light is refracted or scattered by particles in the air, blurring the images captured by the camera, thereby reducing the image quality and affecting the downstream visual tasks. The target detection task has a high requirement for image quality, and the general target detection model will be greatly affected when processing images in adverse weather. Therefore, it is of great significance to explore the precise and rapid traffic target detection method in adverse weather.

[0003] Improving image quality is an intuitive solution to traffic visual tasks in adverse environments. Many image enhancement methods can improve the visual quality of images, but the enhanced images may blur some key features of the original images and introduce new noise, which is not conducive to downstream visual tasks. Therefore, exploring a method to improve the visual quality and detail feature quality of images is a research hotspot to solve the influence of adverse weather on traffic target detection.

[0004] Effective image enhancement methods can improve the visual quality of images in adverse weather. At present, the mainstream image enhancement methods include digital image processing, image enhancement based on prior model, and image reconstruction based on deep learning. Digital image processing methods start from pixel filtering and use spatial domain enhancement, transform domain enhancement and other methods to improve the quality of images. Prior-based image enhancement methods mainly use prior knowledge, such as depth of field information, atmospheric scattering model, etc., to perform inverse restoration on the picture. The image reconstruction method based on deep learning mainly learns the mapping relationship between low-quality images and high-quality images through the powerful fitting ability of neural networks, or learns the prior model parameters to reconstruct clear pictures. The deep learning-based method is divided into supervised learning and unsupervised learning, the former needs pairs of low-quality and clear images, and the cost of collecting data sets is relatively high; the latter assumes that there is a distribution bias between images in adverse weather and clear images, generally adopts the domain self-adaptive principle, and focuses on fitting the distribution characteristics and potential information between low-quality images and clear images.

[0005] In order to better complete the target detection task, combining image enhancement with the target detection task is also one of the mainstream solutions. The method of combining image enhancement and target detection mainly adopts two subnets to realize visual effect enhancement and target detection respectively, and reduces the feature degradation caused by image enhancement through sharing the feature extraction layer. However, in the training process of the combined enhancement and target detection method, it is difficult to adjust the parameters to balance the weights between the two subnets. Another method is to adaptively detect the image, such as IAYOLO, which adds the hyperparameters of digital image processing into the neural network of target detection for unified optimization, but this method may have problems such as over-enhancement and reduced visual effect.

[0006] Therefore, how to solve the problems of over-enhancement and reduced visual effect is an urgent research for those skilled in the art. SUMMARY

[0007] Therefore, the present application provides an enhanced refinement network device and method for target detection in adverse weather to solve the problems in the background art.

[0008] In order to achieve the above purpose, the present application adopts the following technical solutions:

[0009] An enhanced refinement network device for target detection in adverse weather, the enhanced refinement network comprises an adaptive image enhancement module, a feature refinement module and a target detection module connected in sequence;

[0010] The adaptive image enhancement module comprises a limited contrast adaptive histogram equalization filter, an adaptive illumination enhancement filter and a Laplacian pyramid sharpening structure connected in sequence.

[0011] The feature refinement module comprises a down-sampling layer, a feature refinement layer and an up-sampling layer connected in sequence through a residual block.

[0012] Optionally, the target detection module adopts single-stage YOLOv5 as the detection network.

[0013] Optionally, the feature refinement module balances the refined features and visual features through a plurality of loss functions, and the plurality of loss functions comprise an image similarity loss function, a pixel feature loss function and a semantic feature loss function.

[0014] Optionally, the calculation formula of the plurality of loss functions is as follows:

[0015] ;

[0016] L content represents the semantic feature loss function, L ms-ssim represents the image similarity loss function, L pixel represents the pixel feature loss function, wcontent , w ms-ssim , w pixel respectively represent corresponding weights.

[0017] Optionally, the formula of the semantic feature loss function is as follows:

[0018] ;

[0019] wherein j represents the jth layer, C, H and W respectively represent the channel, height and width of the feature map, represents the picture after refinement enhancement, y represents the original picture, represents the jth layer feature map of VGG19.

[0020] Optionally, the formula of the image similarity loss function is as follows:

[0021] ;

[0022] wherein MS-SSIM() represents multi-scale SSIM.

[0023] Optionally, the formula of the pixel feature loss function is as follows:

[0024] ;

[0025] Gau() represents Gaussian blur, C, H and W are respectively the channel number, height and width of the feature map.

[0026] An enhanced refinement method for target detection in adverse weather, using any one of the enhanced refinement network devices for target detection in adverse weather, comprising the following steps:

[0027] Obtain traffic target detection image data set;

[0028] Input the traffic target detection image data set into the enhanced refinement network;

[0029] The enhanced refinement network uses an adaptive image enhancement module to enhance the image, adjusts the brightness of the image and enhances the edge effect; through a feature refinement module, the detail features are enhanced; through a target detection module, target image detection is completed.

[0030] Compared with the prior art, the application provides an enhanced refinement network device and method for target detection in bad weather, which uses a digital image processing technology to perform image enhancement processing, uses a Unet-based feature refinement module to enhance blurred detail features in order to retain more detail features of the image during image enhancement, and proposes a multi-loss function with pixel loss and feature loss in order to further improve the visual quality and detail feature quality of the image, so that a proper feature refinement model can be learned to improve the target detection performance. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings.

[0032] Figure 1 It is a schematic diagram of the overall network structure of the present application.

[0033] Figure 2 It is a self-adaptive image enhancement module diagram of the present application.

[0034] Figure 3 It is a residual block diagram of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] Referring to Figures 1-3 The embodiments of the present application disclose an enhanced refinement network device for target detection in bad weather, which comprises a self-adaptive image enhancement module, a feature refinement module and a target detection module connected in sequence.

[0037] The self-adaptive image enhancement module comprises a limited contrast adaptive histogram equalization filter, a self-adaptive illumination enhancement filter and a Laplacian pyramid sharpening structure connected in sequence.

[0038] The feature refinement module comprises a down-sampling layer, a feature refinement layer and an up-sampling layer connected in sequence through a residual block.

[0039] The residual block consists of two concatenated convolutional modules, each of which first performs a convolutional kernel of size 3. 3. Perform a 2D convolution with a stride of 1, then normalize using batch normalization, and finally perform a non-linear mapping using the ReLU activation function.

[0040] Furthermore, noise caused by severe weather typically has the following characteristics: 1) reduced contrast due to fog, haze, etc.; 2) shading and reflection due to rain, snow, etc.; 3) decreased brightness due to dust, sandstorms, etc.; 4) water vapor generated by vehicles passing by on the road during heavy rain can cause a certain degree of shading. To address these issues, this embodiment proposes an adaptive image enhancement module, such as... Figure 3 As shown, the Adaptive Image Enhancement (ADIE) module consists of three adaptive image filters: contrast-limited adaptive histogram equalization (CLAHE), adaptive gamma correction, and Laplacian pyramid sharpening. These filters can sharpen the image and perform pixel-level filtering.

[0041] Contrast-Limited Adaptive Histogram Equalization Filter (CLAHE): The contrast-limited adaptive histogram equalization filter (CLAHE) adjusts the color space by separately expanding the RGB channels, which can improve image contrast while suppressing noise. CLAHE first divides the original image into multiple sub-regions, then calculates the mapping relationship for each sub-region using the histogram equalization equation (Formula 1), distributing the gray levels exceeding the threshold evenly among the remaining gray levels. Finally, bilinear interpolation is used to calculate the enhanced image.

[0042] (1);

[0043] in, n i The pixel value is i The number of pixels, L This represents the maximum pixel value in the image. Fc( ) Indicates channel c The mapping function.

[0044] Adaptive illumination enhancement filter: Using an adaptive illumination enhancement filter can enhance image colors under different brightness conditions, making the image brightness more balanced. Adaptive gamma correction finds a correction value in the grayscale image of the RGB image, so that the mean of the normalized grayscale image is around 0.5, and then uses Formula 2 to adaptively correct each pixel in the image.

[0045] (2)

[0046] Laplacian pyramid sharpening: The Laplacian pyramid can extract most of the high-frequency information in an image, such as edge details. When the CLAHE algorithm is used for image visibility restoration, it may generate object noise. Using the Laplacian pyramid for edge sharpening can preserve edge details and reduce noise. The expression for the Laplacian pyramid is shown in Equation 3.

[0047] (3)

[0048] in L i The first of the Laplace pyramids i Layer images, G i Indicates the first i The original image of the layer, Gau Indicates a Gaussian filter. λ Represents the sharpening factor, where σ Use Gaussian standard deviation.

[0049] By applying CLAHE filtering, Agamma filtering, and Laplacian sharpening to the image separately, a preliminary adaptive enhanced image can be obtained. Weighted fusion of these preliminary enhanced images yields the final adaptive enhanced image. The adaptive image enhancement module enhances image contrast using the CLAHE method, maintains the naturalness of color and brightness using the Agamma method, and filters noise and enhances edge information using the Laplacian sharpening method, enabling the final enhanced image to reduce image noise caused by harsh environments.

[0050] The feature refinement module (RefineNet) is a lightweight feature refinement network. To reduce artifacts and noise generated after pixel filtering and image sharpening, RefineNet performs feature refinement. The description of RefineNet is as follows: Figure 2 As shown, the input is the enhanced image. UNet fully utilizes the feature maps of images at different resolutions, and finally, multiple loss functions are used to balance the refinement features and visual features. The basic structure of RefineNet will be described in detail below.

[0051] Residual connection: The task goal of RefineNet is to learn the residual image. An image in bad weather can be divided into two parts: low-frequency information and high-frequency information. Low-frequency information refers to the area where the brightness or grayscale value changes slowly in the image, which is represented in the picture as large flat areas such as the sky, road, and heavy fog. Low-frequency information describes the overall contour of the image. High-frequency information refers to areas where the image changes rapidly, which is represented in the picture as edges and noise. High-frequency information describes the local details of the image. The input and output of RefineNet share low-frequency information, directly model the residual image, and refine high-frequency information. Using the residual structure can obtain faster convergence speed and better processing effect.

[0052] Improved Unet feature refinement network: In order to make full use of multi-level features, a neural network based on Unet is used. Unlike the Unet model, multiple residual blocks are used for feature refinement in RefineNet. RefineNet consists of three parts, which are represented by different colors. The first part is the down-sampling layer. The parameters of the down-sampling convolution are denoted as 'Conv k s c', where k represents the size of the convolution kernel, s represents the step, and c represents the number of output channels. The second part is the feature refinement layer. The parameters of the feature refinement layer are denoted as 'ResBlock n', where n represents the number of residual blocks. The structure of ResBlock is shown in Figure 3 Conv represents a standard convolution kernel with k = 3, s = 1, and p = 1. The third part is the up-sampling layer. The parameters of the up-sampling layer are denoted as 'Conv k s c', where k represents the size of the convolution kernel, s represents the step, and c represents the number of output channels. add represents the residual connection, which is used to constrain RefineNet to learn the residual image.

[0053] Multi-item loss function: In order to make the feature refinement result meet the requirements of visual feature enhancement and high-level semantic feature enhancement, RefineNet uses a multi-item loss function that combines low-level feature loss and high-level semantic loss.

[0054] Using semantic feature loss can reduce artifacts caused by pixel-level optimization and improve the visual quality of the picture. Avoiding the use of guided filtering and other artifact processing algorithms speeds up the processing of the image. In addition, using semantic loss can preserve the detailed features needed for downstream feature extraction tasks, avoiding the situation where enhancing the visual effect suppresses the detection effect. The semantic feature loss is shown in equation 4:

[0055] (4)

[0056] Using image similarity loss can reduce the change of image structure in the image reconstruction process, and retain more high-frequency information. Compared with single-scale SSIM, MS-SSIM can maintain stable performance on images of different resolutions. The image similarity loss is shown in formula 5,

[0057] (5);

[0058] Using pixel feature loss can better maintain brightness and color. In order to obtain more objective brightness and color loss, the pixel feature loss of the image is calculated by using the image blurred by Gaussian blur. The calculation of pixel loss is shown in formula 6.

[0059] (6)

[0060] The total loss function of RefineNet is shown in formula 7.

[0061] (7)

[0062] In the present application, a single-stage YOLOv5 is selected as the detection network, and YOLOv5 is one of the most popular target detection models improved and open-sourced from YOLOv3. Compared with previous versions, the network structure and detection head are optimized in the YOLOv5 project, and a feature pyramid is added to realize multi-scale target prediction, thereby further improving the detection accuracy. In addition, YOLOv5 provides a variety of excellent target detection models, including s, m, l, x versions, and specific target detection models can be selected for different tasks. In the present application, the network architecture of the v5s version balanced in precision and speed is adopted. The loss function uses the original loss function of YOLOv5.

[0063] Target detection dataset: In order to better meet the task of traffic target detection, the filtered dataset is used as the target detection dataset to retrain yolov5s. The modification method of the dataset is to only keep the pictures related to traffic targets, and common traffic targets include people, small cars, buses, trucks, bicycles and motorcycles. The filtered dataset is called detect_base dataset.

[0064] Enhanced refinement dataset: the real scene dense fog dataset Dense-Haze and the outdoor training dataset OTS part of the large synthetic fog image dataset RESIDE are used as the training dataset deh_base. The sample size in deh_base is adjusted to 320 320 3, to adapt to the target detection model.

[0065] Test dataset: The target detection dataset RTTS in real-world foggy scenes and the real traffic environment dataset DAWN in different weather conditions are used.

[0066] Training details: RefineNet is implemented based on the Pytorch deep learning framework on the hardware of NVIDIA A10 GPU and Intel Xeon Silver 4210 CPU. Detialnet is trained using the SGD optimizer with an initial learning rate of 0.1 and linearly decreasing to 0.00001 over the training timeline, the momentum is set to 0.9, and the decay factor is set to 0.0001. The training batchsize of RefineNet is set to 16. The loss function weights w content , w ms-ssim , w pixel are set to 100, 0.5 and 0.5 respectively, wherein the semantic feature loss uses the feature map before the 3rd pool of vgg19.

[0067] In order to evaluate the enhancement refinement network in real environment under bad weather. The present application uses real traffic scene data under different weather conditions, and compares with several most advanced target detection models or single image enhancement methods under bad weather. The main comparison is between the enhancement refinement network and the target detection method under bad weather, such as IAYOLO.

[0068] The present application selects the general yolov5 model to directly detect the traffic scene data under bad weather as the baseline, and verifies the effectiveness of the method proposed in the present application.

[0069] Since the present application is to improve the traffic target detection effect under bad environment, the experiment evaluation uses mAP50 as the evaluation standard, mAP50 is a common target detection evaluation standard, P represents the maximum accuracy of a sample, AP represents the average accuracy of a class of samples, that is, the maximum accuracy P is calculated for each recall rate, mAP represents the average accuracy of all classes, and 50 represents the threshold of IOU≥50% under which mAP is calculated. The higher the value of mAP50, the higher the precision and recall rate of the model in the target detection task.

[0070] The present application uses a variety of real foggy scene datasets to verify the effectiveness of the enhancement refinement network. It contains low light, heavy haze, moderate haze, light haze and no haze, etc. Various weather conditions, including road traffic, outdoor square, outdoor grassland, station, etc. A variety of scenes. The enhancement refinement network and other methods are compared under a variety of real foggy scenes. As can be seen from Table 1, the technical scheme of the present application is significantly improved for the target detection effect in common and diverse foggy scenes.

[0071] Table 1 Comparative experiments

[0072]

[0073] The embodiment also discloses an enhanced refinement method for target detection in bad weather, and utilizes any one of the enhanced refinement network devices for target detection in bad weather, and the enhanced refinement network device is characterized by comprising the following steps:

[0074] acquiring a traffic target detection image dataset;

[0075] inputting the traffic target detection image dataset into the enhanced refinement network;

[0076] the enhanced refinement network uses an adaptive image enhancement module to enhance the image, adjusts the brightness of the image and enhances the edge effect; a feature refinement module is used to enhance the detailed features; and a target detection module is used to complete target image detection.

[0077] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other.

[0078] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An enhanced and refined network device for target detection under adverse weather conditions, characterized in that, The enhancement and refinement network consists of an adaptive image enhancement module, a feature refinement module, and an object detection module connected in sequence. The adaptive image enhancement module includes a contrast-limited adaptive histogram equalization filter, an adaptive illumination enhancement filter, and a Laplacian pyramid sharpening structure connected in sequence. The feature refinement module includes a downsampling layer, a feature refinement layer, and an upsampling layer connected in sequence via residual blocks; The feature refinement module balances the refinement of features and visual features through multiple loss functions, including image similarity loss function, pixel feature loss function and semantic feature loss function; The formulas for calculating the multiple loss functions are as follows: ； L content L represents the semantic feature loss function. ms-ssim L represents the image similarity loss function. pixel w represents the pixel feature loss function. content w ms-ssim w pixel These represent the corresponding weights; The formula for the semantic feature loss function is as follows: ； Where j represents the j-th layer, and C, H, and W represent the channels, height, and width of the feature map, respectively. This represents the image after refinement and enhancement, where y represents the original image. This represents the j-th layer feature map of VGG19; The formula for the image similarity loss function is as follows: ； MS-SSIM() represents multi-scale SSIM; The formula for the pixel feature loss function is as follows: ； Gau() represents Gaussian blur, where C, H, and W are the number of channels, height, and width of the feature map, respectively.

2. The enhanced and refined network device for target detection under adverse weather conditions according to claim 1, characterized in that, The target detection module uses a single-stage YOLOv5 as the detection network.

3. A method for enhancing and refining target detection under adverse weather conditions, utilizing the enhanced and refined target detection network device under adverse weather conditions as described in any one of claims 1-2, characterized in that, Includes the following steps: Obtain the traffic target detection image dataset; The traffic target detection image dataset is input into the enhancement and thinning network; The enhancement and thinning network uses an adaptive image enhancement module to enhance the image, adjusting its brightness and enhancing edge effects; a feature thinning module to refine image details and over-enhanced areas; and a target detection module to detect the target image.

Images