Adaptive target detection method in foggy weather based on convolutional neural network

Abstract

The present invention proposes a method for adaptive target detection in foggy conditions based on a convolutional neural network. This method detects targets in scenes with varying fog concentrations with high recognition accuracy. The method comprises: Step S1: Acquiring foggy image data using optical instruments; Step S2: Inputting the foggy image data into an image parameter prediction module based on a convolutional neural network structure to adaptively obtain the prediction parameters required for image defogging and image adjustment; Step S3: Inputting the prediction parameters and the foggy image data into an image processing module, which performs image defogging and image adjustment to obtain a restored image; Step S4: Using the image processed by the image processing module as input to a YOLOv4 detector, and utilizing the YOLOv4 detector to perform joint optimization of the defogging algorithm and target detection algorithm.

Description

Technical Field

[0001] The present invention belongs to the technical field of foggy weather target detection and relates to a foggy weather adaptive target detection method based on a convolutional neural network. Background Art

[0002] Object detection is a computer vision task that locates and labels target regions in videos or images. It then uses algorithms to extract features to identify and locate objects of a specific category. Object detection has a wide range of real-world applications, including face detection, vehicle detection, license plate recognition, pedestrian counting, and autonomous driving. Object detection algorithms learn by extracting features from video or image content and ultimately regressing the target classification and bounding box. Existing object detection models are mostly based on public datasets with no degradation in natural scenes. However, datasets specifically designed for object detection in foggy scenes are limited.

[0003] Scene images or videos captured by outdoor surveillance cameras are often affected by floating atmospheric particles such as fog and haze, resulting in blurred images or videos. This is because atmospheric light is scattered and reflected when passing through these suspended particles, causing changes in image color and contrast, resulting in image quality degradation. Compared to images on clear days, foggy images tend to be whiter and more blurred. In object detection tasks, detection accuracy is closely related to the quality of the image itself; reduced image quality leads to decreased detection accuracy.

[0004] Dehazing algorithms are used to remove fog from foggy images. They can be used as a preprocessing tool for target detection, improving the quality of detected images. By enhancing detected images with dehazing algorithms, higher target detection accuracy is expected.

[0005] Traditional image dehazing algorithms cannot be directly used for target detection. The main reason is that traditional dehazing algorithms and target detection algorithms are independent of each other. Dehazing algorithms can only serve as preprocessing tools for target detection. Target detection algorithms do not serve as input to guide the optimization direction of dehazing algorithms. Target features are not adequately expressed in foggy target detection tasks, and target extraction is difficult. Target detection models have poor robustness to different degrees of fog, and there is no deep fusion between dehazing algorithms and target detection algorithms. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a foggy adaptive target detection method based on convolutional neural network. This method can detect targets in scenes with different fog concentrations with high recognition accuracy.

[0007] A method for adaptive target detection in foggy weather based on a convolutional neural network comprises the following steps:

[0008] Step S1: Acquire foggy image data;

[0009] Step S2: inputting the foggy image data into an image parameter prediction module based on a convolutional neural network structure to obtain the CF-Net prediction parameters required for image defogging and image adjustment;

[0010] Step S3: inputting the CF-Net prediction parameters and the foggy image data into an image processing module, and the image processing module performing image defogging and image adjustment processing based on the CF-Net prediction parameters to obtain a restored image;

[0011] Step S4: The restored image processed by the image processing module is used as the input of the YOLOv4 detector, and the target detection result is output using the YOLOv4 detector, and data is fed back to the image parameter prediction module to optimize the prediction parameters.

[0012] The step S2 comprises: first pre-processing the foggy image data, when the image resolution is greater than 256×256, scaling the image to 256×256; when the image resolution is less than 256×256, keeping the original resolution unchanged;

[0013] A convolutional layer, a dilated convolutional layer, and a fully connected layer are selected as the network structure of the image parameter prediction module, and the CF-Net prediction parameters required for image defogging are predicted by learning the content of the image.

[0014] The image processing module includes three filters: a filter of the defogging processing module, an image pixel-by-pixel filter and an image sharpening processing filter; the filter of the defogging processing module includes: a defogging adjustment factor ω, a defogging bias coefficient C parameter; the image pixel-by-pixel filter includes white balance W, gamma transformation parameter G, contrast t and hue α; the image sharpening processing filter includes a sharpening parameter λ.

[0015] The optimized prediction parameters include:

[0016] Set training parameters: maximum training rounds are 80, and batch size is 6;

[0017] Determining the prediction parameters required by the image processing module through the parameter prediction module;

[0018] performing defogging filtering on the image;

[0019] The processed image is fed into the YOLOv4 detector, and the gradient is calculated by the loss of the YOLOv4 detector, and the prediction parameters of the CF-Net in the parameter prediction module are updated according to the gradient.

[0020] The image parameter prediction module includes a C1 convolution layer, a C2 convolution layer, a C3 convolution layer, a C4 dilated convolution layer, a C5 dilated convolution layer, an F1 fully connected layer, and an F2 fully connected layer;

[0021] The C1 convolutional layer receives the input foggy image data and performs a convolution operation and a LeakyRelu nonlinear activation function activation operation to perform downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 16;

[0022] The C2 convolution layer receives the image processed by the C1 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32;

[0023] The C3 convolution layer receives the image processed by the C2 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32;

[0024] The C4 hole convolution layer receives the image processed by the C3 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, where the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32;

[0025] The C5 dilated convolution layer receives the image processed by the C4 dilated convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, where the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32;

[0026] The F1 fully connected layer is connected to the data downsampled by the C5 void convolution layer for smoothing, and the F2 fully connected layer is connected to the F1 fully connected layer to obtain the prediction parameters, the defogging adjustment factor ω and the defogging bias coefficient C.

[0027] The defogging module represents the foggy image as shown in formula (1):

[0028]

[0029]

[0030] Where I(x) is the foggy image, J(x) represents the image after defogging, and A is the estimated atmospheric light value. is the transmittance, c is the pixel value of the three channels r red, g green and blue.

[0031] Design a contrast conversion function, a white balance mapping function, a gamma conversion mapping function, and a hue conversion mapping function, which constitute the image pixel-by-pixel filter of the image processing module, and input the pixel values ​​of the three channels of red, green and blue J(x)=(r m , g m , b m ), mapping out the new pixel values ​​of the three channels

[0032]

[0033] (r m , g m , b m ) represents the pixel values ​​of the red, green, and blue channels;

[0034] The contrast transformation function expression is shown in formula (2):

[0035] O c =(0.27r m , 0.67g m , 0.06b m ) (2);

[0036] The white balance mapping function expression is shown in formula (3), where W r , W g , W b is the parameter for white balance adjustment, r c , g c , b c are the pixel values ​​of the three channels after contrast transformation;

[0037] r c =0.27r m , g c =0.67g m , b c =0.06b m ;

[0038] O w =(W r r c , W g g c , W b b c ) (3);

[0039] The gamma transform mapping function expression is shown in formula (4), where G is the gamma transform parameter, r w , g w , b w are the pixel values ​​of the three channels after white balance transformation;

[0040] r w =W r r c , g w =W g g c ;

[0041] O gm =(r w G , g w G , b w G ) (4);

[0042] The tone transformation mapping function expression is shown in formula (5), where T is the tone curve, t is the internal parameter of the tone curve T, r gm , g gm , b gm are the pixel values ​​of the three channels after gamma transformation;

[0043] r gm =r w G , g gm =g w G , b gm =b w G ;

[0044]

[0045] The parameters in the pixel-by-pixel filter of the image are optimized using the neural network back propagation of the parameter prediction module.

[0046] The sharpening filter includes a mapping function for sharpening adjustment. The mapping function expression of the sharpening adjustment is shown in formula (6):

[0047] F(x,λ)=O n +λ(O n -Gau(O n )) (6);

[0048] Among them O n is the input image, Gau(I(x)) is the Gaussian filter, λ is the sharpening parameter, and F(x, λ) is the image result processed by the image processing module.

[0049] The embodiment of the present invention provides a method for adaptive target detection in foggy weather based on a convolutional neural network. This method utilizes a parameter prediction module, an image processing module, and a target detection module to implement adaptive parameter acquisition, learning, and prediction based on a convolutional neural network structure, thereby significantly improving the image defogging effect and image attributes such as contrast and hue. Joint learning of image defogging and target detection is achieved in an end-to-end manner. Through the weak supervision constraint method of the target detection box, the parameter prediction module learns image enhancement parameters that are suitable for target detection, thereby improving the performance of the final target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A schematic diagram of a network structure of a parameter prediction module provided by an embodiment of the present invention;

[0051] Figure 2 A schematic diagram of the network structure of an image processing module provided by an embodiment of the present invention;

[0052] Figure 3 A flow chart of a method for adaptive target detection in foggy weather based on a convolutional neural network is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0054] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] See also Figure 3 ,The present invention includes: step S1: collecting foggy image data using optical instruments and equipment;

[0056] Using optical instruments, we collected foggy images, broadly categorizing fog concentrations into three categories: light fog, moderate fog, and heavy fog. The collected foggy image dataset contains 81,110 images and 195,610 detections. Of these, 132,560 are for people, 10,640 for bicycles, 32,640 for cars, 8,220 for buses, and 10,520 for motorcycles.

[0057] Step S2: Input the foggy image data into an image parameter prediction module based on a convolutional neural network structure to adaptively obtain the prediction parameters required for image dehazing and image adjustment. The foggy image data is input into the image parameter prediction module based on a convolutional neural network structure. The input image is a maximum size of 256×256. If the image is larger than 256×256, it is adjusted to 256×256. If the image is smaller than 256×256, the input image size remains unchanged. The C1 convolutional layer receives the input image and performs convolution operations and LeakyRelu nonlinear activation functions for downsampling. The convolution kernel size is 3×3, the convolution stride is 2, and the output channel size is 16. The C2 convolutional layer receives the image processed by the C1 convolutional layer and performs convolution operations and LeakyRelu nonlinear activation functions for downsampling. The convolution kernel size is 3×3, the convolution stride is 2, and the output channel size is 32. The C3 convolutional layer receives the image processed by the C2 convolutional layer and performs convolution operations and LeakyRelu nonlinear activation functions for downsampling. The convolution kernel size is 3×3, the convolution stride is 2, and the output channel size is 32. The C4 atrous convolutional layer receives the image processed by the C3 convolutional layer and performs convolution operations and LeakyRelu nonlinear activation functions for downsampling. The convolution kernel size is 3×3, the convolution stride is 2, and the output channel size is 32. The C5 atrous convolutional layer receives the image processed by the C4 atrous convolutional layer and performs convolution operations and LeakyRelu nonlinear activation functions for downsampling. The convolution kernel size is 3×3, the convolution stride is 2, and the output channel size is 32. The F1 fully connected layer connects the downsampled data of the C5 atrous convolutional layer for smoothing. The F2 fully connected layer connects to the F1 fully connected layer to obtain the predicted values ​​of the final output parameters, the dehazing adjustment factor ω, and the dehazing bias coefficient C.

[0058] Step S3: inputting the prediction parameters and the foggy image data into an image processing module, and the image processing module performs image defogging and image adjustment processing to obtain a restored image;

[0059] Attachment Figure 2 The image processing module's processing flow is shown. The module consists of three differentiable filters. Each filter has different parameters. The dehazing filter consists of a dehazing adjustment factor ω and a dehazing bias coefficient C. The pixel-by-pixel image filter includes four components: white balance, gamma, contrast, and hue. The corresponding parameters are W, G, t, and α. The sharpening parameter of the image sharpening filter is λ.

[0060] The present invention designs a defogging filter with learnable parameters. The defogging parameters obtained from the convolutional neural network are denoted as the defogging adjustment factor ω and the defogging bias coefficient C. The foggy image can be expressed as shown in formula (1):

[0061]

[0062]

[0063] Where I(x) is the foggy image, J(x) represents the image after defogging, and A is the estimated atmospheric light value. is the transmittance map. Since the above operations are differentiable, this paper can optimize ω and C through backpropagation, making the dehazing filter more effective for detecting foggy images. Images processed by the dehazing algorithm have an overall darker hue, requiring the addition of a pixel-by-pixel processing module to adjust the image contrast and improve image quality.

[0064] Design a contrast conversion function, a white balance mapping function, a gamma conversion mapping function, and a hue conversion mapping function, which constitute the image pixel-by-pixel filter of the image processing module, and input the pixel values ​​of the three channels of red, green and blue J(x)=(r m , g m , b m ), mapping out the new pixel values ​​of the three channels

[0065]

[0066] (r m , g m , b m ) represents the pixel values ​​of the red, green, and blue channels;

[0067] The contrast transformation function expression is shown in formula (2):

[0068] O c =(0.27r m , 0.67g m , 0.06b m ) (2)

[0069] The white balance mapping function expression is shown in formula (3), where W r , W g , W b is the parameter for white balance adjustment, r c , g c , b c are the pixel values ​​of the three channels after contrast transformation;

[0070] r c=0.27r m , g c =0.67g m , b c =0.06b m ;

[0071] O w =(W r r c , W g g c , W b b c ) (3)

[0072] The gamma transform mapping function expression is shown in formula (4), where G is the gamma transform parameter, r w , g w , b w are the pixel values ​​of the three channels after white balance transformation;

[0073] r w =W r r c , g w =W g g c , b w =W b b c ;

[0074] O gm =(r w G , g w G , b w G ) (4)

[0075] The tone transformation mapping function expression is shown in formula (5), where T is the tone curve, t is the internal parameter of the tone curve T, r gm , g gm , b gm are the pixel values ​​of the three channels after gamma transformation; r gm =r w G , g gm =g w G , b gm =b w G ;

[0076]

[0077] The parameters in the pixel-by-pixel filter of the image are optimized using the neural network back propagation of the parameter prediction module.

[0078] The image processed by the dehazing algorithm is relatively smooth and the edge information is not obvious. In order to solve this problem, sharpening is performed after dehazing to make the image edge details richer. The mapping function of sharpening adjustment is shown in formula (6):

[0079] F(x,λ)=O n +λ(O n -Gau(O n )) (6);

[0080] Among them O n is the input image, Gan(I(x)) represents the Gaussian filter, λ is the proportional factor, and F(x, λ) is the image result processed by the image processing module. This sharpening operation is differentiable with respect to both x and λ. The degree of sharpening can be adjusted by optimizing λ to achieve better object detection performance.

[0081] Step S4: Using the image processed by the image processing module as the input of the YOLOv4 detector, and using the YOLOv4 detector to complete the joint optimization of the dehazing algorithm and the target detection algorithm;

[0082] As attached Figure 3 As shown, the entire invention consists of a parameter prediction module, an image processing module, and a detection network, YOLOv4. First, the input image is resized to 256×256 and fed into the parameter prediction module to learn and predict the optimal parameters required by the image processing module. The image processing module then performs dehazing and image optimization. The image processed by the image processing module is then used as input to the YOLOv4 detector. YOLOv4's weakly supervised detection box parameter prediction module is used to jointly optimize the dehazing algorithm and the object detection algorithm.

[0083] The specific process of joint optimization is as follows:

[0084] (1) Initialize the parameter prediction module P with pre-trained parameters θ θ , random parameter β initializes the YOLOv4 network D β .

[0085] (2) Set the training phase: num_epochs = 80, batch_size = 6.

[0086] (3) Prepare the foggy dataset VOC_f.

[0087] (4) Through P N =P θ (image_batch) calculates the parameters required by the image processing module.

[0088] (5) Image filtering is performed through image_batch=FP-Net(image_batch,PN), where FP-Net is a collection operation of image processing modules.

[0089] (6) Send image_batch to YOLOv4 network D β ; Update the parameter prediction module P according to the YOLOv4 detection loss θ and YOLOv4 network D β .

[0090] Finally, it should be noted that the above implementation methods are only used to illustrate the technical solutions of the embodiments of the present invention and are not limiting. Although the embodiments of the present invention are described in detail with reference to the above preferred implementation methods, ordinary technicians in this field should understand that the technical solutions of the embodiments of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for adaptive target detection in foggy weather based on convolutional neural network, characterized in that: The following steps are involved: Step S1: Acquire foggy image data; Step S2: inputting the foggy image data into an image parameter prediction module based on a convolutional neural network structure to obtain the CF-Net prediction parameters required for image defogging and image adjustment; Step S3: inputting the CF-Net prediction parameters and the foggy image data into an image processing module, and the image processing module performing image defogging and image adjustment processing based on the CF-Net prediction parameters to obtain a restored image; Step S4: using the restored image processed by the image processing module as the input of the YOLOv4 detector, and using the YOLOv4 detector to complete the target detection result output, and feed back data to the image parameter prediction module to optimize the prediction parameters; The image processing module includes three filters: a filter of the defogging processing module, an image pixel-by-pixel filter, and an image sharpening processing filter; The filter of the defogging processing module includes: a defogging adjustment factor ω, a defogging bias coefficient C parameter; the image pixel-by-pixel filter includes white balance W, gamma transformation parameter G, contrast t and hue α; the image sharpening processing filter includes a sharpening parameter λ; The defogging module represents the foggy image as shown in formula (1): Among them, I(x) is the foggy image, J(x) represents the image after defogging, and A is the estimated atmospheric light value. is the transmittance, c is the pixel value of the three channels r red, g green and blue; Design a contrast conversion function, a white balance mapping function, a gamma conversion mapping function, and a hue conversion mapping function, which constitute the image pixel-by-pixel filter of the image processing module, and input the pixel values ​​of the three channels of red, green and blue J(x)=(r m , g m , b m ), mapping out the new pixel values ​​of the three channels (r m , g m , b m ) represents the pixel values ​​of the red, green, and blue channels; The contrast transformation function expression is shown in formula (2): O c =(0.27r m ,0.67g m ,0.06b m ) (2); The white balance mapping function expression is shown in formula (3), where W r , W g , W b is the parameter for white balance adjustment, r c , g c , b c are the pixel values ​​of the three channels after contrast transformation; r c =0.27r m , g c =0.67g m , b c =0.06b m ; O w =(W r r c ,W g g c ,W b b c ) (3); The gamma transform mapping function expression is shown in formula (4), where G is the gamma transform parameter, r w , g w , b w are the pixel values ​​of the three channels after white balance transformation; r w =W r r c , g w =W g g c , b w =W b b c ; O gm =(r w G ,g w G ,b w G) (4); The tone transformation mapping function expression is shown in formula (5), where T is the tone curve, t is the internal parameter of the tone curve T, r gm , g gm , b gm are the pixel values ​​of the three channels after gamma transformation; r gm =r w G , g gm =g w G , b gm =b w G The parameters in the pixel-by-pixel filter of the image are optimized using the neural network back propagation of the parameter prediction module.

2. The method for adaptive target detection in foggy weather based on convolutional neural network according to claim 1, characterized in that: The step S2 comprises: first pre-processing the foggy image data, when the image resolution is greater than 256×256, scaling the image to 256×256; when the image resolution is less than 256×256, keeping the original resolution unchanged; A convolutional layer, a dilated convolutional layer, and a fully connected layer are selected as the network structure of the image parameter prediction module, and the CF-Net prediction parameters required for image defogging are predicted by learning the content of the image.

3. The method for adaptive target detection in foggy weather based on convolutional neural network according to claim 1, characterized in that: The optimized prediction parameters include: Set training parameters: maximum training rounds are 80, and batch size is 6; Determining the prediction parameters required by the image processing module through the parameter prediction module; performing defogging filtering on the image; The processed image is fed into the YOLOv4 detector, and the gradient is calculated by the loss of the YOLOv4 detector, and the prediction parameters of the CF-Net in the parameter prediction module are updated according to the gradient.

4. The method for adaptive target detection in foggy weather based on convolutional neural network according to claim 2, characterized in that: The image parameter prediction module includes a C1 convolution layer, a C2 convolution layer, a C3 convolution layer, a C4 dilated convolution layer, a C5 dilated convolution layer, an F1 fully connected layer, and an F2 fully connected layer; The C1 convolutional layer receives the input foggy image data and performs a convolution operation and a LeakyRelu nonlinear activation function activation operation to perform downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 16; The C2 convolution layer receives the image processed by the C1 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32; The C3 convolution layer receives the image processed by the C2 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, wherein the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32; The C4 hole convolution layer receives the image processed by the C3 convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, where the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32; The C5 dilated convolution layer receives the image processed by the C4 dilated convolution layer and performs convolution operation and LeakyRelu nonlinear activation function activation operation for downsampling, where the convolution kernel size is 3×3, the convolution step size is 2, and the output channel size is 32; The F1 fully connected layer is connected to the data downsampled by the C5 void convolution layer for smoothing, and the F2 fully connected layer is connected to the F1 fully connected layer to obtain the prediction parameters, the defogging adjustment factor ω and the defogging bias coefficient C.

5. The method for adaptive target detection in foggy weather based on convolutional neural network according to claim 1, characterized in that: The sharpening filter includes a mapping function for sharpening adjustment. The mapping function expression of the sharpening adjustment is shown in formula (6): F(x,λ)=O n +λ(O n -Gau(The n )) (6); Among them O n is the input image, Gau(I(x)) is the Gaussian filter, λ is the sharpening parameter, and F(x, λ) is the image result processed by the image processing module.

Images