A haze image target detection method based on a double-layer alternating deep network

Abstract

The application belongs to the technical field of computer vision, and discloses a haze image target detection method based on a double-layer alternating deep network, which comprises the following steps: step 1, constructing a network training data set based on an atmospheric scattering model, and performing a pretreatment operation on the network training data set; step 2, delivering the haze image after the pretreatment to a shallow haze removal network to generate a shallow haze removal image, wherein the shallow haze removal network is composed of a transmission map generation network and an atmospheric light map generation network double-branch network; step 3, delivering the shallow haze removal image to a YOLOv3 detection network and a multi-scale spatial attention module; step 4, inputting the outputs of step 2 and step 3 into a fusion module to perform feature fusion, and then delivering the outputs to a deep haze removal network to generate a deep haze removal image; and step 5, inputting the deep haze removal image into the YOLOv3 detection network to obtain a target detection result; experiments prove that the application can significantly improve the target detection result of a haze image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a haze image target detection method based on a double-layer alternating deep network. BACKGROUND

[0002] Haze image target detection refers to automatically recognizing and locating target objects in images under low-visibility or adverse weather conditions such as haze, using computer vision technology and image processing methods. This technology aims to accurately detect various targets such as vehicles, pedestrians, and buildings in haze weather by analyzing the features and patterns in the image, and extracting their location information and other related attributes to support various application fields such as traffic safety, urban monitoring, and disaster rescue. In order to cope with the challenges brought by haze weather, haze image target detection tasks have practical application value in many fields.

[0003] Haze image target detection methods can be divided into three categories: 1) directly predicting haze images using target detection methods, 2) first removing haze from images using a haze removal method, and then predicting the haze-removed images using a target detection method, and 3) using a haze removal and target detection combined model to achieve haze image target detection. Directly predicting haze images using target detection methods is difficult to accurately detect targets from haze images because haze images often appear grayish in color, low in visibility, and low in contrast, giving a hazy and blurred feeling. The method of first removing haze from images using a haze removal method and then predicting the haze-removed images using a target detection method uses two steps to complete the target detection task, but the results of the target detection are affected by the haze removal network, and the network is complex and difficult to train, with low robustness and generalization. Although existing haze image target detection methods have made some progress, there are still some defects, the main problem being that it is difficult to meet the requirements of universality and robustness. SUMMARY

[0004] To solve the above technical problems, the present application adopts a haze image target detection network framework based on a double-layer alternating deep network, and designs a double-layer network structure that fuses target detection features in the deep haze removal stage, so that the haze removal network focuses on the detection area and improves the detection performance.

[0005] To achieve the above purpose, the present application is implemented by the following technical scheme: a haze image target detection method based on a double-layer alternating deep network, the method comprising the following steps:

[0006] Step 1, constructing a network training data set based on an atmospheric scattering model and performing preprocessing operations on it;

[0007] Step 2, the preprocessed haze image is transmitted to a shallow haze removal network to generate a shallow haze removal image; the shallow haze removal network is composed of a transmission map generation network and an atmospheric light map generation network double-branch network;

[0008] Step 3, the shallow haze removal image is transmitted to a YOLOv3 detection network and a multi-scale spatial attention module MSSAM;

[0009] Step 4, the outputs of step 2 and step 3 are input into a fusion module for feature fusion, and then transmitted to a deep haze removal network to generate a deep haze removal image;

[0010] Step 5, the deep haze removal image is input into the YOLOv3 detection network to obtain a target detection result.

[0011] Further, the network training data set based on the atmospheric scattering model is constructed in step 1, and a preprocessing operation is performed, and the specific steps are as follows:

[0012] Step 1.1, 8111 images containing human and other species targets in VOC2007 training and prediction data set and VOC2012 training and prediction data set are selected to construct VOC_Norm training data set; 2734 images containing human and other species targets in VOC2007 test data set are selected to construct VOC_Norm_test test data set;

[0013] Step 1.2, the VOC_Norm training data set is randomly fogged based on the atmospheric scattering model formula, and the number of foggy images to clean images is 2:1, a total of 24333 images, to construct a mixed training data set; the VOC_Norm_test test data set is randomly fogged based on the atmospheric scattering model formula to generate 2734 haze images, to construct a VOC_fog_test test data set;

[0014] Step 1.3, select the RTTS data set containing 4322 haze images to construct the RTTS test data set; finally, the image labels of the VOC annotation data in all data sets are converted into YOLO annotation data;

[0015]

[0016] Wherein, x is the image pixel position, d(x) is the depth map, p is the Euler distance of the pixel point to the center point, row and col are the height and width of the image respectively, and the depth map d(x) corresponding to the image size is generated according to the image size; i is a random integer of 0-9, and b is a scattering factor, two i values are randomly initialized for each image to generate a random scattering factor b; t(x) is a transmission map, and the corresponding transmission map t(x) is generated from the depth map d(x) and the scattering factor b; I(x) is a haze image, GT(x) is a clean image, and A(x) is an atmospheric light map;

[0017] Step 1.4, the image size in the training data set is adjusted to 512*512, and multi-scale training is used for training, then the image is randomly scaled or enlarged according to 70%-150%, and the normalized processing is carried out on the image after random scaling or enlargement, the data of the input image is 0-255, and the normalization is 0-1.

[0018] Further, the transmission map generation network and the atmospheric light map generation network, each branch network includes batch normalization BN, two 3*3 convolution, Relu activation layer and Sigmoid activation layer, the preprocessed haze image is transmitted to the shallow haze removal network in step 2 to generate a shallow haze removal image, and the specific operation is as follows:

[0019] An input haze image I(x) is output by the double branch network respectively And The shallow haze removal image feature J(x) is generated by the atmospheric scattering model transformation formula:

[0020]

[0021] Step 2 generates And The network structure of the intermediate variable is as follows:

[0022]

[0023]

[0024]

[0025] Wherein, I(x) is the input image of the shallow haze removal network, and the input image size is 3*512*512; first, the input image I(x) is standardized by batch normalization BN, Is the image feature after BN standardization; then, the two branch networks are output respectively And And The equivalent atmospheric light map and the transmission map output by the shallow defogging network are input, in each branch network, the feature dimension output by the first convolutional layer is 12x512x512, and the feature dimension output by the second convolutional layer is 3x512x512; Conv 3×3 is a 3x3 convolution operation, represents a concatenation operation.

[0026] Further, the step 3 of transmitting the shallow defogging image to the YOLOv3 detection network and the multi-scale spatial attention module MSSAM is as follows:

[0027] Step 3.1, the target detection feature extraction part, the shallow defogging image is input into the YOLOv3 detection network to output target detection, and three size features of target detection are extracted;

[0028] Step 3.2, the multi-scale feature reconstruction part, first, the three size features are converted into a unified size by using up-sampling respectively; then, the multi-scale features are fused; finally, the feature size is unified with the size of the input shallow defogging image by using up-sampling;

[0029] Step 3.3, the spatial attention feature generation part, the spatial attention network SAM is used to generate spatial attention features from the features reconstructed in step 3.2;

[0030] Step 3.4, the generated spatial attention features are multiplied with the input shallow defogging image.

[0031] In step 3.3, the spatial attention network SAM includes batch normalization BN, two 1x1 convolution, Relu activation layer and Sigmoid activation layer, and the specific operation of the spatial attention network SAM is as follows:

[0032]

[0033] Wherein, S in is the input feature of SAM, S out is the output feature of SAM, Conv 1×1 is a 1x1 convolution operation.

[0034] Further, the step 4 of inputting the output of step 2 and step 3 into the fusion module for feature fusion is as follows:

[0035] SF out = XSF in1 + (1-X)SF in2

[0036] X = Sigmoid(Conv 1×1 (Z)·Conv 1×1 (Z))

[0037]

[0038] X h =GAP h (SF in1 +SF in2 )

[0039] X w =GAP w (SF in1 +SF in2 )

[0040] where SF in1 and SF in2 are two input features of the fusion module, SF out is the output feature of the fusion module, GAP h and GAP w are average pooling operations in the vertical and horizontal dimensions, respectively, GAP h converts the feature CxHxW into Cx1xW, GAP w converts the feature CxHxW into CxHx1, and then exchanges the dimensions to become Cx1xH, the Cat operation splices the features together; Sigmoid and Relu are activation layers, and BN is batch normalization.

[0041] Further, the YOLOv3 detection network in step 5 specifically operates as follows:

[0042] The YOLOv3 network is defined as:

[0043] Y out1 ,Y out2 ,Y out3 = YOLOv3(J^(x))

[0044] where J^(x) is the input image of the YOLOv3 detection network, Y out1 , Y out2 , Y out3 are detection results of different sizes of the YOLOv3 detection network; when the input image size is 512x512, Y out1 , Y out2 , Y out3 output feature dimensions are 3x16x16x10, 3x32x32x10, 3x64x64x10, respectively, Y out1 divides the 512x512 input image into 32x32 grids, each grid has 3 prior boxes, a total of 3x16x16 prediction boxes, responsible for predicting the target center in the grid, used to detect large objects in the image; Y out2The input image is divided into a 16x16 grid for detecting medium objects in the image; Y out3 The input image is divided into an 8x8 grid for detecting small objects in the image; each prediction box respectively predicts 10 data, which are target bounding box center offset (t x ,t y ), width and height scaling factor (t w ,t h ), target confidence conf and probability of corresponding 5 categories.

[0045] The beneficial effects of the present application are: (1) the present application designs a joint network of dehazing and target detection, compared with dehazing network + target detection network, the network is easy to train; compared with the target detection network, a small amount of parameters is added while the performance of the network for foggy image target detection is improved.

[0046] (2) the present application designs a two-stage network, the shallow dehazing network restores the image information as a whole, the deep dehazing network restores the image information of the target detection related area from the local, and the image on the shallow dehazing feature reduces the influence of fog and haze on the target detection effect through the deep dehazing.

[0047] (3) the present application designs a target detection multi-scale spatial attention network, extracts the importance of different parts of the image, reduces the processing of irrelevant information for target detection, makes the dehazing network focus on the area needing target detection, removes the background features and useless information of the target detection area, and improves the accuracy of target detection. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the overall network structure diagram of the present application.

[0049] Figure 2 is the dehazing network structure diagram of the present application.

[0050] Figure 3 is the multi-scale spatial attention network (MSSAM) structure diagram of the present application.

[0051] Figure 4 is the fusion module (SF) structure diagram of the present application. DETAILED DESCRIPTION

[0052] The present application is a foggy image target detection method based on a double-layer alternating deep network, and the overall network structure diagram is as follows Figure 1As shown, the process first passes the haze image to a shallow dehazing network to generate shallow dehazing image features. Then, these features are passed to an object detection network and a multi-scale spatial attention module to generate spatial attention features that are fused with the shallow dehazing image. The fused image features are then passed to a deep dehazing network to achieve haze removal based on object detection. Finally, the data is input into the object detection network to perform object detection on the haze image. Specifically, the process includes the following steps:

[0053] Step 1: Construct a network training dataset based on the atmospheric scattering model and perform preprocessing operations on it, including the following steps:

[0054] Step 1.1: Select 8111 images of five targets (humans, etc.) from the VOC2007 and VOC2012 training and prediction datasets to construct the VOC_Norm training dataset; select 2734 images of five targets (humans, etc.) from the VOC2007 test dataset to construct the VOC_Norm_test test dataset.

[0055] Step 1.2: The VOC_Norm training dataset is randomly fogged based on the atmospheric scattering model formula, with a fogged image to clean image ratio of 2:1, totaling 24333 images, to construct a mixed training dataset; the VOC_Norm_test test dataset is randomly fogged based on the atmospheric scattering model formula, generating 2734 haze images, to construct the VOC_fog_test test dataset.

[0056] Step 1.3: Select an RTTS dataset containing 4322 haze images to construct an RTTS test dataset; finally, convert the image labels of VOC-annotated data in all datasets to YOLO-annotated data;

[0057]

[0058] Where x is the position of an image pixel, d(x) is the depth map, ρ is the Eulerian distance from the pixel to the center point, row and col are the height and width of the image, respectively, and a depth map d(x) of the corresponding image size is generated according to the image size; i is a random integer from 0 to 9, β is the scattering factor, two i values ​​are randomly initialized for each image to generate a random scattering factor β; t(x) is the transmission map, which is generated from the depth map d(x) and the scattering factor β; I(x) is the haze image, GT(x) is the clean image, and A(x) is the atmospheric light map, with A(x) set to 0.5, and the haze image dataset is constructed from the atmospheric scattering model;

[0059] Step 1.4, the image size in the training data set is adjusted to 512x512, and multi-scale training is used for training, then the image is randomly scaled or enlarged by 70%-150%, and the normalized image after random scaling or enlargement is subjected to data enhancement operation, the data of the input image is 0-255, and the normalization is 0-1.

[0060] Step 2, the preprocessed haze image is transmitted to the shallow haze removal network to generate a shallow haze removal image; the shallow haze removal network is composed of a transmission map generation network and an atmospheric light map generation network double branch network;

[0061] An input haze image I(x) is output by the double branch network And The shallow haze removal image feature J(x) is generated by the atmospheric scattering model transformation formula:

[0062]

[0063]

[0064] Wherein,

[0065] The transmission map generation network and the atmospheric light map generation network each branch network includes batch normalization BN, two 3x3 convolution, Relu activation layer and Sigmoid activation layer, and generates And The network structure of the intermediate variable is as follows:

[0066] Y in = BN(I(x))

[0067]

[0068]

[0069] Wherein, I(x) is the input image of the shallow haze removal network, and the input image size is 3x512x512; first, the input image I(x) is normalized by batch normalization BN, is the image feature after BN normalization; then, the two branch network outputs And And The equivalent atmospheric light map and transmission map output by the shallow haze removal network, in each branch network, the first convolutional layer outputs a feature dimension of 12x512x512, and the second convolutional layer outputs a feature dimension of 3x512x512; Conv 3×3 is a 3x3 convolution operation, Table concatenation operation.

[0070] Step 3, the shallow dehazing image features are transmitted to the YOLOv3 detection network and the multi-scale spatial attention network module (MSSAM); the YOLOv3 detection network and the multi-scale spatial attention network are composed of four parts:

[0071] S3.1), target detection features are extracted, the shallow dehazing image is input into the YOLOv3 detection network to output target detection, and three size features of target detection are extracted; when the size of the shallow dehazing image is 3×512×512, the feature dimensions are 3×16×16, 3×32×32, and 3×64×64, respectively;

[0072] S3.2), a multi-scale feature reconstruction part, first, the three size features are converted into a unified size using upsampling; when the size of the shallow dehazing image is 3×512×512, the feature is converted into 3×128×128; then, the multi-scale features are fused using a fusion module; finally, the feature size is unified with the size of the input shallow dehazing image 3×512×512 using upsampling;

[0073] S3.3), spatial attention features are generated using a spatial attention network (SAM) to generate spatial attention features from the reconstructed features;

[0074] S3.4), the generated spatial attention features are multiplied with the input shallow dehazing image features.

[0075] In S3.3, the spatial attention network SAM includes batch normalization BN, two 1×1 convolution, Relu activation layer and Sigmoid activation layer, and the specific operation of the spatial attention network SAM is as follows:

[0076]

[0077] Wherein, S in is the input feature of SAM, S out is the output feature of SAM, and Conv 1×1 is a 1×1 convolution operation.

[0078] Step 4, the outputs of step 2 and step 3 are input into a fusion module for feature fusion, and then transmitted to a deep dehazing network to generate a deep dehazing image; the specific operation of feature fusion is as follows:

[0079] SF out =XSF in1 +(1-X)SF in2

[0080] X=Sigmoid(Conv 1×1 (Z)·Conv 1×1 (Z))

[0081]

[0082] X h = GAP h (SF in1 + SF in2 )

[0083] X w = GAP w (SF in1 + SF in2 )

[0084] where SF in1 and SF in2 are two input features of the fusion module, SF out is the output feature of the fusion module, GAP h and GAP w are average pooling operations in the vertical and horizontal dimensions, respectively, GAP h converts the feature CxHxW into Cx1xW, GAP w converts the feature CxHxW into CxHx1, and then exchanges the dimensions to become Cx1xH, and the Cat operation splices the features together.

[0085] Step 5: input the deep haze-removed image into the YOLOv3 detection network to obtain the target detection result. The YOLOv3 network is used:

[0086] Y out1 , Y out2 , Y out3 = YOLOv3(J^(x))

[0087] where J^(x) is the input image of the YOLOv3 detection network, Y out1 , Y out2 , Y out3 are detection results of different sizes of the YOLOv3 detection network. When the size of the input image is 512x512, the feature dimensions of Y out1 , Y out2 , Y out3 are 3x16x16x10, 3x32x32x10, 3x64x64x10, respectively, Y out1 divides the 512x512 input image into 32x32 grids, and there are 3 prior boxes in each grid, a total of 3x16x16 prediction boxes, which are responsible for predicting the target whose center is in the grid, for detecting large objects in the image; Y out2 divides the input image into 16x16 grids, which are used to detect medium objects in the image; Y out3The input image is divided into an 8x8 grid for detecting small objects in the image. Each prediction box predicts 10 data, respectively, target bounding box center offset (t x ,t y ), width and height scaling factor (t w ,t h ), target confidence conf and probability of corresponding 5 categories.

[0088] Step 6, select a variety of haze image data sets to test the performance of the network. The target detection loss function is constructed, the Adam optimizer is selected, the learning rate is set to 0.001, the batch size is set to 6, the number of rounds is set to 50, and the network is started. In order to prove the effectiveness of the haze image target detection method proposed in the present application, three kinds of haze image target detection methods are selected for comparison, which are target detection method (YOLOv3 uses VOC_Norm training set training and mixed training set training), dehazing network + YOLOv3 (dehazing network selects GDNet, MSBDN, FFANet three networks), dehazing and target detection joint model method (IA-YOLOv3), all comparison methods are tested using the default parameters provided in the literature. Select AP and AP 50 objective indicators as quantitative indicators. AP 50 is the average precision when the IoU threshold is 50%, and AP is the average precision when the IoU threshold is 50% to 95%.

[0089] Test on VOC_Norm_test data set, VOC_fog_test data set and RTTS data set, Table 1 is the average index value of various haze image target detection algorithms on three data sets.

[0090] From Table 1, it can be seen that the present application achieves the best result on three data sets. Compared with the dehazing and target detection joint model method IA-YOLOv3, the present application improves AP 50 0.83 on VOC_Norm_test haze-free data set, AP 50 2.25 on VOC_fog_test haze data set, and AP 50 3.41 on RTTS haze data set, which proves the effectiveness and generalization of the haze image target detection network.

[0091] Table 1 Average index of various haze image target detection algorithms on three data sets

[0092]

[0093] In summary, the present application is a feasible and efficient haze image target detection method, which can significantly improve the performance of haze image target detection.

[0094] The above merely illustrates the embodiments of the present application but should not be taken as limitations. Various modifications and changes can be made by those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. A method for target detection in haze images based on a two-layer alternating deep network, characterized in that, The method includes the following steps: Step 1: Construct a network training dataset based on the atmospheric scattering model and perform preprocessing operations on it; Step 2: Pass the preprocessed haze image to the shallow dehazing network to generate a shallow dehazing image. The shallow dehazing network consists of two branches: a transmission map generation network and an atmospheric light map generation network. Each branch of the transmission map generation network and the atmospheric light map generation network includes batch normalization (BN) and two... convolution, Activation layer and To activate the layer, follow these steps: Input a haze image The outputs are obtained through a dual-branch network. and Shallow dehazing image features are generated using atmospheric scattering model transformation formulas. : ； in, Represents transmission graph characteristics; Indicates global atmospheric light characteristics; Step 3: Pass the shallow dehazed image to the YOLOv3 detection network and the multi-scale spatial attention module MSSAM; Step 3.1, Target Detection Feature Extraction: Input the shallow dehazed image into the YOLOv3 detection network to output target detection and extract the three size features of the target detection. Step 3.2, multi-scale feature reconstruction: First, upsampling is used to convert the three sizes of features into a unified size; then, the multi-scale features are fused; finally, upsampling is used to unify the feature size with the size of the input shallow dehazed image. Step 3.3, Spatial Attention Feature Generation: The Spatial Attention Network (SAM) is used to generate spatial attention features from the features reconstructed in Step 3.

2. Step 3.4: Multiply the generated spatial attention features with the input shallow dehazed image; Step 4: Input the outputs of Step 2 and Step 3 into the fusion module for feature fusion, and then pass them to the deep dehazing network to generate a deep dehazing image; The specific steps are as follows: ； ； ； ； ； in, and These are the two input features of the fusion module. For the output characteristics of the fusion module, and These represent average pooling operations along the vertical and horizontal axes, respectively. Features Transform into , Features Transform into Then exchange dimensions to transform , The operation stitches the features together; and For the activation layer, BN is used for batch normalization; Step 5: Input the deep dehazing image into the YOLOv3 detection network to obtain the target detection results.

2. The method for target detection in haze images based on a two-layer alternating deep network according to claim 1, characterized in that, The specific steps for constructing the network training dataset based on the atmospheric scattering model in step 1 and performing preprocessing operations are as follows: Constructing a network training dataset based on an atmospheric scattering model: ； in, The position of an image pixel. This is a depth-of-field image. The distance from the pixel to the center point is the Euler distance. and The image's height and width are given respectively, and a depth map of the corresponding image size is generated based on the image dimensions. ; A random integer between 0 and 9. Two scattering factors are randomly initialized for each image. Values ​​that generate random scattering factors. ; This is a transmission image, derived from a depth image. With scattering factor Generate the corresponding transmission map ; Image of smog. For a clean image, This is an atmospheric light map; Resize the images in the training dataset to... The training adopts a multi-scale training method, and then the images are randomly scaled or enlarged from 70% to 150%, and the images after random scaling or enlargement are normalized by data augmentation.

3. The method for target detection in haze images based on a two-layer alternating deep network according to claim 2, characterized in that: Step 1, which involves constructing the network training dataset based on the atmospheric scattering model, specifically includes: The VOC_Norm training dataset was constructed by selecting 8111 images of human and other species targets from the VOC2007 training and prediction dataset and the VOC2012 training and prediction dataset; the VOC_Norm_test dataset was constructed by selecting 2734 images of human and other species targets from the VOC2007 test dataset. The VOC_Norm training dataset was randomly fogged based on the atmospheric scattering model formula, with a fogged image to clean image ratio of 2:1, totaling 24,333 images, to construct a mixed training dataset; the VOC_Norm_test test dataset was randomly fogged based on the atmospheric scattering model formula, generating 2,734 haze images, to construct the VOC_fog_test test dataset. An RTTS test dataset containing 4322 haze images was selected. Finally, the image labels of the VOC-annotated data in all datasets were converted to YOLO-annotated data.

4. The method for target detection in haze images based on a two-layer alternating deep network according to claim 1, characterized in that: Step 2 generation and The specific steps for constructing the network of intermediate variables are as follows: ； ； ； in, The input image is a shallow dehazing network with a size of [size missing]. First, the input image is processed using batch normalization (BN). standardization, for The standardized image features are then processed through two branch networks to output the results. and , and The equivalent atmospheric light map and transmittance map output by the shallow dehazing network are shown. In each branch network, the first convolutional layer outputs a feature dimension of... The output feature dimension after the second convolutional layer is ; for Convolution operation, This indicates a cascading operation.

5. The method for target detection in haze images based on a two-layer alternating deep network according to claim 1, characterized in that: In step 3.3, the Spatial Attention Network (SAM) includes Batch Normalized Network (BN) and two... convolution, Activation layer and The activation layer, the Spatial Attention Network (SAM), is constructed and operates as follows: ； in, For the input features of SAM, For the output characteristics of SAM, This is a 1×1 convolution operation.

6. The method for target detection in haze images based on a two-layer alternating deep network according to claim 1, characterized in that: The specific operation of the YOLOv3 detection network in step 5 is as follows: The YOLOv3 network is defined as follows: ； in, The input image for the YOLOv3 detection network. , , The detection results of the YOLOv3 detection network at different sizes; when the input image size is hour, , , The output feature dimensions are respectively , , , Will Input image segmentation A grid, with 3 prior boxes in each grid, for a total of Each prediction box is responsible for predicting the target centered in the grid, used to detect large objects in the image; Divide the input image into A grid is used to detect medium-sized objects in an image; Divide the input image into A grid is used to detect small objects in the image; each prediction box predicts 10 data points, which are the center offset of the target bounding box. Width and height scaling factors Target confidence And the probability of prediction for each of the five categories.

Images