Visibility detection method and detection system based on differential network

Abstract

The present invention provides a visibility detection method and detection system based on a differential network, the detection method comprising the following steps: S1, obtaining a highway image and a corresponding reference image; S2, using two VGG16 networks with shared weights to construct a visibility detection differential network; S3, according to the highway image and the corresponding reference image, supervised training is performed on the visibility detection differential network to determine the trained visibility detection differential network; S4, using the trained visibility detection differential network to detect the visibility value of the highway image. The present invention can improve the visibility value regression performance of the real collected highway image by supervised training the visibility detection differential network.

Description

Technical Field

[0001] The invention relates to an image processing method, in particular to a visibility detection method and detection system based on a differential network. Background Art

[0002] Fog is very common in our real life. The appearance of fog affects visual effects and significantly reduces atmospheric visibility. When driving on the highway, the appearance of fog will reduce road visibility, which will have a serious impact on vehicles driving at high speed on the highway, making driving safety difficult to ensure. If visibility can be detected and corresponding driving measures can be taken, it will help improve driving safety.

[0003] At present, there are three ways to detect visibility and their characteristics are: 1. Human eye inspection method, which is easily affected by subjective and objective factors and has low efficiency. 2. Instrument measurement method, which has high deployment and maintenance costs. 3. Image-based visibility detection method, which has the advantages of low cost, high speed and high real-time performance.

[0004] The image-based visibility estimation method obtains visibility information from the foggy images taken. Traditional image methods use manual features such as edges, colors, and contrast or atmospheric scattering models to estimate visibility. At present, most visibility detection methods are mainly based on deep learning, which requires the collection of large-scale image data. How to collect and annotate large-scale high-quality image data is the key to accurate visibility detection. At present, deep learning models mostly use regression models to directly estimate visibility values ​​for visibility value detection. However, due to the wide distribution of visibility values, the visibility meter can detect visibility ranging from a few meters to 30,000 meters, and the scene differences are relatively large. Direct detection of visibility values ​​has problems such as large errors and low accuracy.

[0005] Existing image-based deep learning visibility detection methods mainly perform visibility value detection on synthetic highway scene images, but rarely perform visibility value detection on real highway scene images. In real highway scenes, existing visibility detection methods mainly combine the atmospheric scattering model based on Koschmieder's law with deep learning to directly detect visibility values. This detection method has the problems of complex model and large error. Summary of the invention

[0006] The purpose of the present invention is to provide a visibility detection method and detection system based on a differential network to solve the problems of complex model and large error in highway image detection.

[0007] The object of the present invention is achieved in that:

[0008] A visibility detection method based on a differential network comprises the following steps:

[0009] S1, acquiring a highway image and a corresponding reference image;

[0010] S2, using two VGG16 networks with shared weights to build a visibility detection differential network;

[0011] S3, performing supervised training on the visibility detection differential network according to the highway image and the corresponding reference image, and determining a trained visibility detection differential network;

[0012] S4. Use the trained visibility detection differential network to detect the visibility value of the highway image.

[0013] Furthermore, the visibility detection differential network includes:

[0014] An image acquisition module, including a pre-trained visibility grade classification model, is used to select a corresponding reference image for each input highway image, obtain a highway image pseudo label by inputting the highway image into the pre-trained visibility grade classification model, and randomly select a reference image with the same grade from a reference set according to the pseudo label;

[0015] A feature extraction module, including two VGG16 networks with batch normalization and shared weights, for extracting visibility features from the input highway image that are similar to those of the reference image; and

[0016] The differential regression module includes two fully connected layers, which are used to fuse similar high-dimensional features extracted by the feature extraction module, train the differential network using a supervised training method, regress the visibility difference, and output the difference in the visibility values ​​of the image.

[0017] In the visibility detection differential network, the input of the feature extraction module is an image pair selected by the image acquisition module, which is the highway image for which visibility detection is required and the reference image selected from the reference set. The feature extraction module is used to extract similar visibility features of this set of images. The input of the visibility difference regression module is the visibility feature pair extracted by the feature extraction module, and the final output is the visibility difference of this set of image pairs. The differential loss is introduced in the visibility detection method based on the differential network to guide the differential network to learn the difference information of visibility, so as to achieve better results.

[0018] Furthermore, in step S3, the specific method of conducting supervised training on the visibility detection differential network is: according to the highway image and the corresponding reference image, through the feature extraction module and the differential regression module, determining the loss function of the visibility detection differential network; training the visibility detection differential network according to the loss function to obtain a trained visibility detection differential network.

[0019] The present invention can improve the visibility numerical regression performance of the real collected highway images by performing supervised training on the visibility detection difference network.

[0020] Furthermore, the differential loss of the visibility detection differential network is calculated as:

[0021]

[0022] Among them, L diff is the difference loss, I is the input image; J is the reference image; D is the training set; d est (I, J) is the visibility difference predicted by the network; v gt (I) is the true value of the visibility of the input image; v gt (J) is the true value of visibility of the reference image.

[0023] Furthermore, the specific method of step S4 is: inputting the collected highway image into a pre-trained visibility level classification model to obtain a pseudo label, randomly selecting a reference image with the same visibility level from the training set according to the pseudo label, inputting the collected highway image and the corresponding reference image into a trained visibility detection difference network, detecting the visibility difference between the two images, and determining the visibility value of the input highway image according to the visibility value of the reference image and the detected visibility difference.

[0024] The detection method of the present invention mainly performs supervised training on the differential network. For each highway image, the highway image is first input into the pre-trained visibility level classification model to obtain a pseudo label, and a reference image with the same visibility level is randomly selected from the training set according to the pseudo label. Then, a differential network is designed based on the idea of ​​contrastive learning, which is mainly divided into two modules: feature extraction and visibility difference regression. Two VGG16 networks with shared weights are used to extract similar visibility features of the input image and the reference image. The difference loss is used to make the visibility value difference between the highway image and the reference image smaller, thereby improving the performance of visibility value detection and reducing detection errors.

[0025] The present invention uses two VGG16 networks to construct a differential network with shared weights. For each highway image, a reference image with the same level is randomly selected from a reference set using a strategy for obtaining pseudo labels. Next, the differential network is supervised trained, and the visibility numerical difference regression is realized by utilizing similar visibility features of the highway image and the corresponding reference image. By performing supervised training on the differential network, the visibility numerical regression performance of the actually collected highway images can be improved.

[0026] The present invention adopts a visibility detection method based on a differential network to detect the difference in visibility, and finally obtains the visibility value of the highway image. The differential network includes two parts, one is a feature extraction module, and the other is a visibility difference regression module. The input of the feature extraction module is an image pair, which is a highway image whose visibility needs to be detected and a reference image selected from a reference set. The feature extraction module is used to extract visibility features related to this group of images respectively. The input of the visibility difference regression module is the visibility feature pair extracted by the feature extraction module, and the final output is the visibility difference of this group of image pairs. Differential loss is introduced in the visibility detection method based on the differential network to guide the differential network to learn the difference information of visibility, so as to achieve better results.

[0027] The purpose of the present invention can also be achieved like this:

[0028] A visibility detection system based on a differential network, comprising:

[0029] An image acquisition module, connected to the differential network construction module, is used to acquire highway images and corresponding reference images;

[0030] The differential network construction module is connected to the image acquisition module and the differential network training module respectively, and is used to construct a visibility detection differential network using two VGG16 networks with shared weights;

[0031] A differential network training module is connected to the differential network construction module and the highway image visibility detection module, respectively, and is used to train a visibility detection differential network according to the highway image and the corresponding reference image to obtain a trained visibility detection differential network; and

[0032] The highway image visibility detection module is connected to the difference network training module and is used to detect the visibility difference between the highway image and the reference image based on the trained visibility detection difference network, so as to determine the visibility value of the highway image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flow chart of the visibility detection method of the present invention.

[0034] Figure 2It is the structural block diagram of the visibility detection differential network.

[0035] Figure 3 It is a flow chart of the training method of the visibility detection differential network.

[0036] Figure 4 It is a structural block diagram of the visibility detection system of the present invention. DETAILED DESCRIPTION

[0037] Example 1

[0038] like Figure 1 As shown, the visibility detection method of the present invention comprises the following steps:

[0039] Step 101: Acquire a highway image and a corresponding reference image; input the acquired highway image into a pre-trained visibility level classification model to obtain a pseudo label, randomly select a reference image with the same visibility level from a training set according to the pseudo label, and determine the highway image and the corresponding reference image.

[0040] Step 102: Use two VGG16 networks with shared weights to build a visibility detection differential network.

[0041] Step 103: According to the highway image and the corresponding reference image, supervised training is performed on the visibility detection differential network to determine a trained visibility detection differential network.

[0042] Step 104: Determine the visibility value of the highway image according to the trained visibility detection difference network. Specifically, the collected highway image is input into the trained visibility detection difference network to detect the visibility difference between the two images, and the visibility value of the input highway image is determined according to the visibility value of the reference image and the detected visibility difference.

[0043] like Figure 2 As shown in the figure, the visibility detection differential network includes an image acquisition module, a feature extraction module and a differential regression module; the image acquisition module obtains the pseudo label of the highway image by inputting the highway image into the pre-trained visibility grade classification model, and randomly selects a reference image with the same grade from the reference set based on the pseudo label. The feature extraction module uses two pre-trained VGG16 networks with Batch Normalization layers with shared weights to extract similar high-dimensional features of the highway image and the reference image for subsequent visibility difference regression. The differential regression module uses two fully connected layers to fuse the similar high-dimensional features extracted by the feature extraction module, and uses supervised training to train the differential network to regress the visibility difference information to further improve the performance of visibility numerical detection.

[0044] The supervised training of the visibility detection differential network is to determine the loss function of the visibility detection differential network through the feature extraction module and the differential regression module based on the highway image and the corresponding reference image; the differential network is trained according to the loss function to obtain the trained visibility detection differential network. Figure 3 As shown in Figure 2, the specific training method of the visibility detection differential network is:

[0045] Step 301: According to the highway image, a pre-trained visibility level classification model is used to obtain a highway image pseudo label, and a reference image with the same level is randomly selected from a reference set according to the pseudo label to determine a reference image;

[0046] Step 302: constructing a differential network using two VGG16s with shared weights; the differential network includes: an image acquisition module, a feature extraction module, and a visibility difference regression module;

[0047] Step 303: determining a differential network loss function according to the differential network, the highway image and the corresponding reference image;

[0048] Step 304: training the differential network according to the differential network loss function to obtain a trained differential network;

[0049] Step 305: Input the highway image into the trained differential network to obtain the visibility difference and determine the visibility value of the highway image.

[0050] The loss function of the visibility detection differential network and the determination method of the differential loss are:

[0051]

[0052] Among them, L diff is the difference loss, I is the input image; J is the reference image; D is the training set; d est (I, J) is the visibility difference predicted by the network; v gt (I) is the true value of the visibility of the input image; v gt (J) is the true value of visibility of the reference image.

[0053] Example 2

[0054] like Figure 4As shown, the visibility detection system based on the differential network of the present invention includes an image acquisition module 401, a differential network construction module 402, a differential network training module 403 and a highway image visibility detection module 404, which are connected in sequence. Among them, the image acquisition module 401 is used to determine the highway image and the corresponding reference image; the differential network construction module 402 is used to use two VGG16 networks with shared weights to construct a visibility detection differential network; the differential network training module 403 is used to perform supervised training on the differential network according to the highway image and the corresponding reference image, and determine the trained visibility detection differential network; the highway image visibility detection module 404 is used to detect the visibility difference between the highway image and the reference image according to the trained visibility detection differential network, and determine the visibility value of the highway image according to the visibility value of the reference image and the detected visibility difference.

[0055] The present invention uses video surveillance equipment along the expressway to collect images of the Beijing-Shijiazhuang Expressway and the Daguan Expressway, including images of expressways with different visibility levels. On the Beijing-Shijiazhuang Expressway, 6263 images are selected as the training set and 1536 images are selected as the test set; the data set is named: JFHVI. On the Daguan Expressway, 5689 images are selected as the training set and 1086 images are selected as the test set; the data set is named: DFHVI.

[0056] 1. Experimental results on the JFHVI dataset

[0057] The detection method of the present invention is compared with the Baseline method and the visibility detection method based on ResNet-18. The Baseline is a method that directly regresses the visibility value using only a VGG16 network. For the above method, the code is re-implemented and the model is retrained on the training set, and then tested on the test set. When the visibility is greater than 1000 meters, the visual environment is relatively good, people can clearly see the surrounding environment and road conditions, and it is unlikely to have a significant impact on the safety of driving or pedestrians. Therefore, in this case, the specific visibility value is usually not the focus of attention. The detection method of the present invention performs visibility detection for situations where visibility is less than 1000 meters.

[0058] The detection method of the present invention uses mean absolute error (MAE) as an evaluation index, and the lower the mean absolute error, the better.

[0059]

[0060] Among them, y i is the true difference between the test sample and the reference image, is the predicted difference between the test sample and the reference image, and n is the total number of test samples.

[0061] Table 1 shows the experimental results on the JFHVI dataset. As can be seen from Table 1, based on the MAE evaluation index, the detection method of the present invention is significantly better than the Baseline method and the ResNet-18 method, and the total error of visibility detection is the smallest. Specifically, at level 0 (0-50 meters), level 2 (100-200 meters), level 3 (200-500 meters), and level 4 (500-1000 meters), the detection method of the present invention is better than the two comparison methods, and the visibility detection error is the smallest, which shows the effectiveness of the detection method of the present invention and can reduce the error of visibility detection.

[0062] Table 1 Experimental results on the JFHVI dataset

[0063]

[0064] 2. Experimental Results on the DFHVI Dataset

[0065] On the DFHVI dataset, the same experiment as the JFHVI dataset was implemented, that is, the detection method of the present invention was compared with the Baseline method and the visibility detection method based on ResNet-18.

[0066] Table 2 shows the experimental results on the DFHVI dataset. As can be seen from Table 2, based on the MAE evaluation index, the detection method of the present invention is far superior to the Baseline method and the ResNet-18 method, and its total visibility detection error is the smallest. Specifically, at level 2 (100-200 meters), level 3 (200-500 meters), and level 4 (500-1000 meters), the detection method of the present invention is superior to the above two comparative detection methods, and the visibility detection error result is optimal, which shows the effectiveness of the detection method of the present invention.

[0067] Table 2 Experimental results on the DFHVI dataset

[0068]

Claims

1. A visibility detection method based on a differential network, characterized in that: The following steps are involved: S1, acquiring a highway image and a corresponding reference image; S2, using two VGG16 networks with shared weights to build a visibility detection differential network; S3, performing supervised training on the visibility detection differential network according to the highway image and the corresponding reference image, and determining a trained visibility detection differential network; S4, using the trained visibility detection difference network to detect the visibility value of the highway image; The visibility detection differential network comprises: An image acquisition module, including a pre-trained visibility grade classification model, is used to select a corresponding reference image for each input highway image, obtain a highway image pseudo label by inputting the highway image into the pre-trained visibility grade classification model, and randomly select a reference image with the same grade from a reference set according to the pseudo label; A feature extraction module, including two VGG16 networks with batch normalization and shared weights, for extracting visibility features from the input highway image that are similar to those of the reference image; and The differential regression module includes two fully connected layers, which are used to fuse similar high-dimensional features extracted by the feature extraction module, train the differential network using a supervised training method, regress the visibility difference, and output the difference in the visibility value of the image; The specific method of performing supervised training on the visibility detection differential network in step S3 is: determining the loss function of the visibility detection differential network through a feature extraction module and a differential regression module according to the highway image and the corresponding reference image; training the visibility detection differential network according to the loss function to obtain a trained visibility detection differential network; The differential loss of the visibility detection differential network is calculated as: Among them, L diff is the difference loss, I is the input image; J is the reference image; D is the training set; d est (I, J) is the visibility difference predicted by the network; v gt (I) is the true value of the visibility of the input image; v gt (J) is the true value of visibility of the reference image; The specific method of step S4 is: input the collected highway image into a pre-trained visibility level classification model to obtain a pseudo label, randomly select a reference image with the same visibility level from the training set according to the pseudo label, input the collected highway image and the corresponding reference image into a trained visibility detection difference network, detect the visibility difference between the two images, and determine the visibility value of the input highway image according to the visibility value of the reference image and the detected visibility difference.

2. A visibility detection system based on a differential network, characterized in that: include: An image acquisition module, connected to the differential network construction module, is used to acquire highway images and corresponding reference images; The differential network construction module is connected to the image acquisition module and the differential network training module respectively, and is used to construct a visibility detection differential network using two VGG16 networks with shared weights; The differential network training module is connected to the differential network construction module and the highway image visibility detection module respectively, and is used to train the visibility detection differential network according to the highway image and the corresponding reference image to obtain a trained visibility detection differential network; as well as The highway image visibility detection module is connected to the difference network training module and is used to detect the visibility difference between the highway image and the reference image according to the trained visibility detection difference network, and determine the visibility value of the highway image; The visibility detection differential network comprises: An image acquisition module, including a pre-trained visibility grade classification model, is used to select a corresponding reference image for each input highway image, obtain a highway image pseudo label by inputting the highway image into the pre-trained visibility grade classification model, and randomly select a reference image with the same grade from a reference set according to the pseudo label; A feature extraction module, including two VGG16 networks with batch normalization and shared weights, for extracting visibility features from the input highway image that are similar to those of the reference image; and The differential regression module includes two fully connected layers, which are used to fuse similar high-dimensional features extracted by the feature extraction module, train the differential network using a supervised training method, regress the visibility difference, and output the difference in the visibility value of the image; The visibility detection differential network is trained in the following manner: according to the highway image and the corresponding reference image, a loss function of the visibility detection differential network is determined through a feature extraction module and a differential regression module; the visibility detection differential network is trained according to the loss function to obtain a trained visibility detection differential network; The differential loss of the visibility detection differential network is calculated as: Among them, L diff is the difference loss, I is the input image; J is the reference image; D is the training set; d est (I, J) is the visibility difference predicted by the network; v gt (I) is the true value of the visibility of the input image; v gt (J) is the true value of visibility of the reference image; The specific method of detecting the visibility value of the highway image is: input the collected highway image into the pre-trained visibility level classification model to obtain a pseudo label, randomly select a reference image with the same visibility level from the training set according to the pseudo label, input the collected highway image and the corresponding reference image into the trained visibility detection difference network, detect the visibility difference between the two images, and determine the visibility value of the input highway image according to the visibility value of the reference image and the detected visibility difference.

Images