Expressway night visibility detection method based on monitoring image

Abstract

The invention relates to an expressway night visibility detection method based on a monitoring image. According to the invention, monitoring image signals are adopted to describe night road visibility characteristics, a multi-characteristic grading model oriented to night visibility conditions is designed, and the night highway visibility detection method is realized. According to the invention, decision support can be provided for a meteorological department and a traffic management department to take corresponding road control measures, and accurate road visibility information is provided for drivers and passengers to facilitate travel planning, so that the occurrence risk of traffic accidents is reduced. According to the invention, the picture of the erected monitoring camera is used as a data source, and compared with a professional visibility meter, the system has the advantages of low equipment cost and high monitoring density, and more importantly, the system is mainly used for detecting the visibility under a non-natural light condition at night, and the system has a wide application prospect. The problem that visibility grading can only be achieved under the natural light condition in the daytime in existing achievements is solved, and the method has wide market application prospects and remarkable economic value.

Description

Technical Field

[0001] The present invention belongs to the field of traffic meteorology and relates to computer vision and deep learning technology, and specifically to a method for detecting visibility of a highway at night based on monitoring images. Background Art

[0002] With the rapid increase in the total mileage of highways and the number of motor vehicles in my country, higher requirements have been put forward for the safety of transportation. Among them, meteorological disasters, as a visual obstacle, can quickly reduce atmospheric visibility and easily cause major traffic accidents on highways; research shows that driving at night is more dangerous than driving during the day. The main reasons affecting night driving safety include glare from artificial light sources, limited illumination range of driving lights, short sight distance, blind spots, low visibility, and drivers are not conducive to observing road traffic conditions. At the same time, the driver's driving proficiency and psychological state will also lead to the driver's misjudgment of the driving environment. Therefore, the detection of night visibility and the classification of visibility levels are of great significance for reducing and preventing traffic accidents, formulating traffic safety measures in real time, and ensuring traffic safety.

[0003] At present, visibility monitoring methods can be roughly divided into three categories: visual inspection, visibility instrument detection method and video image detection method. The visual inspection method is greatly affected by the subjective experience of the judge, and its standardization and objectivity are relatively poor; visibility meters have problems such as high price, difficult maintenance, sparse layout, and low accuracy in visibility detection under non-uniform conditions; while visibility monitoring based on image processing has become the most economical and rapid visibility monitoring method due to the rapid decline in camera costs, which has enabled high density of monitoring equipment and no need to set up other instruments. At present, there have been relatively many studies on road visibility detection during the day, but visibility detection under night conditions is rarely mentioned at this stage. In fact, low visibility caused by fog usually occurs before and after dawn. The night itself is not bright enough, and in foggy conditions, it will cause serious traffic accidents due to low visibility, resulting in irreversible consequences. From a technical perspective, the characteristics of nighttime images are significantly different from those under daytime natural light conditions. Therefore, the visibility classification model constructed based on daytime images is not applicable to nighttime conditions. Based on this, the present invention constructs a nighttime visibility detection method for road images of nighttime highways. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for detecting highway nighttime visibility based on monitoring images, which is used to detect highway road visibility in nighttime scenes, and includes: constructing a nighttime road image dataset, extracting nighttime road image features, training and constructing a nighttime road visibility detection model, and detecting the visibility status of the road section to be analyzed. The specific steps are as follows:

[0005] (1) On the monitoring road, surveillance camera videos under different camera locations, different typical road conditions, and different visibility conditions are selected, and the video stream is preprocessed. An image is extracted every 1 second to form an image set that forms a sequence over time. Image labels are classified into different levels according to the highway foggy road visibility level classification standard, and a nighttime road image dataset with different visibility levels is constructed;

[0006] (2) Downsample the images in the nighttime road image dataset to obtain downsampled feature maps containing spatial features. Then, use discrete cosine transform and Shannon entropy to process the downsampled feature maps, extract discrete cosine transform DCT coefficients and Shannon entropy features, and obtain frequency domain feature maps and spatial domain feature maps of nighttime road visibility images, thereby reducing the impact of nighttime illumination on the model and fully learning the features.

[0007] (3) Superimposing the downsampled feature map with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the nighttime road image;

[0008] (4) Considering the characteristics of nighttime road images, the deep residual network model is improved to form the network to be trained;

[0009] (5) The SRGB feature vector of the nighttime road image is input into the network for feature extraction and classification. The visibility level label is used as the objective function to train and verify the model and provide feedback to the model to update the network weights and optimize the model parameters, thus building a nighttime road visibility detection model.

[0010] (6) Input the image to be tested into the nighttime road visibility detection model, perform feature extraction and classification, and output the visibility level obtained by the detection. Extend the visibility model to the visibility determination of all camera image data in the system, so as to achieve the effect of road visibility monitoring;

[0011] Furthermore, in step (1), for each type of nighttime road scene, a surveillance video sequence of the target road is obtained based on the surveillance camera, and visibility conditions of each type of nighttime road are formed, and visibility videos corresponding to nighttime roads are obtained in combination with the foggy visibility standard. The visibility ranges corresponding to the visibility levels of the four classified images are reasonably divided, and the number of images of each level in the constructed basic image data set is balanced.

[0012] Furthermore, the calculation steps for extracting DCT coefficients in step (2) are:

[0013]

[0014]

[0015] Where C(u,v) is the transformed frequency domain coefficient, which represents the energy distribution of the image in the frequency domain, α(u)α(v) are normalization coefficients, (x,y) represents the pixel position of the image, and f(x,y) is the pixel value of the image. (u,v) represents the coordinates in the frequency domain, which represents the frequency components in the horizontal and vertical directions, and (W,H) represents the width and height of the image.

[0016] Furthermore, the calculation steps for extracting the Shannon entropy feature in step (2) are:

[0017] For each local area of ​​the image, the probability distribution p(i) of each gray level is calculated, and the calculation formula is:

[0018]

[0019] Wherein, count(i) represents the number of pixels with gray level i in the local area, and N is the total number of pixels in the local area.

[0020] Based on the definition of Shannon entropy, the Shannon entropy H(w) of each local area is calculated. The specific formula is as follows:

[0021]

[0022] Where L is the gray level of the image.

[0023] Furthermore, the step (3) fuses the downsampled feature map with the frequency domain feature map and the spatial domain feature map through a concatenate function, and the specific steps are as follows:

[0024] The feature map obtained by downsampling operation, the frequency domain coefficient feature map extracted by discrete cosine transform (DCT), and the spatial domain feature map obtained by Shannon entropy calculation are normalized and superimposed pixel by pixel to generate the SRGB feature vector of the night road image. The SRGB feature vector contains not only the low-frequency feature information in the frequency domain, but also the complexity information in the spatial domain, so that it can fully reflect the characteristics of the night road image under different lighting and visibility conditions. After the size is normalized, they are connected by the concatenate function to obtain the fusion feature X fusion :

[0025] X fusion =concat(X1,X2,X3).

[0026] Furthermore, the process of model improvement in step (4) is as follows: taking Resnet50 as an example, the first-layer convolution block is replaced with a multi-layer convolution sequence block, in addition to the camera image data, the calculated frequency domain features and spatial domain features are combined to assist network classification, the number of channels of the original model is adjusted to match the fusion features, a batch normalization layer is added to enhance the stability of the model, the last fully connected layer is deleted, and then a new fully connected layer is added, whose output node number is equal to the number of visibility levels, while reducing the number of parameters and preventing the model from overfitting.

[0027] Furthermore, the specific process of deep learning of the neural network in step (5) is as follows:

[0028] Conduct network training: divide the training set into training set and validation set in a ratio of 8:2, and then divide the divided training set again for 4-fold cross-validation; perform data augmentation on the image by randomly flipping the image horizontally and randomly transforming the brightness and saturation; set relevant hyperparameters, set the number of samples selected for one training to 16, the learning rate to 0.001, the maximum number of iterations to 80, the L2 regularization coefficient, and the Adam optimization method, and use MSE Loss as the loss function to supervise network learning; conduct iterative training, and use the validation set and test set for accuracy calculation after each epoch, and save the best one in the training process.

[0029] Furthermore, the specific process of model feature extraction and classification in step (6) is as follows:

[0030] The deep convolutional network extracts the visibility feature vector of the input image and feeds it into the softmax function:

[0031]

[0032] Among them, P i (img) represents the probability that the input image is of the i-th category, K represents the number of image categories, i(img) represents the unnormalized score of the image in the i-th category; j(img) represents the unnormalized score of the image in the j-th category; the softmax function calculates the probability of visibility of each category, and the visibility level with the highest probability is the final output level, that is:

[0033] V = argmax{P i (img),i=1,2,3,4}

[0034] Where V is the output visibility level, P i(img) represents the probability that the input image img belongs to the i-th visibility category. Taking the pre-trained model Resnet50 as an example, the sample image is input into the improved Resnet50 model. After being processed by the convolutional network, the probability of the category to which the sample image belongs is output. The one with the largest probability is the grade detected by the model and output.

[0035] Beneficial Effects

[0036] The present invention has significant beneficial effects and can greatly improve the safety of nighttime roads. Through the nighttime road visibility recognition method based on surveillance video images, the present invention provides accurate visibility detection results, provides timely nighttime road visibility information for meteorological departments, traffic management departments, drivers and passengers, and helps to take necessary road control measures to reduce the risk of traffic accidents. The nighttime road visibility characteristics are characterized by visual signals, combined with the national foggy visibility standards, and the nighttime scene road visibility video sequence is obtained through the surveillance camera. The residual network is used for training to form an efficient visibility detection model, which reflects the technical innovation and advancement. The system design has the advantages of low equipment cost and easy installation, significantly reduces the difficulty of nighttime road visibility detection, reduces related equipment and maintenance costs, and has good economic benefits and broad market application prospects. The detection model obtained by combining the visibility visual features of surveillance video images and the residual network has high precision and high reliability, can provide stable visibility detection services in complex nighttime environments, and enhances the practicality of the system. In summary, the present invention not only realizes the accurate detection of nighttime road visibility at the technical level, but also has significant advantages in terms of cost, installation and maintenance, and has high application value and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall workflow diagram of the method of the present invention;

[0038] Figure 2 This is the structure diagram of the global feature extraction module;

[0039] Figure 3 Schematic diagram of the residual unit in the ResNet network;

[0040] Figure 4 This is a schematic diagram of the ResNet50 network architecture. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0042] like Figure 1 As shown, the present invention provides a method for detecting the visibility level of a highway at night based on surveillance camera images. In practical applications, Figure 1As shown, perform the following steps A to D to obtain a multi-scenario road visibility level detection model.

[0043] Step A. For each preset night scene road, based on the surveillance cameras set at the corresponding roadside positions at preset intervals along the scene road direction, the surveillance cameras of each road section respectively collect the visibility conditions of the road section corresponding to its set position, that is, obtain surveillance camera videos of as many surveillance cameras as possible under different visibility conditions, pre-process the video stream, extract an image every 1s, and then form an image set that forms a sequence over time, classify the image labels into different levels according to the classification standard of visibility levels of foggy roads on highways, and construct a basic image data set of different visibility levels.

[0044] The visibility level classification standard is divided into four categories. The image label of the lowest visibility level is 1, and the image label of the highest visibility level is 4. A basic image dataset containing four visibility levels is constructed. After classification, the visibility ranges corresponding to the visibility levels of the four categories are reasonably divided, and the number of images of each level in the constructed basic image dataset is balanced.

[0045] Step B. For the images corresponding to the road sections of various visibility conditions on the night scene road, execute the following steps B1 to B3, perform discrete cosine transform and Shannon entropy processing, and extract the frequency domain and spatial domain image features of each image. The process is as follows: Figure 2 As shown;

[0046] Step B1. Downsample the tensor size to obtain feature map X1: the input dimension is C×H×W; Kernel Size is 3×3, Stride is 2, and Padding is 1. After downsampling, the dimension becomes C×H / 2×W / 2.

[0047] Step B2. Perform a discrete cosine transform operation on the feature map X1 to obtain a feature map X2 after frequency domain processing;

[0048] Step B3. Using X1 as input, Shannon entropy extraction is performed to obtain the spatial domain feature map X3:

[0049] Step C. Design a feature fusion module to superimpose the downsampled feature map obtained in step B with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the night road image and merge and connect them as follows:

[0050] Normalize the size of the final output results X1, X2 and X3 of step B, and then connect them through the concatenate function to obtain the fusion feature X_fusion;

[0051] Step D. Select the deep residual network ResNet model. The ResNet model contains multiple Figure 3 The residual unit shown in Figure 2 selects ResNet50 as the network design, and its network architecture diagram is as follows: Figure 4 As shown, considering the extracted image features, the input and output channels of the model are improved to form the network to be trained;

[0052] Step E. Input the above data into the neural network, take the visibility level label as the objective function and the visibility level on the corresponding scene road as the output, perform model training, verification and feedback to the model to update the network weights and optimize the model parameters, and form a multi-scenario road visibility level detection model.

[0053] Based on the multi-scenario road visibility level detection model, the following steps i to iii are executed in real time to realize real-time road visibility condition detection for the section to be analyzed in the target scene road.

[0054] Step i. Obtain a surveillance video of the target road at night, capture video frames at fixed time intervals in the surveillance video stream, obtain basic data such as video surveillance images and shooting time, and then proceed to step ii.

[0055] Step ii. According to the method of step B, obtain the down-sampled feature map, DCT coefficient feature map, and Shannon entropy feature map corresponding to the image under the visibility condition, and then proceed to step iii.

[0056] Step iii. According to the method of step C, the down-sampled feature map, the DCT coefficient feature map, and the Shannon entropy feature map are input into the feature fusion module to obtain the SRGB feature vector of the night road image.

[0057] Step iiii. Apply the multi-scenario road visibility level detection model to the SRGB feature vector of the night road image, that is, obtain the visibility level of the road to be analyzed in the target scene road.

[0058] The specific process of model feature extraction and classification is as follows: the deep convolutional network extracts the visibility feature vector of the input image and inputs it into the softmax function:

[0059]

[0060] Among them, P i (img) represents the probability that the input image is of the i-th category, K represents the number of image categories, and e J(img) Represents the jth dimension of the visibility feature vector of image img; the softmax function calculates the probability of each type of visibility, and the visibility level with the highest probability is the final output level, that is:

[0061] V = argmax{P i (img),i=1,2,3,4}

[0062] Among them, V is the output visibility level, and Pi(img) represents the probability that the input image img belongs to the i-th visibility category.

[0063] According to the above-mentioned design of the road visibility level recognition method based on video images, the road section to be analyzed in the actual target scene road is detected, the road visibility status of the road section to be analyzed is obtained, and then a warning prompt is given in time to the direction of the oncoming vehicle. The specific design operations are as follows:

[0064] Use electronic warning signs to warn passing vehicles. When a vehicle is approaching, turn on the electronic display screen or sound to output road visibility conditions to warn vehicles that there is bad weather on the road section they are passing through.

[0065] In practical applications, the above-mentioned road visibility condition identification method based on video image analysis is applied based on a further designed system, which includes a control module, and the control module includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the method are implemented.

[0066] The above technical scheme designs a road visibility condition recognition method based on video image analysis. The design adopts the characteristics of night road visibility of video image signals, and is based on various road weather conditions corresponding to scene video images on various scene roads. The road weather conditions corresponding to different visibility on various scene roads are analyzed and obtained. Combined with the Resnet network for training, a multi-scene road weather condition detection model is obtained, which is then used in practical applications to realize multi-scene road weather condition detection, provide accurate road weather condition information for meteorology, transportation, traffic control, and drivers and passengers, take road control measures in time, and reduce the occurrence of accidents. The entire design system has the characteristics of low equipment cost and easy installation, which can greatly reduce the difficulty of road weather condition detection, and has broad market application prospects and economic value.

[0067] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. A method for detecting visibility of a highway at night based on monitoring images, characterized in that: The method is used to detect the road visibility of highways in night scenes, including: constructing a night road image dataset, extracting night road image features, training and constructing a night road visibility detection model, and detecting the visibility status of the road section to be analyzed. The specific steps are as follows: S1: Select surveillance camera video streams at different camera locations and under different visibility conditions on the night road of the highway, extract an image every 1s from the video stream, and form an image set that forms a sequence over time. According to the highway foggy road visibility level classification standard, it is divided into different levels of night road image data sets; S2: Downsample the images in the nighttime road image dataset to obtain a downsampled feature map containing spatial features, then extract the discrete cosine transform DCT coefficients and Shannon entropy features of the downsampled feature map respectively, and process them to obtain the frequency domain feature map and spatial domain feature map of the nighttime road image; S3: Superimpose the downsampled feature map with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the night road image; S4: According to the characteristics of nighttime road images, the deep residual network model is improved to form a network to be trained; S5: Input the SRGB feature vector into the network for feature extraction and classification, take the visibility level label as the objective function, train and verify the model, and feed back to the model to update the network weights, optimize the model parameters, and build a nighttime road visibility detection model; S6: Input the image to be tested into the constructed nighttime road visibility detection model, perform feature extraction and classification, and output the visibility level obtained by the detection.

2. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1, characterized in that: In step S1, for each type of nighttime road scene, a surveillance video sequence of the target road is obtained based on the surveillance camera to form various types of nighttime road visibility conditions, and the visibility video corresponding to the nighttime road is obtained in combination with the foggy visibility standard.

3. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1 is characterized in that: In step S2, the step of extracting discrete cosine transform DCT coefficients and Shannon entropy features includes: 1) Downsample the original nighttime road image to obtain the feature map X1: the input dimension is C×H×W, the kernel size is 3×3, the stride is 2, and the padding is 1. After downsampling, the dimension of the feature map becomes C×H / 2×W / 2; where C represents the number of image channels, and W and H represent the width and height of the image respectively; 2) Perform discrete cosine transform operation on the feature map X1 to obtain the feature map X2 after processing in the frequency domain, which is expressed as follows: Where C(u,v) is the transformed frequency domain coefficient, which represents the energy distribution of the image in the frequency domain, α(u) and α(v) are normalization coefficients, (x,y) represents the pixel position of the image, and f(x,y) is the pixel value of the image; (u,v) represents the coordinates in the frequency domain, which represents the frequency components in the horizontal and vertical directions, and (W,H) represents the width and height of the image; 3) Using X1 as input, the feature map X3 after processing in the spatial domain is obtained, and the image is divided into local areas using a sliding window operation according to the following steps; for each local area, the probability distribution p(i) of each gray level is calculated, and the calculation formula is: Where count(i) represents the number of pixels with gray level i in the local area, and N is the total number of pixels in the local area; Based on the definition of Shannon entropy, the Shannon entropy H(w) of each local area is calculated. The specific formula is as follows: Where L is the grayscale level of the image.

4. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 3 is characterized in that: The specific steps of step S3 are: normalize the final output results X1, X2 and X3 of step S2, and then connect them through the concatenate function to obtain the fusion feature X fusion : X fusion =concat(X1,X2,X3)。 5. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1 is characterized in that: In the step S4, the deep residual network ResNet50 is used as the backbone model, and the specific improvements include: in addition to the camera image data, the calculated frequency domain features and spatial domain features are combined to assist network classification, the number of input channels is adjusted to the number of channels of the fused features, a batch normalization layer is added after each convolution layer of the deep learning pre-training model to enhance the stability of the model, the last fully connected layer is deleted, and then a new fully connected layer is added, whose output node number is equal to the number of visibility levels.

6. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1, characterized in that: In step S5, the model training steps include: 1) Divide the training set into a training set and a validation set in a ratio of 8:2, and then divide the divided training set again for 4-fold cross validation; 2) Image augmentation based on random flipping, random brightness and saturation transformation; 3) Set the hyperparameters of BatchSize, learning rate, and maximum number of iterations, use the Adam optimization method, and use MSELoss as the loss function to supervise network learning; 4) Perform iterative training. After each epoch, the validation set and test set will be used to calculate the accuracy, and the best one in the training process will be saved.

7. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1, characterized in that: In step S6, the specific process of model feature extraction and classification is as follows: the deep convolutional network extracts the visibility feature vector of the input image and inputs it into the softmax function: Among them, P i (img) represents the probability that the input image is of the i-th category, K represents the number of image categories, i(img) represents the unnormalized score of the image in the i-th category; j(img) represents the unnormalized score of the image in the j-th category; the softmax function calculates the probability of visibility of each category, and the visibility level with the highest probability is the final output level, that is: V=argmax{P i (img),i=1,2,3,4} Where V is the visibility level of the output.

Images