A highway night visibility detection method based on monitoring images

Abstract

The present application relates to a kind of expressway night visibility detection method based on monitoring image.The present application designs to use monitoring image signal to describe night road visibility feature, designs the multi-feature classification model for night visibility condition, realizes night expressway visibility detection method.The present application can provide decision support for meteorological department, traffic management department to take corresponding road control measures, provide accurate road surface visibility information for driver and passenger to facilitate their travel planning, thereby reduce the risk of traffic accident.The present application uses the monitoring camera picture that has been erected as data source, compared with professional visibility instrument has the advantages of low equipment cost, monitoring density is big, especially important is, the present application is mainly aimed at the detection of night non natural light condition visibility, solves the visibility classification problem in daytime natural light condition in prior art, has broad market application prospect and significant economic value.

Description

Technical Field

[0001] The present invention belongs to the field of traffic meteorology and relates to computer vision and deep learning technology, and specifically to a method for detecting nighttime visibility on highways based on monitoring images. Background Art

[0002] With the rapid increase in the total mileage of expressways and the number of motor vehicles in my country, higher requirements have been placed on transportation safety. Meteorological disasters, acting as visual obstructions, can rapidly reduce atmospheric visibility, making them highly susceptible to major traffic accidents on highways. Research has shown that nighttime driving is more dangerous than daytime driving. The main factors affecting nighttime driving safety include glare from artificial light sources, limited illumination range of driving lights, short sight distances, blind spots, and low visibility, which hinder drivers' ability to observe road traffic conditions. Furthermore, drivers' proficiency and psychological state can also lead to misjudgment of the driving environment. Therefore, the detection and classification of nighttime visibility are crucial for reducing and preventing traffic accidents, formulating real-time traffic safety measures, and ensuring traffic safety.

[0003] Currently, visibility monitoring methods can be broadly categorized into three categories: visual inspection, visibility instrumentation, and video-based methods. Visual inspection is significantly influenced by the subjective experience of the assessor and is relatively inadequate in terms of standardization and objectivity. Visibility meters are expensive, difficult to maintain, and sparsely deployed, with low accuracy in non-uniform visibility conditions. Image processing-based visibility monitoring, however, has become the most economical and rapid visibility monitoring method due to the rapid decline in camera costs, enabling high-density deployment of monitoring equipment and eliminating the need for additional equipment. While there has been relatively extensive research on daytime road visibility monitoring, nighttime visibility monitoring is currently less explored. In reality, low visibility caused by fog typically occurs around dawn. The inherent lack of light at night, combined with fog, can lead to serious traffic accidents, resulting in irreversible consequences. From a technical perspective, the characteristics of nighttime images are significantly different from those under daytime natural light conditions. Therefore, the visibility classification model constructed based on daytime images is not applicable to nighttime conditions. Based on this, the present invention constructs a nighttime visibility detection method for road images of nighttime highways. Summary of the Invention

[0004] The technical problem to be solved by this invention is to provide a method for detecting highway nighttime visibility based on surveillance images. The method is used to detect highway road visibility in nighttime scenes. The method comprises the following steps: constructing a nighttime road image dataset; extracting nighttime road image features; training and constructing a nighttime road visibility detection model; and detecting the visibility conditions of the road section to be analyzed. The specific steps are as follows:

[0005] (1) On the monitoring road, surveillance camera videos under different camera locations, typical road conditions, and visibility conditions were selected. The video stream was preprocessed and an image was extracted every 1 second to form an image set that formed a sequence over time. The image labels were classified into different levels according to the highway foggy road visibility level classification standard, and a nighttime road image dataset with different visibility levels was constructed.

[0006] (2) Down-sample the images in the nighttime road image dataset to obtain a down-sampled feature map containing spatial features. Then, the down-sampled feature map is processed using discrete cosine transform and Shannon entropy to extract discrete cosine transform (DCT) coefficients and Shannon entropy features. The frequency domain feature map and spatial domain feature map of the nighttime road visibility image are obtained to reduce the impact of nighttime illumination on the model and fully learn the features.

[0007] (3) Superimposing the downsampled feature map with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the nighttime road image;

[0008] (4) Considering the characteristics of nighttime road images, the deep residual network model is improved to form the network to be trained;

[0009] (5) The SRGB feature vectors of the nighttime road image are input into the network for feature extraction and classification. The visibility level label is used as the objective function to train and verify the model and provide feedback to the model to update the network weights and optimize the model parameters, thus building a nighttime road visibility detection model.

[0010] (6) Input the image to be tested into the nighttime road visibility detection model, perform feature extraction and classification, and output the visibility level obtained by the detection. Extend the visibility model to the visibility determination of all camera image data in the system to achieve the effect of road visibility monitoring;

[0011] Furthermore, in step (1), for each type of nighttime road scene, surveillance video sequences of the target road are obtained based on surveillance cameras to construct visibility conditions for each type of nighttime road, and visibility videos corresponding to the nighttime road are obtained in combination with foggy visibility standards. The visibility ranges corresponding to the visibility levels of the four classified images are reasonably divided, and the number of images of each level in the constructed basic image dataset is balanced.

[0012] Furthermore, the calculation steps for extracting DCT coefficients in step (2) are:

[0013]

[0014]

[0015] Where C(u,v) is the transformed frequency domain coefficient, representing the energy distribution of the image in the frequency domain, α(u) and α(v) are normalization coefficients, (x,y) represents the pixel position of the image, and f(x,y) is the pixel value of the image. (u,v) represents the coordinates in the frequency domain, indicating the frequency components in the horizontal and vertical directions, and (W,H) represents the width and height of the image.

[0016] Furthermore, the calculation steps for extracting the Shannon entropy feature in step (2) are:

[0017] For each local area of ​​the image, the probability distribution p(i) of each gray level is calculated, and the calculation formula is:

[0018]

[0019] Wherein, count(i) represents the number of pixels with gray level i in the local area, and N is the total number of pixels in the local area.

[0020] Based on the definition of Shannon entropy, the Shannon entropy H(w) of each local area is calculated. The specific formula is as follows:

[0021]

[0022] Where L is the grayscale of the image.

[0023] Furthermore, in step (3), the downsampled feature map is fused with the frequency domain feature map and the spatial domain feature map through a concatenate function. The specific steps are as follows:

[0024] The feature map obtained by downsampling, the frequency domain coefficient feature map extracted by discrete cosine transform (DCT), and the spatial domain feature map obtained by Shannon entropy calculation are normalized and superimposed pixel by pixel to generate the SRGB feature vector of the night road image. The SRGB feature vector contains not only the low-frequency feature information in the frequency domain, but also the complexity information in the spatial domain, so it can fully reflect the characteristics of the night road image under different lighting and visibility conditions. After the size is normalized, they are connected by the concatenate function to obtain the fusion feature X fusion :

[0025] X fusion =concat(X1,X2,X3).

[0026] Furthermore, the process of model improvement in step (4) is as follows: taking Resnet50 as an example, the first-layer convolution block is replaced by a multi-layer convolution sequence block, in addition to the camera image data, the calculated frequency domain features and spatial domain features are combined to assist network grading, the number of original model channels is adjusted to match the fusion features, a batch normalization layer is added to enhance the stability of the model, the last fully connected layer is deleted, and then a new fully connected layer is added, the number of output nodes of which is equal to the number of visibility levels, while reducing the number of parameters and preventing the model from overfitting.

[0027] Furthermore, the specific process of deep learning of the neural network in step (5) is as follows:

[0028] Perform network training: Split the training set into a training set and a validation set in an 8:2 ratio, and then further divide the divided training set to perform 4-fold cross-validation; perform data augmentation on the image by randomly flipping the image horizontally and randomly transforming the brightness and saturation; set relevant hyperparameters, set the number of samples selected for one training to 16, the learning rate to 0.001, the maximum number of iterations to 80, the L2 regularization coefficient, and the Adam optimization method, using MSE Loss as the loss function to supervise network learning; perform iterative training, and calculate the accuracy using the validation set and test set after each epoch, and save the best one in the training process.

[0029] Furthermore, the specific process of model feature extraction and classification in step (6) is as follows:

[0030] The deep convolutional network extracts the visibility feature vector of the input image and inputs it into the softmax function:

[0031]

[0032] Among them, P i (img) represents the probability that the input image belongs to the i-th category, K represents the number of image categories, i(img) represents the unnormalized score of the image in the i-th category; j(img) represents the unnormalized score of the image in the j-th category; the softmax function calculates the probability of visibility of each category, and the visibility level with the highest probability is the final output level, that is:

[0033] V=argmax{P i (img),i=1,2,3,4}

[0034] Where V is the output visibility level, P i(img) represents the probability that the input image img belongs to the i-th visibility category. Taking the pre-trained Resnet50 model as an example, a sample image is input into the improved Resnet50 model. After convolutional network processing, the output is the probability of the sample image belonging to the category. The highest probability is the grade detected by the model and output.

[0035] Beneficial effects

[0036] The present invention has significant beneficial effects and can greatly improve the safety of nighttime roads. Through the nighttime road visibility recognition method based on surveillance video images, the present invention provides accurate visibility detection results, provides timely nighttime road visibility information to meteorological departments, traffic management departments, drivers and passengers, and helps to take necessary road control measures to reduce the risk of traffic accidents. By using visual signals to characterize the visibility characteristics of nighttime roads, combined with the national foggy visibility standards, the nighttime scene road visibility video sequence is obtained through surveillance cameras, and the residual network is used for training to form an efficient visibility detection model, which embodies the innovation and advancement of the technology. The system design has the advantages of low equipment cost and easy installation, which significantly reduces the difficulty of nighttime road visibility detection, reduces related equipment and maintenance costs, and has good economic benefits and broad market application prospects. The detection model obtained by combining the visibility visual features of surveillance video images and the residual network has high precision and high reliability, can provide stable visibility detection services in complex nighttime environments, and enhances the practicality of the system. In summary, the present invention not only realizes the accurate detection of nighttime road visibility at the technical level, but also has significant advantages in terms of cost, installation and maintenance, and has high application value and promotion potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is the overall workflow diagram of the method of the present invention;

[0038] Figure 2 This is the structure diagram of the global feature extraction module;

[0039] Figure 3 Schematic diagram of the residual unit in the ResNet network;

[0040] Figure 4 This is a diagram of the ResNet50 network architecture. DETAILED DESCRIPTION

[0041] The technical solution of the present invention is further described in detail below with reference to the accompanying drawings.

[0042] like Figure 1 As shown, the present invention provides a method for detecting the visibility level of highways at night based on surveillance camera images. In practical applications, Figure 1As shown, perform the following steps A to D to obtain a multi-scenario road visibility level detection model.

[0043] Step A. For each preset nighttime road scene, surveillance cameras are installed at corresponding roadside locations at preset intervals along the scene road direction. The surveillance cameras on each road section respectively collect visibility conditions for the road section corresponding to their respective locations. That is, surveillance camera videos from as many surveillance cameras as possible under different visibility conditions are obtained. The video stream is preprocessed, and an image is extracted every 1 second to form an image set that forms a sequence over time. The image labels are classified into different levels according to the highway foggy road visibility level classification standard, and a basic image dataset with different visibility levels is constructed.

[0044] The visibility level classification standard is divided into four categories. The image label of the lowest visibility level is 1, and the image label of the highest visibility level is 4. A basic image dataset containing four visibility levels is constructed. After classification, the visibility ranges corresponding to the visibility levels of the four categories are reasonably divided, and the number of images of each level in the constructed basic image dataset is balanced.

[0045] Step B. For each image corresponding to each road section with various visibility conditions on the night scene, perform the following steps B1 to B3, perform discrete cosine transform and Shannon entropy processing, and extract the frequency domain and spatial domain image features of each image. The process is as follows: Figure 2 As shown;

[0046] Step B1. Downsample the tensor to obtain feature map X1: the input dimension is C×H×W; the kernel size is 3×3, the stride is 2, and the padding is 1. After downsampling, the dimension becomes C×H / 2×W / 2.

[0047] Step B2. Perform a discrete cosine transform operation on the feature map X1 to obtain a feature map X2 after frequency domain processing;

[0048] Step B3. Using X1 as input, perform Shannon entropy extraction to obtain the spatial domain feature map X3:

[0049] Step C. Design a feature fusion module to superimpose the downsampled feature map obtained in step B with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the nighttime road image. The method for merging and connecting them is as follows:

[0050] Normalize the size of the final output results X1, X2, and X3 of step B, and then connect them through the concatenate function to obtain the fusion feature X_fusion;

[0051] Step D. Select the deep residual network ResNet model. The ResNet model contains multiple Figure 3 The residual unit shown in the figure selects ResNet50 as the network design, and its network architecture diagram is as follows Figure 4 As shown, considering the extracted image features, the input and output channels of the model are improved to form the network to be trained;

[0052] Step E. Input the above data into the neural network, use the visibility level label as the objective function, and the visibility level on the corresponding scene road as the output. Perform model training, verification, and feedback to the model to update the network weights and optimize the model parameters to form a multi-scenario road visibility level detection model.

[0053] Based on the obtained multi-scenario road visibility level detection model, the following steps i to iii are executed in real time to realize real-time road visibility condition detection for the road section to be analyzed in the target scene road.

[0054] Step i. Obtain a surveillance video of the target road at night, capture video frames at fixed time intervals in the surveillance video stream, obtain basic data such as video surveillance images and shooting time, and then proceed to step ii.

[0055] Step ii. According to the method of step B, obtain the down-sampled feature map, DCT coefficient feature map, and Shannon entropy feature map corresponding to the image under the visibility condition, and then enter step iii.

[0056] Step iii. According to the method of step C, the downsampled feature map, the DCT coefficient feature map, and the Shannon entropy feature map are input into the feature fusion module to obtain the SRGB feature vector of the night road image.

[0057] Step iiii. Apply the multi-scene road visibility level detection model to the SRGB feature vector of the night road image, that is, obtain the visibility level of the road to be analyzed in the target scene road.

[0058] The specific process of model feature extraction and classification is as follows: the deep convolutional network extracts the visibility feature vector of the input image and inputs it into the softmax function:

[0059]

[0060] Among them, P i (img) represents the probability that the input image is of the i-th category, K represents the number of image categories, e J(img) Represents the j-th dimension of the visibility feature vector of the image img; the softmax function calculates the probability of each type of visibility, and the visibility level with the highest probability is the final output level, that is:

[0061] V=argmax{P i (img),i=1,2,3,4}

[0062] wherein V is the output visibility level, and Pi(img) represents the probability that the input image img belongs to the i-th visibility class.

[0063] According to the above design, the road visibility level recognition method based on video images realizes detection on the to-be-analyzed road section in the actual target scene road, obtains the road visibility condition of the to-be-analyzed road section, and then timely warns and prompts the oncoming vehicles. The specific design operation is as follows:

[0064] The electronic warning board is used to warn the oncoming vehicles. When it is judged that a vehicle is approaching, the electronic display screen or sound is turned on to output the road visibility condition, so as to warn that the road section passed by the vehicle has adverse weather.

[0065] In actual application, the above road visibility condition recognition method based on video image analysis is applied based on the further designed system. The system includes a control module, and the control module includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the method are realized.

[0066] The above technical solution designs a road visibility condition recognition method based on video image analysis. The design uses the characteristics of video image signals of night road visibility, analyzes the road surface meteorological conditions corresponding to each type of scene video image on each type of scene road, obtains the road surface meteorological conditions corresponding to different visibility on each type of scene road, trains in combination with the Resnet network, obtains a multi-scene road surface meteorological condition detection model, and is then used in actual application to realize the detection of the road surface meteorological conditions of multi-scene roads, provide accurate road surface meteorological condition information for meteorology, traffic, traffic control, and drivers, take timely road control measures, reduce accidents, and the entire design system has the characteristics of low equipment cost and easy installation, can greatly reduce the difficulty of road surface meteorological condition detection, has broad market application prospect and economic value.

[0067] The embodiments of the present application are described in detail above in combination with the drawings, but the present application is not limited to the above embodiments. Within the knowledge of those skilled in the art, various changes can be made without departing from the purpose of the present application.

Claims

1. A method for detecting highway nighttime visibility based on monitoring images, characterized in that: The method is used to detect highway road visibility in nighttime scenes. It includes: constructing a nighttime road image dataset, extracting nighttime road image features, training and building a nighttime road visibility detection model, and detecting the visibility conditions of the road section to be analyzed. The specific steps are as follows: S1: We select surveillance camera video streams from different camera locations and visibility conditions on highways at night. We extract an image every 1 second from the video streams to form a time-series image dataset. This dataset is then divided into different levels of nighttime road image datasets based on the highway foggy visibility classification standard. S2: Downsample the images in the nighttime road image dataset to obtain a downsampled feature map containing spatial features. Then, the discrete cosine transform (DCT) coefficients and Shannon entropy features of the downsampled feature map are extracted to obtain the frequency domain feature map and spatial domain feature map of the nighttime road image. S3: Superimpose the downsampled feature map with the frequency domain feature map and the spatial domain feature map to obtain the SRGB feature vector of the nighttime road image; S4: According to the characteristics of nighttime road images, the deep residual network model is improved to form the network to be trained; S5: Input the SRGB feature vector into the network for feature extraction and classification. Using the visibility level label as the objective function, the model is trained and verified, and feedback is given to the model to update the network weights and optimize the model parameters to build a nighttime road visibility detection model. S6: Input the image to be tested into the constructed nighttime road visibility detection model, perform feature extraction and classification, and output the visibility level obtained by detection; In step S4, the deep residual network ResNet50 is used as the backbone model. Specific improvements include: in addition to the camera image data, the calculated frequency domain features and spatial domain features are combined to assist network classification, the number of input channels is adjusted to the number of channels of the fused features, a batch normalization layer is added after each convolutional layer of the deep learning pre-training model to enhance the stability of the model, the last fully connected layer is deleted, and then a new fully connected layer is added, whose output node number is equal to the number of visibility levels.

2. The method for detecting highway visibility at night based on monitoring images according to claim 1, characterized in that: In step S1, for each type of nighttime road scene, a surveillance video sequence of the target road is obtained based on the surveillance camera, and visibility conditions of each type of nighttime road are constructed. In combination with the foggy visibility standard, a visibility video corresponding to the nighttime road is obtained.

3. The method for detecting highway visibility at night based on monitoring images according to claim 1, characterized in that: In step S2, the step of extracting discrete cosine transform (DCT) coefficients and Shannon entropy features includes: 1) Downsample the original nighttime road image to obtain the feature map X1: the input dimension is C×H×W, the kernel size is 3×3, the stride is 2, and the padding is 1. After downsampling, the dimension of the feature map becomes C×H / 2×W / 2, where C represents the number of image channels, and W and H represent the width and height of the image, respectively. 2) Perform discrete cosine transform on the feature map X1 to obtain the feature map X2 after processing in the frequency domain, which is expressed as follows: Where C(u,v) is the transformed frequency domain coefficient, which represents the energy distribution of the image in the frequency domain, α(u) and α(v) are normalization coefficients, (x,y) represents the pixel position of the image, and f(x,y) is the pixel value of the image; (u,v) represents the coordinates in the frequency domain, which represents the frequency components in the horizontal and vertical directions, and (W,H) represents the width and height of the image; 3) Using X1 as input, the feature map X3 after spatial domain processing is obtained. The image is divided into local regions using a sliding window operation according to the following steps; for each local region, the probability distribution p(i) of each gray level is calculated, and the calculation formula is: Where count(i) represents the number of pixels with gray level i in the local area, and N is the total number of pixels in the local area; Based on the definition of Shannon entropy, the Shannon entropy H(w) of each local area is calculated. The specific formula is as follows: Where L is the grayscale level of the image.

4. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 3, characterized in that: The specific steps of step S3 are: normalize the size of the final output results X1, X2 and X3 of step S2, and then connect them through the concatenate function to obtain the fusion feature X fusion : X fusion =concat(X1,X2,X3)。 5. The method for detecting nighttime visibility of a highway based on monitoring images according to claim 1, characterized in that: In step S5, the model training steps include: 1) Divide the training set into a training set and a validation set in a ratio of 8:2, and then divide the divided training set again to perform 4-fold cross-validation; 2) Image augmentation based on random flipping, random brightness and saturation transformation; 3) Set the hyperparameters of BatchSize, learning rate, and maximum number of iterations, use the Adam optimization method, and use MSELoss as the loss function to supervise network learning; 4) Perform iterative training. After each epoch, the validation set and test set will be used to calculate the accuracy, and the best result in the training process will be saved.

6. The method for detecting highway visibility at night based on monitoring images according to claim 1, characterized in that: In step S6, the specific process of model feature extraction and classification is as follows: the deep convolutional network extracts the visibility feature vector of the input image and inputs it into the softmax function: Among them, P i (img) represents the probability that the input image belongs to the i-th category, K represents the number of image categories, i(img) represents the unnormalized score of the image in the i-th category; j(img) represents the unnormalized score of the image in the j-th category; the softmax function calculates the probability of visibility of each category, and the visibility level with the highest probability is the final output level, that is: V=argmax{P i (img),i=1,2,3,4} Where V is the visibility level of the output.

Images