A method and device for identifying severe weather on highways

Abstract

This invention discloses a method and device for identifying severe weather on highways, comprising: A) a severe weather video dataset; B) labeling the dataset and dividing it into seven categories: sunny, cloudy, rainy, snowy, foggy, dusty, and night; C) converting it to the HSV color channel; D) detecting overexposed areas; E) extracting scene illumination components, extracting texture features, estimating scene transmittance, and calculating moving target vectors; F) classifying weather conditions based on illumination components, texture features, scene transmittance, motion information, chroma components, saturation components, and brightness components; and G) summarizing detection results from multiple frames and voting to determine the weather type. This method addresses the low severe weather recognition rate and limited number of recognized types in related technologies, achieving accurate recognition of seven major weather categories.

Description

Technical Field

[0001] The invention relates to a method and device for identifying severe weather on highways. Background Art

[0002] Inclement weather conditions such as fog, rain, snow, and dust can make highway roads slippery and reduce visibility, increasing the risk of traffic accidents. Therefore, effective methods for identifying inclement weather on highways are needed to ensure early warning of adverse weather conditions, thereby reducing accidents and casualties. Currently, traditional methods for detecting inclement weather on highways rely primarily on deploying meteorological sensors along both sides of the highway or at specific locations along the highway, observing meteorological elements to detect inclement weather. However, this approach is not feasible due to the high cost of meteorological sensors, which prevent their widespread deployment.

[0003] To address this issue, a series of severe weather recognition methods have been proposed based on surveillance cameras widely deployed across highway networks. These methods analyze and process images captured by surveillance cameras to classify and identify weather conditions. However, due to the diversity, complexity, and variability of severe weather conditions, existing technologies are unable to accurately classify all types of severe weather, such as fog, rain, snow, and dust. Consequently, the classification accuracy falls short of practical application requirements. Summary of the Invention

[0004] Based on the above problems in the prior art, it is necessary to provide a method and device for identifying severe weather on highways with comprehensive identification types and high accuracy to address the above problems.

[0005] According to one aspect of the present invention, a method for identifying severe weather on a highway is provided, which is characterized by comprising:

[0006] A) Collect videos shot in highway environments to build a severe weather dataset;

[0007] B) Label and classify the collected data set into seven categories: night, dust, fog, rain, snow, cloudy, and sunny;

[0008] C) converting the input video frame I into HSV color channels to obtain chroma component, saturation component and brightness component;

[0009] D) Input the brightness component, detect the overexposed area, and obtain the mask of the non-overexposed area;

[0010] E) extracting scene illumination components, extracting texture features, estimating scene transmittance, and calculating a moving target vector in a non-overexposed area of ​​video frame I;

[0011] F) performing weather recognition on the video based on the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component, and brightness component of the non-overexposed area of ​​the video frame, and sequentially identifying night, dusty, foggy, rainy, snowy, cloudy, and sunny weather;

[0012] G) Count the detection results of multiple frames and use multi-frame voting to finally determine the weather type of the current scene.

[0013] Step E includes

[0014] E1) inputting the brightness component extracted in step C) into a least squares filtering algorithm to obtain an illumination component,

[0015] E2) Convert the image into grayscale image, construct grayscale co-occurrence matrix, calculate the angular second moment, entropy, contrast and other indicators to obtain texture features,

[0016] E3) Calculate the difference between adjacent pixels to extract local gradient texture features, use the brightness component, estimate the initial transmittance based on the CEP ellipsoid prior, and use the local gradient texture information as the confidence weight to correct the initial transmittance to obtain the scene transmittance.

[0017] E4) Generate a series of images of different resolutions using a Gaussian pyramid to construct a pyramid; detect and track feature points using FAST corner points; and then use the Lucas-Kanade optical flow estimation algorithm to estimate the optical flow of the feature points to obtain motion information.

[0018] The step F) comprises:

[0019] F1) Setting a brightness threshold based on experience, calculating the percentage of dark pixels below the threshold in the non-overexposed area and the average brightness of the non-overexposed scene, setting a percentage threshold and an average brightness threshold, and determining whether the scene is night. If it is night, identify the next frame;

[0020] F2) If it is not night, continue to determine whether it is a dusty day. For dusty days, obtain the overall color information of the current scene based on the scene's chromaticity components, set a chromaticity threshold, calculate the average chromaticity of the non-overexposed area, and determine whether the current scene is a dusty day based on the color. If it is a dusty day, identify the next frame;

[0021] F3) If it is not a dusty day, continue to determine whether it is foggy. For foggy days, calculate the proportion of low transmittance values ​​in the upper half of the non-overexposed area, the average transmittance intensity and the average saturation, and set the corresponding threshold. Based on the threshold, determine whether the current scene is foggy. If it is foggy, identify the next frame;

[0022] F4) If it is not foggy, continue to determine whether it is raining or snowing. For rainy and snowy conditions, the SVM model is trained offline, combining motion information, color information, and texture features of the non-overexposed area. Online inference is performed to determine whether the current scene is rainy or snowy. If it is rainy or snowy, the next frame is identified.

[0023] F5) If it is not rainy or snowy, continue to determine whether it is cloudy or sunny. For cloudy and sunny days, an SVM model is trained offline based on the average brightness and average light intensity of the non-overexposed area. Through online inference, it determines whether the current scene is cloudy or sunny.

[0024] F6) To ensure the stability of the detection results, the detection results of multiple frames are counted and a multi-frame voting method is adopted to finally determine the weather type of the current scene.

[0025] According to another aspect of the present invention, there is provided a device for identifying severe weather on a highway, characterized by comprising:

[0026] An acquisition module is used to acquire videos of highways;

[0027] A feature extraction module is used to extract the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component and brightness component of the non-overexposed area of ​​the video frame;

[0028] The recognition module is used to classify the video into a weather category based on the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component and brightness component of the non-overexposed area of ​​the video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 The figure is a schematic diagram of the overall process of a method for identifying severe weather conditions on highways according to the present invention.

[0030] Figure 2 This is a structural block diagram of a device for identifying severe weather on highways according to the present invention.

[0031] Specific implementation

[0032] According to an embodiment of the present invention, a method for identifying severe weather is as follows: Figure 1 As shown, it mainly includes the following steps:

[0033] a) Convert the video frame from RGB color space to HSV and obtain the chroma component, saturation component, and brightness component.

[0034] b) Input the brightness component, detect the overexposed area, and obtain the mask of the non-overexposed area

[0035] c) Input the brightness component and use the least squares filtering algorithm to calculate the illumination component of the non-overexposed area.

[0036] d) Grayscale the image, construct the grayscale co-occurrence matrix of the non-overexposed area, and calculate texture features such as angular second moment, entropy, and contrast.

[0037] e) Calculating the difference between adjacent pixels to extract local gradient texture features, and combining the brightness component, using the modified ellipsoid prior designed by the present invention to calculate the scene transmittance of the non-overexposed area.

[0038] f) Use the pyramid Lucas-Kanade to obtain the target motion information of the non-overexposed area.

[0039] g) Set a brightness threshold based on experience. Count the proportion of dark pixels below the threshold to the pixels in the non-overexposed area, as well as the average brightness of the non-overexposed scene. Set a percentage threshold and an average brightness threshold to determine whether the scene is night. If it is night, identify the next frame.

[0040] h) If it is not night, continue to determine whether it is a dusty day. Set a chromaticity threshold, calculate the average chromaticity of the non-overexposed area, and determine whether the current scene is a dusty day. If it is a dusty day, identify the next frame.

[0041] i) If the scene is not dusty, the next step is to determine whether it is foggy. Based on the scene transmittance, the percentage of low transmittance values ​​in the upper half of the non-overexposed area is calculated. Combined with the scene transmittance and saturation components, the current scene is determined to be foggy. If so, the next frame is identified.

[0042] j) If it's not foggy, the system continues to determine whether it's raining or snowing. For rainy and snowy conditions, the SVM model is trained offline, combining target motion information, color information, and texture features in non-overexposed areas. Online inference is then performed to determine whether the current scene is rainy or snowy. If so, the next frame is identified.

[0043] k) If it's not raining or snowing, continue determining whether it's cloudy or sunny. Train an SVM model offline based on the average brightness and average light intensity of the non-overexposed areas. Use online inference to determine whether the scene is cloudy or sunny.

[0044] l) Count the detection results of multiple frames and use multi-frame voting to finally determine the weather type of the current scene.

[0045] According to a further embodiment of the present invention, in the above step b), the overexposed area in the image is detected according to the overexposed pixel detection method of reference [1], and the detection method is as follows:

[0046] S th =F v(v)*F h (h)

[0047] Among them, S th is the overexposure threshold, F v (v) is to adjust S according to the brightness of the pixel th The scaling factor of the value has a Gaussian function with respect to the brightness v. F h (h) is the chromaticity adjustment S according to the pixel th The scaling factor of the value is composed of three Gaussian functions about the chromaticity h. For each pixel in the image, its overexposure threshold is obtained according to its brightness and chromaticity. When its saturation s is greater than S th , the point is considered overexposed. Finally, the mask of the non-overexposed area is obtained.

[0048] According to a further embodiment of the present invention, in the above step c), a least squares filtering algorithm is used to obtain the illumination component of the scene:

[0049]

[0050]

[0051] Among them, x is the pixel position index, L is the final solution of the illumination component, V represents the brightness component of the image, α is the regularization coefficient, set to 0.1, and W is the smoothing weight, which is the first-order difference of the brightness component. The calculation is obtained, including both horizontal and vertical directions, β is the smoothing coefficient, set to 1.5, ε is set to 1e-4 to prevent the denominator from being 0, is the first-order difference of the illumination component.

[0052] According to a further embodiment of the present invention, in step d) above, the image is first grayscaled, and then a gray-level co-occurrence matrix is ​​constructed to statistically analyze the image's texture features. Because the gray-level co-occurrence matrix is ​​a correlation function between different grayscale values ​​of an image at specific angles and distances, it reflects the two-dimensional statistical characteristics of the image's texture. The following describes the construction of the gray-level co-occurrence matrix and the statistical analysis of texture features.

[0053] Constructing a gray-level co-occurrence matrix: The gray-level co-occurrence matrix is ​​an N*N square matrix, where N is the number of grayscale quantization levels in the image. The rows and columns of the matrix represent different grayscale levels. Each element in the matrix represents the number of times the grayscale level represented by the row and column co-occurs for all pairs of pixels at a specific orientation and distance. The generation process is as follows.

[0054] (1) In an image, any point (x, y) and a point (x+a, y+b) offset from it form a point pair. Let the grayscale value of this point pair be (f1, f2). Assuming the maximum grayscale level of the image is L, there are L*L combinations of f1 and f2.

[0055] (2) For the entire image, count the number of times each (f1, f2) value appears, and then arrange it into a square matrix.

[0056] (3) The total number of occurrences of (f1, f2) is normalized to obtain the probability P(f1, f2). The resulting matrix is ​​the gray-level co-occurrence matrix.

[0057] Statistical texture features: After constructing the gray-level co-occurrence matrix, statistics should be performed on it, and the statistical features of the gray-level co-occurrence matrix should be used to reflect the spatial correlation characteristics of the image. Several commonly used statistical features are as follows:

[0058] Angular Second Moment (ASM): This is the sum of the squares of the elements in the grayscale co-occurrence matrix. It measures the stability of the grayscale variation of the image texture and reflects the uniformity of the image grayscale distribution and the coarseness of the texture. A large energy value indicates that the current texture is a regular and stable texture. The calculation formula is as follows:

[0059]

[0060] Among them, P(i,j) represents the element of the gray-level co-occurrence matrix, and i and j represent the row coordinates and column coordinates of the element respectively.

[0061] Entropy (ENT): A measure of the randomness of the information contained in an image. Entropy is maximum when all values ​​in the co-occurrence matrix are equal or when pixel values ​​exhibit the greatest randomness. Therefore, the entropy value indicates the complexity of the image's grayscale distribution; the larger the entropy value, the more complex the image.

[0062] ENT=-∑ i ∑ j P(i,j)log(P(i,j)).

[0063] Contrast (Con): This measures the distribution of matrix values ​​and the amount of local variation within an image, reflecting image clarity and the depth of texture grooves. Deeper texture grooves create a greater contrast and a clearer image; conversely, lower contrast values ​​create shallower grooves and a blurred image.

[0064]

[0065] According to a further embodiment of the present invention, in step e) above, the initial transmittance is estimated based on the CEP ellipsoid prior published in reference [2], and guided filtering is applied to the original image to ensure that the estimated transmittance is consistent with the original image structure. To improve the problem of underestimation of transmittance in the sky and strongly reflective areas, the present invention uses local gradient texture information as a confidence weight to correct it, which is defined as follows:

[0066]

[0067] Among them, t(x) and t(x)′ are the initial transmittance and the corrected transmittance, respectively. max is the maximum transmittance of the entire image, C(x) is the gradient texture of the original image, I(x) is the brightness of the original image, It means that the rough result of transmittance is normalized by the gradient information of brightness. The smaller it is, the closer the area is to the high-brightness weak-texture area, which may be a strong light or overexposed area, and its transmittance is supplemented.

[0068] According to a further embodiment of the present invention, in the above step f), the present invention first grayscales the image sequence. Secondly, a Gaussian pyramid is used to generate a series of images of different resolutions to construct a pyramid. Then, on each pyramid layer, the FAST corner point is used to detect feature points. Further, for each selected feature point, its position is found in the current frame, and the corresponding position is found in the next frame. Between two frames, the pyramid Lucas-Kanade optical flow estimation algorithm is used to calculate the motion information of the feature point. Specifically, for each feature point, a 2x2 spatial derivative matrix (Ix and Iy) is constructed to represent the image gradient in the x and y directions, respectively. A time derivative matrix (It) is constructed to represent the image difference between the two frames. On each pyramid layer, these derivative matrices are used to solve the Lucas-Kanade equation to estimate the motion information of the feature point.

[0069] According to a further embodiment of the present invention, in the above step g), the dark pixel ratio is defined as

[0070]

[0071] N represents the total number of pixels in the video frame, mask is the mask of the non-overexposed area, N mask Indicates the total number of pixels after excluding the overexposed area, V i represents the brightness of the i-th pixel, T dark is the dark pixel brightness threshold, which is set to 50 after offline statistics. Indicates the total number of dark pixels in the non-overexposed area. Then, based on the dark pixel ratio and the average brightness of the scene, it is determined whether the current scene is a low-light scene:

[0072]

[0073] in, represents the average brightness of the scene. The threshold in the condition is obtained by offline statistical analysis based on Raida's law. Represents a discriminant, when the discriminant condition When satisfied, the discriminant result is 1, that is, D dark If it is 1, it indicates a low-light scene; otherwise, it is 0, which indicates a non-low-light scene.

[0074] According to a further embodiment of the present invention, in the above step h), the dusty environment has a yellowish chromaticity and a low chromaticity, and whether the current scene is a dusty scene is determined based on the scene average chromaticity information:

[0075]

[0076] in, represents the average chromaticity of the scene. The threshold in the condition is obtained by offline statistical analysis based on Raida's law. Represents a discriminant, when the discriminant condition When satisfied, the discriminant result is 1, that is, D dust If it is 1, it indicates a dusty scene; otherwise, it is 0, which indicates a non-dusty scene.

[0077] According to a further embodiment of the present invention, in the above step i), since the diffuse transmittance is lower than that of general scenes in foggy weather, the average transmittance of the scene is very helpful for judging foggy scenes with uniform fog. At the same time, according to the prior knowledge of the camera installation angle on the highway, the probability of fog in the upper half of the scene is relatively high, so the proportion of low transmittance values ​​P in the upper half of the scene is further calculated. t :

[0078]

[0079] Among them, t i represents the transmittance of the i-th pixel in the scene, T t is the transmittance threshold, which is set to 80 for offline statistical analysis according to Raida's law, and mask i *(t i <T t ) indicates that the transmittance of the non-overexposed area is less than T t pixels.

[0080] In addition, scattering will cause the saturation of the area to decrease. Therefore, whether it is foggy or not is determined based on the transmittance and saturation:

[0081]

[0082] in, represents the average transmitted intensity of the scene, It represents the average saturation of the scene. The threshold in the condition is obtained by offline statistical analysis based on Raida's law. Represents a discriminant, when the discriminant condition When satisfied, the discriminant result is 1, that is, D t The value is 1, indicating foggy weather, otherwise it is 0, indicating no fog.

[0083] In combination with the severe weather identification method for highways in the above embodiment, a severe weather identification device for highways is also provided in this embodiment to implement the above embodiment and preferred implementation mode, and the details that have been explained will not be repeated here. The term "module" used below refers to a device composed of software and / or hardware that can realize a predetermined function. Although it is ideal that the device described in the following embodiments is mainly implemented by software, it is also possible to consider implementing it by hardware or a combination of software and hardware. Figure 2 As shown, the device includes: an acquisition module for acquiring a video of a highway; a feature extraction module for extracting the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component and brightness component of the non-overexposed area of ​​the video frame; and an identification module for classifying the video to obtain a weather category based on the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component and brightness component of the non-overexposed area of ​​the video frame.

[0084] In the aforementioned severe weather identification device for highways, each module may be implemented in whole or in part using software, hardware, or a combination of both. These modules may be embedded in or independent of a computer processor in the form of hardware, or they may be stored in the computer memory in the form of software, so that the processor can call and execute operations corresponding to each module.

[0085] [1]Yoon,Y.-J.;Byun,K.-Y.;Lee,D.-H.;Jung,S.-W.;Ko,S.-JA New HumanPerception-Based Over-Exposure Detection Method for Color Images.Sensors2014,14,17159-17173.https: / / doi.org / 10.3390 / s140917159

[0086] [2]Berman D,Avidan S.Non-local image dehazing[A].Proceedings of theIEEE conference on computer vision and pattern recognition[C].2016:1674-1682.

Claims

1. A method for identifying severe weather on highways, characterized in that The steps include: A) Collect videos shot in highway environments and build a severe weather video dataset; B) Identify and classify the weather types of the above-collected dataset into seven categories: night, dust, fog, rain, snow, cloudy, and sunny. C) converting the input video frame I into HSV color channels to obtain chroma component, saturation component and brightness component; D) Input the brightness component, detect the overexposed area, and obtain the mask of the non-overexposed area; E) extracting scene illumination components, extracting texture features, estimating scene transmittance, and calculating a moving target vector in a non-overexposed area of ​​video frame I; F) performing weather recognition on the video based on the illumination component, texture feature, scene transmittance, motion information, chroma component, saturation component, and brightness component of the non-overexposed area of ​​the video frame, and sequentially identifying night, dusty, foggy, rainy, snowy, cloudy, and sunny weather; G) Count the results of multiple frames and use multi-frame voting to finally determine the weather type of the current scene. in: The operations for extracting scene illumination components in step E include: Extract the brightness component of the video frame, input the brightness component into the least squares filtering algorithm to obtain the illumination component, The operations for extracting texture features in step E include: Convert the image to grayscale; Construct gray-level co-occurrence matrix; Calculate the angular second moment, entropy, contrast index, and obtain texture features. The operation of estimating the scene transmittance in step E includes: Calculate the difference between adjacent pixels to extract local gradient texture features; Input brightness component and estimate initial transmittance based on CEP ellipsoid prior; Use local gradient texture information as confidence weight to correct the initial transmittance. The operation of obtaining motion information in step E includes: A Gaussian pyramid is used to generate a series of images with different resolutions to construct a pyramid. The FAST corner points are used to detect and track feature points. The Lucas-Kanade optical flow estimation algorithm is then used to estimate the optical flow of the feature points to obtain motion information.

2. The method for identifying severe weather on highways according to claim 1, characterized in that: The operations to identify night include: Based on experience, a brightness threshold is set. The percentage of dark pixels below the threshold in the non-overexposed area and the average brightness of the non-overexposed scene are counted. The percentage threshold and average brightness threshold are then set to determine whether the scene is night. Operations to identify dusty days include: Set the chroma threshold, calculate the average chroma of the non-overexposed area, and determine whether the current scene is a dusty scene. The operations for identifying foggy days include: Calculate the proportion of low transmittance values ​​in the upper half of the non-overexposed area, the average transmittance intensity, and the average saturation, set corresponding thresholds, and use these thresholds to determine whether the current scene is foggy. Operations to identify rain and / or snow include: Combine the motion information, color information, and texture features of the non-overexposed area to train the SVM model offline, and then use online inference to determine whether the current scene is rainy or snowy. Operations to identify cloudy and / or sunny days include: An SVM model is trained offline based on the average brightness and average light intensity of the non-overexposed area, and then judged whether the current scene is cloudy or sunny through online inference.

Images