Road icing detection system and method based on video processing

Abstract

The present invention discloses a road icing detection system and method based on video processing. The road icing detection system includes a video input module, a visible light and thermal infrared color image fusion module, a road line extraction and recognition module, an icing area extraction module, a processor module, and a GPS and video frame synchronization module; the video input module obtains road monitoring video images for a period of time and extracts video key frames; the road line extraction and recognition module extracts road lines and locks road areas; the icing area extraction module detects road icing based on images; the processor module restores the original image based on the received road icing image, and transmits the original image to the GPS and video frame synchronization module to find the longitude and latitude position of the road icing. The present invention can monitor the real-time, accurate and fixed-point icing in a small range of key urban areas, conduct long-term risk assessment, and provide immediate warning and monitoring for key urban areas.

Description

Technical Field

[0001] The present invention belongs to the technical field of road inspection, and relates to a detection system, and in particular to a road icing detection system and method based on video processing. Background Art

[0002] Traffic road icing refers to the icing phenomenon that occurs when precipitation (such as rain, snow, sleet, fog droplets, etc.) hits the road surface with a temperature below 0°C. It usually includes frozen residual snow, bumpy ice ruts, snowmelt or other reasons that cause road water to accumulate, and hard ice layers formed in cold seasons. When the road is icy, the friction coefficient between the vehicle tires and the road surface is greatly reduced, which makes it very easy for the vehicle to slip, thus causing road traffic accidents. Road icing generally occurs from December to April of the following year (i.e. from winter to early spring). Road icing often occurs in northern my country and higher altitude areas in the south.

[0003] So far, there is no reliable system to efficiently and accurately warn of road icing. At present, most areas use the method of manually observing whether the road is icy. Although this method is intuitive and effective, for the huge road network, manual warning is not only time-consuming and laborious, but also prone to negligence, especially in some areas where the traffic department does not supervise strictly, and often the seriousness of the problem is not realized until a traffic accident occurs. Therefore, real-time monitoring of road icing has become an urgent need to protect the safety of life and property and avoid traffic jams. Summary of the invention

[0004] The present invention provides a road icing detection system and method based on video processing, which can monitor the real-time, accurate and fixed-point icing in a small range of key urban areas, conduct long-term risk assessment, and provide instant warning and monitoring for key urban areas.

[0005] To solve the above technical problem, according to one aspect of the present invention, the following technical solution is adopted:

[0006] A road icing detection system based on video processing, the road icing detection system comprising: a video input module, a visible light and thermal infrared color image fusion module, a road line extraction and recognition module, an icing area extraction module, a processor module, and a GPS and video frame synchronization module;

[0007] The video input module acquires a period of road monitoring video images, extracts video key frames, and transmits the extracted key frames to the visible light and thermal infrared color image fusion module, obtains the visible light and thermal infrared color fusion image and transmits it to the road line extraction and recognition module;

[0008] The road line extraction and recognition module extracts road lines and locks the road area;

[0009] The icing area extraction module obtains road surface image information based on the monitoring image image processing of the road line extraction and recognition module, and performs road icing detection on the image;

[0010] The processor module restores the original image based on the received road icing image, and transmits the original image to the GPS and video frame synchronization module to find the longitude and latitude positions of the road icing.

[0011] As an implementation manner of the present invention, the video input module includes a visible light video input imaging unit and a thermal infrared video input imaging unit.

[0012] As an implementation mode of the present invention, the video input module includes a visible light video input imaging unit and extracts video key frames by clustering method, and searches for corresponding images in thermal infrared video after obtaining the visible light key frames.

[0013] As an implementation mode of the present invention, the clustering method is used to extract video key frames, including:

[0014] Use formula (1) to maintain a centroid for each class;

[0015]

[0016] Use formula (2) to calculate the similarity of its cluster centroid for each frame; (2)

[0018]

[0019]

[0020]

[0021] Use formula (3) to calculate the image with the largest entropy in each cluster and use it as the key frame;

[0022] (3)

[0023] As an implementation manner of the present invention, the video input module includes a visible light video input imaging unit and a thermal infrared video input imaging unit, both of which are high-speed cameras carried by a drone.

[0024] A detection method of the road icing detection system based on video processing, the detection method comprises the following steps:

[0025] (1) Obtaining visible light and thermal infrared color surveillance video images through the video input module, transmitting the obtained visible light and thermal infrared color key frame images to the visible light and thermal infrared color image fusion module, and performing color image fusion based on YUV space color transfer;

[0026] (2) The color fusion image obtained by the visible light and thermal infrared color image fusion module is transmitted to the road line extraction and recognition module for processing to obtain the road area, and the other areas of the image are weakened, and the processed image is transmitted to the ice area extraction module;

[0027] (3) The road line extraction module processes the received background image to obtain road area information, and transmits the road area information to the processor module;

[0028] (4) The icing area extraction module performs grayscale processing on the image, detects the edge of the ice surface using an image edge detection algorithm, and transmits the image showing road icing to the processor module;

[0029] (5) The processor module restores the original visible light image based on the received image and transmits it to the GPS and video frame synchronization module.

[0030] A detection method of the road icing detection system based on video processing, the detection method comprises the following steps:

[0031] (1) Obtaining visible light and thermal infrared color images through the video input module, and the video input module uses VideoCapture and VideoWriter in the cv2 module to extract each frame of monitoring image in the visible light video and thermal infrared video;

[0032] (2) The video input module uses clustering-based video key frame extraction based on the visible light monitoring image provided by the input module, and first clusters the video into n classes. The video frames within the n classes are similar, while the video frames between classes are not similar; the second step is to extract a representative from each class as a key frame. In addition, if the number of frames in a class is too small, then this class is not representative and is directly merged with adjacent frames; the video input module extracts the key frame of the visible light video, searches for the thermal infrared image corresponding to the key frame, and transmits the video key frame image to the visible light and thermal infrared color image fusion module;

[0033] (3) The visible light and thermal infrared color image fusion module uses color transfer based on YUV space to perform color image fusion. First, the visible light and thermal infrared dual-band images are linearly combined in the YUV color space to generate an initial color fusion image S, and its mean and standard deviation in the YUV channel are calculated; the combination coefficient ri is calculated based on the mean and standard deviation of the S image and the basic reference image in the U and V channels; 6 statistical values ​​of the combined reference image are obtained, and color transfer is performed in the YUV space according to formula (3) to obtain a natural color fusion image in the YUV space; the YUV color fusion image is converted back to the RGB space for display and observation; and the fused image is transmitted to the road line extraction module;

[0034] (4) The road line extraction module uses the inter-frame difference method to divide the image into binary images; uses the Canny edge detection method to extract the road line edges of the binary image with the best effect, weakens the non-road part of the image, and transmits the image to the ice area extraction module.

[0035] (5) The ice region extraction module uses an image edge detection algorithm to detect the ice region. The image edge detection algorithm differentiates or calculates the second-order differential of each pixel point of the image to determine the edge pixel point; the peak of the first-order differential image corresponds to the edge point of the image; the zero-crossing point of the second-order differential image corresponds to the edge point of the image; according to the characteristics of digital images, differentials are often used to replace derivative operations in the image processing process. For simple first-order derivative operations of the image, the gradient operator Sobel operator is used; the ice region extraction module transmits the image detected with ice to the processor module;

[0036] (6) The processor module restores the processed road icing image to the original image and transmits it to the GPS and video frame synchronization module;

[0037] (7) The GPS and video frame synchronization module uses GPS positioning to correspond the image to the latitude and longitude of the video shooting.

[0038] The beneficial effects of the present invention are as follows: the road icing detection system and method based on video processing proposed in the present invention can monitor the real-time, accurate and fixed-point icing in a small range in key urban areas, conduct long-term risk assessment, and provide immediate warning and monitoring for key urban areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 The figure is a schematic diagram of the composition of a road icing detection system based on video processing in one embodiment of the present invention.

[0040] Figure 2 The figure is a flow chart of a road icing detection method based on video processing in one embodiment of the present invention.

[0041] Figure 3 The figure is a flowchart of some steps of a road icing detection system based on video processing in one embodiment of the present invention. DETAILED DESCRIPTION

[0042] The preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0043] In order to further understand the present invention, preferred embodiments of the present invention are described below in conjunction with examples. However, it should be understood that these descriptions are only for further illustrating the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0044] The description in this section is only for several typical embodiments, and the present invention is not limited to the scope of the embodiments. The same or similar prior art means and some technical features in the embodiments are mutually replaced within the scope of the present invention.

[0045] The present invention discloses a road icing detection system based on video processing. Figure 1 FIG. 1 is a schematic diagram of the composition of a road icing detection system based on video processing in one embodiment of the present invention; please refer to FIG. Figure 1 In one embodiment of the present invention, the road icing detection system includes: a video input module 1, a visible light and thermal infrared color image fusion module 2, a road line extraction and recognition module 3, an icing area extraction module 4, a processor module 5, and a GPS and video frame synchronization module 6.

[0046] The video input module 1 obtains road monitoring video images for a period of time, extracts video key frames, and transmits the extracted key frames to the visible light and thermal infrared color image fusion module 2, obtains the visible light and thermal infrared color fusion image and transmits it to the road line extraction and recognition module 3. The road line extraction and recognition module 3 extracts road lines and locks the road area. The icing area extraction module 4 obtains road surface image information based on the monitoring image image processing of the road line extraction and recognition module 3, and performs road icing detection on this image. The processor module 5 restores the original image based on the received road icing image, and transmits the original image to the GPS and video frame synchronization module 6 to find the longitude and latitude position of the road icing.

[0047] In one embodiment of the present invention, the video input module 1 includes a visible light video input imaging unit and a thermal infrared video input imaging unit. In one embodiment of the present invention, the video input module includes a visible light video input imaging unit and a thermal infrared video input imaging unit, both of which are high-speed cameras carried by drones.

[0048] In one embodiment of the present invention, the video input module includes a visible light video input image unit, and extracts video key frames by clustering method, and searches for corresponding images in thermal infrared video after obtaining visible light key frames. In one embodiment of the present invention, the above clustering method extracts video key frames, including:

[0049] Use formula (1) to maintain a centroid for each class;

[0050]

[0051] Use formula (2) to calculate the similarity of its cluster centroid for each frame;

[0052]

[0053]

[0054]

[0055] Use formula (3) to calculate the image with the largest entropy in each cluster and use it as the key frame;

[0056]

[0057] The present invention also discloses a detection method of the road icing detection system based on video processing. Figure 2 is a flow chart of a road icing detection method based on video processing in one embodiment of the present invention; please refer to Figure 2 In one embodiment of the present invention, the detection method comprises the following steps:

[0058] (1) Obtaining visible light and thermal infrared color surveillance video images through the video input module, transmitting the obtained visible light and thermal infrared color key frame images to the visible light and thermal infrared color image fusion module, and performing color image fusion based on YUV space color transfer;

[0059] (2) The color fusion image obtained by the visible light and thermal infrared color image fusion module is transmitted to the road line extraction and recognition module for processing to obtain the road area, and the other areas of the image are weakened, and the processed image is transmitted to the ice area extraction module;

[0060] (3) The road line extraction module processes the received background image to obtain road area information, and transmits the road area information to the processor module;

[0061] (4) The icing area extraction module performs grayscale processing on the image, detects the edge of the ice surface using an image edge detection algorithm, and transmits the image showing road icing to the processor module;

[0062] (5) The processor module restores the original visible light image based on the received image and transmits it to the GPS and video frame synchronization module.

[0063] The present invention discloses a detection method of the road icing detection system based on video processing. Figure 3 This is a flow chart of some steps of a road icing detection system based on video processing in one embodiment of the present invention; Figure 2 , Figure 3 In one embodiment of the present invention, the detection method comprises the following steps:

[0064] (1) Obtaining visible light and thermal infrared color images through the video input module, and the video input module uses VideoCapture and VideoWriter in the cv2 module to extract each frame of monitoring image in the visible light video and thermal infrared video;

[0065] (2) The video input module uses clustering-based video key frame extraction based on the visible light monitoring image provided by the input module, and first clusters the video into n classes. The video frames within the n classes are similar, while the video frames between classes are not similar; the second step is to extract a representative from each class as a key frame. In addition, if the number of frames in a class is too small, then this class is not representative and is directly merged with adjacent frames; the video input module extracts the key frame of the visible light video, searches for the thermal infrared image corresponding to the key frame, and transmits the video key frame image to the visible light and thermal infrared color image fusion module;

[0066] (3) The visible light and thermal infrared color image fusion module uses color transfer based on YUV space to perform color image fusion. First, the visible light and thermal infrared dual-band images are linearly combined in the YUV color space to generate an initial color fusion image S, and its mean and standard deviation in the YUV channel are calculated; the combination coefficient ri is calculated based on the mean and standard deviation of the S image and the basic reference image in the U and V channels; 6 statistical values ​​of the combined reference image are obtained, and color transfer is performed in the YUV space according to formula (3) to obtain a natural color fusion image in the YUV space; the YUV color fusion image is converted back to the RGB space for display and observation; and the fused image is transmitted to the road line extraction module;

[0067] (4) The road line extraction module uses the inter-frame difference method to divide the image into binary images; uses the Canny edge detection method to extract the road line edges of the binary image with the best effect, weakens the non-road part of the image, and transmits the image to the ice area extraction module.

[0068] (5) The ice region extraction module uses an image edge detection algorithm to detect the ice region. The image edge detection algorithm differentiates or calculates the second-order differential of each pixel point of the image to determine the edge pixel point; the peak of the first-order differential image corresponds to the edge point of the image; the zero-crossing point of the second-order differential image corresponds to the edge point of the image; according to the characteristics of digital images, differentials are often used to replace derivative operations in the image processing process. For simple first-order derivative operations of the image, the gradient operator Sobel operator is used; the ice region extraction module transmits the image detected with ice to the processor module;

[0069] (6) The processor module restores the processed road icing image to the original image and transmits it to the GPS and video frame synchronization module;

[0070] (7) The GPS and video frame synchronization module uses GPS positioning to correspond the image to the latitude and longitude of the video shooting.

[0071] In summary, the road icing detection system and method based on video processing proposed in the present invention can monitor the real-time, accurate and fixed-point icing in a small range in key urban areas, conduct long-term risk assessment, and provide instant warning and monitoring for key urban areas.

[0072] The description and application of the present invention here are illustrative and are not intended to limit the scope of the present invention to the above-described embodiments. The deformation and changes of the embodiments disclosed here are possible, and the replacement of the embodiments and the various components equivalent to those of ordinary skill in the art are well known. It should be clear to those skilled in the art that the present invention can be realized in other forms, structures, arrangements, proportions, and with other components, materials and parts without departing from the spirit or essential characteristics of the present invention. Other deformations and changes can be made to the embodiments disclosed here without departing from the scope and spirit of the present invention.

Claims

1. A road icing detection system based on video processing, characterized in that: The road icing detection system includes: a video input module, a visible light and thermal infrared color image fusion module, a road line extraction and recognition module, an icing area extraction module, a processor module, and a GPS and video frame synchronization module; The video input module acquires a period of road monitoring video images, extracts video key frames, and transmits the extracted key frames to the visible light and thermal infrared color image fusion module, obtains the visible light and thermal infrared color fusion image and transmits it to the road line extraction and recognition module; The road line extraction and recognition module extracts road lines and locks the road area; The icing area extraction module obtains road surface image information based on the monitoring image image processing of the road line extraction and recognition module, and performs road icing detection on the image; The processor module restores the original image based on the received road icing image, and transmits the original image to the GPS and video frame synchronization module to find the longitude and latitude positions of the road icing.

2. The road icing detection system based on video processing according to claim 1, characterized in that: The video input module includes a visible light video input imaging unit and a thermal infrared video input imaging unit.

3. The road icing detection system based on video processing according to claim 2, characterized in that: The video input module includes a visible light video input image unit and extracts video key frames through a clustering method, and searches for corresponding images in the thermal infrared video after obtaining the visible light key frames.

4. The road icing detection system based on video processing according to claim 3, characterized in that: Clustering method is used to extract video key frames, including: Use formula (1) to maintain a centroid for each class; (1) Use formula (2) to calculate the similarity of its cluster centroid for each frame; (2) Use formula (3) to calculate the image with the largest entropy in each cluster and use it as the key frame; (3) 5. The road icing detection system based on video processing according to claim 2, characterized in that: The video input module includes a visible light video input imaging unit and a thermal infrared video input imaging unit, both of which are high-speed cameras carried by the drone.

6. A detection method of a road icing detection system based on video processing according to any one of claims 1 to 5, characterized in that: The detection method comprises the following steps: (1) Obtaining visible light and thermal infrared color surveillance video images through the video input module, transmitting the obtained visible light and thermal infrared color key frame images to the visible light and thermal infrared color image fusion module, and performing color image fusion based on YUV space color transfer; (2) The color fusion image obtained by the visible light and thermal infrared color image fusion module is transmitted to the road line extraction and recognition module for processing to obtain the road area, and the other areas of the image are weakened, and the processed image is transmitted to the ice area extraction module; (3) The road line extraction module processes the received background image to obtain road area information, and transmits the road area information to the processor module; (4) The icing area extraction module performs grayscale processing on the image, detects the edge of the ice surface using an image edge detection algorithm, and transmits the image showing road icing to the processor module; (5) The processor module restores the original visible light image based on the received image and transmits it to the GPS and video frame synchronization module.

7. A detection method of a road icing detection system based on video processing according to any one of claims 1 to 5, characterized in that: The detection method comprises the following steps: (1) Obtaining visible light and thermal infrared color images through the video input module, and the video input module uses VideoCapture and VideoWriter in the cv2 module to extract each frame of monitoring image in the visible light video and thermal infrared video; (2) The video input module uses clustering-based video key frame extraction based on the visible light monitoring image provided by the input module, and first clusters the video into n classes. The video frames within the n classes are similar, while the video frames between classes are not similar; the second step is to extract a representative from each class as a key frame. In addition, if the number of frames in a class is too small, then this class is not representative and is directly merged with adjacent frames; the video input module extracts the key frame of the visible light video, searches for the thermal infrared image corresponding to the key frame, and transmits the video key frame image to the visible light and thermal infrared color image fusion module; (3) The visible light and thermal infrared color image fusion module uses color transfer based on YUV space to perform color image fusion. First, the visible light and thermal infrared dual-band images are linearly combined in the YUV color space to generate an initial color fusion image S, and its mean and standard deviation in the YUV channel are calculated; the combination coefficient ri is calculated based on the mean and standard deviation of the S image and the basic reference image in the U and V channels; 6 statistical values ​​of the combined reference image are obtained, and color transfer is performed in the YUV space according to formula (3) to obtain a natural color fusion image in the YUV space; the YUV color fusion image is converted back to the RGB space for display and observation, and the fused image is transmitted to the road line extraction module; (4) The road line extraction module uses the inter-frame difference method to divide the image into binary images; uses the Canny edge detection method to extract the road line edge of the binary image with the best effect, weakens the non-road part of the image, and transmits the image to the ice area extraction module; (5) The ice region extraction module uses an image edge detection algorithm to detect the ice region. The image edge detection algorithm differentiates or calculates the second-order differential of each pixel point of the image to determine the edge pixel point; the peak of the first-order differential image corresponds to the edge point of the image; the zero-crossing point of the second-order differential image corresponds to the edge point of the image; according to the characteristics of digital images, differentials are often used to replace derivative operations in the image processing process. For simple first-order derivative operations of the image, the gradient operator Sobel operator is used; the ice region extraction module transmits the image detected with ice to the processor module; (6) The processor module restores the processed road icing image to the original image and transmits it to the GPS and video frame synchronization module; (7) The GPS and video frame synchronization module uses GPS positioning to correspond the image to the latitude and longitude of the video shooting.

Images