Road video monitoring equipment anomaly detection method based on video semantics
Through video semantic recognition and deep learning technology, the angle, occlusion and lens destruction problems in road monitoring equipment are identified, which solves the problems of high error detection rate and insufficient flexibility of traditional detection methods, and realizes automated operation and maintenance management and efficient maintenance .
Patent Information
- Application Number
- CN202411104011.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional video image abnormality detection technology is difficult to effectively detect focal length, angle, occlusion and lens debris in road video surveillance equipment. The error detection rate is high, which affects operation and maintenance efficiency, and lacks flexibility and personalized and rapid judgment capabilities.
Using a detection method based on video semantics, by acquiring road monitoring images, identifying specific targets and areas, calculating proportions to judge angle abnormalities, identifying occlusions and determining their location and area, combining deep learning models and image processing algorithms, lens debris is determined. Generate maintenance alarm information.
It realizes automatic detection of abnormal status of road video surveillance equipment, reduces manual operation and maintenance, improves maintenance timeliness and equipment proper rate, and effectively monitors and records road traffic conditions.
Smart Images

Figure CN118942056A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image recognition, and more specifically, to a method for detecting anomalies of road video monitoring equipment based on video semantics. Background Art
[0002] Common faults of road video surveillance equipment include hardware problems such as power failure, network disconnection, equipment damage, network packet loss, etc., as well as setting problems such as focus, pan / tilt, white balance, and maintenance problems such as lens occlusion and lighting problems. The former is directly manifested in the back-end video as black screen, white screen, blue screen, and other no picture failures and picture freezes, intermittent, mosaic, etc., while the latter is manifested as blurred pictures, angle problems (facing the sky, facing the road or facing outside the road), color cast, occlusion, lens contamination (frost, fog, water droplets, dust), too dark pictures, loss of details, low contrast, etc.
[0003] Traditional video image anomaly detection technology, based on image and image quality analysis, is more suitable for the detection of no picture, mosaic, color cast, too dark picture, and low contrast. However, the detection effect is not ideal for problems such as focal length, angle, occlusion, lens contamination (frost, fog, water droplets, dust), etc., and the false detection rate is relatively high, which seriously affects the operation and maintenance efficiency.
[0004] To this end, many researchers have tried to use deep learning-based classification technology to identify abnormal video image states, and have achieved certain results. However, this method still has some problems: First, the boundaries of abnormal state classification are unknown and uncontrollable. Take the abnormal state of facing the sky in the angle problem as an example. The proportion of the sky in the image that indicates abnormality varies with time, place, and needs. However, with the classification algorithm based on deep learning training, the boundaries of this classification have been internalized in the algorithm and become a black box. It is neither known to the user nor can it be changed by the user. This greatly limits the flexibility of the application of abnormal state detection algorithms of video surveillance equipment.
[0005] Second, it is difficult to integrate personalized quick judgment skills based on experience. Take the video surveillance equipment on some bridges and viaducts as an example. When some equipment is working normally, most of the area is the sky or water similar to the sky. It is very easy to be identified as facing the sky using the detection algorithm that is common to various scenes. Of course, a large number of image samples on bridges can be collected and labeled to train special recognition algorithms for bridges, but this method is too expensive. However, for the same picture, manual recognition is very fast because humans have many quick judgment skills based on experience. How to flexibly apply these skills to the detection of abnormal conditions of monitoring equipment has become a topic. Summary of the invention
[0006] In order to solve the deficiencies in the prior art, the present invention provides a road video surveillance equipment anomaly detection method based on video semantics, which can automatically discover the abnormal state of road video surveillance equipment during the operation and maintenance management of road video surveillance equipment, and improve the timeliness of road video surveillance equipment maintenance.
[0007] As a first aspect of the present invention, a method for detecting anomalies of road video surveillance equipment based on video semantics is provided, comprising the following steps: Step S1: Acquire the current road monitoring image captured by the road video monitoring equipment; Step S2: using a video semantic recognition algorithm to respectively identify all specific targets and all specific areas from the current road monitoring image; Step S3: Calculate the proportion of each specific area in the current road monitoring image, and determine whether the proportion of each specific area is within a reasonable proportion range; if it is not within the reasonable proportion range, determine that the road video monitoring device has an abnormal monitoring angle; if it is within the reasonable proportion range, identify the obstruction from the current road monitoring image, and determine the position, shape and area of the obstruction, and then execute step S4; Step S4: determining whether the position of the obstruction is close to the center of the current road monitoring image and whether the area of the obstruction is greater than a preset value; if so, determining that the road video monitoring device is obstructed; otherwise, identifying the unknown stain in the current road monitoring image, and when the unknown stain in the current road monitoring image cannot be identified, executing step S5; Step S5: Obtain road monitoring images taken by the road video monitoring device at different time points. When the position of the unknown stain in the road monitoring images at the different time points remains fixed, it is determined that the road video monitoring device has lens contamination.
[0008] Furthermore, the specific targets include basic elements of road traffic, road facilities and off-road targets, and the specific areas include road areas, building areas, sky areas and farmland areas, wherein the basic elements of road traffic include traffic signs, markings, pedestrians and vehicles, the road facilities include overpasses, gantries, poles, signs and guardrails, and the off-road targets include houses, trees, people, animals and objects.
[0009] Furthermore, after acquiring the current road monitoring image captured by the road video monitoring device, the method further includes: Use the image calibration tool to calibrate the normal road monitoring images taken by each road video surveillance device to determine the reasonable proportion range of each specific area.
[0010] Furthermore, the step of obtaining the current road monitoring image captured by the road video monitoring device further includes: A timed task is set or a cyclic polling mechanism is adopted to obtain the road monitoring images captured by the road video monitoring device in turn.
[0011] Furthermore, the calculating of the proportion of each specific area in the current road monitoring image further includes: The proportion of each specific area is determined by using the ratio of the area of each specific area in the current road monitoring image to the total area of the current road monitoring image.
[0012] Furthermore, the step of identifying an obstruction from the current road monitoring image further includes: The obstruction is identified from the current road monitoring image using a target detection algorithm, wherein the obstruction includes a pole, a tree, a wire or a wire bridge.
[0013] Furthermore, the identifying of unknown stains in the current road monitoring image further includes: An image processing algorithm and a deep learning model are used to identify unknown stains in the current road monitoring image, wherein the unknown stains include water droplets, mist or dust.
[0014] Furthermore, the step S5 further includes: When the position of the unknown stain in the road monitoring image at different time points is fixed, it is determined that the unknown stain is stain on the lens or stain on the glass cover, so that it is determined that the road video monitoring device has lens stain or glass stain; then it is confirmed by the stain detection algorithm and a maintenance alarm message is generated; When the unknown stain disappears or its position changes significantly in the road monitoring images at different time points, the image change detection algorithm is used to determine that the unknown stain is a moving object in the background.
[0015] The road video surveillance equipment anomaly detection method based on video semantics provided by the present invention has the following advantages: it can automatically discover the abnormal state of the road video surveillance equipment during the operation and maintenance management of the road video surveillance equipment, thereby solving the problem of automated operation and maintenance management of the road video surveillance equipment, reducing the workload of manual operation and maintenance, improving the timeliness of maintenance, improving the proper rate of road video surveillance equipment, and effectively monitoring and recording road traffic conditions. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.
[0017] Figure 1 A flowchart of a road video surveillance equipment anomaly detection method based on video semantics provided by the present invention.
[0018] Figure 2 A flowchart of a specific implementation of a road video surveillance equipment anomaly detection method based on video semantics provided by the present invention. DETAILED DESCRIPTION
[0019] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0020] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation method, structure, characteristics and effects of a road video surveillance equipment anomaly detection method based on video semantics proposed by the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so as to describe the embodiments of the present invention described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0022] In this embodiment, a method for detecting abnormalities in road video surveillance equipment based on video semantics is provided. Figure 1 As shown, the road video surveillance equipment anomaly detection method based on video semantics includes: Step S1: Acquire the current road monitoring image captured by the road video monitoring equipment; It should be noted that by connecting the surveillance camera, ensure that its field of view covers the road area that needs to be monitored, use a high-resolution camera to obtain clear images, and access the real-time video stream to the monitoring system.
[0023] Preferably, Figure 2As shown, after obtaining the current road monitoring image captured by the road video monitoring device, it also includes: Use image calibration tools to calibrate normal road monitoring images captured by each road video monitoring device to determine the reasonable scale range of each specific area. Use image processing technology GrabCut algorithm to segment the area and record the calibration results.
[0024] It should be noted that by collecting and labeling relevant target samples, training video semantic target recognition algorithms for various targets based on deep learning, and identifying various targets and their positions, shapes, and sizes in surveillance videos, experience-based logical judgments can be used to detect anomalies in video surveillance equipment.
[0025] Preferably, Figure 2 As shown, the acquisition of the current road monitoring image captured by the road video monitoring device also includes: Set a scheduled task or adopt a cyclic polling mechanism to obtain the road monitoring images captured by the road video monitoring equipment in turn, ensuring that each video stream can be detected regularly to avoid monitoring blind spots.
[0026] Step S2: using a video semantic recognition algorithm to respectively identify all specific targets and all specific areas from the current road monitoring image; Specifically, a deep learning model (U-Net, DeepLab) is used to perform semantic segmentation on the current road monitoring image to identify all specific targets and all specific areas; wherein the specific targets include basic elements of road traffic, road facilities and off-road targets, and the specific areas include road areas, building areas, sky areas and farmland areas. wherein the basic elements of road traffic include traffic signs, markings, pedestrians, vehicles, etc., the road facilities include overpasses, gantries, poles, signs, guardrails, etc., and the off-road targets include houses, trees, people, animals, objects, etc.
[0027] Step S3: Calculate the proportion of each specific area in the current road monitoring image, and determine whether the proportion of each specific area is within a reasonable proportion range; if it is not within the reasonable proportion range, determine that the road video monitoring device has an abnormal monitoring angle; if it is within the reasonable proportion range, identify the obstruction from the current road monitoring image, and determine the position, shape and area of the obstruction, and then execute step S4; Preferably, the calculating the proportion of each specific area in the current road monitoring image further includes: The proportion of each specific area is determined by using the ratio of the area of each specific area in the current road monitoring image to the total area of the current road monitoring image.
[0028] Preferably, the step of identifying an obstruction from the current road monitoring image further includes: The object detection algorithm (YOLO, Faster R-CNN) is used to identify the occluders from the current road monitoring image, wherein the occluders include poles, trees, wires, overpasses, etc. The semantic segmentation technology is combined to determine the position, shape and area of these occluders.
[0029] It should be noted that for each specific video surveillance point, when the surveillance video is set normally, the approximate proportion facing the sky can be determined, and the normal sky proportion range can be manually calibrated. During video detection, the video semantic segmentation algorithm is used to calculate the proportion of the sky in the current image and compare it with the calibrated proportion to determine whether there is any abnormality in the surveillance angle.
[0030] Step S4: determining whether the position of the obstruction is close to the center of the current road monitoring image and whether the area of the obstruction is greater than a preset value; if so, determining that the road video monitoring device is obstructed by using a logic judgment algorithm (such as rule-based reasoning); otherwise, identifying the unknown stain in the current road monitoring image, and when the unknown stain in the current road monitoring image cannot be identified, executing step S5; Preferably, the step of identifying unknown stains in the current road monitoring image further includes: An image processing algorithm (such as morphological operation) and a deep learning model (such as ResNet) are used to identify unknown stains in the current road monitoring image, wherein the unknown stains include water droplets, fog or dust.
[0031] It should be noted that common obstructions to road video surveillance equipment include poles, trees, wires, overpasses, etc. Through target recognition and semantic segmentation, these targets can be identified and their positions, shapes, and sizes can be obtained. Combining this information, it is possible to quickly determine whether there is an obstruction (it cannot be certain that there is no obstruction). For example, if a video surveillance section with roadside trees identifies leaves and branches, and the distribution area is relatively large and the location is close to the center of the video, it can be determined as an obstruction based on experience.
[0032] Step S5: Obtain road monitoring images taken by the road video monitoring device at different time points. When the position of the unknown stain in the road monitoring images at the different time points remains fixed, it is determined that the road video monitoring device has lens contamination.
[0033] Preferably, the step S5 further includes: When the position of the unknown stain in the road monitoring image at different time points is fixed, it is determined that the unknown stain is stain on the lens or stain on the glass cover, so that it is determined that the road video monitoring device has lens stain or glass stain; then it is confirmed by a stain detection algorithm (such as background subtraction) and a maintenance alarm message is generated; When the unknown stain disappears or its position changes significantly in the road monitoring images at different time points, an image change detection algorithm (such as a frame difference method) is used to determine that the unknown stain is not a stain on the lens but a moving object in the background (such as clouds, vehicles, etc.).
[0034] It should be noted that some lens contamination, such as water droplets and fog, can be identified, but there are also some "stains" that cannot be determined and cannot be distinguished from the background in the video. For example, a gray stain is difficult to distinguish at first glance, whether it is dust, clouds, or other unrecognizable objects in the background. For this problem, you can compare whether there are highly consistent stains at the same location through video images captured at different times (including daytime and nighttime). Use image registration technology (such as feature point matching) to ensure image alignment and analyze the changes of stains at different time points. Highly consistent "stains" always appear at the same location to determine lens contamination. Because if it is a moving background object such as clouds, it will definitely disappear or change its position significantly in a large number of images at different times (including daytime and nighttime). Even relatively fixed background objects such as road debris will show significant differences between daytime and nighttime, especially when the lens moves. The only thing that can remain fixed is the stain on the lens or camera glass cover.
[0035] In the embodiment of the present invention, the image semantic target recognition and semantic segmentation technology based on deep learning is used to identify the basic elements of road traffic (traffic signs, markings, pedestrians and vehicles, etc.), road facilities (overpasses, gantries, poles, signs, guardrails, etc.), off-road targets (such as houses, trees, people, animals, objects, etc.), intersections and adjacent roads and other main targets and roads, sky, farmland and other main areas in the road monitoring video, and judge the abnormality of the video monitoring equipment according to the type, quantity, location, shape, size and other characteristics of the targets and areas. A series of combined algorithms and logical judgments are used to determine whether the signal lights in the video frame images are working abnormally.
[0036] The present invention provides a method for detecting abnormalities of road video surveillance equipment based on video semantics. The method comprises the following steps: (1) using a video semantic recognition algorithm to identify the main targets and main areas in the road surveillance video, and detecting abnormalities in the state of the road video surveillance equipment through logical judgment; (2) judging abnormalities in the monitoring angle such as facing the sky through the approximate proportion of a specific area in a specific scene; (3) judging abnormalities in the video occlusion through the position and area of common occlusions; and (4) judging lens contamination through the fixation of unknown "stains" at different times.
[0037] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the technical solution of the present invention.
Claims
1. A road video surveillance equipment anomaly detection method based on video semantics, characterized in that: The road video monitoring equipment anomaly detection method based on video semantics comprises the following steps: Step S1: Acquire the current road monitoring image captured by the road video monitoring equipment; Step S2: using a video semantic recognition algorithm to respectively identify all specific targets and all specific areas from the current road monitoring image; Step S3: Calculate the proportion of each specific area in the current road monitoring image, and determine whether the proportion of each specific area is within a reasonable proportion range; if it is not within the reasonable proportion range, determine that the road video monitoring device has an abnormal monitoring angle; if it is within the reasonable proportion range, identify the obstruction from the current road monitoring image, and determine the position, shape and area of the obstruction, and then execute step S4; Step S4: determining whether the position of the obstruction is close to the center of the current road monitoring image and whether the area of the obstruction is greater than a preset value; if so, determining that the road video monitoring device is obstructed; otherwise, identifying the unknown stain in the current road monitoring image, and when the unknown stain in the current road monitoring image cannot be identified, executing step S5; Step S5: Obtain road monitoring images taken by the road video monitoring device at different time points. When the position of the unknown stain in the road monitoring images at the different time points remains fixed, it is determined that the road video monitoring device has lens contamination.
2. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The specific targets include basic elements of road traffic, road facilities and off-road targets, and the specific areas include road areas, building areas, sky areas and farmland areas. The basic elements of road traffic include traffic signs, markings, pedestrians and vehicles, the road facilities include overpasses, gantries, poles, signs and guardrails, and the off-road targets include houses, trees, people, animals and objects.
3. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: After the current road monitoring image captured by the road video monitoring device is obtained, the method further includes: Use the image calibration tool to calibrate the normal road monitoring images taken by each road video surveillance device to determine the reasonable proportion range of each specific area.
4. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The step of obtaining the current road monitoring image captured by the road video monitoring device also includes: A timed task is set or a cyclic polling mechanism is adopted to obtain the road monitoring images captured by the road video monitoring device in turn.
5. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The step of calculating the proportion of each specific area in the current road monitoring image also includes: The proportion of each specific area is determined by using the ratio of the area of each specific area in the current road monitoring image to the total area of the current road monitoring image.
6. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The step of identifying an obstruction from the current road monitoring image further includes: The obstruction is identified from the current road monitoring image using a target detection algorithm, wherein the obstruction includes a pole, a tree, a wire or a wire bridge.
7. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The step of identifying unknown stains in the current road monitoring image further includes: An image processing algorithm and a deep learning model are used to identify unknown stains in the current road monitoring image, wherein the unknown stains include water droplets, mist or dust.
8. The method for detecting anomalies of road video surveillance equipment based on video semantics according to claim 1, characterized in that: The step S5 further includes: When the position of the unknown stain in the road monitoring image at different time points is fixed, it is determined that the unknown stain is stain on the lens or stain on the glass cover, so that it is determined that the road video monitoring device has lens stain or glass stain; then it is confirmed by the stain detection algorithm and a maintenance alarm message is generated; When the unknown stain disappears or its position changes significantly in the road monitoring images at different time points, the image change detection algorithm is used to determine that the unknown stain is a moving object in the background.
