Event Dynamic Inspection Method, Device and Electronic Equipment for Pan-Tilt-Zoom Cameras

Abstract

The present invention provides an event dynamic inspection method, device and electronic device for a pan-tilt camera. The method includes: when the pan-tilt camera is adjusted from the current preset position to the next preset position, acquiring video data of the monitoring area corresponding to the next preset position, where the image frames in the video data include a first object and a second object, the first object includes: a road area, lane lines and road traffic signs, and the second object is a moving object; determining the image segmentation result of the first object and the trajectory information of the second object; based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs, or based on the PTZ information of the pan-tilt camera at different times, determining whether the pan-tilt camera has shifted; in the case of no shift, determining whether a traffic anomaly event has occurred based on the trajectory information of the second object. The event dynamic inspection method for a pan-tilt camera provided by the present invention improves the detection accuracy and efficiency of traffic anomaly events.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent transportation, and particularly to an event dynamic inspection method, device and electronic device for a pan-tilt camera. Background Art

[0002] Most of the monitoring cameras used for traffic event detection on the basic sections of existing expressways are bullet cameras. The main difficulties in applying current traffic event detection algorithms to pan-tilt cameras such as dome cameras are mainly as follows: lack of automatic recognition of the effective road area; false alarms are likely to occur during the shifting process of the pan-tilt camera.

[0003] In terms of identifying the effective road area, existing solutions are divided into two categories: one category is that most existing event detection systems are based on setting detection areas and related auxiliary lines in advance in the monitoring screen of each camera, detecting and tracking vehicles and pedestrians, and judging abnormal events according to preset rules. This method depends on the accuracy of manual line-drawing rules. If the pan-tilt camera shifts or the manually set rules are inaccurate, a large number of false alarm events will occur, and the application in large-scale monitoring scenarios requires a large amount of manpower for maintenance, and the operation and maintenance workload is huge. The other category is to construct the road area through background modeling. This method depends on the modeling of multiple fixed frames and cannot distinguish the difference between the main concerned road and side branch roads.

[0004] In terms of the shifting detection of the pan-tilt camera, related technologies compare in the spatio-temporal domain the straight lines obtained from the lane line detection results to determine whether the pan-tilt camera has shifted. This method is only applicable to straight road sections and cannot be applied to the actual road scenarios including complex road areas such as curves, ramps, and intersections. Summary of the Invention

[0005] The present invention provides an event dynamic inspection method, device and electronic device for a pan-tilt camera to solve the problem that image changes during the shifting process of the pan-tilt camera in the prior art are likely to cause event false alarms.

[0006] The present invention provides an event dynamic inspection method for a pan-tilt camera, including:

[0007] When the pan-tilt camera is adjusted from the current preset position to the next preset position, acquiring video data of the monitoring area corresponding to the next preset position, the video data of the monitoring area is collected by the pan-tilt camera, and the image frames in the video data include a first object and a second object, the first object includes: a road area, lane lines and road traffic signs, and the second object is a moving object in the monitoring area;

[0008] Processing the image frames to determine the image segmentation result of the first object and the trajectory information of the second object;

[0009] Based on the image segmentation result of the lane line and the image segmentation result of the road traffic sign, determine whether the pan-tilt camera has shifted; or, based on the PTZ information of the pan-tilt camera at different times, determine whether the pan-tilt camera has shifted;

[0010] When the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane line;

[0011] When the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object.

[0012] In some embodiments, the processing of the image frame to determine the image segmentation result of the first object and the trajectory information of the second object includes:

[0013] Perform panoramic segmentation on the image frame through a multi-task convolutional neural network based on improved YOLOPv2 to obtain the image segmentation result of the first object;

[0014] The head network of the multi-task convolutional neural network based on improved YOLOPv2 includes a segmentation head corresponding to the road area, a segmentation head corresponding to the lane line, a segmentation head corresponding to the edge line of the carriageway, and a segmentation head corresponding to the edge area of the road.

[0015] In some embodiments, the processing of the image frame to determine the image segmentation result of the first object and the trajectory information of the second object includes:

[0016] Use an object detector to perform multi-object tracking on the second object in the video data to determine the trajectory information of the second object.

[0017] In some embodiments, the determining whether the pan-tilt camera has shifted based on the image segmentation result of the lane line and the image segmentation result of the road traffic sign includes:

[0018] Based on the image segmentation result of the lane line and the image segmentation result of the road traffic sign, determine a first distance and a second distance, where the first distance is the distance between the pixel point coordinates corresponding to the same lane line in two adjacent video frames, and the second distance is the distance between the pixel point coordinates corresponding to the same road traffic sign in two adjacent video frames;

[0019] Based on the first distance and the second distance, determine whether the pan-tilt camera has shifted.

[0020] In some embodiments, when the pan-tilt-zoom camera does not shift, based on the trajectory information of the second object in the effective road area, determining whether a traffic anomaly event occurs includes:

[0021] When the second object is a pedestrian and the trajectory information of the pedestrian first appears in the effective road area, determining that the traffic anomaly event is a pedestrian intrusion event;

[0022] When the second object is a non-motor vehicle and the trajectory information of the non-motor vehicle first appears in the effective road area, determining that the traffic anomaly event is a non-motor vehicle intrusion event;

[0023] When the second object is a motor vehicle, and the time that the trajectory information of the motor vehicle in the effective road area remains stationary exceeds a time threshold, and the number of parked vehicles in the effective road area does not exceed a vehicle threshold, determining that the traffic anomaly event is a parking event;

[0024] When the second object is a motor vehicle, and the time that the trajectory information of the motor vehicle in the effective road area remains stationary exceeds a time threshold, and the number of parked vehicles in the effective road area exceeds the vehicle threshold, determining that the traffic anomaly event is a congestion event.

[0025] In some embodiments, determining whether the pan-tilt-zoom camera shifts based on the PTZ information of the pan-tilt-zoom camera at different times includes:

[0026] Obtaining the PTZ information of the pan-tilt-zoom camera at the current moment and the PTZ information at the previous moment;

[0027] Based on the PTZ information at the current moment and the PTZ information at the previous moment, determining whether the pan-tilt-zoom camera shifts.

[0028] The present invention further provides an event dynamic inspection device for a pan-tilt-zoom camera, including:

[0029] An acquisition module, configured to obtain video data of a monitoring area corresponding to the next preset position when the pan-tilt-zoom camera is adjusted from the current preset position to the next preset position, the video data of the monitoring area is collected by the pan-tilt-zoom camera, and the image frames in the video data include a first object and a second object, the first object includes: a road area, lane lines, and road traffic signs, and the second object is a moving object in the monitoring area;

[0030] A first determination module, configured to process the image frame to determine an image segmentation result of the first object and trajectory information of the second object;

[0031] A first judgment module, configured to judge whether the pan-tilt camera has shifted based on the image segmentation result of the lane line and the image segmentation result of the road traffic sign; or, judge whether the pan-tilt camera has shifted based on the PTZ information of the pan-tilt camera at different times;

[0032] A second judgment module, configured to judge whether a traffic anomaly event occurs based on the trajectory information of the second object in the effective road area when the pan-tilt camera has not shifted; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane line;

[0033] A second determination module, configured to re-determine the image segmentation result of the first object and the trajectory information of the second object when the pan-tilt camera has shifted.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the event dynamic inspection method for a pan-tilt camera as described in any one of the above is implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the event dynamic inspection method for a pan-tilt camera as described in any one of the above is implemented.

[0036] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the event dynamic inspection method for a pan-tilt camera as described in any one of the above is implemented.

[0037] The event dynamic inspection method, device, and electronic device for a pan-tilt camera provided by the present invention process the video data of the monitoring area collected by the pan-tilt camera to determine the image segmentation result of the first object and the trajectory information of the second object. Based on the image segmentation result of the first object and the trajectory information of the second object, or based on the PTZ information of the pan-tilt camera at different times, it is judged whether the pan-tilt camera has shifted. It can also automatically extract the effective road area in the monitoring area, without relying on manual setting of rules every time the viewing angle is switched to complete event detection, solving the limitation that the current event detection system cannot be applied to pan-tilt cameras. At the same time, the image segmentation result of the road area can avoid the interference of targets in the surrounding irrelevant areas and false alarms, improving the detection accuracy and efficiency of traffic anomaly events. Description of the Drawings

[0038] To more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic flowchart of the event dynamic patrol inspection method for a pan-tilt camera provided by the present invention;

[0040] Figure 2 It is a schematic structural diagram of the event dynamic patrol inspection system for a pan-tilt camera provided by the present invention;

[0041] Figure 3 It is a schematic structural diagram of the event dynamic patrol inspection device for a pan-tilt camera provided by the present invention;

[0042] Figure 4 It is a schematic structural diagram of the electronic device provided by the present invention. Specific embodiments

[0043] To make the objectives, technical solutions and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0044] It can be understood that the variable field of view and variable focal length of a pan-tilt camera can expand the monitoring range, but the following difficulties are brought to traffic event detection based on a pan-tilt camera: (1) Event detection depends on manually setting a rule area, and the rules set manually become invalid after the pan-tilt camera is displaced; (2) Image changes during the dynamic displacement of the pan-tilt camera are likely to cause false alarms of events.

[0045] The following will be combined with Figures 1-4 Describe the event dynamic patrol inspection method, device and electronic device for a pan-tilt camera of the present invention.

[0046] Figure 1 It is a schematic flowchart of the event dynamic patrol inspection method for a pan-tilt camera provided by the present invention.

[0047] The execution subject of the event dynamic inspection method for pan-tilt cameras provided by the present invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a handheld computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a Network Attached Storage (NAS), a personal computer (PC), a television (TV), a teller machine, or a self-service machine, etc. The present invention does not make specific limitations.

[0048] Taking a computer executing the event dynamic inspection method for pan-tilt cameras provided by the present invention as an example, the technical solution of the present invention will be described in detail below.

[0049] Refer to Figure 1 , the event dynamic inspection method for pan-tilt cameras provided by the present invention may include the following steps:

[0050] Step 110: When the pan-tilt camera is adjusted from the current preset position to the next preset position, obtain the video data of the monitoring area corresponding to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area;

[0051] Step 120: Process the image frames to determine the image segmentation result of the first object and the trajectory information of the second object;

[0052] Step 130: Based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs, determine whether the pan-tilt camera has shifted; or, based on the PTZ information of the pan-tilt camera at different times, determine whether the pan-tilt camera has shifted;

[0053] Step 140: When the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines;

[0054] Step 150: When the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object.

[0055] It should be noted that a pan-tilt camera refers to a camera with a pan-tilt. The pan-tilt can drive mechanical devices to control the posture of the camera, expanding the monitoring angle and range of the camera. The main function of the pan-tilt camera is to receive control signals, drive the camera to rotate horizontally or vertically, and drive the camera lens to perform actions such as zooming in, zooming out, and opening and closing the aperture.

[0056] In actual execution, it is necessary to initialize the pan-tilt camera first.

[0057] Optionally, load the basic information, preset positions, and patrol schedule of the pan-tilt camera, and initialize the preset algorithm models, including but not limited to algorithm models such as road area segmentation models, lane line detection models, object detection models, multi-object tracking models, and object re-identification models. By setting preset positions, the camera can be scheduled to perform polling according to a time plan, and event detection can be performed at each preset position, which can expand the monitoring perspective of the pan-tilt camera and cover a more comprehensive detection range. It should be noted that the preset position is a way to associate the key areas to be monitored with the operating conditions of the PTZ camera.

[0058] In step 110, the monitoring area of the pan-tilt camera is the target highway, and dynamic patrol of the highway is realized through the pan-tilt camera. In actual execution, by searching for the next preset position that matches the current preset position in the patrol schedule, the pan-tilt camera is controlled to adjust to the next preset position. When the adjustment is completed, subsequent video streaming and analysis are performed. Among them, any section of the target highway is not specifically limited in the present invention.

[0059] Obtain the video data corresponding to the next preset position through the Real-Time Streaming Protocol (RTSP) and start video analysis. The video data is generated after the pan-tilt camera performs video acquisition on the target highway. It can be understood that the video data of the monitoring area includes continuous image frames, and the image frames may include a first object and a second object. The first object includes but is not limited to: road area, lane lines, and road traffic signs. The second object includes but is not limited to: moving objects in the monitoring area such as motor vehicles, non-motor vehicles, pedestrians, animals, and common obstacles.

[0060] It can be understood that the above-mentioned moving objects can all be used to detect traffic abnormal events.

[0061] In step 120, the video data is processed. Optionally, the video processing specifically includes: performing image segmentation on the road area and road traffic signs in the monitoring area and performing lane line detection, and performing object detection and tracking on the second object in each image frame. Thereby, the image segmentation result of the first object and the trajectory information of the second object can be determined.

[0062] The image segmentation results of the road area are used to automatically extract the effective road area to avoid interference from targets in irrelevant areas. The effective road area is the effective area corresponding to the traffic event.

[0063] In step 130, based on the image segmentation results of the lane lines and the image segmentation results of the road traffic signs, it is possible to detect whether the pan-tilt camera moves or rotates, that is, to determine whether the pan-tilt camera is displaced.

[0064] In step 140, when the PTZ camera is not displaced, the trajectory information of the second object in the effective area of ​​the road is judged to detect abnormal traffic events such as pedestrian intrusion, non-motor vehicle intrusion, parking, reverse driving, congestion, etc., and save the event alarm picture and alarm message. In addition, in this step, a method for judging whether the PTZ camera is displaced is also provided.

[0065] The PTZ information of the PTZ camera is obtained through the manufacturer's private protocol (Software Development Kit, SDK) protocol, ONVIF protocol or other private protocols. Based on the PTZ information, the current position and posture coordinates of the PTZ camera can be determined, and then the PTZ camera can be compared to see if it has shifted.

[0066] It should be noted that PTZ is the abbreviation of Pan / Tilt / Zoom, which represents the all-round (left / right / up / down) movement of the pan / tilt and the zoom control of the lens.

[0067] The ONVIF protocol is used for communication between network cameras and servers, and mainly provides a standardized network open interface for network video products.

[0068] The ONVIF protocol adopts the browser / server (B / S) architecture mode of Web Services Description Language (WSDL) and Extensible Markup Language (XML), encapsulates the control message as an HTTP request (request message from the client to the server), and controls the ONVIF video service flow through the Real-time Transport Protocol (RTP) / Real-time Streaming Protocol (RTSP) protocol.

[0069] In step 150, it is determined that the position of the current PTZ camera has changed, ie, has shifted, and that road segmentation and lane line detection need to be performed again.

[0070] After step 150, when the time reaches the next moment in the patrol inspection schedule, the dynamic patrol inspection module continues to switch to the next preset position, and continues to perform video data analysis and traffic event detection.

[0071] The event dynamic patrol inspection method for pan-tilt cameras provided by the present invention processes the video data of the monitored area collected by the pan-tilt camera to determine the image segmentation result of the first object and the trajectory information of the second object. Based on the image segmentation result of the first object and the trajectory information of the second object, or based on the PTZ information of the pan-tilt camera at different times, it is judged whether the pan-tilt camera has shifted. It can also automatically extract the effective road area in the monitored area, without relying on manual rule setting every time after switching the perspective to complete event detection, solving the limitation that the current event detection system cannot be applied to pan-tilt cameras. At the same time, the image segmentation result of the road area can avoid the interference of targets in the surrounding irrelevant areas and false alarms, improving the detection accuracy and efficiency of traffic abnormal events.

[0072] In some embodiments, step 120 may include:

[0073] Performing panoramic segmentation on the image frame through a multi-task convolutional neural network based on improved YOLOPv2 to obtain the image segmentation result of the first object;

[0074] The head network of the multi-task convolutional neural network based on improved YOLOPv2 includes a segmentation head corresponding to the road area, a segmentation head corresponding to the lane line, a segmentation head corresponding to the edge line of the carriageway, and a segmentation head corresponding to the edge area of the road.

[0075] In actual execution, each image frame in the video data can be input into the multi-task convolutional neural network based on improved YOLOPv2 for panoramic segmentation, that is, panoramic segmentation of the road area, lane line, and road traffic signs in the video picture, so as to obtain the image segmentation result of the first object output by the multi-task convolutional neural network.

[0076] Optionally, verify the image segmentation results of the lane line and the road area, and determine whether the extracted effective road area and the lane line detection result conform to the prior of the actual road structure according to the road prior rules.

[0077] It should be noted that panoramic segmentation includes semantic segmentation and instance segmentation, which distinguish individual object instances; the instances do not overlap. Panoramic segmentation assigns a semantic label and an instance number to each pixel point in the image.

[0078] This model is based on the YOLOPv2 multi-task network, and uses the prior knowledge of road lane lines and road areas to train the model to output prediction results that meet the structural characteristics of lane lines and road areas, so as to realize the mutual enhancement of the lane line segmentation and road area segmentation results.

[0079] Specifically, two auxiliary supervision tasks are designed in the training stage.

[0080] On the basis of the segmentation head corresponding to the road area and the segmentation head corresponding to the lane line, two segmentation heads are added, namely, the segmentation head corresponding to the edge line of the carriageway and the segmentation head corresponding to the edge area of the road.

[0081] The segmentation head corresponding to the edge area of the road is used to detect the edge of the road, and the segmentation head corresponding to the edge line of the carriageway is used to detect the lane line on the roadside. The lane line on the roadside is used to indicate the edge of the motor vehicle lane or to divide the boundary between motor vehicles and non-motor vehicles. Among them, the edge line of the carriageway is a subset of the lane line and is close to the road edge.

[0082] Through the above segmentation heads, the context clues from the road edge and the edge line of the carriageway can be used to predict the road area and the lane line, aiming to constrain the lane line within the road area. In addition to the peer information transmission, the reverse information flow is added, and the higher-level semantic features that have been decoded in the road area branch are input into the bottom layer of the lane line branch to realize the decoding of the lane line by combining the road information.

[0083] The event dynamic patrol inspection method for a pan-tilt camera provided by the present invention realizes the automatic recognition of the effective road area and limits the effective range of traffic anomaly event detection through the road area segmentation and lane line detection algorithms based on a multi-task convolutional network.

[0084] In some embodiments, processing the image frame to determine the image segmentation result of the first object and the trajectory information of the second object includes:

[0085] Using a target detector to perform multi-object tracking on the second object in the video data to determine the trajectory information of the second object.

[0086] In actual execution, a target detector is used to obtain pedestrians and vehicles in the video frame, perform multi-object tracking on the pedestrians and vehicles, and record and update the trajectory information of each pedestrian and vehicle.

[0087] The event dynamic patrol inspection method for a pan-tilt camera provided by the present invention can obtain the trajectory information of pedestrians and vehicles in the effective road area by performing multi-object tracking on the pedestrians and vehicles, providing important data support for judging whether the pan-tilt camera has shifted.

[0088] In some embodiments, step 130 may include:

[0089] Based on the image segmentation results of lane lines and road traffic signs, determine a first distance and a second distance. The first distance is the distance between the pixel coordinates corresponding to the same lane line in two adjacent video frames, and the second distance is the distance between the pixel coordinates corresponding to the same road traffic sign in two adjacent video frames;

[0090] Based on the first distance and the second distance, determine whether the pan-tilt camera has shifted.

[0091] In actual implementation, when performing shift detection on the pan-tilt camera, it can be determined by comparing key targets in the current video frame and historical video frames.

[0092] By measuring the distances between the pixel coordinates of key targets such as lane lines and road traffic signs, for example: Intersection over Union (IoU) and Mean Intersection over Union (MIoU), and comparing with a judgment threshold to determine whether the pan-tilt camera has moved or rotated.

[0093] It can be understood that based on the image segmentation results of lane lines and road traffic signs, the pixel coordinates of the feature points corresponding to the lane lines and road traffic signs in the image frame can be determined, the first distance and the second distance can be determined, and further, the comparison of the feature point distributions can be realized.

[0094] IoU can calculate the ratio of the intersection and union of the predicted lane line region in the current video frame and the predicted lane line region in the historical video frame.

[0095] It can also calculate the ratio of the intersection and union of the predicted road traffic sign in the current video frame and the predicted road traffic sign in the historical video frame. The intersection over union is expressed by the following formula:

[0096]

[0097] where, is the value of pixel i in the mask predicted at time t1, is the value of pixel i in the mask predicted at time t2.

[0098] mIoU can target the differences between regions of different categories. It measures the overlapping degree of the predicted segmentation regions in different video frames at two moments. In addition, the Mean Absolute Error (MAE) or Mean Squared Error (MSE) can be used to quantify the distance error between the predicted positions of lane lines and traffic signs in the current video frame and the historical video frame. This method is applicable to both offline video analysis and online real-time video analysis.

[0099] The event dynamic inspection method for a pan-tilt camera provided by the present invention can automatically determine whether the pan-tilt camera has undergone displacement rotation by comparing the detected lane lines with road traffic signs, adjust the algorithm to dynamically adapt to regional changes, and avoid false alarms of events caused by image changes during the rotation of the pan-tilt.

[0100] In some embodiments, after step 110, the method further includes:

[0101] Obtain the PTZ information of the pan-tilt camera at the current moment and the PTZ information at the previous moment;

[0102] Based on the PTZ information at the current moment and the PTZ information at the previous moment, determine whether the pan-tilt camera has shifted.

[0103] In actual implementation, for the displacement detection of the pan-tilt camera, the PTZ information of the pan-tilt camera can also be obtained through the SDK protocol, ONVIF protocol or other private protocols. If there is a change compared with the PTZ information saved at the previous moment, it is determined that the current position of the pan-tilt camera has changed, and road area segmentation and lane line detection need to be re-performed. This method is applicable to real-time video analysis under cameras that support ONVIF PTZ-related protocols.

[0104] In some embodiments, step 140 may include:

[0105] In the case where the second object is a pedestrian and the trajectory information of the pedestrian first appears in the effective road area, determine that the traffic abnormal event is a pedestrian intrusion event;

[0106] In the case where the second object is a non-motor vehicle and the trajectory information of the non-motor vehicle first appears in the effective road area, determine that the traffic abnormal event is a non-motor vehicle intrusion event;

[0107] In the case where the second object is a motor vehicle, the trajectory information of the motor vehicle in the effective road area remains stationary for a time exceeding the time threshold, and the number of parked vehicles in the effective road area does not exceed the vehicle threshold, determine that the traffic abnormal event is a parking event;

[0108] In the case where the second object is a motor vehicle, the trajectory information of the motor vehicle in the effective road area remains stationary for a time exceeding the time threshold, and the number of parked vehicles in the effective road area exceeds the vehicle threshold, determine that the traffic abnormal event is a congestion event.

[0109] In actual implementation, the second object may be a motor vehicle, a non-motor vehicle, a pedestrian, an animal, a common obstacle, etc.

[0110] When the trajectory information of a pedestrian first appears in the effective area of the road, that is, when the pedestrian first appears in the area divided by the road, it is determined as a pedestrian intrusion event.

[0111] If the trajectory information of the vehicle remains stationary for a time exceeding the time threshold T, and the number of vehicles stopped in the current section does not exceed the vehicle threshold N, it is determined as a parking event. Among them, the time threshold T and the vehicle threshold N in the present invention can be set according to actual needs, and no specific limitations are made here.

[0112] It can be understood that traffic abnormal events include, but are not limited to, pedestrian intrusion events and parking events. For example, it can also include: non-motor vehicle intrusion events, reverse driving events or congestion events, etc. Different event detection conditions can be set according to actual situations, and no specific limitations are made in the present invention.

[0113] For example: When the trajectory information of the motor vehicle in the effective area of the road remains stationary for a time exceeding the time threshold T, and the number of parked vehicles in the effective area of the road exceeds the vehicle threshold N, the traffic abnormal event is determined as a congestion event.

[0114] When the trajectory information of a non-motor vehicle first appears in the effective area of the road, the traffic abnormal event is determined as a non-motor vehicle intrusion event.

[0115] The event dynamic inspection method for a pan-tilt camera provided by the present invention realizes the automatic recognition of the effective area of the road through road area segmentation and lane line detection, as the scope of traffic event detection; by comparing the masks of the lane lines and road traffic signs detected in the current frame with the mask results of the historical frame, it automatically detects whether the pan-tilt camera has shifted.

[0116] In actual implementation, the event dynamic inspection method for a pan-tilt camera provided by the present invention can be applied to an event dynamic inspection system for a pan-tilt camera.

[0117] Figure 2 It is a schematic structural diagram of an event dynamic inspection system for a pan-tilt camera provided by the present invention.

[0118] Such as Figure 2 shown, the system includes:

[0119] An initialization module 210, which is used to load the basic information of the pan-tilt camera, preset positions and inspection plans, and initialize the algorithm model. By setting the preset positions, the camera is scheduled to perform polling according to the time plan, and event detection is carried out at each preset position, which can expand the monitoring angle of the camera and cover a more comprehensive detection range.

[0120] The dynamic inspection module 220 is used to control the camera pan / tilt to adjust to the next preset position by searching the next preset position matching the current preset position information in the inspection plan table. When the adjustment is completed, subsequent video stream acquisition and analysis are performed.

[0121] The video stream acquisition module 230 is used to obtain the camera video stream of the preset position through the real-time streaming protocol and start video analysis.

[0122] The road segmentation and lane detection module 240 is used to perform panoramic segmentation of the road area, lane lines, and road traffic signs in the video screen through a multi-task convolutional neural network based on the improved YOLOPv2. Specifically, the lane line and road segmentation results are verified, and it is determined whether the extracted road valid area and lane line detection results conform to the actual road structure prior according to the road prior rules.

[0123] The target detection and tracking module 250 is used to use a target detector to obtain human and vehicle targets in the video frame, perform multi-target tracking on the human and vehicle targets, and record and update the trajectory of each human and vehicle target.

[0124] The displacement detection module 260 is used to determine whether target detection and tracking need to be performed again by detecting whether the camera moves or rotates.

[0125] The event analysis module 270 is used to detect abnormal traffic events such as pedestrian intrusion, non-motor vehicle intrusion, parking, wrong-way driving, congestion, etc. by judging the trajectory information of the target object in the effective area of ​​the road, and save the event alarm picture and alarm message.

[0126] In some embodiments, when the time reaches the next moment in the inspection schedule, the dynamic inspection module 220 switches to the next preset position to continue road segmentation and traffic event detection.

[0127] The event dynamic inspection system for PTZ cameras provided by the present invention realizes automatic identification of effective road areas through road area segmentation and lane line detection, which serves as the scope of traffic event detection; and automatically detects whether the PTZ camera is displaced by comparing the distribution of feature points of detected lane lines and road signs.

[0128] The event dynamic inspection device for a pan-tilt camera provided by the present invention is described below. The event dynamic inspection device for a pan-tilt camera described below and the event dynamic inspection method for a pan-tilt camera described above can be referenced to each other.

[0129] Figure 3 Schematic diagram of the structure of the event dynamic inspection device for the PTZ camera provided by the present invention. Figure 3 The event dynamic inspection device for a PTZ camera provided by the present invention comprises:

[0130] An acquisition module 310, configured to obtain video data of a monitoring area corresponding to the next preset position when the pan-tilt camera adjusts from the current preset position to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera, and the image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs, and the second object is a moving object in the monitoring area;

[0131] A first determination module 320, configured to process the image frame to determine an image segmentation result of the first object and trajectory information of the second object;

[0132] A first judgment module 330, configured to judge whether the pan-tilt camera has shifted based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs; or, judge whether the pan-tilt camera has shifted based on the PTZ information of the pan-tilt camera at different times;

[0133] A second judgment module 340, configured to judge whether a traffic anomaly event occurs based on the trajectory information of the second object in the effective road area when the pan-tilt camera has not shifted; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines;

[0134] A second determination module 350, configured to re-determine the image segmentation result of the first object and the trajectory information of the second object when the pan-tilt camera has shifted.

[0135] The event dynamic patrol inspection device for a pan-tilt camera provided by the present invention processes the video data of the monitoring area collected by the pan-tilt camera to determine the image segmentation result of the first object and the trajectory information of the second object. Based on the image segmentation result of the first object and the trajectory information of the second object, or based on the PTZ information of the pan-tilt camera at different times, it judges whether the pan-tilt camera has shifted, and can also automatically extract the effective road area in the monitoring area, without relying on manual rule setting every time the viewing angle is switched to complete event detection, solving the limitation that the current event detection system cannot be applied to pan-tilt cameras. At the same time, the image segmentation result of the road area can avoid the interference of targets in surrounding irrelevant areas and false alarms generated, improving the detection accuracy and efficiency of traffic anomaly events.

[0136] In some embodiments, the first determination module 320 is specifically configured to:

[0137] Perform panoramic segmentation on the image frame through a multi-task convolutional neural network based on improved YOLOPv2 to obtain the image segmentation result of the first object;

[0138] The head network of the multi-task convolutional neural network based on the improved YOLOPv2 includes a segmentation head corresponding to the road area, a segmentation head corresponding to the lane line, a segmentation head corresponding to the road edge line of the carriageway, and a segmentation head corresponding to the road edge area.

[0139] In some embodiments, the first determination module 320 is specifically configured to:

[0140] Use an object detector to perform multi-object tracking on the second object in the video data, and determine the trajectory information of the second object.

[0141] In some embodiments, the first judgment module 330 is specifically configured to:

[0142] Based on the image segmentation results of the lane line and the image segmentation results of the road traffic signs, determine a first distance and a second distance, where the first distance is the distance between the pixel coordinates corresponding to the same lane line in two adjacent video frames, and the second distance is the distance between the pixel coordinates corresponding to the same road traffic sign in two adjacent video frames;

[0143] Based on the first distance and the second distance, determine whether the pan-tilt camera has shifted.

[0144] In some embodiments, the second judgment module 340 is specifically configured to:

[0145] In the case where the second object is a pedestrian and the trajectory information of the pedestrian first appears in the effective area of the road, determine that the traffic abnormal event is a pedestrian intrusion event;

[0146] In the case where the second object is a non-motor vehicle and the trajectory information of the non-motor vehicle first appears in the effective area of the road, determine that the traffic abnormal event is a non-motor vehicle intrusion event;

[0147] In the case where the second object is a motor vehicle, and the time for the trajectory information of the motor vehicle in the effective area of the road to remain stationary exceeds a time threshold, and the number of parked vehicles in the effective area of the road does not exceed a vehicle threshold, determine that the traffic abnormal event is a parking event;

[0148] In the case where the second object is a motor vehicle, and the time for the trajectory information of the motor vehicle in the effective area of the road to remain stationary exceeds a time threshold, and the number of parked vehicles in the effective area of the road exceeds the vehicle threshold, determine that the traffic abnormal event is a congestion event.

[0149] In some embodiments, the first judgment module 330 is further configured to:

[0150] After obtaining the video data of the monitoring area corresponding to the next preset position, obtain the PTZ information of the pan-tilt camera at the current moment and the PTZ information at the previous moment;

[0151] Based on the PTZ information at the current moment and the PTZ information at the previous moment, determine whether the pan-tilt camera has shifted.

[0152] Figure 4 An example of the physical structure diagram of an electronic device is shown as Figure 4 shown. The electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call the logical instructions in the memory 430 to execute the event dynamic inspection method for the pan-tilt camera. The method includes:

[0153] In the case where the pan-tilt camera is adjusted from the current preset position to the next preset position, obtain the video data of the monitoring area corresponding to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area;

[0154] Process the image frame to determine the image segmentation result of the first object and the trajectory information of the second object;

[0155] Based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs, determine whether the pan-tilt camera has shifted; or, based on the PTZ information of the pan-tilt camera at different moments, determine whether the pan-tilt camera has shifted;

[0156] In the case where the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines;

[0157] In the case where the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object.

[0158] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0159] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the event dynamic patrol inspection method for a pan-tilt camera provided by the above-mentioned various methods. The method includes:

[0160] In the case where the pan-tilt camera is adjusted from the current preset position to the next preset position, obtain the video data of the monitoring area corresponding to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area;

[0161] Process the image frame to determine the image segmentation result of the first object and the trajectory information of the second object;

[0162] Based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs, determine whether the pan-tilt camera has shifted; or, based on the PTZ information of the pan-tilt camera at different times, determine whether the pan-tilt camera has shifted;

[0163] In the case where the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines;

[0164] In the case where the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements an event dynamic inspection method for a pan-tilt camera, and the method includes:

[0166] When the pan-tilt camera is adjusted from the current preset position to the next preset position, obtain video data of the monitoring area corresponding to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area;

[0167] Process the image frame to determine the image segmentation result of the first object and the trajectory information of the second object;

[0168] Based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs, determine whether the pan-tilt camera has shifted; or, based on the PTZ information of the pan-tilt camera at different times, determine whether the pan-tilt camera has shifted;

[0169] When the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines;

[0170] When the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0172] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An event dynamic inspection method for a pan-tilt camera, characterized in that Including: When the pan-tilt camera is adjusted from the current preset position to the next preset position, obtain the video data of the monitoring area corresponding to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area; Process the image frames to determine the image segmentation result of the first object and the trajectory information of the second object. The determination method of the image segmentation result of the first object is: perform panoramic segmentation on the image frames through a multi-task convolutional neural network based on improved YOLOPv2 to obtain the image segmentation result of the first object. The head network of the multi-task convolutional neural network based on improved YOLOPv2 includes a segmentation head corresponding to the road area, a segmentation head corresponding to the lane lines, a segmentation head corresponding to the edge lines of the carriageway, and a segmentation head corresponding to the road edge area. Based on the image segmentation results of the lane lines and the road traffic signs, determine whether the pan-tilt camera has shifted. Among them, calculate the ratio of the intersection and union of the predicted lane line area in the current video frame and the predicted lane line area in the historical video frame, or calculate the ratio of the intersection and union of the predicted road traffic signs in the current video frame and the predicted road traffic signs in the historical video frame, and compare it with the judgment threshold to determine whether the pan-tilt camera has moved or rotated; When the pan-tilt camera has not shifted, based on the trajectory information of the second object in the effective road area, determine whether a traffic anomaly event has occurred. The effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines; When the pan-tilt camera has shifted, re-determine the image segmentation result of the first object and the trajectory information of the second object; The processing of the image frames to determine the image segmentation result of the first object and the trajectory information of the second object includes: Use an object detector to perform multi-object tracking on the second object in the video data to determine the trajectory information of the second object.

2. The event dynamic inspection method for a pan-tilt camera according to claim 1, wherein, The determination of whether the pan-tilt camera has shifted based on the image segmentation results of the lane lines and the road traffic signs includes: Based on the image segmentation results of the lane lines and the road traffic signs, determine a first distance and a second distance. The first distance is the distance between the pixel point coordinates corresponding to the same lane line in two adjacent video frames. The second distance is the distance between the pixel point coordinates corresponding to the same road traffic sign in two adjacent video frames; Based on the first distance and the second distance, determine whether the pan-tilt camera has shifted.

3. The event dynamic inspection method for a pan-tilt camera according to claim 1, wherein The determination of whether a traffic anomaly event has occurred based on the trajectory information of the second object in the effective road area when the pan-tilt camera has not shifted includes: When the second object is a pedestrian and the trajectory information of the pedestrian first appears in the effective road area, determine that the traffic abnormal event is a pedestrian intrusion event; When the second object is a non-motor vehicle and the trajectory information of the non-motor vehicle first appears in the effective road area, determine that the traffic abnormal event is a non-motor vehicle intrusion event; When the second object is a motor vehicle, the stationary time of the trajectory information of the motor vehicle in the effective road area exceeds a time threshold, and the number of parked vehicles in the effective road area does not exceed a vehicle threshold, determine that the traffic abnormal event is a parking event; When the second object is a motor vehicle, the stationary time of the trajectory information of the motor vehicle in the effective road area exceeds a time threshold, and the number of parked vehicles in the effective road area exceeds the vehicle threshold, determine that the traffic abnormal event is a congestion event.

4. An event dynamic inspection device for a pan-tilt camera, characterized in that, Including: An acquisition module, configured to obtain video data of a monitoring area corresponding to the next preset position when the pan-tilt camera is adjusted from the current preset position to the next preset position. The video data of the monitoring area is collected by the pan-tilt camera. The image frames in the video data include a first object and a second object. The first object includes: a road area, lane lines, and road traffic signs. The second object is a moving object in the monitoring area; A first determination module, configured to process the image frames to determine an image segmentation result of the first object and trajectory information of the second object; the method for determining the image segmentation result of the first object is: performing panoramic segmentation on the image frames through a multi-task convolutional neural network based on improved YOLOPv2 to obtain the image segmentation result of the first object; the head network of the multi-task convolutional neural network based on improved YOLOPv2 includes a segmentation head corresponding to the road area, a segmentation head corresponding to the lane lines, a segmentation head corresponding to the edge lines of the carriageway, and a segmentation head corresponding to the road edge area; processing the image frames to determine the image segmentation result of the first object and the trajectory information of the second object includes: using an object detector to perform multi-object tracking on the second object in the video data to determine the trajectory information of the second object; A first judgment module, configured to judge whether the pan-tilt camera has shifted based on the image segmentation result of the lane lines and the image segmentation result of the road traffic signs; wherein, calculate the ratio of the intersection and union of the predicted lane line area in the current video frame and the predicted lane line area in the historical video frame, or calculate the ratio of the intersection and union of the predicted road traffic signs in the current video frame and the predicted road traffic signs in the historical video frame, and compare it with a judgment threshold to judge whether the pan-tilt camera has moved or rotated; A second judgment module, configured to judge whether a traffic abnormal event occurs based on the trajectory information of the second object in the effective road area when the pan-tilt camera has not shifted; the effective road area is determined based on the image segmentation result of the road area and the image segmentation result of the lane lines; A second determination module, configured to re-determine the image segmentation result of the first object and the trajectory information of the second object when the pan-tilt camera is displaced.

5. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the event dynamic inspection method for the pan-tilt camera according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the event dynamic inspection method for the pan-tilt camera according to any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the event dynamic inspection method for the pan-tilt camera according to any one of claims 1 to 3.

Images