AI-powered intelligent patrol and early warning method and system for highway incidents

Abstract

This invention relates to the fields of computer vision and intelligent video surveillance technology, specifically to a method and system for intelligent patrol and early warning of highway events using AI. The method includes: S1: acquiring real-time video streams of the highway using a PTZ camera; S2: performing PTZ motion stability detection based on the video stream to obtain a stability score; S3: when the PTZ stability score is less than a stability threshold, performing road structure recognition on the video frames acquired by the current PTZ camera to obtain a road structure segmentation mask; S4: updating the virtual detection area consistent with the current PTZ camera's viewpoint based on the segmentation mask. This invention solves the technical problem in the prior art of how to automatically and accurately update the virtual detection area of ​​any unknown new scene when the PTZ camera rotates to it without prior physical location information.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent video surveillance technology, and more specifically, to a method and system for intelligent patrol and early warning of AI-driven events on highways. Background Technology

[0002] In large-scale monitoring scenarios such as highways and urban arterial roads, using PTZ cameras for video patrol is an effective technical means to expand monitoring coverage and improve equipment utilization. However, this method has the following technical problems: when the PTZ rotates to switch perspectives, the spatial correspondence between the virtual monitoring area pre-calibrated in the image pixel coordinate system and the actual road scene becomes invalid and mismatched, causing intelligent monitoring functions such as illegal parking and traffic flow statistics to fail to function properly. To solve this technical problem, existing technologies mainly offer the following two solutions.

[0003] One approach involves manually calibrating the virtual detection area after each pan-tilt-zoom (PTZ) camera viewpoint change. This method requires manual observation and calibration of the virtual monitoring area every time the PTZ rotates to a new position. This is tedious, time-consuming, and inefficient, and cannot adapt to scenarios requiring frequent rotation or dynamic adjustment of monitoring points. Another approach is an adaptive scheme based on prior models and geometric calculations. Chinese patent document CN105957340A discloses a traffic monitoring method based on adaptive virtual coils. This method pre-determines the actual location of the detection area of ​​interest in the physical world and uses it as a baseline monitoring position. When the PTZ rotates, the system uses the prior information of the baseline monitoring position, the PTZ rotation angle, and road line features extracted through image processing algorithms to perform geometric modeling and calculations, thereby updating and setting the virtual detection area at the presumed position of the original area of ​​interest in the new image. This method heavily relies on prior measurement information of the physical location of the detection target, essentially tracking and repositioning a known fixed area, and cannot meet the needs of new scenarios where the PTZ rotates without prior measurement. Secondly, the solution lacks a sensing element for the stability of the gimbal's mechanical movement. If geometric calculations are performed when the gimbal has not completely stopped and the image is still shaking, errors are easily introduced, affecting the accuracy of subsequent detection.

[0004] In summary, a current technical bottleneck is how to enable the gimbal to automatically and accurately update the virtual detection area of ​​any unknown new scene when rotating to it without prior physical location information. Summary of the Invention

[0005] The purpose of this application is to provide a method and system for intelligent patrol and early warning of AI-powered events on highways. It solves the technical problems existing in the prior art, such as how to enable the gimbal to automatically and accurately update the virtual detection area of ​​any unknown new scene when it rotates to any unknown new scene without prior physical location information.

[0006] To solve the above-mentioned technical problems, the solution adopted in this application is as follows:

[0007] A highway AI-based intelligent patrol and early warning method for incidents, characterized by the following steps:

[0008] S1: Real-time video stream of the highway is acquired via a PTZ camera;

[0009] S2: Based on the video stream, perform gimbal motion stability detection and obtain a score for gimbal stability;

[0010] S3: When the stability score of the PTZ camera is less than the stability threshold, perform road structure recognition on the video frames captured by the current PTZ camera to obtain the segmentation mask of the road structure.

[0011] S4: Update the virtual detection area that matches the current PTZ camera's viewpoint based on the segmentation mask;

[0012] S5: Real-time AI event detection of the video stream based on the updated virtual detection area;

[0013] S6: When a preset event is detected, a warning message is generated and output.

[0014] Preferably, the gimbal motion stability detection in S2 includes rapid detection and precise detection.

[0015] Preferably, the implementation of the rapid detection specifically includes the following steps:

[0016] S2.1: Convert the RGB image to YUV format and extract the Y component as a grayscale image;

[0017] S2.2: Downsample the grayscale image to obtain a low-resolution grayscale image;

[0018] S2.3: Fast detection: Calculate the grayscale difference of each pixel in the grayscale images of frame t-1 and frame t. Pixels with a grayscale difference greater than the threshold T are considered as moving regions, and the binary moving regions are output. If no binary moving regions exist, repeat S2.3 to detect whether there are binary moving regions in the remaining image frames. If there are binary moving regions, execute S2.4.

[0019] Preferably, the implementation of the precise detection specifically includes the following steps:

[0020] S2.4: Perform seamless processing on the binarized motion region obtained from rapid detection to allow for precise detection;

[0021] S2.5: Precise Detection: Use the three-frame difference method to detect whether there is a motion region in the grayscale images of frame t-1, frame t, and frame t+1. If there is no motion region, the gimbal is in a stable state; if there is a motion region, proceed to S2.6.

[0022] S2.6: Calculate the gimbal stability score using the optical flow method.

[0023] Preferably, the implementation of S2.4 specifically includes the following steps:

[0024] S2.4.1: Map the binarized motion region from low resolution to high resolution;

[0025] The mapping formula is as follows: ; , The coordinates before mapping; These are the mapped coordinates;

[0026] S2.4.2: Based on the coordinates of the mapped pixels, crop the ROI image from the 1920×1080 image;

[0027] Among them, the rapid detection detected the motion area as The high-resolution cropping region is:

[0028]

[0029]

[0030] in , To increase the number of pixels.

[0031] Preferably, the implementation of S2.5 specifically includes the following steps:

[0032] S2.5.1: Calculate the difference image between frame t and frame (t-1). ;

[0033] in, ;

[0034] S2.5.2: Calculate the difference image between frame t and frame t+1. :

[0035] in, ;

[0036] S2.5.3: For difference images Sum and difference images Perform a logical AND operation to obtain the final motion region image. ;

[0037] in, ;

[0038] S2.5.4: Image of the moving region The motion region is extracted by comparing it with a threshold T. ;

[0039] in, .

[0040] Preferably, the implementation of S2.6 specifically includes the following steps:

[0041] S2.6.1: Using the ROI images of frame t−1 and frame t, calculate the pixel motion vector ( );

[0042] Among them, pixel motion vector ( The calculation formula is: ;( () is a motion vector, representing the direction and magnitude of the pixel's displacement; , For image spatial gradient, For time gradient;

[0043] S2.6.2: Calculate the amplitude of each motion vector Retention range Greater than the threshold The ROI image region;

[0044] The formula for calculating the amplitude is as follows: ; It can be 5 pixels per frame;

[0045] S2.6.3: Calculate the average magnitude of motion vectors within the preserved ROI image region. If the average amplitude Greater than the threshold If the optical flow result is positive, then it is determined that there is valid motion within the preserved ROI image region; the optical flow result is recorded. Otherwise, record it as 0;

[0046] S2.6.4: Within the retained ROI image regions of frame t-1 and frame t, use the SIFT algorithm to detect feature points; use the FLANN matcher to match the feature points of the current frame with the feature points of the previous frame.

[0047] S2.6.5: Perform a matching distance threshold verification between the feature points of the current frame and the feature points of the previous frame, and only retain those with a matching distance less than the threshold. Feature point pairs;

[0048] S2.6.6: Remove anomalous feature point pairs using the RANSAC algorithm, and retain feature point pairs that conform to the motion model;

[0049] S2.6.7: If the number of feature point pairs remaining Greater than the threshold Then record the feature point matching result. Otherwise, record it as 0;

[0050] S2.6.8: Result of optical flow method Matching results with feature points Weighted fusion is performed to obtain a score for gimbal stability. The score of gimbal stability for the ROI image in frame t. Greater than the threshold Then, repeat S2.6.1-S2.6.8, and score the gimbal stability of the ROI image in frames t-1, t, and t+1. Less than the stability threshold At that time, execute S3;

[0051] in, , .

[0052] Preferably, the implementation of S3 specifically includes the following steps:

[0053] S3.1: Capture a frame of the video stream at the current moment, and normalize the image to obtain a standard image frame;

[0054] S3.2: Input the standardized image frames after standardization into the OneFormer image segmentation model for model task configuration;

[0055] S3.3: The OneFormer image segmentation model performs forward inference processing through its internal feature extraction network, Transformer decoder and other structures, and outputs the panoramic segmentation result of the current image frame.

[0056] S3.4: Analyze the panoramic segmentation results of the current image frame and extract the binarization mask corresponding to the semantic category related to the road structure.

[0057] Preferably, the implementation of S4 specifically includes the following steps:

[0058] S4.1: Parse the segmentation mask and extract the binarized mask corresponding to the semantic categories related to the road structure;

[0059] S4.2: For each type of extracted binary mask, generate the corresponding virtual detection region geometric parameters;

[0060] S4.3: Based on the geometric parameters of the virtual detection area, map it to the world coordinate system / monitoring plane coordinate system through perspective transformation to obtain its coordinate description in physical space.

[0061] The highway AI event intelligent patrol and early warning system adopts the highway AI event intelligent patrol and early warning method as described in claims 1-7, including:

[0062] The video stream acquisition module is used to acquire real-time video streams of highways through a PTZ camera;

[0063] The gimbal motion detection module, connected to the video stream acquisition module, is used to perform multi-level motion detection on the video stream, determine the stability of the gimbal, and generate trigger signals.

[0064] The road structure recognition module, connected to the gimbal motion detection module, is used to respond to a trigger signal by calling the OneFormer image segmentation model to process the current video frame and output a semantic segmentation mask of the road structure.

[0065] The virtual detection region generation module, connected to the road structure recognition module, is used to generate a virtual detection region that matches the current road structure based on the semantic segmentation mask.

[0066] The AI ​​event detection module is connected to the virtual detection area generation module and is used to perform real-time event recognition within the updated virtual detection area.

[0067] The early warning output module, connected to the AI ​​event detection module, is used to generate and output early warning information;

[0068] The data management and communication module is used to coordinate the data flow between modules, store system parameters and event logs, and provide external communication interfaces.

[0069] The technical solution of this application has at least the following advantages and beneficial effects:

[0070] 1. This invention processes video stream data acquired by a PTZ camera frame by frame. For each frame of RGB image, a low-resolution grayscale image is first rapidly detected to identify moving regions. These moving regions are then mapped to the coordinates of the original high-resolution image. A three-frame difference method is used for precise detection of the high-resolution image, further distinguishing whether changes in pixels within the moving region are due to changes in the monitoring scene caused by PTZ movement within the same scene, or changes caused by vehicles, weather, etc., thus determining whether the PTZ is moving. After determining that the PTZ is moving, the image frames of the video stream are continuously monitored until the PTZ stabilizes. If the stability score obtained from the image frame processing is less than a set threshold, it indicates that the PTZ has stabilized after moving. At this point, road recognition and segmentation are triggered to identify and segment roads and other features in the new monitoring scene, generating a virtual detection area corresponding to the scene. This method solves the problem in traditional patrol systems where the PTZ movement detection function fails, requiring manual recalibration of the virtual detection area, without the need for prior measurements of various parameters in the physical world. Attached Figure Description

[0071] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0072] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0073] It should be noted that similar reference numerals and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. The terms "center," "upper," "lower," "inner," and "outer," indicating orientation or positional relationships based on the orientation or positional relationships shown in the figures, or the orientation or positional relationships commonly used when the product is in use, are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or operated in a specific orientation, and therefore should not be construed as a limitation on this application. It should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," and "connect" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; mechanical connections or electrical connections; direct connections or indirect connections through an intermediate medium; and internal communication between two elements. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0074] See Figure 1 This invention discloses a method for intelligent patrol and early warning of highway events using AI, which includes the following steps:

[0075] S1: Real-time video stream of the highway is acquired via a PTZ camera;

[0076] The camera can be an industrial camera, mounted on a movable pan-tilt head to form a pan-tilt camera. A pan-tilt head is a mechanical support and motion device that controls the horizontal rotation, vertical tilt, and zoom of the camera. It is mounted below the camera and is driven by a motor to achieve multi-angle and multi-range field of view adjustment of the camera. The pan-tilt camera captures highway images at a frame rate of 30fps to obtain a video stream about the highway. The resolution of the captured image is 1920*1080, and the image captured by the pan-tilt camera is an RGB image.

[0077] S2: Based on the video stream, perform gimbal motion stability detection and obtain a score for gimbal stability;

[0078] Specifically, the gimbal motion stability detection operation is as follows: the gimbal stability score is calculated using the multi-level frame difference method and optical flow method based on the frame images of the video stream.

[0079] The implementation of S2 specifically includes the following steps:

[0080] S2.1: Convert the RGB image to YUV format and extract the Y component as a grayscale image;

[0081] The conversion formula used in the above operation is as follows: .

[0082] The video stream consists of several RGB image frames. Instead of detecting the entire video stream, we perform fast detection and processing on each of the several RGB image frames separately.

[0083] S2.2: Downsample the grayscale image to obtain a low-resolution grayscale image;

[0084] The grayscale image obtained by downsampling is 320×240 in size.

[0085] S2.3: Fast detection: Calculate the grayscale difference of each pixel in the grayscale images of frame t-1 and frame t. Pixels with a grayscale difference greater than the threshold T are considered as moving regions, and the binary moving regions are output. If no binary moving regions exist, repeat S2.3 to detect whether there are binary moving regions in the remaining image frames. If there are binary moving regions, execute S2.4.

[0086] The threshold T can be set to 70, and the t-1 frame is the grayscale image of the frame preceding the t frame.

[0087] It is worth emphasizing that instead of detecting the entire video stream, several RGB image frames are converted to grayscale images and downsampled before being quickly detected separately. The downsampled grayscale images are used to initially label the motion regions, reducing the original 1920*1080 image size by nearly 30 times. Processing grayscale images also requires less computation than processing RGB images, which reduces the amount of data processed and lowers the computational cost.

[0088] S2.4: Perform seamless processing on the binarized motion region obtained from rapid detection to allow for precise detection;

[0089] Furthermore, when transitioning from fast detection to precise detection, the binarized motion region output by the fast detection layer is a low-resolution 320×240, which needs to be converted to a high-resolution 1920×1080 through linear scaling. The processing steps include:

[0090] S2.4.1: Map the binarized motion region from low resolution to high resolution;

[0091] The mapping formula is as follows: . , The coordinates before mapping; These are the mapped coordinates.

[0092] S2.4.2: Based on the coordinates of the mapped pixels, crop the ROI image from the 1920×1080 image;

[0093] For example, rapid detection detects the motion region as The high-resolution cropping region is:

[0094]

[0095]

[0096] in , To increase the number of pixels.

[0097] It's important to emphasize that the rapid detection performed before precise detection aims to conserve computational resources and preliminarily identify the motion region. The resulting coordinates of this motion region are at low resolution, while the precise detection image is at high resolution. Directly using these coordinates from rapid detection for precise detection would result in significant positional inaccuracies. Therefore, a linear scaling mechanism is used to map the coordinates of the motion region detected during rapid detection to the coordinates of the original high-resolution image. During precise detection, only the mapped motion region from the high-resolution image needs to be detected.

[0098] S2.5: Precise Detection: Use the three-frame difference method to detect whether there is a motion region in the grayscale images of frame t-1, frame t, and frame t+1. If there is no motion region, the gimbal is in a stable state; if there is a motion region, proceed to S2.6.

[0099] Furthermore, the S2.5 method utilizes a three-frame difference method for accurate detection, and its steps are as follows:

[0100] S2.5.1: Calculate the difference image between frame t and frame (t-1). ;

[0101] in, . Let be the gray value of the pixel at coordinates (x, y) in the grayscale image of frame t; Let be the gray value of the pixel at coordinates (x, y) in the (t-1)th frame of the grayscale image; This represents the intensity of the grayscale value change of the pixel at image coordinates (𝑥,𝑦).

[0102] S2.5.2: Calculate the difference image between frame t and frame t+1. :

[0103] in, .

[0104] S2.5.3: For difference images Sum and difference images Perform a logical AND operation to obtain the final motion region image. ;

[0105] in, .

[0106] S2.5.4: Image of the moving region The motion region is extracted by comparing it with a threshold T. ;

[0107] in, ;

[0108] It is worth emphasizing that the fast detection step is relatively coarse in detecting moving regions. Directly using the fast detection result as a condition for triggering subsequent steps may lead to false detections. However, if precise detection is performed on every frame, many frames do not actually show any change in pixel grayscale values. Therefore, calculating and comparing precise pixel grayscale values ​​for these frames wastes a significant amount of computational resources. In this embodiment, a fast detection method is used to initially confirm the existence of moving regions in the image, and a precise detection method is used to further verify the results of the fast detection, thus saving computational power.

[0109] S2.6: Calculate the gimbal stability score using the optical flow method;

[0110] Furthermore, S2.6 The calculation of the gimbal stability score using the optical flow method specifically includes the following steps:

[0111] S2.6.1: Calculate pixel motion vectors using the ROI images of frame t−1 and frame t;

[0112] The calculation formula is as follows: . ( () is a motion vector, representing the direction and magnitude of the pixel's displacement; , For image spatial gradient, This represents the time gradient.

[0113] S2.6.2: Calculate the amplitude of each motion vector Retention range Greater than the threshold The ROI image region;

[0114] The formula for calculating the amplitude is as follows: .

[0115] It can be 5 pixels per frame.

[0116] S2.6.3: Calculate the average magnitude of motion vectors within the preserved ROI image region. If the average amplitude Greater than the threshold If the optical flow result is positive, then it is determined that there is valid motion within the preserved ROI image region; the optical flow result is recorded. Otherwise, record it as 0;

[0117] S2.6.4: Within the retained ROI image regions of frame t-1 and frame t, use the SIFT algorithm to detect feature points; use the FLANN matcher to match the feature points of the current frame with the feature points of the previous frame.

[0118] S2.6.5: Perform a matching distance threshold verification between the feature points of the current frame and the feature points of the previous frame, and only retain those with a matching distance less than the threshold. Feature point pairs;

[0119] S2.6.6: Remove anomalous feature point pairs using the RANSAC algorithm, and retain feature point pairs that conform to the motion model;

[0120] It's worth noting that the SIFT algorithm extracts local feature points in an image that remain stable despite scale, rotation, and brightness changes. These points typically correspond to prominent parts of physical entities in the scene. When changes in lighting or weather cause an image to darken or brighten, the grayscale values ​​of image pixels also change. If this is not excluded, the system will assume that all image pixel changes are caused by gimbal movement, leading to false detections and unnecessary road structure recognition operations, which undoubtedly wastes computing resources. In this embodiment, the SIFT algorithm's descriptor is based on the gradient orientation histogram, which is insensitive to lighting changes, thus distinguishing between physical displacement and brightness changes. The RANSAC algorithm is then used to fit motion models to the matched feature points in the two frames, differentiating feature changes caused by gimbal movement from those caused by non-gimbal movement factors. Therefore, the SIFT and RANSAC algorithms are used to verify whether image changes originate from the gimbal's own movement.

[0121] S2.6.7: If the number of feature point pairs remaining Greater than the threshold Then record the feature point matching result. Otherwise, record it as 0.

[0122] S2.6.8: Result of optical flow method Matching results with feature points Weighted fusion is performed to obtain a score for gimbal stability. The score of gimbal stability for the ROI image in frame t. Greater than the threshold Then, repeat S2.6.1-S2.6.8, and score the gimbal stability of the ROI image in frames t-1, t, and t+1. Less than the stability threshold At that time, execute S3;

[0123] in, , .

[0124] It is worth emphasizing that the scoring is performed when the pan-tilt camera is in motion. Greater than the threshold If consecutive image frames Less than the threshold If the PTZ camera has stabilized after changing its state, then proceed to the next step of road structure recognition.

[0125] S3: When the stability score of the PTZ camera is less than the stability threshold, perform road structure recognition on the video frames captured by the current PTZ camera to obtain the segmentation mask of the road structure.

[0126] The implementation of S3 specifically includes the following steps:

[0127] S3.1: Capture a frame of the video stream at the current moment, and normalize the image to obtain a standard image frame;

[0128] Standardization processes include size normalization and data standardization.

[0129] S3.2: Input the standardized image frames after standardization into the OneFormer image segmentation model for model task configuration;

[0130] During model task configuration, task prompt text such as "panopticsegmentation" can be input into the model.

[0131] S3.3: The OneFormer image segmentation model performs forward inference processing through its internal feature extraction network, Transformer decoder and other structures, and outputs the panoramic segmentation result of the current image frame.

[0132] The feature extraction network is ConvNeXt; the panoramic segmentation result includes a semantic segmentation map; in the semantic segmentation map, each pixel in the standard image frame is classified into a predefined semantic category, which includes background, main lane, emergency lane, shoulder, etc.

[0133] S3.4: Analyze the panoramic segmentation results of the current image frame and extract the binarization mask corresponding to the semantic category related to the road structure;

[0134] Each binarized mask identifies the pixel location of that semantic category in the image.

[0135] S4: Update the virtual detection area that matches the current PTZ camera's viewpoint based on the segmentation mask;

[0136] The implementation of S4 specifically includes the following steps:

[0137] S4.1: Parse the segmentation mask and extract the binarized mask corresponding to the semantic categories related to the road structure;

[0138] The semantic categories include at least the main driving lane and the emergency lane.

[0139] S4.2: For each type of extracted binary mask, generate the corresponding virtual detection region geometric parameters;

[0140] S4.3: Based on the geometric parameters of the virtual detection area, map it to the world coordinate system / monitoring plane coordinate system through perspective transformation to obtain its coordinate description in physical space.

[0141] S5: Real-time AI event detection of the video stream based on the updated virtual detection area;

[0142] In this step, the YOLOv8 detection model is used to perform image recognition on the video stream, identifying targets such as vehicles, pedestrians, and obstacles in the video stream. A multi-target tracking algorithm is then used to track the motion trajectory of the identified targets. Based on the motion trajectory and event rules, it is determined whether events such as traffic accidents, vehicle stops, pedestrians running in, and vehicles driving in the wrong direction have occurred in the highway scenario. In this invention, these events are collectively referred to as AI events.

[0143] The implementation of S5 includes the following steps:

[0144] S5.1: Real-time image recognition of video streams using the YOLOv8 detection model;

[0145] The identified image targets include vehicles, pedestrians, obstacles, and other targets;

[0146] It should be noted that the YOLOv8 detection model is a target detection model that can identify targets such as vehicles, pedestrians, traffic signs, and obstacles in images in real time.

[0147] S5.2: Combining multi-target tracking algorithm and event rules, track and judge the image identified in S5.1 to determine whether the preset event has occurred;

[0148] Specifically, the DeepSORT algorithm can be used for multi-object tracking. This algorithm is a standard multi-object tracking algorithm in the field of computer vision, used to maintain the ID of the same target in a video and obtain its motion trajectory.

[0149] The event rules are based on the logical judgment of the rule "if the target is in area A and the speed is less than the threshold for N frames, then a parking event is triggered". This is a common method for implementing event detection in the industry, so it will not be elaborated on here.

[0150] Among the identifiable events are: traffic accidents, vehicles stopped, pedestrians trespassing, and vehicles driving in the wrong direction.

[0151] S6: When a preset event is detected, a warning message is generated and output.

[0152] Preset events include traffic accidents, vehicle stops, pedestrians trespassing, and vehicles driving in the wrong direction. When any of these events are detected, an early warning message is generated and sent to the alarm platform, where it is displayed via sound and light, SMS, and platform pop-ups.

[0153] Another aspect of this invention discloses a highway AI event intelligent patrol and early warning method, wherein the highway AI event intelligent patrol and early warning system adopts the above-mentioned highway AI event intelligent patrol and early warning method, including:

[0154] The video stream acquisition module is used to acquire real-time video streams of highways through a PTZ camera;

[0155] The gimbal motion detection module, connected to the video stream acquisition module, is used to perform multi-level motion detection on the video stream, determine the stability of the gimbal, and generate trigger signals.

[0156] The road structure recognition module, connected to the gimbal motion detection module, is used to respond to a trigger signal by calling the OneFormer image segmentation model to process the current video frame and output a semantic segmentation mask of the road structure.

[0157] The virtual detection region generation module, connected to the road structure recognition module, is used to generate a virtual detection region that matches the current road structure based on the semantic segmentation mask.

[0158] The AI ​​event detection module is connected to the virtual detection area generation module and is used to perform real-time event recognition within the updated virtual detection area.

[0159] The early warning output module, connected to the AI ​​event detection module, is used to generate and output early warning information;

[0160] The data management and communication module is used to coordinate the data flow between modules, store system parameters and event logs, and provide external communication interfaces.

[0161] The various embodiments of the present invention have now been described in detail. To avoid obscuring the concept of the invention, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions of this invention based on the above description, and the scope of the invention is defined by the appended claims.

Claims

1. A method for intelligent patrol and early warning of highway events using AI, characterized in that, Includes the following steps: S1: Real-time video stream of the highway is acquired via a PTZ camera; S2: Based on the video stream, perform gimbal motion stability detection and obtain a score for gimbal stability; S3: When the stability score of the PTZ camera is less than the stability threshold, perform road structure recognition on the video frames captured by the current PTZ camera to obtain the segmentation mask of the road structure. S4: Update the virtual detection area that matches the current PTZ camera's viewpoint based on the segmentation mask; S5: Real-time AI event detection of the video stream based on the updated virtual detection area; S6: When a preset event is detected, a warning message is generated and output.

2. The highway AI event intelligent patrol and early warning method according to claim 1, characterized in that, The gimbal motion stability detection in S2 includes both rapid detection and precise detection.

3. The intelligent patrol and early warning method for highway AI events according to claim 2, characterized in that, The implementation of the rapid detection specifically includes the following steps: S2.1: Convert the RGB image to YUV format and extract the Y component as a grayscale image; S2.2: Downsample the grayscale image to obtain a low-resolution grayscale image; S2.3: Fast detection: Calculate the grayscale difference of each pixel in the grayscale images of frame t-1 and frame t. Pixels with a grayscale difference greater than the threshold T are considered as moving regions, and the binary moving regions are output. If no binary moving regions exist, repeat S2.3 to detect whether there are binary moving regions in the remaining image frames. If there are binary moving regions, execute S2.

4.

4. The highway AI event intelligent patrol and early warning method according to claim 2, characterized in that, The implementation of the precise detection specifically includes the following steps: S2.4: Perform seamless processing on the binarized motion region obtained from rapid detection to allow for precise detection; S2.5: Precise Detection: Use the three-frame difference method to detect whether there is a motion region in the grayscale images of frame t-1, frame t, and frame t+1. If there is no motion region, the gimbal is in a stable state; if there is a motion region, proceed to S2.

6. S2.6: Calculate the gimbal stability score using the optical flow method.

5. The intelligent patrol and early warning method for highway AI events according to claim 2, characterized in that, The implementation of S2.4 specifically includes the following steps: S2.4.1: Map the binarized motion region from low resolution to high resolution; The mapping formula is as follows: ; , The coordinates before mapping; These are the mapped coordinates; S2.4.2: Based on the coordinates of the mapped pixels, crop the ROI image from the 1920×1080 image; Among them, the rapid detection detected the motion area as The high-resolution cropping region is: in , To increase the number of pixels.

6. The intelligent patrol and early warning method for highway AI events according to claim 2, characterized in that, The implementation of S2.5 specifically includes the following steps: S2.5.1: Calculate the difference image between frame t and frame (t-1). ; in, ; S2.5.2: Calculate the difference image between frame t and frame t+1. : in, ; S2.5.3: For difference images Sum and difference images Perform a logical AND operation to obtain the final motion region image. ; in, ; S2.5.4: Image of the moving region The motion region is extracted by comparing it with a threshold T. ; in, .

7. The intelligent patrol and early warning method for highway AI events according to claim 2, characterized in that, The implementation of S2.6 specifically includes the following steps: S2.6.1: Using the ROI images of frame t−1 and frame t, calculate the pixel motion vector ( ); Among them, pixel motion vector ( The calculation formula is: ;( () is a motion vector, representing the direction and magnitude of the pixel's displacement; , For image spatial gradient, For time gradient; S2.6.2: Calculate the amplitude of each motion vector Retention range Greater than the threshold The ROI image region; The formula for calculating the amplitude is as follows: ; It can be 5 pixels per frame; S2.6.3: Calculate the average magnitude of motion vectors within the preserved ROI image region. If the average amplitude Greater than the threshold If the optical flow result is positive, then it is determined that there is valid motion within the preserved ROI image region; the optical flow result is recorded. Otherwise, record it as 0; S2.6.4: Within the retained ROI image regions of frame t-1 and frame t, use the SIFT algorithm to detect feature points; use the FLANN matcher to match the feature points of the current frame with the feature points of the previous frame. S2.6.5: Perform a matching distance threshold verification between the feature points of the current frame and the feature points of the previous frame, and only retain those with a matching distance less than the threshold. Feature point pairs; S2.6.6: Remove anomalous feature point pairs using the RANSAC algorithm, and retain feature point pairs that conform to the motion model; S2.6.7: If the number of feature point pairs remaining Greater than the threshold Then record the feature point matching result. Otherwise, record it as 0; S2.6.8: Result of optical flow method Matching results with feature points Weighted fusion is performed to obtain a score for gimbal stability. The score of gimbal stability for the ROI image in frame t. Greater than the threshold Then, repeat S2.6.1-S2.6.8, and score the gimbal stability of the ROI image in frames t-1, t, and t+1. Less than the stability threshold At that time, execute S3; in, , .

8. The intelligent patrol and early warning method for highway AI events according to claim 1, characterized in that, The implementation of S3 specifically includes the following steps: S3.1: Capture a frame of the video stream at the current moment, and normalize the image to obtain a standard image frame; S3.2: Input the standardized image frames after standardization into the OneFormer image segmentation model for model task configuration; S3.3: The OneFormer image segmentation model performs forward inference processing through its internal feature extraction network, Transformer decoder and other structures, and outputs the panoramic segmentation result of the current image frame. S3.4: Analyze the panoramic segmentation results of the current image frame and extract the binarization mask corresponding to the semantic category related to the road structure.

9. The intelligent patrol and early warning method for highway AI events according to claim 1, characterized in that, The implementation of S4 specifically includes the following steps: S4.1: Parse the segmentation mask and extract the binarized mask corresponding to the semantic categories related to the road structure; S4.2: For each type of extracted binary mask, generate the corresponding virtual detection region geometric parameters; S4.3: Based on the geometric parameters of the virtual detection area, map it to the world coordinate system / monitoring plane coordinate system through perspective transformation to obtain its coordinate description in physical space.

10. A highway AI event intelligent patrol and early warning system, employing the highway AI event intelligent patrol and early warning method of claims 1-9, comprising: The video stream acquisition module is used to acquire real-time video streams of highways through a PTZ camera; The gimbal motion detection module, connected to the video stream acquisition module, is used to perform multi-level motion detection on the video stream, determine the stability of the gimbal, and generate trigger signals. The road structure recognition module, connected to the gimbal motion detection module, is used to respond to a trigger signal by calling the OneFormer image segmentation model to process the current video frame and output a semantic segmentation mask of the road structure. The virtual detection region generation module, connected to the road structure recognition module, is used to generate a virtual detection region that matches the current road structure based on the semantic segmentation mask. The AI ​​event detection module is connected to the virtual detection area generation module and is used to perform real-time event recognition within the updated virtual detection area. The early warning output module, connected to the AI ​​event detection module, is used to generate and output early warning information; The data management and communication module is used to coordinate the data flow between modules, store system parameters and event logs, and provide external communication interfaces.

Images