Fence climbing behavior recognition method and electronic device

Abstract

The application discloses a fence climbing behavior recognition method and electronic equipment, adopts a YOLOv7 model as a pedestrian detection algorithm, uses an HRNet pedestrian key point detection algorithm for improvement and optimization, removes useless key points of a data set and performs data enhancement, accelerates a model inference speed, improves generalization ability of the model, enhances detection precision of the model, and based on adaptive distance judgment of upper and lower fence lines and multi-angle posture recognition algorithms based on topological relations, posture recognition is performed on the basis of a key point detection model, key frames of climbing and crossing of pedestrians when crossing the fence are recognized and alarmed, threshold definition of the algorithm is reduced, and robustness and algorithm accuracy of the algorithm are improved. Compared with the algorithm based on the increase of the behavior recognition model or the human body posture classification model, the algorithm complexity is also reduced, and the calculation amount is smaller.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and in particular to a method and electronic device for recognizing fence climbing behavior. Background Technology

[0002] Various types of fencing devices are widely present in cities, such as road fences, plaza fences, and fences around residential areas, schools, and factories. These fencing devices can clearly indicate and convey relevant safety and traffic information regarding pedestrian and vehicular traffic, thereby protecting public order. However, even within these widely available fencing areas, some illegal acts of crossing the fences still occur, such as crossing urban road fences, which frequently leads to safety accidents. With the development of deep learning technology and the widespread and high-density installation of various surveillance cameras in cities, it has become possible to detect illegal human crossings of fences in real time based on video. Through deep learning algorithms and video surveillance, illegal fence-climbing behavior can be quickly detected, and keyframes can be obtained to generate alarm image information, which helps relevant agencies to investigate and track down violations, enabling better management and law enforcement.

[0003] With the development of technology, the currently known published patents for detecting pedestrian crossing of fences are mainly divided into methods based on physical sensing functions, target detection trajectories, and behavior classification.

[0004] Physical sensing-based fencing uses photoelectric sensors. When the sensors detect signal blockage and pedestrian movement, measures such as sound alarms are triggered to prevent pedestrians from climbing over. Compared to ordinary fencing, this type of fencing has higher deployment and maintenance costs and cannot record information about violators. Target detection trajectory-based methods, with advancements in computer vision technology, extract pedestrian features through traditional image processing or deep learning for target detection, tracking, recording, and prediction of pedestrian trajectories, determining whether the target appears on either side of the fence to achieve a tripping rule. This method is fast but cannot accurately identify key behaviors such as climbing and traversing postures when crossing the fence, and cannot provide accurate keyframe alarms. Furthermore, overlapping pedestrians on both sides can easily lead to false detections and is difficult to handle complex situations. Behavior classification-based methods, after pedestrian detection and identification, further input keypoint detection data for pedestrians near the fence, and then feed the detection results into a classification network to detect fence-climbing behavior. This type of method can precisely define actions and accurately identify corresponding behaviors. However, this method involves cumbersome and complex data collection and training, and takes a long time for recognition and inference. Furthermore, its robustness is poor when dealing with various perspectives and occlusions around the fence during action classification. There are also methods that use human keypoints and fence distances to determine the presence of fence-related actions, such as invention patent 2020110260725. However, these algorithms often use the height difference between fence lines and the height difference between the pedestrian's ankle or knee points for judgment, involving many custom thresholds. Moreover, when the angle is closer to vertical, the judgment of the knee point being higher than the fence line is severely biased due to angle projection, easily leading to various false alarms. It basically does not support detection at almost vertical angles and can only be used in scenarios directly facing the fence, resulting in poor applicability.

[0005] Furthermore, our invention patent CN115424342 A discloses a method, device, and electronic device for detecting guardrail climbing behavior. However, it includes subjectively set hyperparameter thresholds of 0.7 and 0.15, but these thresholds are fixed and cannot be adaptively adjusted for different scenarios. This leads to misjudgments when the angle changes significantly, such as when two fences are about to overlap, rendering these two thresholds largely meaningless. In addition, for fences of different heights, especially at different angles, the corresponding t1 and t2 errors gradually increase with the tilt angle of the guardrail, resulting in low detection accuracy and a tendency for false alarms. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a method and electronic device for recognizing fence climbing behavior. The objective of this invention is achieved through the following technical solution:

[0007] A method for recognizing fence climbing behavior includes the following steps:

[0008] Step S1: Obtain the fence monitoring video stream to get the monitoring image. Mark the upper fence line and lower fence line in the monitoring image and obtain the coordinates of the upper fence line and lower fence line.

[0009] Step S2: Detect pedestrians in the monitored images;

[0010] Step S3: Identify the key points of the pedestrian's body, the coordinates of the key points, and the target detection box and coordinates of the target detection box;

[0011] Step S4: Determine if the pedestrian has approached the fence. If so, proceed to the next step; otherwise, track the pedestrian.

[0012] The first judgment threshold is obtained by using human key points, and the judgment distance between the pedestrian and the fence is obtained based on the target detection box coordinates of the pedestrian, the upper fence line coordinates, and the lower fence line coordinates. If any of the judgment distances is less than the first judgment threshold, it is determined that the pedestrian is close to the fence.

[0013] Step S5: Determine if the pedestrian is climbing the fence. If yes, proceed to the next step; otherwise, track the pedestrian.

[0014] Step S6: Issue a climbing alarm.

[0015] In a further improvement, in step S4, the distance determination includes D1 and D2, and the methods for obtaining D1 and D2 are as follows:

[0016] The midpoint of the line segment between the key points of the left hip and the right hip of the pedestrian is obtained as the center point P1 of the middle of the human body; the midpoint of the line segment between the two points of the lower edge of the target detection box of the pedestrian is obtained as the center point P2 of the bottom of the human body; the vertical distance D1 from P1 to the upper fence line L1 and the vertical distance D2 from P2 to the lower fence line L2 are obtained.

[0017] In a further improvement, the method for obtaining the first judgment threshold in step S4 is as follows:

[0018] Calculate the distances between the ankle keypoint and the knee keypoint and the knee keypoint and the hip keypoint of the pedestrian's two legs respectively, and use the maximum value of the two distances as the first judgment threshold J1.

[0019] In a further improvement, in step S5, if the line connecting the two key points of a person's legs intersects with the upper fence line L1, and the distance between the key point of the ankle and the key point of the knee on a certain leg is less than the second judgment threshold, then it indicates that the pedestrian has climbing behavior; otherwise, there is no climbing behavior.

[0020] In a further improvement, the method for determining whether a pedestrian is climbing the fence in step S5 is as follows:

[0021] Line segment LA is obtained by connecting the left and right ankle key points of the pedestrian, and line segment LK is obtained by connecting the left and right knee key points. The vertical distances H_L_AK between the left ankle key point and the left knee key point, and H_R_AK between the right ankle key point and the right knee key point are also considered. If at least one of LA and LK intersects with the upper fence line L1, and both H_L_AK and H_R_AK are less than the second judgment threshold, then it is determined that the pedestrian is climbing; otherwise, climbing behavior is not observed.

[0022] A further improvement is that the second judgment threshold is obtained as follows: the vertical distance between the key points of the pedestrian's left and right ankles is the second judgment threshold HA.

[0023] In a further improvement, in step S3, the YOLOv7 object detection model is used to obtain the object detection box for pedestrians. If one of the following conditions is met, the pedestrians are filtered out:

[0024] 1) The target detection bounding box for pedestrians is smaller than 70*120 pixels;

[0025] 2) The target detection box of the pedestrian does not intersect with the upper and lower fence lines.

[0026] In a further improvement, in step S3, key points of the human body are identified through the HRNet network.

[0027] Further improvements include the following key points on the human body: right shoulder key point, right elbow key point, right wrist key point, left shoulder key point, left elbow key point, left wrist key point, right hip key point, right knee key point, right ankle key point, left hip key point, left knee key point, and left ankle key point.

[0028] In a further improvement, in step S6, the alarm method is to use the image of the pedestrian climbing as the alarm image, and output the alarm image, alarm timestamp, location information and alarm description to the alarm terminal.

[0029] A further improvement is that if a pedestrian is still detected climbing the fence 3 seconds after the alarm is triggered, the alarm will continue to be triggered.

[0030] An electronic device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-9.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention is based on adaptive distance judgment of upper and lower fence lines and multi-angle posture recognition algorithm based on topological relationship. It performs posture recognition on the basis of key point detection model, and identifies and alarms the key frames of pedestrian climbing and crossing fences. It reduces the threshold definition of the algorithm, improves the robustness and accuracy of the algorithm. Compared with the algorithm based on adding behavior recognition model or human posture classification model, it also reduces the algorithm complexity, has less computation, and improves the speed and accuracy of recognition.

[0033] Improvements and optimizations were made using the HRNet pedestrian keypoint detection algorithm, and useless keypoints in the dataset were removed and data augmentation was performed. This accelerated the model's inference speed, improved its generalization ability, and enhanced its detection accuracy.

[0034] The YOLOv7 model was used as the pedestrian detection algorithm, and human detection boxes smaller than the threshold were filtered out, which effectively overcame the interference of pedestrians at a distance. Attached Figure Description

[0035] The invention will be further illustrated with reference to the accompanying drawings, but the contents of the drawings do not constitute any limitation on the invention.

[0036] Figure 1 This is a schematic diagram of the process of the present invention;

[0037] Figure 2 A schematic diagram of key point detection on the human body;

[0038] Figure 3-1 A key point detection diagram for determining whether a person is close to a fence from a parallel angle;

[0039] Figure 3-2 This is an actual detection diagram for determining whether a person is close to a fence from a parallel angle;

[0040] Figure 4-1 A key point detection diagram for determining whether a person is close to a fence at an inclined angle;

[0041] Figure 4-2 This is an actual detection diagram for determining whether a person is close to the fence at an inclined angle;

[0042] Figure 5 This is a schematic diagram of parallel angle pose recognition based on the topological relationship between the upper and lower fence lines and key points of the human body.

[0043] Figure 6 This is a schematic diagram of tilt angle pose recognition based on the topological relationship between the upper and lower fence lines and human key points.

[0044] Figure 7-1 Image 1 showing an alarm when a pedestrian crosses a fence;

[0045] Figure 7-2Image 2 showing an alarm for pedestrians crossing a fence;

[0046] Figure 7-3 Image 3 shows an alarm for pedestrians crossing a fence. Implementation

[0047] To make the purpose, technical solution, and advantages of the invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and examples.

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] In the description of this application, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more features. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0050] In this application, the term "exemplary" is used to mean "serving as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to implement and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0051] This invention proposes a method for identifying fence climbing behavior based on keypoint detection. First, the YOLOv7 model is used as the human detection algorithm, and targeted model enhancements are performed based on fence scenarios and various crossing and climbing postures to detect pedestrians near the fence. The human keypoint dataset is modified to use only human keypoints from key postures for model training and inference, improving the detection speed of the keypoint model. Various data augmentation training methods are applied to fence scenarios to obtain the keypoint detection model. Based on the detected pedestrian keypoints, an adaptive distance judgment based on upper and lower fence lines and a multi-angle posture recognition algorithm based on topological relationships are proposed to address the insensitivity of existing algorithms to angle transformation information and distance and height information, thereby improving detection accuracy.

[0052] This invention patent addresses existing problems by proposing a method for identifying fence climbing behavior based on key point detection, referencing... Figure 1 As shown, it includes the following steps:

[0053] Step S101: Initialize the system;

[0054] First, there's the data input configuration and fence configuration. Users select online or offline video data to input, configure the video frame interval, and then configure the fence. Users can draw the upper and lower fence lines on the interface. After completing this, the input video can be processed by the algorithm. The algorithm uses a model enhanced for fence-crossing scenarios to detect pedestrians, then filters them using the detected pedestrian bounding boxes and the upper and lower fence line data. The coordinates of the filtered pedestrian bounding boxes are then input into the optimized keypoint detection model to obtain information on 12 key points of the pedestrian (e.g., ...). Figure 2 (As shown). Then, based on the information of these 12 key points and the topological positional relationship of the upper and lower fence lines, adaptive distance judgment and fence climbing posture recognition are performed to determine whether the pedestrian has crossed the fence and climbed it. If the judgment confirms that the pedestrian has climbed the fence, the key frame image of the climb is captured to generate alarm information for saving and display.

[0055] Optimize the target detection model and key point model for fence crossing scenarios.

[0056] This invention addresses the specific data augmentation and model optimization required for training YOLOv7 and HRNet detection models in fence-crossing scenarios. For the detection model, in addition to conventional data augmentation, the size of pedestrian bounding boxes in multiple datasets is adjusted to locate the waist and leg positions of pedestrians. Numerous fence images are randomly flipped, scaled, rotated, and their transparency is randomly increased to occlude normal pedestrian walking postures and fence-crossing postures, thus enhancing pedestrian detection performance in fence-crossing scenarios. For the pedestrian keypoint recognition model, targeted keypoint pedestrian annotation data is used. Various collected fence images are used to accurately locate the human torso (focusing on areas such as the hips, knees, and ankles) in the sample set. Random occlusion of the fence images is then applied to synthesize samples. These synthesized special cases are input into the network to improve the model's accuracy in recognizing human keypoints when occluded by fences during fence crossings. In addition to occluding images of normal pedestrian walking and fence crossing postures by randomly flipping, scaling, rotating, and increasing the opacity of a large number of fence images, the number of keypoints in the keypoint dataset is reduced to 12. This improves the training and inference speed and enhances the model's recognition accuracy while ensuring the recognition of the main key behavioral postures. For the existing sample data, there are many human body samples in various common postures, but very few samples in various crossing and climbing postures. During object detection, it is difficult to select all human bodies in these postures. This synthesis method can significantly improve the accuracy of human body recognition in various crossing and climbing postures and the accuracy of completely selecting the body into the target bounding box.

[0057] Step S102, Data Input:

[0058] This involves loading the video stream captured by the camera. The invention targets various urban fence scenarios. Users can set corresponding online or offline monitoring video streams and configure the video frame extraction frequency. When user data configuration is incorrect, such as an incorrect video address or video stream acquisition failure, the algorithm will provide corresponding prompt information.

[0059] After configuring the video, to detect fence crossing behavior in the video area, the protective fence needs to be set. The upper and lower fence lines of the specific fence area to be detected are drawn by the mouse. These two lines must be configured.

[0060] Step S103: Determine whether the image frames of the video stream can be acquired normally. If they can be acquired normally, proceed to the next step; otherwise, end the process and prompt an error message, requiring maintenance by staff.

[0061] Step S104: Perform human target detection.

[0062] This invention patent utilizes the YOLOv7 object detection model for human object detection. The dataset is preprocessed, and then the YOLOv7 object detection model is trained to obtain a trained YOLOv7 object detection model. This trained model can completely select pedestrians in various climbing and crossing postures. By extracting frames from online or offline video streams at set intervals and inputting the images into the trained pedestrian detection model, the coordinates and position of the rectangular bounding box of the entire pedestrian are obtained.

[0063] Step S105: Determine whether a human body has been detected. If a human body is detected, proceed to the next step; otherwise, return to step S103.

[0064] The detection process requires target filtering of the human body:

[0065] Specifically, the filtering is performed based on the size of the detected pedestrian rectangle. If the width and height of the detected pedestrian are less than 70*120, the target is filtered out, and no further steps are performed.

[0066] Step S106: Determine whether the human detection box overlaps with the fence area:

[0067] Perform topological analysis on the quadrilateral area enclosed by the pedestrian frame, the upper fence line, and the lower fence line. If there is no intersecting area, filter the target; otherwise, proceed to the next step.

[0068] Step S107: Perform human body key point detection:

[0069] This invention patent uses the HRNet network for human keypoint detection. The HRNet network model, as a 2D human pose estimation task network for a single individual in 2019, mainly lies in its parallel connection of high-resolution to low-resolution convolutions to maintain high-resolution representation, and the idea of ​​enhancing high-resolution representation by repeatedly performing multi-scale fusion across parallel convolutions.

[0070] The fundamental principle of HRNet is to build a multi-layered feature map to capture more image details while maintaining high resolution. HRNet's network structure is very simple; it generates images of multiple resolutions through multiple operations and then combines them to form a single high-resolution image. This allows HRNet to handle difficult-to-process locations such as sides and tops, while also improving image transitions and sharpness. HRNet's network structure comprises four stages: high-resolution input, high-resolution output, low-resolution feature extraction, and high-resolution output combination. Four parallel subnetworks gradually reduce the resolution to half, while the corresponding width (number of channels) doubles. The first stage contains four residual units, each identical to ResNet-50, consisting of a 64-width bottleneck followed by a 3×3 convolution that reduces the feature map width to c. The second, third, and fourth stages contain 1, 4, and 3 swapped blocks, respectively. Each swapped block contains four residual units, with each unit containing two 3×3 convolutions at each resolution, and one swapped unit spans multiple resolutions.

[0071] Compared to commonly used pose estimation networks, HRNet offers two advantages: 1. It connects high-resolution to low-resolution subnets in parallel, rather than in series as most existing solutions do. Therefore, it maintains high resolution, rather than recovering resolution through a low-to-high process, resulting in potentially more spatially accurate predicted heatmaps. 2. Most existing fusion schemes aggregate low-level and high-level representations. In contrast, HRNet performs repeated multi-scale fusion to enhance high-resolution representations using low-resolution representations of similar depth and level.

[0072] After obtaining the input human detection bounding box information, the keypoint detection model of the corresponding human image in the bounding box is used for inference and prediction to obtain the coordinate information of 12 keypoints. The 12 keypoints are: right shoulder keypoint, right elbow keypoint, right wrist keypoint, left shoulder keypoint, left elbow keypoint, left wrist keypoint, right hip keypoint, right knee keypoint, right ankle keypoint, left hip keypoint, left knee keypoint, and left ankle keypoint. After obtaining the human keypoint information, combined with the line segment information of the upper and lower fences, adaptive distance judgment based on the upper and lower fence lines and multi-angle pose recognition based on topological relationships are performed through the following steps.

[0073] Step S108: Determine whether the human body is close to the fence (i.e., determine whether the human body is near the fence). If yes, proceed to step S109; otherwise, proceed to step S113 to continuously track the human target.

[0074] First, adaptive distance determination is performed based on the detection results of the upper and lower fence lines and human key points:

[0075] After obtaining key information about the human body, due to the differences in the projection of the camera onto the fence from different angles, although the obtained human target detection box overlaps with the area enclosed by the upper and lower fence lines, the human body is still far away from the fence. At this time, if the pedestrian performs certain simulated behaviors, it will be projected as crossing the fence from the camera's perspective, causing false alarms. Therefore, it can be filtered by distance judgment.

[0076] Specifically, it includes two steps:

[0077] Step S1081: Determine if the line connecting the knee and ankle points of the human body intersects with the upper fence line. If there is an intersection, proceed to step S1082; otherwise, proceed to step S113 to continuously track the human target.

[0078] Step S1082: Determine whether the height difference between the two feet is greater than half of the maximum distance from the ankle to the hip.

[0079] like Figure 3-1 and Figure 4-1 As shown, by obtaining the coordinates of the key points on the left and right hips of the pedestrian, the midpoint P1 between the two points is calculated and regarded as the center point of the human body. Then, by using the coordinates of the two points on the lower edge of the target detection box, the center point P2 is calculated and regarded as the center point of the bottom of the human body. Then, the distance D1 from P1 to the upper fence line segment L1 is calculated, and the distance D2 from P2 to the lower fence line L2 is calculated. The maximum distance between the left ankle, left knee, and left hip points is used as the human body adaptive scale J1 under each human body detection box, which is analogous to half of the maximum distance between the two ankles and the hips when a person is fully standing, as the judgment criterion.

[0080] D1 and D2 represent the height difference between the two feet. If either D1 or D2 is less than J1, it indicates that the person is approaching the fence; otherwise, they are moving away from the fence. Adaptive judgment indicators are calculated using key point information from each human detection target, thus eliminating the need for various custom thresholds and increasing the algorithm's applicability.

[0081] The corresponding steps in invention patent CN115424342 A are as follows: "First, add the x-coordinates of the pedestrian's left and right shoulders and divide by 2 to obtain the center x-coordinate. Then, substitute this coordinate into the equations of the upper and lower lines of the guardrail to calculate the corresponding points p1 and p2 of the upper and lower lines of the guardrail. Calculate the distance between p1 and p2 and multiply this distance by a weight w1 to obtain the guardrail threshold t1. In this patent, w1 is set to 0.7. If the height difference between the pedestrian's two feet is greater than the threshold t1, the next round of judgment is performed. First, obtain the coordinate f1 of the pedestrian's raised foot, then calculate the distance difference d1 between f1 and p1. If the value of d1 is less than the guardrail threshold t2, it is determined that the pedestrian has climbed over the guardrail. The value of the guardrail threshold t2 is equal to the guardrail threshold t1 * 0.15." This includes subjectively set hyperparameter thresholds of 0.7 and 0.15. These thresholds need to be adjusted accordingly for different scenarios. Without adjustment, this threshold becomes largely meaningless when the angle changes significantly, such as when two fences are about to overlap. This is not a fully adaptive method. For fences of different heights, especially at different angles, the corresponding t1 and t2 errors gradually increase with the tilt angle of the fence. Determining the weights requires continuous experimentation under different fence conditions. Furthermore, this threshold setting is only applicable when the fence's tilt angle in the image is small, and a custom weight w1 is needed. In many scenarios, such as from a more overhead angle or with a larger tilt angle, the two fence lines almost overlap, making it difficult to achieve good accuracy. Also, this distance is not the true relative distance between the fences; it should be the distance from a point to a line between the two lines, not the t1 described in patent CN115424342 A. Moreover, the distance value is a custom weight, making it difficult to adapt well to multi-angle pose recognition tasks. When the distance is close, its accuracy is low in some unsuitable scenarios, leading to a large number of false alarms. The adaptive distance judgment and topology-based attitude recognition proposed in this invention patent do not require additional manual thresholds. All parameters are adaptively calculated based on the identified key points and the fence itself, thus overcoming the above-mentioned defects, improving the accuracy by 4-6%, and greatly reducing false alarms.

[0082] Step S109: Determine whether the current target has been tracked. If yes, proceed to step S110 and step S113 to continuously track the human target.

[0083] Step S110: Determine whether there is fence climbing in two or more consecutive frames. If yes, proceed to step S111; otherwise, proceed to step S113 to continuously track the human target.

[0084] After determining whether a person is approaching or moving away from the fence, further posture recognition is needed for targets approaching the fence. Because the projection of the fence view from different angles varies—for example, the height difference between the knee or ankle and the fence line can lead to misjudgments due to significant differences in projection on the fence line—the topological relationship between key points on the human body and the upper fence line is used to determine whether keyframes depicting climbing or crossing the fence have been created.

[0085] Line segment LA is obtained by connecting the keypoint coordinates of the pedestrian's left and right ankles, and line segment LK is obtained by connecting the keypoint coordinates of the left and right knees. Although the fence angle changes, the person's posture in the vertical direction does not change much from the camera's perspective. The difference HA between the two ankle points in the Y direction, the vertical distance H_L_AK from the left ankle to the left knee, and the vertical distance H_R_AK from the right ankle to the right knee joint are calculated simultaneously. The cross product is used to determine if LA intersects with the upper fence line L1, and if LK intersects with the upper fence line L1. If the line connecting the ankle points intersects with the upper fence line, or if the line connecting the two knee points intersects with the upper fence line, and the maximum value of H_L_AK and H_R_AK is taken and compared with HA, if HA is greater than the maximum value of H_L_AK and H_R_AK, it indicates that the pedestrian is lifting their foot to climb, and this is considered fence climbing. Figure 5 and Figure 6 As shown. This judgment logic also calculates adaptive judgment indicators based on the key point information of each human detection target, eliminating various custom thresholds and increasing the algorithm's applicability.

[0086] Step S111: Determine if the time since the last alarm is greater than 3 seconds. If yes, proceed to step S112; otherwise, proceed to step S113 to continuously track the human target.

[0087] Step S112: Issue an alarm and save the alarm image, location information, and alarm description, then return to step S103. The alarm method is as follows:

[0088] When the algorithm detects that a pedestrian is climbing the fence during their first crossing, it immediately generates an alarm and outputs an alarm image, such as... Figures 7-1 to 7-3 As shown, it outputs the alarm timestamp, location information, and alarm description. If the pedestrian is still crossing or climbing the fence after 3 seconds, it continues to output the alarm.

[0089] Step S113: Continuously track the human target.

[0090] This invention patent uses the ByteTrack target tracker to track pedestrian detection results, outputting the trajectory of the current frame and the lost trajectory retained by the tracker, and continuously updating the pedestrian trajectory. Tracking is initiated when a pedestrian enters a fenced area and stops when they leave. In target tracking, if a target exhibits fence-climbing behavior for two or more consecutive frames, it is considered that the target has participated in a fence-crossing event.

[0091] ByteTrack is a multi-target tracker that matches low-confidence detection bounding boxes with tracklets. This effectively reduces the problem of missed targets or rejections due to low detection confidence caused by issues such as target occlusion and motion blur, thereby reducing the frequency of IDs and tracklets and improving the performance of the MOT tracker.

[0092] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for recognizing fence climbing behavior, characterized in that: Includes the following steps: Step S1: Obtain the fence monitoring video stream to get the monitoring image. Mark the upper fence line and lower fence line in the monitoring image and obtain the coordinates of the upper fence line and lower fence line. Step S2: Detect pedestrians in the monitored images; Step S3: Identify the key points of the pedestrian's body, the coordinates of the key points, and the target detection box and coordinates of the target detection box; Step S4: Determine if the pedestrian has approached the fence. If so, proceed to the next step; otherwise, track the pedestrian. The first judgment threshold is obtained by using human key points, and the judgment distance between the pedestrian and the fence is obtained based on the target detection box coordinates of the pedestrian, the upper fence line coordinates, and the lower fence line coordinates. If any of the judgment distances is less than the first judgment threshold, it is determined that the pedestrian is close to the fence. The judgment spacing includes D1 and D2, which are obtained as follows: the midpoint of the line segment between the key point of the left hip and the key point of the right hip of the pedestrian is obtained as the center point P1 of the middle of the human body; the midpoint of the line segment between the two points of the lower edge of the target detection box of the pedestrian is obtained as the center point P2 of the bottom of the human body; the vertical distance D1 from P1 to the upper fence line L1 and the vertical distance D2 from P2 to the lower fence line L2 are obtained. The method for obtaining the first judgment threshold is as follows: calculate the distance between the ankle key point and the knee key point and the distance between the knee key point and the hip key point of the pedestrian's two legs respectively, and use the maximum value of the two distances as the first judgment threshold; Step S5: Determine if the pedestrian is climbing the fence. If yes, proceed to the next step; otherwise, track the pedestrian. Specifically, line segment LA is obtained by connecting the left and right ankle key points of the pedestrian, and line segment LK is obtained by connecting the left and right knee key points; the vertical distance between the left ankle key point and the left knee key point is H_L_AK, and the vertical distance between the right ankle key point and the right knee key point is H_R_AK; if at least one of LA and LK intersects with the upper fence line L1, and both H_L_AK and H_R_AK are less than the second judgment threshold, it indicates that the pedestrian has climbed; otherwise, there is no climbing behavior. The second judgment threshold is obtained as follows: the vertical distance between the key points of the pedestrian's left ankle and right ankle is the second judgment threshold; Step S6: Issue a climbing alarm.

2. The fence climbing behavior recognition method as described in claim 1, characterized in that: In step S3, the YOLOv7 object detection model is used to obtain the target detection box for pedestrians. If any of the following conditions are met, the pedestrians are filtered out: 1) The target detection box size for pedestrians is less than 70. 120 pixels; 2) The target detection box of the pedestrian does not intersect with the upper and lower fence lines.

3. The fence climbing behavior recognition method as described in claim 1, characterized in that: In step S6, the alarm method is to use the image of the pedestrian climbing as the alarm image, and output the alarm image, alarm timestamp, location information and alarm description to the alarm terminal.

4. The fence climbing behavior recognition method as described in claim 3, characterized in that: If, after 3 seconds of the alarm being triggered, the pedestrian is still detected climbing the fence, the alarm will continue to be triggered.

5. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-4.

Images