Guardrail and isolation fence crossing behavior detection method based on visual calculation

Abstract

The invention discloses a visual calculation-based guardrail and isolation fence crossing behavior detection method. A YOLOv8 model is adopted as a pedestrian detection algorithm, a DeepLabv3 model is adopted as a guardrail and isolation fence position detection algorithm, and an HRNet model is adopted as a human body key point detection algorithm. The method for detecting the behavior of crossing over the guardrail and the isolation fence based on visual calculation is wide in actual city application range, can be applied to scenes such as road guardrails, isolation fences, lake fences, community fences, cultural and sports activity guardrails, external traffic alert and city constraint governance, reduces the possibility of accidents and the possibility of suspicious personnel entering, and improves the safety of people. The city operation efficiency and the city management quality are improved; meanwhile, by customizing the threshold value of the human body posture, compared with a traditional behavior recognition model, the algorithm complexity is reduced, the calculation amount is smaller, and implementation and popularization are easy.

Description

Technical Field

[0001] The invention belongs to the field of image recognition, and in particular relates to a method for detecting behaviors of climbing over guardrails and isolation barriers based on visual computing. Background Art

[0002] In modern urban construction, municipal guardrails and isolation barriers are mainly used to provide safety protection and guide the orderly passage of people and vehicles. The practice of climbing over guardrails and isolation barriers itself has risks to personal life and property, urban property risks, and urban operation and maintenance risks, which are important matters. Accidents caused by illegal climbing often occur in daily life. Climbing over guardrails and isolation barriers without safety protection measures is prone to accidents, which may cause minor injuries such as falls or serious injuries to major parts. In some specific situations, climbing over behavior may even endanger life or cause significant losses to social industries. The detection, discovery and governance of such behavior are part of urban operation management, but in actual work, due to the limitations of time and space conditions and high labor costs, it is often difficult to detect and stop.

[0003] In power construction sites, many workers have poor safety awareness and often climb over guardrails and isolation barriers instead of entering and exiting according to the preset entrances and exits of guardrails and isolation barriers, which poses a great safety hazard to construction work; in tourist attractions, people often ignore the safety reminders of the scenic area, climb over guardrails and isolation barriers to pick up mobile phones and accidentally fall to their death; in the traffic monitoring system, people often climb over guardrails and isolation barriers in traffic and cross the road for various reasons, etc. Such behaviors are sudden and occur infrequently, but once they occur, they are likely to affect people's lives and cause serious economic and property losses.

[0004] Since the 20th century, with the popularization of various electronic detection equipment, many places have been equipped with surveillance cameras. These surveillance cameras that almost fully cover the city have also become an important part of the city's security system. The surveillance video screen is huge and continuous. The traditional behavior detection method relies heavily on human resources. A large amount of manpower must be invested in a long time to highly concentrate on viewing the monitor to identify small probability accidents. At present, the monitoring system of key areas such as scenic spots, construction sites, traffic scenes, schools, and factories still uses traditional supervision methods. The supervision cost is high, the efficiency is low, and it is impossible to fully supervise all areas. In addition, some abnormal events are often missed due to fatigue of monitoring personnel or other situations, which brings great harm to society.

[0005] Therefore, it is very necessary to use artificial intelligence technology represented by machine learning, especially deep learning, to analyze and identify video images of climbing over guardrails and isolation fences captured by surveillance videos, and to study a detection method for dangerous behaviors of climbing over guardrails and isolation fences. Real-time monitoring is performed to determine whether there is any behavior of climbing over guardrails and isolation fences in the area. If so, an alarm signal is immediately issued to avoid the occurrence of safety accidents. The existing common detection methods for climbing over guardrails and isolation fences are mainly divided into two categories. One is to use a behavior detection model to directly detect the behavior of climbing over guardrails and isolation fences. The model comprehensively judges the type of continuous behavior by learning the time domain information and spatial domain information of the video. This method has high accuracy but large calculation amount and consumes a lot of memory resources. The second category generates the motion trajectory of pedestrians through continuous target detection, and determines whether pedestrians have climbed over guardrails and isolation fences through the motion trajectory. That is, when it is found that the motion trajectory of pedestrians intersects with guardrails and isolation fences, it is determined that the pedestrian has climbed over the guardrails and isolation fences. However, this method can only judge after the pedestrian has completed climbing over the guardrail or isolation barrier. It is unable to identify the pedestrian in the process of climbing over the guardrail or isolation barrier and stop the behavior of climbing over the guardrail or isolation barrier in time. Summary of the invention

[0006] The purpose of the present invention is to provide a method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing, which can improve the detection speed and real-time performance, and improve the accuracy of recognition by judging the posture of human body based on the key point detection model and identifying and alarming the key frames of the key actions of pedestrians when climbing over guardrails and isolation barriers. The purpose of the present invention is achieved by the following technical solutions:

[0007] A method for judging the behavior of climbing over guardrails and isolation barriers based on visual computing includes the following steps:

[0008] S1. Obtain the specified surveillance camera image and obtain the detection area of ​​the guardrail and isolation fence based on historical monitoring data;

[0009] S2. Detect pedestrians based on the real-time monitoring image and obtain the target frame of the pedestrians;

[0010] S3. Perform semantic segmentation of guardrails and isolation fences based on real-time monitoring images to obtain segmentation results of guardrail and isolation fence positions;

[0011] S4. calculating the intersection of the two regions based on the target frame of the pedestrian and the segmented positions of the guardrail and the isolation fence, and determining the positional relationship;

[0012] S5. Further detect key points of pedestrians, calculate key point geometric parameters, and compare them within a predetermined parameter range to determine human body posture;

[0013] S6. Continue to process and detect the above steps on the surveillance video stream. When the behavior of climbing over the guardrail or isolation fence is detected, compare it with the previous detection result. If the behavior occurs to a new target, an alarm is triggered.

[0014] Furthermore, in step S1, the method for obtaining the detection area of ​​the guardrail and isolation fence is as follows:

[0015] Determine the cameras that need to be inspected and the positions of the guardrails and isolation barriers in the cameras; collect historical monitoring image data, which requires different weather, different lighting conditions and different shooting angles, at least including day, night, sunny, rainy, cloudy, snowy, etc., and the camera has been rotated; perform denoising and image enhancement operations on the collected image data, mark the positions of the guardrails and isolation barriers in the image, and scale the image to a preset size, and uniformly input it into the semantic segmentation model for training.

[0016] As a further improvement, in step S2, the method for obtaining the pedestrian target frame is as follows:

[0017] The YOLOv8 model is used to detect pedestrians and obtain the target frame positions of multiple pedestrians in the monitoring screen. If one of the following situations occurs, the pedestrian is filtered out:

[0018] 1) The target frame size of the pedestrian is less than 50*100 pixels;

[0019] 2) If there is an overlapping detection result of "big frame inside small frame" in the target frame of the pedestrian, the small frame is ignored and the large frame is retained.

[0020] As a further improvement, in step S3, the method for obtaining the segmentation result of the guardrail and isolation fence position is as follows:

[0021] Use the trained DeepLabv3 model to perform semantic segmentation on the image and extract irregular shaped areas representing the positions of guardrails and isolation fences. If multiple guardrails and isolation fences are segmented, they are considered to be different guardrails and isolation fences and saved separately; if a "hole" appears in the segmented area, it is considered to be a guardrail or isolation fence area, and it is filled to make the area complete and correct the segmentation result.

[0022] As a further improvement, in step S4, the method for obtaining the positional relationship between the pedestrian and the guardrail or isolation barrier is as follows:

[0023] The target frame position of each pedestrian is taken out from the target detection results, and its target frame and the guardrail and isolation fence area in the semantic segmentation detection results are regarded as two polygons. It is calculated whether the two have an intersection. If so, it is considered that the pedestrian is close to the guardrail and isolation fence and may climb over the guardrail and isolation fence. Further detection of human key points is required to make a judgment.

[0024] As a further improvement, in step S5, the method for obtaining the key points of the human body is as follows:

[0025] The HRNet model is used to detect key points of pedestrians near guardrails and isolation barriers. The key points of concern are the left ankle key point, the right ankle key point, the left knee key point, the right knee key point, the left hip key point, the right hip key point, the left shoulder key point, and the right shoulder key point.

[0026] As a further improvement, in step S5, the method for obtaining the geometric parameters of the key points is as follows:

[0027] The first geometric parameter θ1 is the angle formed by the left ankle key point, the left knee key point, and the left hip key point. The second geometric parameter θ2 is the angle formed by the right ankle key point, the right knee key point, and the right hip key point. The third geometric parameter θ3 is the angle formed by the midpoint of the line segment connecting the left and right hip key points, the midpoint of the line segment connecting the left and right shoulder key points, and any point on the plumb line passing through the midpoint of the line segment connecting the left and right hip key points. The sizes of these three angles are all between 0 and 180 degrees.

[0028] As a further improvement, in step S5, the method for obtaining the judgment result of the behavior of climbing over the guardrail or isolation fence by using the human body posture is as follows:

[0029] After calculating these three geometric parameters, they are compared with the predetermined threshold range, which is divided into a first-class posture threshold judgment standard and a second-class posture threshold judgment standard. The first-class posture considers the key posture of the human body when it is about to climb over the guardrail or isolation fence, and the judgment standard is that θ1 or θ2 is less than 120 degrees. The second-class posture considers the key posture of the human body when climbing over the guardrail or isolation fence, and the judgment standard is that θ1 or θ2 is less than 150 degrees and θ3 is greater than 45 degrees. If the currently obtained geometric parameters meet the parameter range of any human posture, it is considered that climbing over the guardrail or isolation fence has occurred, otherwise it has not occurred.

[0030] As a further improvement, in step S6, the method for determining whether to issue an alarm is as follows:

[0031] The real-time behavior of pedestrians is judged. If the behavior of climbing over guardrails and isolation barriers is detected, it is recorded as state 1. If there is no state 1 in the current detection result, it means that there is no climbing behavior and directly enters the next detection; if there is state 1, the previous detection result is obtained. If the previous detection result is not 1, it means that the current detection result is the behavior of a new target and an alarm is triggered; if the previous detection result is 1, the positions of the pedestrian prediction boxes with state 1 in the two detection results are compared in turn to determine whether the continuous behavior of the same tourist is detected; if it is found to be the same target, directly enter the next detection, otherwise an alarm is triggered.

[0032] As a further improvement, in step S6, the content of the alarm is a picture of the detected behavior of climbing over the guardrail or isolation fence, the picture timestamp, the camera information to which the picture belongs, and a description of the alarm content.

[0033] A device for detecting climbing over guardrails and isolation barriers, characterized in that it includes a memory and a processor coupled to each other, and the processor is used to execute program instructions stored in the memory to implement the method for detecting climbing over guardrails and isolation barriers based on visual computing as described in any one of claims 1 to 6.

[0034] A computer-readable storage medium having program instructions stored thereon, characterized in that when the program instructions are executed by a processor, the method for detecting behaviors of climbing over guardrails and isolation barriers based on visual computing as described in any one of claims 1 to 6 is implemented.

[0035] The beneficial effects of the present invention are:

[0036] The present invention judges the positional relationship between pedestrians and guardrails and isolation barriers based on the position detection results of the two, only judges pedestrians close to guardrails and isolation barriers, and then performs human posture judgment based on the key point detection model, and identifies and alarms the key frames of key actions when pedestrians climb over guardrails and isolation barriers. Compared with the method based on behavior recognition model, the algorithm complexity is reduced, the calculation amount is smaller, the speed is faster, and it is more real-time. Compared with the method based on motion trajectory judgment, the recognition accuracy is improved, and the behavior can be detected and alarmed when it occurs, which is less likely to be missed or late. This method can be applied to scenes such as road guardrails, isolation barriers, bridge guardrails, isolation barriers, lake fences, community fences, etc., to reduce the possibility of accidents and the entry of suspicious persons. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the process of the present invention;

[0038] Figure 2 A schematic diagram of key points of the human body detected by the method of the present invention;

[0039] Figure 3 A schematic diagram of the human body posture and key point geometric parameters when a human body is about to climb over a guardrail or isolation fence proposed by the method of the present invention;

[0040] Figure 4 A schematic diagram of human body posture and key point geometric parameters when a human body is climbing over a guardrail or isolation fence, as proposed by the method of the present invention;

[0041] Figure 5 This is a schematic diagram of the results of detecting the positions of guardrails and isolation fences using the method of the present invention;

[0042] Figure 6This is a schematic diagram of the results of the method of the present invention detecting the behavior of pedestrians climbing over guardrails and isolation barriers. DETAILED DESCRIPTION

[0043] In order to better understand the technical content of the present invention, specific embodiments are given and described as follows in conjunction with the accompanying drawings.

[0044] like Figure 1 As shown, the vision-based detection method for climbing over guardrails and isolation barriers provided by the present invention comprises the following steps:

[0045] S1. Obtain the specified surveillance camera image and obtain the detection area of ​​the guardrail and isolation fence based on historical monitoring data;

[0046] S2. Detect pedestrians based on the real-time monitoring image and obtain the target frame of the pedestrians;

[0047] S3. Perform semantic segmentation of guardrails and isolation fences based on real-time monitoring images to obtain segmentation results of guardrail and isolation fence positions;

[0048] S4. calculating the intersection of the two regions based on the target frame of the pedestrian and the segmented positions of the guardrail and the isolation fence, and determining the positional relationship;

[0049] S5. Further detect key points of pedestrians, calculate key point geometric parameters, and compare them within a predetermined parameter range to determine human body posture;

[0050] S6. Continue to process and detect the above steps on the surveillance video stream. When the behavior of climbing over the guardrail or isolation fence is detected, compare it with the previous detection result. If the behavior occurs to a new target, an alarm is triggered.

[0051] The present invention is further described in detail below with reference to a practical case:

[0052] First, select the camera that needs to be detected and specify the guardrails and isolation fences that need to be detected, and collect the historical monitoring data of the camera, including data for night, early morning, morning, noon, afternoon, dusk, and sunny, cloudy, rainy, and snowy days. After denoising and image enhancement, mark the guardrails and isolation fences in the image, and finally scale it to a uniform size and feed it into the DeepLabv3 model for training.

[0053] Secondly, obtain the real-time video stream. The scenario here is the protective fence on the river bank in the scenic area. It will determine whether the picture frame of the video stream can be obtained normally. If it can be obtained, proceed to the next step, otherwise an error prompt will be given.

[0054] The acquired real-time video stream uses the YOLOv8 target detection model to detect pedestrians. Through preprocessing of the data set, a trained YOLOv8 target detection model is obtained, which can detect tourists in the scenic area. By extracting key frames and inputting them into the model for detection, the rectangular frame coordinates and confidence of all pedestrians in the picture are obtained. If a pedestrian is detected, proceed to the next step, otherwise continue to extract the next frame for judgment.

[0055] Among them, human targets need to be filtered during detection. If the target frame size is less than 50*100 pixels, it will be filtered, that is, the subsequent steps will not be performed. If the target frame overlaps with the "large frame inside the small frame", the small frame will be ignored and the large frame will be retained.

[0056] Next, the trained DeepLabv3 model is used to perform semantic segmentation on the image and extract irregular shape areas representing the locations of guardrails and isolation barriers, such as Figure 5 If multiple guardrails or isolation fences are segmented, they are considered as different guardrails or isolation fences and saved separately; if a "hole" appears in the segmented area, it is considered as a guardrail or isolation fence area, and it is filled to make the area complete and correct the segmentation result.

[0057] Subsequently, multithreading is used to determine the positional relationship of multiple pedestrians. The target frame position of each pedestrian is taken out respectively, and its target frame and the guardrail and isolation fence area in the semantic segmentation detection result are regarded as two polygons. Whether the two intersect is calculated. If so, it is considered that the pedestrian is close to the guardrail and isolation fence and may climb over the guardrail and isolation fence, and the next step is carried out. Otherwise, the target is filtered, that is, the subsequent steps are not carried out.

[0058] Next, the HRNet model is used to detect the key points of pedestrians. After obtaining the input human body detection frame information, the human body image in the corresponding frame is input into the key point detection model for inference prediction to obtain the coordinate information of 8 key points. The 8 key points are the left ankle key point, the right ankle key point, the left knee key point, the right knee key point, the left hip key point, the right hip key point, the left shoulder key point, and the right shoulder key point. Figure 2 shown.

[0059] After obtaining the key point information, three geometric parameters are calculated. The first geometric parameter θ1 is the angle formed by the left ankle key point, the left knee key point, and the left hip key point. The second geometric parameter θ2 is the angle formed by the right ankle key point, the right knee key point, and the right hip key point. The third geometric parameter θ3 is the angle formed by the midpoint of the line segment connecting the left and right hip key points, the midpoint of the line segment connecting the left and right shoulder key points, and any point on the plumb line passing through the midpoint of the line segment connecting the left and right hip key points. The sizes of these three angles are all between 0 and 180 degrees.

[0060] After calculating these three geometric parameters, they are compared with the predetermined threshold range, which is divided into the first type of posture threshold judgment standard and the second type of posture threshold judgment standard. The first type of posture considers the key posture of the human body when it is about to climb over the guardrail or isolation fence, such as Figure 3 As shown in , the judgment standard is that θ1 or θ2 is less than 120 degrees. The second type of posture considers the key posture of the human body when climbing over the guardrail or isolation fence, such as Figure 4 As shown in the figure, the judgment criteria are that θ1 or θ2 is less than 150 degrees and θ3 is greater than 45 degrees. If the currently obtained geometric parameters meet the parameter range of any human posture, it is considered that the guardrail or isolation fence has been climbed over, otherwise it has not occurred. The climbing behavior is recorded as state 1, and the target frame coordinates are saved.

[0061] If the current detection result does not have status 1, it means there is no climbing behavior and directly enters the detection of the next frame; if it has status 1, obtain the previous detection result. If the previous detection result does not have status 1, it means that a new target is detected this time and an alarm is triggered; if the previous detection result has status 1, compare the distance between the target frames of the two detection results in turn. If it is less than 10 pixels, it is considered to be the same tourist, indicating that his continuous behavior has been continuously detected; if it is found to be the same target, directly enter the detection of the next frame, otherwise an alarm is triggered.

[0062] Finally, the system outputs an alarm, which contains a picture of the person climbing over the guardrail or isolation fence, the picture timestamp, the camera information to which the picture belongs, and a description of the alarm content. If the surveillance camera is turned off, the judgment stops, otherwise the camera's surveillance images will continue to be obtained, and this detection process will be repeated.

[0063] Although the present invention has been disclosed as above with preferred embodiments, it is not intended to limit the present invention. A person with ordinary knowledge in the technical field to which the present invention belongs may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be determined by the definition of the claims.

Claims

1. A method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing, comprising the following steps: S1. Obtain the specified surveillance camera image and obtain the detection area of ​​the guardrail and isolation fence based on historical monitoring data; S2. Detect pedestrians based on the real-time monitoring image and obtain the target frame of the pedestrians; S3. Perform semantic segmentation of guardrails and isolation fences based on real-time monitoring images to obtain segmentation results of guardrail and isolation fence positions; S4. calculating the intersection of the two regions based on the target frame of the pedestrian and the segmented positions of the guardrail and the isolation fence, and determining the positional relationship; S5. Further detect key points of pedestrians, calculate key point geometric parameters, and compare them within a predetermined parameter range to determine human body posture; S6. Continue to process and detect the above steps on the surveillance video stream. When the behavior of climbing over the guardrail or isolation fence is detected, compare it with the previous detection result. If the behavior occurs to a new target, an alarm is triggered.

2. The method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing according to claim 1, Features: In step S1, by determining the camera to be detected and the position of the guardrail and isolation fence in the camera; collecting historical monitoring image data, performing denoising and image enhancement operations, marking the position of the guardrail and isolation fence in the image, and scaling the image to a preset size, inputting it into the semantic segmentation model for training, thereby obtaining the detection area of ​​the guardrail and isolation fence.

3. The method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing according to claim 1, Features: In step S2, the YOLOv8 model is used to perform target detection on pedestrians, and the target frame positions of multiple pedestrians in the monitoring picture are obtained, thereby obtaining the target frames of the pedestrians.

4. The method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing according to claim 1, Features: In step S3, the DeepLabv3 model is used to perform semantic segmentation on the guardrail and isolation fence, obtain the irregular shapes of the guardrail and isolation fence in the monitoring picture, and obtain the segmentation results of the positions of the guardrail and isolation fence.

5. The method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing according to claim 1, Features: In step S4, the target frame position of each pedestrian is taken out from the target detection results of multiple pedestrians, the target frame and the semantic segmentation area are regarded as two polygons, and it is calculated whether the two have an intersection. If so, it is regarded that the pedestrian is close to the guardrail or isolation barrier and may climb over the guardrail or isolation barrier, and further detection is needed to determine.

6. The method for detecting guardrail and isolation fence climbing behavior based on visual computing according to claim 1, Features: In step S5, the HRNet model is used to detect key points of pedestrians close to the guardrail and the isolation barrier, and three key point geometric parameters are calculated. The first geometric parameter is the angle formed by the left ankle, the left knee, and the left hip. The second geometric parameter is the angle formed by the right ankle, the right knee, and the right hip. The third geometric parameter is the angle formed by the midpoint of the left and right hips, the midpoint of the left and right shoulders, and any point on the plumb line passing through the midpoint of the left and right hips. These three geometric parameters are compared with a predetermined threshold range to determine whether the human body posture at this time meets the two predetermined human body postures for climbing over the guardrail and the isolation barrier. If so, it is considered that climbing over the guardrail and the isolation barrier has occurred.

7. The method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing according to claim 1, Features: In step S6, the real-time behavior of the pedestrian is judged. If the behavior of climbing over the guardrail or isolation fence is detected, it is recorded as state 1. If the current detection result does not have state 1, it means that there is no climbing behavior, and the next detection is directly entered; if there is state 1, the previous detection result is obtained. If the previous detection result does not have 1, it means that the current detection result is a behavior of a new target, and an alarm is triggered; If the last detection result is 1, the positions of the pedestrian prediction boxes with status 1 in the two detection results are compared in turn to determine whether the continuous behavior of the same tourist is detected; If the same target is found, it will directly proceed to the next detection, otherwise it will trigger an alarm.

8. A detection device for climbing over guardrails and isolation barriers, It is characterized in that It comprises a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing as described in any one of claims 1 to 6.

9. A computer-readable storage medium having program instructions stored thereon, It is characterized in that When the program instructions are executed by the processor, the method for detecting the behavior of climbing over guardrails and isolation barriers based on visual computing as described in any one of claims 1 to 6 is implemented.

Images