Guardrail climbing behavior detection method, device and electronic equipment

By using a combination of Yolov5 model and key point model in computer vision detection and combining multiple data enhancement technologies, the problem of high error rate in the behavior detection of pedestrians climbing over guardrails is solved, and higher detection accuracy and generalization capabilities are achieved , support effective law enforcement by traffic management departments.

CN115424342BActive Publication Date: 2024-11-19CHANGSHA HISENSE INTELLIGENT SYST RES INST CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211043990.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-08-30
Publication Date
2024-11-19
Estimated Expiration
2042-08-30

AI Technical Summary

Technical Problem

The existing computer vision detection of pedestrians' behavior overturning guardrails has the problem of high error rate. Traditional image algorithms are too simple to select pedestrian behavior characteristics and are difficult to deal with complex situations.

Method used

The guardrail overpass behavior detection method is adopted, including data input and preprocessing, target detection, key point detection and classification, target tracking and alarm. Target detection is carried out through the Yolov5 model, combined with key point model and classification detection model, and multiple data enhancements are used. Methods improve model generalization ability and judge the behavior of pedestrians over the guardrail.

Benefits of technology

It improves detection accuracy and generalization capabilities, can quickly detect crossing guardrails, reduce error rates, and provide accurate image information, which is conducive to law enforcement by traffic management departments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115424342B_ABST
    Figure CN115424342B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device and electronic device for detecting guardrail climbing behavior, including the following steps: data input and preprocessing; target detection; key point detection and classification, extracting multiple key human body point information including at least the feet of pedestrian targets from pedestrian targets within the detection frame coordinates through a key point model, inputting a classification detection model, combining the guardrail upper end line and the guardrail bottom line in the monitoring data to output classification information, and judging the behavior posture of pedestrians climbing over the guardrail: target tracking, tracking the characteristics of pedestrians determined to have climbed over the guardrail through a tracking model; alarm, alarming the pedestrians who are detected to have climbed over the guardrail for the first time. The present invention realizes the automatic detection of pedestrians climbing over guardrails on urban roads, can quickly detect the behavior of climbing over guardrails, obtain image information of pedestrians climbing over guardrails, and is conducive to better law enforcement by traffic management departments.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention discloses a guardrail climbing behavior detection method, device and electronic equipment, which belong to the specific application of computer vision field in traffic behavior monitoring. Background Art

[0002] Guardrails are safety protection isolation devices that separate sidewalks from motor vehicle lanes in road traffic. For safety reasons, the height of guardrails on traffic roads is generally not high. Pedestrians who lack traffic safety awareness are prone to dangerous behaviors such as climbing over guardrails and entering the area of ​​motor vehicles. Pedestrians climbing over the middle guardrails of urban roads not only violate relevant laws and regulations, but are also very likely to cause safety accidents.

[0003] The current mainstream methods for detecting pedestrians climbing over guardrails include intelligent alarm guardrails and using image recognition technology from traffic monitoring cameras to identify the behavior of climbing over guardrails.

[0004] The former is to improve the hardware of the guardrail itself, and set a sensor on the guardrail to detect climbing over. When the sensor senses that a pedestrian is climbing over, it takes certain measures to prevent the pedestrian from climbing over, or sounds an alarm, such as a municipal road guardrail disclosed in Chinese patent application number CN201922288682.1. Compared with general guardrails, this kind of guardrail has higher manufacturing cost, deployment cost and maintenance cost, and it cannot collect information about the person climbing over.

[0005] The latter is to use computer vision to perform image recognition on the behavior of climbing over guardrails in traffic monitoring images. With the popularization of urban surveillance cameras, it has become possible to automatically detect pedestrians climbing over the middle guardrails of urban roads. The behavior of pedestrians climbing over guardrails is detected through urban road monitoring video streams. Traditional image algorithms are often too simple in selecting pedestrian behavior features, resulting in a high error rate in actual use. Some methods based on deep learning have simple logic and are difficult to handle complex situations. Summary of the invention

[0006] The technical problem solved by the present invention is: in view of the high error rate of existing computer vision detection of pedestrian climbing over guardrails, a new guardrail climbing over behavior detection method, device and electronic equipment are provided.

[0007] The present invention is implemented by the following technical solutions:

[0008] The guardrail climbing behavior detection method includes the following steps:

[0009] S1, data input and preprocessing, obtaining input monitoring data containing guardrails, and marking the upper end line and the bottom line of the guardrails in the input monitoring data;

[0010] S2, target detection, detecting pedestrian targets in the monitoring data through a target detection model to obtain the coordinates of a detection frame containing the pedestrian targets;

[0011] S3, key point detection and classification, extract multiple key point information of the human body including at least the feet of the pedestrian target from the pedestrian target in the detection frame coordinates through the key point model, input the deep features of the key point information of the human body into the classification detection model, combine the upper end line and the bottom line of the guardrail in the monitoring data to output classification information, and judge the behavior posture of the pedestrian climbing over the guardrail through the following logic:

[0012] If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has made a climbing action of stepping over the guardrail with one leg, and whether the pedestrian has climbed over the guardrail is determined by the key point information of the human body; if the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has made a climbing action of straddling the guardrail, and whether the pedestrian has climbed over the guardrail is determined by the classification information;

[0013] S4, target tracking, tracking the characteristics of pedestrians who are determined to have climbed over the guardrail through the tracking model;

[0014] S5: Alarm: Alarm the pedestrian who is detected climbing over the guardrail for the first time.

[0015] Furthermore, the guardrail climbing behavior detection method of the present invention includes the following steps in the data input and preprocessing in step S1:

[0016] S11, selecting a real-time video or offline video or surveillance picture taken by a surveillance camera as input surveillance data;

[0017] S12, drawing the guardrail upper end line and the guardrail bottom line in the input monitoring data by drawing lines;

[0018] S13, resampling the size of the input monitoring data into a uniform size;

[0019] S14, performing image sharpening on the input monitoring data using data;

[0020] S15. Use data enhancement method to create a copy image for the input monitoring data.

[0021] Furthermore, in the guardrail climbing behavior detection method of the present invention, in the step S11, the video monitoring data in the input monitoring data is frame-extracted to obtain a monitoring picture.

[0022] Furthermore, the data enhancement method in the step S15 of the guardrail climbing behavior detection method of the present invention includes at least one of MixUp, Mosaic, Random Mask, MultiplyBrightness, and Flip.

[0023] Furthermore, in the guardrail climbing behavior detection method of the present invention, in step S2, the yolov5 target detection model is used to perform target detection.

[0024] Furthermore, in the step S3 of the guardrail climbing behavior detection method of the present invention, the key point model extracts 14 key points of the human body, including the top of the head, chin, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, and left ankle. The above 14 key points are predicted to obtain the coordinates of the 14 key points of the pedestrian.

[0025] Preferably, in the guardrail climbing over behavior detection method of the present invention, the key point model and classification detection model used in step S3 are the alphapose detection model with an added guardrail climbing over action classification branch, and the input of the classification branch is the human key point information extracted by the key point model. This information is input into a convolution block including three convolution structures and a BN structure, and a convolution operation is performed. The information is then input into a convolution layer to obtain a onehot encoding result, and finally the classification information is output through an average pooling layer.

[0026] Furthermore, in the guardrail climbing behavior detection method of the present invention, in step S3, the first threshold t1 corresponds to the height of the guardrail, and the center point of the line connecting the left shoulder and the right shoulder of the pedestrian target corresponds to the point p1 of the upper end line of the guardrail and the projection point p2 of the bottom line of the guardrail to obtain the distance between p1 and p2, and the distance is multiplied by the weight w1 to obtain the first threshold t1, and w1 is 0.7.

[0027] Furthermore, in the guardrail climbing behavior detection method of the present invention, in step S3,

[0028] If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, first obtain the coordinate f1 of the pedestrian's raised foot, calculate the distance difference d1 between f1 and p1, if the value of d1 is less than the second threshold t2, it is determined that the pedestrian target has climbed over the guardrail, if the value of d1 is greater than the second threshold t2, it is determined that the pedestrian target has not climbed over the guardrail, t2=0.15*t1;

[0029] If the height difference between the two feet of the pedestrian target is less than the first threshold value t1, the classification detection model is used to classify the pedestrian information to determine whether the pedestrian target has climbed over the guardrail.

[0030] Furthermore, in the guardrail climbing behavior detection method of the present invention, in step S4 and step S5, the tracking model determines whether the pedestrian target has been tracked by the tracking model.

[0031] If the pedestrian target has been tracked before, no new alarm information will be output;

[0032] If the pedestrian target has not been tracked before, the initial alarm information is output.

[0033] The present invention also provides a guardrail climbing behavior detection device, comprising:

[0034] An acquisition module is used to acquire input monitoring data containing guardrails, and mark the upper end line and the bottom line of the guardrails in the input monitoring data;

[0035] A detection module, used to detect pedestrian targets in the monitoring data and obtain the coordinates of the detection frame containing the pedestrian targets;

[0036] An analysis module is used to extract multiple key human body points including at least the feet of the pedestrian target within the detection frame coordinates, and output classification information based on the guardrail upper end line and guardrail bottom line in the monitoring data to determine the behavior posture of the pedestrian climbing over the guardrail;

[0037] A tracking module, used to track the characteristics of pedestrians who are determined to have climbed over the guardrail;

[0038] The alarm module is used to output an alarm message to the pedestrian who is detected climbing over the guardrail for the first time.

[0039] The present invention further provides an electronic device, comprising: a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the guardrail climbing behavior detection method of the present invention.

[0040] In the guardrail climbing behavior detection method of the present invention, the first step is data input and initialization. The user can input the video data or picture data to be detected, and offline video input and online video input are supported. After the user enters the data, the guardrail can be selected in the operation interface. The position of the guardrail is determined by two lines. The first line is the upper end line of the guardrail, and the second line is the bottom line of the guardrail. The user can complete the line drawing operation through the mouse. After that, the input data is processed, and the video data is framed according to the frame rate specified by the user. The pre-processed data is input into the detection model, and the deep features of the target are mined through deep feature extraction to determine the target type. If it is determined that the detected target is a pedestrian, the detection frame coordinates of the target are input into the key point and classification detection model. The key point and classification detection model detects the input pedestrian target, outputs 14 key point information and classification information of the pedestrian, and then performs logical judgment to determine whether the pedestrian has climbed over the guardrail. If it is determined that the pedestrian has climbed the guardrail, an alarm is issued and an alarm picture is generated.

[0041] The present invention has the following beneficial effects:

[0042] The present invention adopts the Yolov5 model as the pedestrian detection algorithm, and the algorithm has a fast reasoning speed; at the same time, a combination of multiple (MixUp, Mosaic, Random Mask, MultiplyBrightness, Flip, etc.) data enhancement methods is considered to further improve the generalization ability of the model; the present invention combines the human body posture estimation algorithm with the human body posture classification algorithm by coupling the key point model and the classification detection model, and adds a human body posture classification branch on the basis of the original single human body posture estimation algorithm, so that the algorithm outputs the human body posture and the category of the posture at the same time, which greatly enhances the detection accuracy of the calculation model.

[0043] In summary, the present invention uses surveillance cameras to realize automatic detection of pedestrians climbing over guardrails on urban roads, can quickly detect the behavior of climbing over guardrails, and obtain image information of pedestrians climbing over guardrails, which is conducive to better law enforcement by traffic management departments. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is a flow chart of the guardrail climbing behavior detection method of the present invention.

[0045] Figure 2 Schematic diagram of the obtained pedestrian target key points.

[0046] Figure 3 Schematic diagram of the classification branch for climbing over guardrail action in the detection model.

[0047] Figure 4 This is a schematic diagram of the climbing action of stepping over the guardrail with one leg.

[0048] Figure 5 Schematic diagram of the climbing action of riding over the guardrail.

[0049] Figure 6 It is a schematic diagram of a guardrail climbing behavior detection device of the present invention.

[0050] Figure 7 A schematic diagram of an electronic device for detecting guardrail climbing behavior according to the present invention. DETAILED DESCRIPTION

[0051] To further illustrate the technical means and effects of the present invention, the following is a detailed description in conjunction with the preferred embodiments of the present invention and the accompanying drawings.

[0052] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be more comprehensive and complete and will fully convey the concept of the example embodiments to those skilled in the art.

[0053] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0054] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0055] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0056] Embodiment 1

[0057] like Figure 1 As shown, the guardrail climbing behavior detection method of the present invention includes the following steps:

[0058] S1. Data input and preprocessing. Obtain input monitoring data containing guardrails, and mark the guardrail upper end line and guardrail bottom line in the input monitoring data.

[0059] S2. Target detection: Detect pedestrian targets in the monitoring data using a target detection model to obtain the coordinates of a detection frame containing the pedestrian targets.

[0060] This embodiment uses the yolov5 target detection model for target detection. First, read the parameter file, parse the yolov5 model, and load the trained yolov5 model. Detect pedestrian targets in the input monitoring data. After the model detects pedestrians, the obtained detection frame coordinates are input into the key point and classification detection model.

[0061] S3. Key point detection and classification. The key point model is used to extract multiple key point information of the pedestrian target within the detection frame coordinates, including at least the pedestrian target's feet. The deep features of the extracted key point information are input into the classification detection model. The classification information is output in combination with the guardrail upper end line and guardrail bottom line in the monitoring data. The behavior posture of the pedestrian climbing over the guardrail is judged through the following logic.

[0062] If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail with one leg. The key point information of the human body is used to determine whether the pedestrian has climbed over the guardrail. If the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail by straddling the guardrail. The classification information is used to determine whether the pedestrian has climbed over the guardrail.

[0063] S4. Target tracking: The features of the pedestrian who is determined to have climbed over the guardrail are tracked through the tracking model.

[0064] S5. Alarm. Alarm the pedestrian who is detected climbing over the guardrail for the first time. Alarm includes but is not limited to generating an alarm and saving the initial alarm image, outputting the alarm timestamp, point information, and alarm description.

[0065] The tracking model determines whether the pedestrian target has been tracked by the tracking model. For pedestrian targets that have been tracked before, no new alarm information will be output; for pedestrian targets that are tracked for the first time, the initial alarm information will be output and no repeated alarm will be issued.

[0066] The data input and preprocessing in step S1 specifically include the following sub-steps:

[0067] S11. Select real-time video or offline video or surveillance picture taken by the surveillance camera as input surveillance data. The scene targeted by this embodiment is an urban road scene, and the data used is the surveillance camera data of the urban road scene. To ensure that the algorithm can run normally, the user needs to correctly input the surveillance data, including real-time video, offline video, and surveillance pictures. If the user enters the wrong data format, the algorithm will generate a warning message and prompt the correct data format. In addition, the video surveillance data is framed to obtain the surveillance picture. The user can specify the frame extraction frequency of the video, such as extracting 1 frame for every 5 frames of the video file for detection.

[0068] S12. Draw the guardrail upper end line and guardrail bottom line in the input monitoring data by drawing lines. The user sets the target guardrail obtained in the input monitoring data, and supports drawing the guardrail that the user wants to detect on the video screen by drawing lines with the mouse. The position of the guardrail is determined by two lines, the first line is the guardrail upper end line, and the second line is the guardrail bottom line.

[0069] S13, resampling the size of the input monitoring data to a uniform size. First, the size of the input monitoring data is resampled to a size that meets the algorithm requirements. In this embodiment, the input monitoring data is resampled to a size of 640*640 to reduce the occupancy of the memory and video memory.

[0070] S14, performing image sharpening on the input monitoring data.

[0071] S15. Use data enhancement methods to create a copy image of the input monitoring data. Use data enhancement methods to create a copy image, and perform data enhancement operations such as MixUp, Mosaic, Random Mask, MultiplyBrightness, and Flip on the image copy.

[0072] Further, such as Figure 2 , Figure 4 and Figure 5 As shown, in step S3, this embodiment extracts 14 key points of the human body from the pedestrian target through the key point model, including the top of the head, chin, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, and left ankle. The above 14 key points are predicted to obtain the coordinates of the 14 key points of the pedestrian.

[0073] Specifically, the key point model and classification detection model used in step S3 of this embodiment is an alphapose detection model with an added classification branch for climbing over guardrails. By adding a classification branch for pedestrian climbing over guardrails, it is possible to better cope with the omission of key frames caused by video frame extraction, which results in the failure of key point discrimination logic to detect and judge the key points and action classification of pedestrians climbing over guardrails. Figure 3 As shown in the figure, the input of the classification branch is the human key point information extracted by the alphapose detection model. This information is input into a convolution block containing three convolution structures and a BN structure for convolution operation. After that, the information is input into a convolution layer to obtain the onehot encoding result, and finally the classification information is output through an average pooling layer.

[0074] See again Figure 1 , after obtaining the key point information and classification information of the pedestrian target. For the two classic postures of pedestrians when climbing over the guardrail, key points and classification are used to make judgments respectively. First, it is judged whether the height difference between the two feet of the pedestrian target is greater than the first threshold t1. If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has stepped over the guardrail with one leg, such as Figure 4 As shown in FIG, in this case, the key point data of the human body is used to determine whether the pedestrian has climbed over the guardrail. If the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian may have ridden over the guardrail. Figure 5 As shown, in this case, the classification information is used to determine whether the pedestrian has climbed over the guardrail.

[0075] Due to the influence of the lens in the surveillance camera, the height of the guardrail appears larger when it is near and smaller when it is far away in the picture. In view of this situation, the first threshold t1 uses a dynamic method to correspond to the height of the guardrail at different positions. First, the center point of the line connecting the left shoulder and the right shoulder of the pedestrian target corresponds to the point p1 of the upper end line of the guardrail and the projection point p2 of the bottom line of the guardrail. The x-coordinates of the left shoulder and the right shoulder of the pedestrian target are added to the center x-coordinate obtained by dividing by 2, and substituted into the equations of the upper end line of the guardrail and the bottom line of the guardrail to calculate the corresponding points p1 and p2. The distance between point p1 and point p2 is calculated, and the distance is multiplied by a weight w1 to obtain the first threshold t1. In this embodiment, the weight w1 is set to 0.7.

[0076] Furthermore, if the height difference between the two feet of the pedestrian target is greater than the first threshold t1, the next round of judgment is performed.

[0077] First, the coordinate f1 of the pedestrian's raised foot is obtained, and then the distance difference d1 between f1 and the point p1 on the upper end line of the guardrail is calculated. If the value of d1 is less than the second threshold t2, it is determined that the pedestrian has climbed over the guardrail. In this embodiment, the value of the second threshold t2 is equal to the first threshold t1*0.15. At this time, the facial features of the pedestrian target are input into the tracking model to determine whether the person has been tracked by the tracking model. If the person has been tracked before, it means that the person has been detected to have climbed over the guardrail, and an alarm has been issued and an alarm message has been generated, and no new alarm message is output. If the person has not been tracked before, it means that this person is detected to have climbed over the guardrail for the first time. At this time, an alarm is issued and the initial alarm picture is saved, and the alarm timestamp, point information, and alarm description are output to facilitate the staff to quickly locate the pedestrian target.

[0078] If the value of d1 is greater than the second threshold value t2, it means that although the pedestrian has raised his legs, he is far enough away from the guardrail, so it is judged that the pedestrian has not climbed over the guardrail. The characteristics of the pedestrian are input into the tracking model to determine whether the person has been tracked by the tracking model. If the person has been tracked before, continue to track the pedestrian to avoid re-reading alarms.

[0079] If the height difference between the two feet of the pedestrian target is less than the first threshold t1, the classification information output by the pedestrian target is judged. If the classification result output by the alphapose detection model is climbing over the guardrail, it is determined that the pedestrian has climbed over the guardrail. At this time, the facial features of the pedestrian are input into the tracking model to determine whether the person has been tracked by the tracking model. If the person has been tracked before, it means that the person has been detected to have climbed over the guardrail before, and an alarm has been issued and an alarm message has been generated, and no new alarm message is output. If the person has not been tracked before, it means that the person has been detected to have climbed over the guardrail for the first time. At this time, an alarm is issued and the initial alarm picture is saved, and the alarm timestamp, point information, and alarm description are output to facilitate the staff to quickly locate the pedestrian target. If the classification result output by the classification model is not climbing over the guardrail, it is determined that the pedestrian has not climbed over the guardrail. If the person has been tracked before, continue to track the pedestrian to avoid re-reading alarms.

[0080] Specifically, the tracking model of this embodiment uses the DeepSort model for target tracking. First, read the parameter file, parse the tracking model, and load the trained tracking model. Use the features of the pedestrians who were determined to have climbed over the guardrail in the previous step as input, assign a unique ID to each pedestrian who has climbed over the guardrail, and stop tracking when the pedestrian leaves the monitoring screen.

[0081] Embodiment 2

[0082] This embodiment provides a device for detecting guardrail climbing behavior. Figure 6 As shown, it includes an acquisition module 10, a detection module 20, an analysis module 30, a tracking module 40 and an alarm module 50.

[0083] The acquisition module 10 is used to acquire input monitoring data containing guardrails and mark the upper end line and the bottom line of the guardrails in the input monitoring data. The specific steps are as follows:

[0084] S11, select the real-time video or offline video or monitoring picture taken by the monitoring camera as the input monitoring data. Extract frames from the video monitoring data to obtain the monitoring picture. The user can specify the frequency of video frame extraction, such as extracting 1 frame out of every 5 frames of the video file for detection.

[0085] S12. Draw the guardrail upper end line and guardrail bottom line in the input monitoring data by drawing lines. The user sets the target guardrail obtained in the input monitoring data, and supports drawing the guardrail that the user wants to detect on the video screen by drawing lines with the mouse. The position of the guardrail is determined by two lines, the first line is the guardrail upper end line, and the second line is the guardrail bottom line.

[0086] S13, resampling the size of the input monitoring data to a uniform size. First, the size of the input monitoring data is resampled to a size that meets the algorithm requirements. In this embodiment, the input monitoring data is resampled to a size of 640*640 to reduce the occupancy of the memory and video memory.

[0087] S14, performing image sharpening on the input monitoring data.

[0088] S15. Use data enhancement methods to create a copy image of the input monitoring data. Use data enhancement methods to create a copy image, and perform data enhancement operations such as MixUp, Mosaic, Random Mask, MultiplyBrightness, and Flip on the image copy.

[0089] The detection module 20 is used to detect pedestrian targets in the monitoring data and obtain the coordinates of the detection frame containing the pedestrian targets. After the pedestrian is detected, the obtained detection frame coordinates are input into the analysis module 30.

[0090] Among them, the analysis module 30 is used to extract multiple human body key point information including at least the feet of the pedestrian target within the detection frame coordinates, and output classification information based on the guardrail upper line and guardrail bottom line in the monitoring data to determine the behavior posture of the pedestrian climbing over the guardrail.

[0091] The specific judgment logic is as follows:

[0092] If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail with one leg. The key point information of the human body is used to determine whether the pedestrian has climbed over the guardrail. If the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail by straddling the guardrail. The classification information is used to determine whether the pedestrian has climbed over the guardrail.

[0093] The tracking module 40 is used to track the characteristics of pedestrians who are determined to have climbed over the guardrail.

[0094] The alarm module 50 is used to output alarm information to the pedestrian who is detected climbing over the guardrail for the first time. The alarm includes but is not limited to generating an alarm and saving the initial alarm picture, outputting the alarm timestamp, point information, and alarm description.

[0095] The tracking module 40 determines whether the pedestrian target has been tracked. For pedestrian targets that have been tracked before, no new alarm information is output; for pedestrian targets that are tracked for the first time, the initial alarm information is output without repeated alarm.

[0096] Please refer to the above method for the specific implementation process of each module, which will not be repeated here. It should be understood that the division of the above functional modules is only a division of logical functions, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. At the same time, the above integrated units can be implemented in the form of hardware or software functional units.

[0097] Embodiment 3

[0098] This embodiment provides an electronic device for detecting guardrail climbing behavior, such as Figure 7 As shown, it includes a memory 100 and a processor 200, the memory 100 stores a computer program, and when the computer program is executed by the processor 200, the processor 200 performs the following steps.

[0099] S1. Data input and preprocessing. Obtain input monitoring data containing guardrails, and mark the guardrail upper end line and guardrail bottom line in the input monitoring data.

[0100] S2. Target detection: Detect pedestrian targets in the monitoring data using a target detection model to obtain the coordinates of a detection frame containing the pedestrian targets.

[0101] This embodiment uses the yolov5 target detection model for target detection. First, read the parameter file, parse the yolov5 model, and load the trained yolov5 model. Detect pedestrian targets in the input monitoring data. After the model detects pedestrians, the obtained detection frame coordinates are input into the key point and classification detection model.

[0102] S3. Key point detection and classification. The key point model is used to extract multiple key point information of the pedestrian target within the detection frame coordinates, including at least the pedestrian target's feet. The deep features of the extracted key point information are input into the classification detection model. The classification information is output in combination with the guardrail upper end line and guardrail bottom line in the monitoring data. The behavior posture of the pedestrian climbing over the guardrail is judged through the following logic.

[0103] If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail with one leg. The key point information of the human body is used to determine whether the pedestrian has climbed over the guardrail. If the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has climbed over the guardrail by straddling the guardrail. The classification information is used to determine whether the pedestrian has climbed over the guardrail.

[0104] S4. Target tracking: The features of the pedestrian who is determined to have climbed over the guardrail are tracked through the tracking model.

[0105] S5. Alarm. Alarm the pedestrian who is detected climbing over the guardrail for the first time. Alarm includes but is not limited to generating an alarm and saving the initial alarm image, outputting the alarm timestamp, point information, and alarm description.

[0106] The tracking model determines whether the pedestrian target has been tracked by the tracking model. For pedestrian targets that have been tracked before, no new alarm information will be output; for pedestrian targets that are tracked for the first time, the initial alarm information will be output and no repeated alarm will be issued.

[0107] The data input and preprocessing in step S1 specifically include the following sub-steps:

[0108] S11. Select real-time video or offline video or surveillance picture taken by the surveillance camera as input surveillance data. The scene targeted by this embodiment is an urban road scene, and the data used is the surveillance camera data of the urban road scene. To ensure that the algorithm can run normally, the user needs to correctly input the surveillance data, including real-time video, offline video, and surveillance pictures. If the user enters the wrong data format, the algorithm will generate a warning message and prompt the correct data format. In addition, the video surveillance data is framed to obtain the surveillance picture. The user can specify the frame extraction frequency of the video, such as extracting 1 frame for every 5 frames of the video file for detection.

[0109] S12. Draw the guardrail upper end line and guardrail bottom line in the input monitoring data by drawing lines. The user sets the target guardrail obtained in the input monitoring data, and supports drawing the guardrail that the user wants to detect on the video screen by drawing lines with the mouse. The position of the guardrail is determined by two lines, the first line is the guardrail upper end line, and the second line is the guardrail bottom line.

[0110] S13, resampling the size of the input monitoring data to a uniform size. First, the size of the input monitoring data is resampled to a size that meets the algorithm requirements. In this embodiment, the input monitoring data is resampled to a size of 640*640 to reduce the occupancy of the memory and video memory.

[0111] S14, performing image sharpening on the input monitoring data.

[0112] S15. Use data enhancement methods to create a copy image of the input monitoring data. Use data enhancement methods to create a copy image, and perform data enhancement operations such as MixUp, Mosaic, Random Mask, MultiplyBrightness, and Flip on the image copy.

[0113] In the specific implementation process, the computer program corresponding to the method of the present invention is loaded into the memory, so that the processor of the electronic terminal calls the computer program to implement the above process.

[0114] The electronic device also includes: a communication interface for communicating with external devices and performing data interactive transmission.

[0115] The memory may include a high-speed RAM memory, for example, at least one disk memory.

[0116] If the memory, processor and communication interface are implemented independently, the memory, processor and communication interface can be connected to each other through a bus and communicate with each other. The bus can be an industrial standard architecture bus, a peripheral device interconnection bus or an extended industrial standard architecture bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0117] Optionally, in a specific implementation, if the memory, processor and communication interface are integrated on a chip, the memory, processor and communication interface can communicate with each other through an internal interface.

[0118] For the specific implementation process of each step, please refer to the description of the above method.

[0119] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.

[0120] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

[0121] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.

Claims

1. A method for detecting guardrail climbing behavior, characterized in that The steps include: S1. Obtain input monitoring data containing guardrails, and mark the upper end line and the bottom line of the guardrails in the input monitoring data; S2. Detect pedestrian targets in the monitoring data through the target detection model to obtain the coordinates of the detection frame containing the pedestrian targets; S3. Extract multiple key points of the pedestrian target within the detection frame coordinates, including at least the foot of the pedestrian target, through the key point model, and input them into the classification detection model. Combined with the guardrail upper end line and guardrail bottom line in the monitoring data, the classification information is output, and the behavior posture of the pedestrian climbing over the guardrail is judged by the following logic: If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has made a climbing action of stepping over the guardrail with one leg, and whether the pedestrian has climbed over the guardrail is determined by the key point information of the human body; if the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has made a climbing action of straddling the guardrail, and whether the pedestrian has climbed over the guardrail is determined by the classification information; S4, tracking the characteristics of the pedestrian determined to have climbed over the guardrail through the tracking model; S5. Alarm the pedestrian who is detected climbing over the guardrail for the first time.

2. According to the guardrail climbing behavior detection method of claim 1, the step S1 comprises the following steps: S11, selecting a real-time video or offline video or surveillance picture taken by a surveillance camera as input surveillance data. If video surveillance data is selected as input surveillance data, extracting frames from the video surveillance data to obtain a surveillance picture; S12, drawing the guardrail upper end line and the guardrail bottom line in the input monitoring data by drawing lines; S13, resampling the size of the input monitoring data into a uniform size; S14, performing image sharpening on the input monitoring data using data; S15. Use data enhancement method to create a copy image for the input monitoring data.

3. According to the guardrail climbing behavior detection method of claim 2, the data enhancement method in step S15 includes at least one of MixUp, Mosaic, Random Mask, MultiplyBrightness, and Flip.

4. According to the guardrail climbing behavior detection method of claim 1, in the step S3, the key point model extracts 14 key points of the human body, including the top of the head, chin, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, and left ankle. The above 14 key points are predicted to obtain the coordinates of the 14 key points of the pedestrian.

5. According to the guardrail climbing over behavior detection method according to claim 4, the key point model and classification detection model used in step S3 are the alphapose detection model with an added guardrail climbing over action classification branch, the input of the classification branch is the human key point information extracted by the key point model, the information is input into a convolution block including three convolution structures and a BN structure, a convolution operation is performed, and then the information is input into a convolution layer to obtain a onehot encoding result, and finally the classification information is output through an average pooling layer.

6. According to the guardrail climbing behavior detection method of claim 1, in the step S3, the first threshold t1 corresponds to the height of the guardrail, the center point of the line connecting the left shoulder and the right shoulder of the pedestrian target corresponds to the point p1 of the upper end line of the guardrail and the projection point p2 of the bottom line of the guardrail, the distance between p1 and p2 is obtained, and the distance is multiplied by the weight w1 to obtain the first threshold t1, and w1 is 0.

7.

7. The guardrail climbing behavior detection method according to claim 6, wherein in step S3, If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, first obtain the coordinate f1 of the pedestrian's raised foot, calculate the distance difference d1 between f1 and p1, if the value of d1 is less than the second threshold t2, it is determined that the pedestrian target has climbed over the guardrail, if the value of d1 is greater than the second threshold t2, it is determined that the pedestrian target has not climbed over the guardrail, t2=0.15*t1; If the height difference between the two feet of the pedestrian target is less than the first threshold value t1, the classification detection model is used to classify the pedestrian information to determine whether the pedestrian target has climbed over the guardrail.

8. The guardrail climbing behavior detection method according to claim 1, in step S4 and step S5, the tracking model determines whether the pedestrian target has been tracked by the tracking model, If the pedestrian target has been tracked before, no new alarm information will be output; If the pedestrian target has not been tracked before, the initial alarm information is output.

9. A device for detecting guardrail climbing behavior, characterized in that include: An acquisition module is used to acquire input monitoring data containing guardrails, and mark the upper end line and the bottom line of the guardrails in the input monitoring data; The detection module is used to detect pedestrian targets in the monitoring data through the target detection model and obtain the coordinates of the detection frame containing the pedestrian targets; The analysis module is used to extract multiple key points of the human body including at least the feet of the pedestrian target from the pedestrian target in the detection frame coordinates through the key point model, and input the information into the classification detection model. The classification information is output in combination with the guardrail upper end line and guardrail bottom line in the monitoring data, and the behavior posture of the pedestrian climbing over the guardrail is judged by the following logic: If the height difference between the two feet of the pedestrian target is greater than the first threshold t1, it is suspected that the pedestrian has made a climbing action of stepping over the guardrail with one leg, and whether the pedestrian has climbed over the guardrail is determined by the key point information of the human body; if the height difference between the two feet of the pedestrian target is less than the first threshold t1, it is suspected that the pedestrian has made a climbing action of straddling the guardrail, and whether the pedestrian has climbed over the guardrail is determined by the classification information; A tracking module, used to track the characteristics of pedestrians determined to have climbed over the guardrail through a tracking model; The alarm module is used to alarm the pedestrian who is detected climbing over the guardrail for the first time.

10. An electronic device, characterized in that include: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 8.