Escalator pedestrian converse driving detection method and system based on human skeleton

Abstract

The invention relates to the technical field of computers, and discloses a human skeleton-based escalator pedestrian retrograde driving detection method, which comprises the following steps of: 1, acquiring a video stream of an escalator through a camera, and establishing a pedestrian bbox frame and a plurality of skeleton points after a pedestrian enters an escalator area; 2, judging whether the upper body of the pedestrian is shielded or not through the skeleton points; if the pedestrians are shielded all the time in the escalator area, the pedestrians are filtered; if the pedestrians are not shielded all the time in the escalator area, the next step is carried out; 3, judging whether the track direction of the midpoint of the top of the bbox is consistent with the ladder walking direction or not; if not, judging that the vehicle runs in a retrograde manner, and giving an alarm; and if so, no retrograde motion exists. According to the pedestrian retrogradation detection method, the accuracy of escalator retrogradation detection can be improved when the pedestrians are partially shielded. Meanwhile, the invention further provides an escalator pedestrian converse driving detection system based on the human skeleton.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and system for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton. Background Art

[0002] With the development of artificial intelligence, human body detection methods have been widely used in security, video surveillance, abnormal behavior analysis and other fields. Human body detection methods can detect one or more human bodies in the image captured by the camera equipment, and mark the position of each human body, so as to obtain the movement trajectory of each human body. Especially in places where people gather in high-speed railways and airports, there are great safety hazards in going against the flow on escalators. Traditional reverse detection is generally carried out by human body detection. However, since there are many pedestrians on escalators and there are many cases of occlusion, the bbox frame of human body detection cannot judge whether it is blocked. Secondly, the midpoint of the bottom of the bbox frame is used as the point for judging the position of the pedestrian. When the legs of the pedestrian are blocked, the size of the bbox frame will change, which is easy to cause misjudgment, thereby increasing the number of misjudgments and the workload of the staff. Therefore, it is necessary to improve the accuracy of reverse judgment to reduce the workload of the staff.

[0003] CN202210807278.4 discloses a passenger retrograde detection method, device and computer storage medium, including: obtaining a video stream containing the escalator; setting at least one reference line for the escalator in the image frame of the video stream; performing position detection on the passengers on the escalator to obtain position change information of each passenger; analyzing the direction of travel of the escalator based on the position change information of the passenger; for each passenger, based on the position change information of the passenger and the direction of travel of the escalator, performing a retrograde crossing line detection step of determining whether the passenger retrogradely crosses the reference line, and in response to determining that the passenger retrogradely crosses the reference line, determining that the passenger retrogrades. The method obtains the position of the passenger through a detection frame, compares it with the direction of travel of the escalator, and then cooperates with detecting whether it crosses the reference line to realize retrograde detection, but uses the bottom midpoint of the detection frame as the position point. When the passenger is blocked, the size of the detection frame will change, which is easy to cause misjudgment.

[0004] CN202110586379 discloses a method and system for judging retrograde based on image recognition, comprising the following steps: S1, obtaining a video stream in the area to be detected, and performing RIO processing on the video stream to obtain data of the area to be detected; S2, preprocessing the data of the area to be detected to obtain multiple frames of images; S3, inputting all the frame images into a pre-trained VGG convolutional image face recognition model for image recognition to obtain the orientation of pedestrians in all the frame images; S4, judging whether there is retrograde in the area to be detected based on the orientation of pedestrians in all the frame images. The retrograde judgment method judges whether the pedestrian is retrograde by identifying the orientation of the pedestrian, but when the pedestrian is sideways or turns his head, it is easy to be identified as retrograde, resulting in misjudgment.

[0005] The technical problem to be solved by the present invention is: how to improve the accuracy of escalator reverse detection when pedestrians are partially blocked. Summary of the invention

[0006] The main purpose of the present invention is to provide an escalator pedestrian reverse detection method based on human skeleton. It judges whether the pedestrian is blocked by detecting the pedestrian skeleton points. If the range of pedestrian blockage is too large, the pedestrian is filtered to avoid affecting the detection of other pedestrians. Secondly, the position point of the pedestrian is determined by the midpoint of the top of the bbox frame, which avoids the change of the bbox frame size, thereby affecting the judgment of the pedestrian position point and improving the accuracy of reverse detection.

[0007] At the same time, a human skeleton-based escalator pedestrian reverse detection system is also provided.

[0008] To achieve the above objectives, the technical solutions adopted in this application are:

[0009] A method for detecting pedestrians walking in reverse on an escalator based on human skeleton comprises the following steps:

[0010] Step 1: Obtain the video stream of the escalator through the camera. After the pedestrian enters the escalator area, establish the pedestrian bbox frame and multiple skeleton points;

[0011] Step 2: Determine whether the upper body of the pedestrian is blocked by the skeleton points; if the pedestrian is always blocked in the escalator area, filter the pedestrian; if the pedestrian is not always blocked in the escalator area, proceed to the next step;

[0012] Step 3: Determine whether the trajectory direction of the midpoint at the top of the bbox is consistent with the direction of the ladder; if not, it is determined to be reverse and an alarm is issued; if consistent, there is no reverse.

[0013] Preferably, the multiple skeleton points include points corresponding to joints on the limbs, hip joints, shoulder joints, and the head.

[0014] Preferably, the step 1 is specifically as follows: acquiring an image of the escalator through a camera; determining the ROI area of ​​the escalator through image semantic segmentation; determining the escalator direction through an optical flow method; and establishing a pedestrian bbox frame and human skeleton points when a pedestrian enters the ROI area.

[0015] Preferably, the method further includes step 4: determining whether the relative positions of the left and right skeleton points of the pedestrian change; if a change occurs, determining that the pedestrian has turned around, and if the pedestrian is in the ROI area at this time, issuing an alarm; if there is no change, continuing detection.

[0016] Preferably, in step 2, whether the pedestrian is blocked is determined by detecting the confidence of the pedestrian skeleton points; when the confidence is less than 0.7, it is determined to be blocked.

[0017] Preferably, in step 3, the trajectory direction of the midpoint of the top of the bbox box is obtained by the NVDCF algorithm.

[0018] Preferably, a pedestrian bbox, multiple skeleton points and the confidence of the skeleton points are obtained through a posture estimation algorithm.

[0019] It should be noted that:

[0020] Bbox box: It is the abbreviation of BoundingBox, which refers to a directed rectangular box used to represent the position and size of an object in target detection.

[0021] ROI area: region of interest, is the area to be processed in the image processing, which is outlined in the form of a box, circle, ellipse, irregular polygon, etc. This area is the focus of image analysis. By delineating this area for further processing, the processing time can be reduced and the accuracy can be improved.

[0022] At the same time, a human skeleton-based escalator pedestrian reverse detection system is also provided, including a camera and a server; the server includes the following modules:

[0023] Pedestrian recognition module: used to establish pedestrian bbox and multiple skeleton points after pedestrians enter the escalator area;

[0024] Occlusion judgment module: used to judge whether the upper body of the pedestrian is occluded through the skeleton points; if it is occluded, the pedestrian is filtered out;

[0025] Retrograde judgment module: used to judge whether the trajectory direction of the top midpoint of the bbox box is consistent with the direction of the ladder. If not, it is judged as retrograde and an alarm is issued.

[0026] Preferably, the server further includes:

[0027] Turn detection module: determines whether the relative position of the left and right skeletal points of the pedestrian has changed; if there is a change, it is determined that the pedestrian has turned, and an alarm is issued when the pedestrian is in the escalator area.

[0028] Compared with the prior art, this solution has the following beneficial effects:

[0029] The escalator pedestrian reverse detection method of this scheme detects the obscured parts of pedestrians through skeleton points. When the upper body of the pedestrian is obscured, the pedestrian is filtered out, such as: a baby in an embrace, only the hands or feet are not obscured, etc., to avoid these situations affecting the detection of other pedestrians or detecting these situations and causing false alarms. In addition, the midpoint trajectory direction of the top of the bbox frame and the direction of the escalator are judged. Since the legs of pedestrians are easily obscured in the escalator, the bottom of the bbox frame moves up, so the midpoint of the top of the bbox frame is used as the position point for judgment. By detecting the obscured conditions of pedestrians and the direction of the position points, the misjudgment caused by the obscured pedestrians is reduced, and the accuracy of reverse detection is improved.

[0030] Secondly, by judging whether the relative positions of the left and right skeleton points have changed, it is determined whether the pedestrian has turned around and detected whether the pedestrian is walking in the opposite direction. Combined with the pedestrian's moving direction and turning situation, the accuracy of pedestrian walking in the opposite direction is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the escalator pedestrian reverse detection method of Example 1;

[0032] Figure 2 This is a block diagram of an escalator pedestrian reverse detection system according to Example 2. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. The components of the present application implemented as described and shown in the drawings here can be arranged and designed in various different configurations.

[0034] Example 1

[0035] refer to Figure 1 , a method for detecting pedestrians walking in reverse on an escalator based on human skeleton, comprising the following steps:

[0036] Step 1: Obtain the video stream of the escalator through the camera. After the pedestrian enters the escalator area, establish the pedestrian bbox frame and multiple skeleton points;

[0037] Preferably, the step 1 is specifically as follows: acquiring an image of the escalator through a camera; determining the ROI area of ​​the escalator through image semantic segmentation; determining the escalator direction through an optical flow method; and establishing a pedestrian bbox frame and human skeleton points when a pedestrian enters the ROI area.

[0038] In this embodiment, the camera is placed diagonally above the escalator and can capture the entire or part of the escalator path. After acquiring the video stream of the escalator, the ROI area (detection area) is determined for the video image through image semantic segmentation. In this solution, the image semantic segmentation uses the FCN-HRNetW18 algorithm to divide the left baffle, right baffle, path, and floor plate of the escalator area in the video image.

[0039] For the usage of FCN-HRNetW18 algorithm, please refer to HRNet-FCN in the semantic segmentation model of product matrix in https: / / github.com / PaddlePaddle / PaddleSeg?tab=readme-ov-file. We will not describe it in detail here.

[0040] After obtaining the roi area of ​​the escalator, it is also necessary to determine the direction of the escalator. The direction of the escalator can be determined manually or by using the optical flow method. In this embodiment, the optical flow method is used to determine the direction of the escalator, specifically, the sparse LK optical flow algorithm is used to determine the direction of the escalator. According to the escalator scene and demand analysis, since the sparse optical flow is fast and relatively stable, it is more appropriate to use the sparse LK optical flow algorithm to determine the direction of the escalator. The Lucas Kanade sparse optical flow algorithm is implemented in opencv, and the function is cv2.calcOpticalFlowPyrLK(), which can be called directly.

[0041] The usage of the sparse LK optical flow algorithm can be found at https: / / blog.csdn.net / qq_41368247 / article / details / 82562165, which will not be described in detail here.

[0042] The ROI area and direction of the escalator only need to be set when the escalator is used for the first time or the escalator is changed during detection. Moreover, the ROI area and direction of the escalator are determined respectively by image semantic segmentation and optical flow method. After obtaining the video stream of the camera, the ROI area and direction of the escalator can be automatically determined without manual configuration by the staff, thus reducing the workload of the staff and improving efficiency.

[0043] After determining the ROI area and the direction of the stairs, when a pedestrian enters the ROI area, a bbox frame and skeleton points will be established for the pedestrian. The bbox frame is determined by the skeleton points, and the bbox frame and skeleton points are combined to analyze whether the pedestrian is walking in the opposite direction. By combining multiple detections, the accuracy of the reverse detection is improved.

[0044] Step 2: Determine whether the upper body of the pedestrian is blocked by the skeleton points; if the pedestrian is always blocked in the escalator area, filter the pedestrian; if the pedestrian is not always blocked in the escalator area, proceed to the next step;

[0045] Preferably, in step 2, whether the pedestrian is blocked is determined by detecting the confidence of the pedestrian skeleton points; when the confidence is less than 0.7, it is determined to be blocked.

[0046] In this embodiment, the skeleton points are used to determine which parts of the pedestrian are blocked. When the lower body (hip joint and legs) of the pedestrian is blocked, it does not affect the subsequent bbox frame detection, and the next step of detection can be performed. However, when the upper body (shoulders and head) of the pedestrian is blocked, since the pedestrian stands from top to bottom in the escalator, in other words, most of the pedestrian's area is blocked. At this time, the hands, elbows or part of the legs may be exposed. In this case, it is necessary to filter these parts that may affect other pedestrians, and avoid the subsequent detection of the bbox frame. The bbox frame is established for only these parts, thereby causing a misjudgment of retrograde. Therefore, only the upper body of the pedestrian needs to be blocked. If the pedestrian is not blocked within a certain period of time, the pedestrian will continue to be detected.

[0047] The occlusion is judged by the confidence of the skeleton points. When the confidence of the skeleton points is less than 0.7, it is occluded. Furthermore, when the confidence of the skeleton points of the upper body is less than 0.7, the retrograde detection of this pedestrian is filtered to avoid misjudgment or affect the detection of other pedestrians.

[0048] Step 3: Determine whether the trajectory direction of the midpoint at the top of the bbox is consistent with the direction of the ladder; if not, it is determined to be reverse and an alarm is issued; if consistent, there is no reverse.

[0049] In this embodiment, the midpoint of the top of the bbox frame is used as the detection point of the pedestrian position. The advantages are: first, this method can avoid the pedestrian's legs being blocked, causing the bbox frame to move upward and mistakenly believe that the pedestrian is moving upward, thereby misjudging the pedestrian's retrograde movement; second, under normal circumstances, the upper body of the pedestrian will not be blocked, and the midpoint of the top of the bbox frame will not change abnormally. Combined with the occlusion detection in step 2, the accuracy of the detection of the top midpoint of the bbox frame can be guaranteed. The trajectory direction of the top midpoint of the bbox frame can be adjusted according to actual needs by setting the distance between the front and rear positions of the top midpoint of the bbox frame to adjust the measured trajectory direction, and then compared with the direction of the stairs, which can avoid the influence of special situations such as pedestrians suddenly squatting or jumping, and improve the accuracy of retrograde detection.

[0050] Preferably, in step 3, the trajectory direction of the midpoint of the top of the bbox box is obtained by the NVDCF algorithm.

[0051] In this embodiment, the NVDCF algorithm can be used to track the target trajectory to ensure that the same person is tracked each time. For example, each pedestrian has an ID. The ID of pedestrian A is A, and the ID of pedestrian B is B. The NVDCF algorithm will keep tracking pedestrian A, and the ID of pedestrian A will always be A and will not change to B. In addition, the NVDCF algorithm can filter out the impact of special situations such as pedestrians squatting or jumping suddenly on the trajectory direction detection of the top midpoint of the bbox frame.

[0052] For the usage of the NVDCF algorithm, please refer to https: / / blog.csdn.net / weixin_41817841 / article / details / 133977715, which will not be described in detail here.

[0053] Preferably, the method further includes step 4: determining whether the relative positions of the left and right skeleton points of the pedestrian change; if a change occurs, determining that the pedestrian has turned around, and if the pedestrian is in the ROI area at this time, issuing an alarm; if there is no change, continuing detection.

[0054] In this embodiment, after determining the trajectory direction and the staircase direction, it is also possible to determine whether the pedestrian turns around to further enhance the accuracy of the reverse detection.

[0055] The method for judging whether a pedestrian turns is to judge whether the relative positions of the left and right skeleton points of the pedestrian change. In other words, whether the positions of the left and right skeleton points are swapped. The current posture estimation algorithm can detect the difference between the left and right skeleton points of the pedestrian. Therefore, when the relative positions of the left and right skeleton points of the pedestrian change, it can be judged that the pedestrian turns. Then, if the pedestrian is in the ROI area at this time, an alarm is issued. If the pedestrian has walked out of the ROI area, he has left the escalator range. At this time, the pedestrian turning will not be dangerous, and there is no need to issue an alarm.

[0056] Preferably, the multiple skeleton points include points corresponding to joints on the limbs, hip joints, shoulder joints, and the head.

[0057] Preferably, a pedestrian bbox, multiple skeleton points and the confidence of the skeleton points are obtained through a posture estimation algorithm.

[0058] The pose estimation algorithm may be an algorithm that can obtain skeleton points, such as YOLO-Pose, Kapao, LitePose, or light-weight OpenPose. In this embodiment, the YOLO-Pose algorithm is used to obtain the pedestrian bbox, multiple skeleton points, and the confidence of the skeleton points.

[0059] For the usage of the YOLO-Pose algorithm, please refer to https: / / blog.csdn.net / wqthaha / artic le / details / 125249721, which will not be described in detail here.

[0060] The advantages of this method are:

[0061] 1. By adding pedestrian occlusion detection, the accuracy of using the top midpoint of the bbox box as the pedestrian position point is improved to avoid abnormal changes in the top midpoint of the bbox box that affect the judgment of the pedestrian trajectory direction. Combining pedestrian occlusion detection and using the top midpoint of the bbox box as the pedestrian position point greatly improves the accuracy of retrograde detection.

[0062] 2. By adding pedestrian turn detection, the reliability of reverse detection is further enhanced. Combined with the bbox frame and turn detection, the bbox frame detection will show that the pedestrian is walking down the escalator with the opposite direction of the escalator, which cannot be judged. Therefore, judging that the pedestrian turns around can determine that this situation is reverse. The accuracy of reverse detection is greatly improved through the bbox frame and turn detection.

[0063] Example 2

[0064] refer to Figure 2 , an escalator pedestrian reverse detection system based on human skeleton, including a camera and a server; the server includes the following modules:

[0065] Pedestrian recognition module: used to establish pedestrian bbox and multiple skeleton points after pedestrians enter the escalator area;

[0066] Occlusion judgment module: used to judge whether the upper body of the pedestrian is occluded through the skeleton points; if it is occluded, the pedestrian is filtered out;

[0067] Retrograde judgment module: used to judge whether the trajectory direction of the top midpoint of the bbox box is consistent with the direction of the ladder. If not, it is judged as retrograde and an alarm is issued.

[0068] Preferably, the server further includes:

[0069] Turn detection module: determines whether the relative position of the left and right skeletal points of the pedestrian has changed; if there is a change, it is determined that the pedestrian has turned, and an alarm is issued when the pedestrian is in the escalator area.

[0070] In this embodiment, the specific working process of the escalator pedestrian reverse detection system is as follows:

[0071] The camera obtains the video stream image of the escalator. The staff first determines the ROI area and the direction of the escalator. The staff can set it manually or add image semantic segmentation and optical flow method to the retrograde detection system to determine the ROI area and the direction of the escalator. When a pedestrian enters the escalator area, the pedestrian recognition module establishes a bbox frame and multiple skeleton points for the pedestrian entering; and transmits the bbox frame and multiple skeleton points to the occlusion judgment module. The occlusion judgment module determines the occluded part of the pedestrian through the skeleton points. When the upper body of the pedestrian is occluded, the pedestrian is filtered. When the upper body of the pedestrian is not occluded, the pedestrian is re-detected. The occlusion judgment module sends the bbox frame of the pedestrian whose upper body is not occluded to the retrograde judgment module. The retrograde judgment module judges the trajectory direction of the top midpoint of the bbox frame and the direction of the stairs. If the trajectory direction is consistent with the direction of the stairs, there is no retrograde movement; if the trajectory direction is inconsistent with the direction of the stairs, the bbox frame and skeleton points of the pedestrian are sent to the turn judgment module. The turn judgment module judges the relative positions of the left and right skeleton points of the pedestrian. If the relative positions of the left and right skeleton points change, it is judged that the pedestrian has turned. If the pedestrian is within the ROI area, the pedestrian is retrograde and an alarm is issued; if the relative positions of the left and right skeleton points do not change, the pedestrian is not retrograde.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton, characterized in that: The following steps are involved: Step 1: Obtain the video stream of the escalator through the camera. After the pedestrian enters the escalator area, establish the pedestrian bbox frame and multiple skeleton points; Step 2: Determine whether the upper body of the pedestrian is blocked by the skeleton points; If the pedestrian is always blocked in the escalator area, filter the pedestrian; if the pedestrian is not always blocked in the escalator area, proceed to the next step; Step 3: Determine whether the trajectory direction of the midpoint at the top of the bbox is consistent with the direction of the ladder; if not, it is determined to be reverse and an alarm is issued; if consistent, there is no reverse.

2. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 1, characterized in that: The multiple skeleton points include points corresponding to joints on limbs, hip joints, shoulder joints, and the head.

3. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 1, characterized in that: The step 1 is specifically as follows: obtaining an image of the escalator through a camera; determining the ROI area of ​​the escalator through image semantic segmentation; determining the escalator direction through the optical flow method; and establishing the pedestrian bbox frame and human skeleton points when a pedestrian enters the ROI area.

4. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 3 is characterized in that: The method also includes step 4: determining whether the relative positions of the left and right skeleton points of the pedestrian change; if there is a change, determining that the pedestrian has turned around, and if the pedestrian is in the ROI area at this time, issuing an alarm; if there is no change, continuing detection.

5. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 1, characterized in that: In the step 2, whether the pedestrian is blocked is determined by detecting the confidence of the pedestrian skeleton points; when the confidence is less than 0.7, it is determined to be blocked.

6. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 1, characterized in that: In step 3, the trajectory direction of the midpoint of the top of the bbox frame is obtained by the NVDCF algorithm.

7. The method for detecting pedestrians walking in the opposite direction on an escalator based on human skeleton according to claim 1, characterized in that: The pedestrian bbox, multiple skeleton points and the confidence of the skeleton points are obtained through the posture estimation algorithm.

8. An escalator pedestrian reverse detection system based on human skeleton, characterized in that: It includes a camera and a server; the server includes the following modules: Pedestrian recognition module: used to establish pedestrian bbox and multiple skeleton points after pedestrians enter the escalator area; Occlusion judgment module: used to judge whether the upper body of the pedestrian is blocked by skeleton points; If it is blocked, filter the pedestrian; Retrograde judgment module: used to judge whether the trajectory direction of the top midpoint of the bbox box is consistent with the direction of the ladder. If not, it is judged as retrograde and an alarm is issued.

9. The escalator pedestrian reverse detection system based on human skeleton according to claim 8, characterized in that: The server also includes: Turn detection module: determines whether the relative position of the left and right skeletal points of the pedestrian has changed; if there is a change, it is determined that the pedestrian has turned, and an alarm is issued when the pedestrian is in the escalator area.

Images