Escalator pedestrian reverse direction detection method and system based on human skeleton

Abstract

The application relates to the computer technical field and discloses a human body skeleton-based escalator pedestrian reverse running detection method, which comprises the following steps: step 1: obtaining a video stream of an escalator through a camera; after a pedestrian enters an escalator area, a pedestrian bbox frame and a plurality of skeleton points are established; step 2: judging whether the upper body of the pedestrian is shielded through the skeleton points; if the pedestrian is shielded all the time in the escalator area, the pedestrian is filtered; if the pedestrian is not shielded all the time in the escalator area, the next step is performed; step 3: judging whether the trajectory direction of the midpoint of the top of the bbox frame is consistent with the escalator running direction; if not, the pedestrian is determined to run reversely, and an alarm is sent; if yes, the pedestrian does not run reversely. The present pedestrian reverse running detection method can improve the accuracy of escalator reverse running detection when the pedestrian is partially shielded. Meanwhile, a human body skeleton-based escalator pedestrian reverse running detection system is provided.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and system for detecting pedestrians going against the flow on escalators based on the human skeleton. Background Technology

[0002] With the development of artificial intelligence, human detection methods have been widely applied in security, video surveillance, and abnormal behavior analysis. Human detection methods can detect one or more human bodies in images captured by camera equipment and mark the position of each body, thus obtaining the movement trajectory of each person. Especially in crowded places like high-speed rail stations and airports, walking against the flow of traffic on escalators poses a significant safety hazard. Traditional methods for detecting this movement generally use human body detection. However, due to the large number of pedestrians on escalators and the numerous instances of obstruction, the bounding box (bbox) of human body detection cannot accurately determine whether there is obstruction. Furthermore, using the midpoint of the bottom of the bbox as the point for determining the pedestrian's position means that the bbox size changes when the pedestrian's legs are obstructed, easily leading to false positives and increasing the workload of staff. Therefore, it is necessary to improve the accuracy of reverse movement detection to reduce the workload of staff.

[0003] CN202210807278.4 discloses a method, apparatus, and computer storage medium for detecting passengers traveling in the wrong direction. The method includes: acquiring a video stream containing an escalator; setting at least one reference line for the escalator in image frames of the video stream; detecting the position of passengers on the escalator to obtain position change information for each passenger; analyzing the escalator's direction of travel based on the passenger's position change information; and for each passenger, based on the passenger's position change information and the escalator's direction of travel, performing a step to determine whether the passenger has traveled in the wrong direction by crossing the reference line, and determining that the passenger has traveled in the wrong direction in response to determining that the passenger has crossed the reference line in the wrong direction. This method obtains the passenger's position through a detection frame and compares it with the escalator's direction of travel, then detects whether the passenger has crossed the reference line to achieve reverse travel detection. However, using the bottom midpoint of the detection frame as the position point means that the size of the detection frame will change when the passenger is obstructed, which can easily lead to misjudgment.

[0004] CN202110586379 discloses a method and system for determining inverse pedestrian movement based on image recognition, comprising the following steps: S1, acquiring a video stream within a region to be detected and performing RIO processing on the video stream to obtain data of the region to be detected; S2, preprocessing the data of the region to be detected to obtain multiple frames of images; S3, inputting all the frames of images into a pre-trained VGG convolutional image face recognition model for image recognition to obtain the facing direction of pedestrians in all the frames of images; S4, determining whether there is inverse pedestrian movement within the region to be detected based on the facing direction of pedestrians in all the frames of images. This method determines whether a pedestrian is moving in the wrong direction by recognizing their facing direction; however, when a pedestrian is to the side or turning their head, it is easily identified as moving in the wrong direction, leading to misjudgment.

[0005] The technical problem that this invention aims to solve is: how to improve the accuracy of escalator reverse movement detection when pedestrians are partially obscured. Summary of the Invention

[0006] The main objective of this invention is to provide a method for detecting reverse pedestrian movement on escalators based on human skeletons. By detecting pedestrian skeleton points, it is determined whether a pedestrian is obscured. If the obscured area is too large, the pedestrian is filtered out to avoid affecting the detection of other pedestrians. Secondly, the pedestrian's position is determined by the midpoint of the top of the bounding box, which avoids the impact of changes in the size of the bounding box on the determination of the pedestrian's position, thereby improving the accuracy of reverse movement detection.

[0007] In addition, a pedestrian reverse movement detection system for escalators based on the human skeleton is also provided.

[0008] To achieve the above objectives, the technical solution adopted in this application is as follows:

[0009] A method for detecting pedestrians going against the flow on escalators based on the human skeleton includes the following steps:

[0010] Step 1: Acquire the video stream of the escalator through the camera. After the pedestrian enters the escalator area, create the pedestrian's bounding box and multiple skeleton points;

[0011] Step 2: Determine whether the upper body of the pedestrian is obscured by the skeletal points; if the pedestrian is always obscured in the escalator area, filter out this pedestrian; if the pedestrian is not always obscured in the escalator area, proceed to the next step.

[0012] Step 3: Determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator; if they are inconsistent, it is determined to be reverse movement and an alarm is issued; if they are consistent, there is no reverse movement.

[0013] Preferably, the plurality of skeletal points include points corresponding to joints on the limbs, hip joints, shoulder joints, and head.

[0014] Preferably, step 1 specifically involves: acquiring an image of the escalator through a camera; determining the ROI region of the escalator through image semantic segmentation; determining the escalator's direction of travel using optical flow; and establishing a pedestrian bounding box and human skeleton points after a pedestrian enters the ROI region.

[0015] Preferably, the method further includes step 4: determining whether the relative positions of the left and right skeletal points of the pedestrian have changed; if a change occurs, the pedestrian is determined to turn around, and if the pedestrian is in the ROI area at this time, an alarm is issued; if there is no change, the detection continues.

[0016] Preferably, in step 2, the confidence level of the pedestrian skeleton points is used to determine whether the occlusion is detected; when the confidence level is less than 0.7, it is determined to be occluded.

[0017] Preferably, in step 3, the trajectory direction of the top midpoint of the bbox is obtained by the NVDCF algorithm.

[0018] Preferably, the pedestrian bounding box, multiple skeletal points, and the confidence level of the skeletal points are obtained through a pose estimation algorithm.

[0019] It should be noted that:

[0020] bbox: short for BoundingBox, refers to the directed rectangular box used in object detection to represent the position and size of an object.

[0021] ROI (Region of Interest) is a region of interest defined in an image during image processing, using shapes such as rectangles, circles, ellipses, or irregular polygons. This region is the focus of image analysis; defining it for further processing can reduce processing time and improve accuracy.

[0022] Meanwhile, a pedestrian reverse-movement detection system for escalators based on the human skeleton is also provided, including a camera and a server; the server includes the following modules:

[0023] Pedestrian recognition module: used to create a pedestrian bounding box and multiple skeleton points after a pedestrian enters the escalator area;

[0024] Occlusion detection module: Used to determine whether a pedestrian's upper body is occluded by using skeletal points; if occluded, the pedestrian is filtered out.

[0025] Reverse movement detection module: Used to determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator. If they are inconsistent, it is determined to be reverse movement and an alarm is issued.

[0026] Preferably, the server further includes:

[0027] Turning detection module: Determines whether the relative positions of the left and right skeletal points of a pedestrian have changed; if a change occurs, it determines that the pedestrian has turned around, and issues an alarm when the pedestrian is in the escalator area.

[0028] Compared with existing technologies, this solution has the following advantages:

[0029] This escalator pedestrian reverse movement detection method uses skeletal points to detect obstructed parts of the pedestrian's body. When the upper body of a pedestrian is obstructed, that pedestrian is filtered out, such as when they are holding an infant, or when only their hands or feet are not obstructed. This avoids these situations affecting the detection of other pedestrians or causing false alarms. Furthermore, the method determines the trajectory direction of the midpoint of the top of the bbox (body box) relative to the escalator's direction of travel. Since a pedestrian's legs are easily obstructed on escalators, causing the bottom of the bbox to shift upwards, the midpoint of the top of the bbox is used as the location point for judgment. By detecting pedestrian obstruction and the direction of the location point, false alarms due to pedestrian obstruction are reduced, improving the accuracy of reverse movement detection.

[0030] Secondly, by judging whether the relative positions of the left and right skeletal points change, it can determine whether the pedestrian has turned around and detect whether the pedestrian is going against the flow of traffic. Combining the pedestrian's direction of movement and turning situation further improves the accuracy of pedestrians going against the flow of traffic. Attached Figure Description

[0031] Figure 1 This is a flowchart of the pedestrian reverse movement detection method for escalators in Example 1;

[0032] Figure 2 This is a block diagram of the escalator pedestrian reverse movement detection system in Example 2. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application implemented as described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0034] Example 1

[0035] refer to Figure 1 A method for detecting pedestrians going against the flow on escalators based on the human skeleton includes the following steps:

[0036] Step 1: Acquire the video stream of the escalator through the camera. After the pedestrian enters the escalator area, create the pedestrian's bounding box and multiple skeleton points;

[0037] Preferably, step 1 specifically involves: acquiring an image of the escalator through a camera; determining the ROI region of the escalator through image semantic segmentation; determining the escalator's direction of travel using optical flow; and establishing a pedestrian bounding box and human skeleton points after a pedestrian enters the ROI region.

[0038] In this embodiment, the camera is placed diagonally above the escalator, capable of capturing the entirety or part of the escalator's path. After acquiring the escalator's video stream, the ROI region (detection region) is determined from the video image through image semantic segmentation. In this scheme, the image semantic segmentation uses the FCN-HRNetW18 algorithm to divide the escalator area in the video image into the left baffle, right baffle, path, and floor slabs.

[0039] For instructions on using the FCN-HRNetW18 algorithm, please refer to the HRNet-FCN section in the semantic segmentation model of the product matrix at https: / / github.com / PaddlePaddle / PaddleSeg?tab=readme-ov-file. A detailed description will not be provided here.

[0040] After obtaining the ROI region of the escalator, it is also necessary to determine the escalator's travel direction. This direction can be determined manually or using optical flow. In this embodiment, optical flow is used to determine the travel direction, specifically employing the sparse LK optical flow algorithm. Based on the escalator scenario and requirements analysis, the sparse LK optical flow algorithm is suitable for determining the escalator direction due to its relatively fast and stable speed. The Lucas Kanade sparse optical flow algorithm is implemented in OpenCV using the function `cv2.calcOpticalFlowPyrLK()`, which can be directly called.

[0041] For instructions on using the sparse LK optical flow algorithm, please refer to https: / / blog.csdn.net / qq_41368247 / article / details / 82562165. It will not be described in detail here.

[0042] The ROI area and travel direction of the escalator only need to be set when the escalator is used for the first time or when the escalator is changed during testing. Moreover, the ROI area and travel direction are determined by image semantic segmentation and optical flow respectively. After acquiring the video stream from the camera, the ROI area and travel direction can be automatically determined without manual configuration by the staff, reducing the workload of the staff and improving efficiency.

[0043] After determining the ROI region and the direction of the stairs, when a pedestrian enters the ROI region, a bounding box and skeleton points are created for the pedestrian. The bounding box is determined by the skeleton points. By combining the bounding box and skeleton points, we can analyze whether the pedestrian is going against the flow of traffic. By combining multiple detection methods, we can improve the accuracy of going against the flow of traffic detection.

[0044] Step 2: Determine whether the upper body of the pedestrian is obscured by the skeletal points; if the pedestrian is always obscured in the escalator area, filter out this pedestrian; if the pedestrian is not always obscured in the escalator area, proceed to the next step.

[0045] Preferably, in step 2, the confidence level of the pedestrian skeleton points is used to determine whether the occlusion is detected; when the confidence level is less than 0.7, it is determined to be occluded.

[0046] In this embodiment, skeletal points are used to determine which parts of a pedestrian are obscured. When the lower body (hips and legs) is obscured, it does not affect subsequent bounding box detection, and the next step of detection can proceed. However, when the upper body (shoulders and head) is obscured, since pedestrians stand from top to bottom on escalators, in other words, most of the pedestrian's area is obscured. In this case, hands, elbows, or parts of the legs may be exposed. Therefore, these parts that might affect other pedestrians need to be filtered out. This also avoids creating bounding boxes for only these exposed parts during subsequent bounding box detection, which could lead to false positives for walking in the wrong direction. Therefore, only the upper body of the pedestrian needs to be occluded. If the pedestrian is not obscured for a certain period of time, the detection of that pedestrian continues.

[0047] The occlusion is determined by the confidence level of the skeletal points. When the confidence level of a skeletal point is less than 0.7, it is considered occluded. Furthermore, when the confidence level of the skeletal points of the upper body is less than 0.7, the detection of the pedestrian going against the flow is filtered out to avoid misjudgment or to avoid affecting the detection of other pedestrians.

[0048] Step 3: Determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator; if they are inconsistent, it is determined to be reverse movement and an alarm is issued; if they are consistent, there is no reverse movement.

[0049] In this embodiment, the midpoint of the top of the bounding box is used as the detection point for the pedestrian's position. The advantages are: First, this method avoids misinterpreting the pedestrian's movement as upwards due to the bounding box moving upwards while the pedestrian's legs are obstructed, thus preventing a false judgment of the pedestrian moving in the wrong direction. Second, under normal circumstances, the upper body of the pedestrian is not obstructed, and the midpoint of the top of the bounding box will not show abnormal changes. Combined with the occlusion detection in step 2, this ensures the accuracy of the midpoint detection. The trajectory direction of the midpoint can be adjusted by setting the distance before and after the midpoint, and then compared with the direction of the staircase. This avoids the influence of special situations such as the pedestrian suddenly crouching or jumping, improving the accuracy of the wrong-way detection.

[0050] Preferably, in step 3, the trajectory direction of the top midpoint of the bbox is obtained by the NVDCF algorithm.

[0051] In this embodiment, the NVDCF algorithm can track the target trajectory, ensuring that each tracking is of the same person. For example, each pedestrian has an ID; pedestrian A's ID is A, and pedestrian B's ID is B. Through the NVDCF algorithm, the tracking will continue indefinitely, and pedestrian A's ID will always remain A and will not change to B. Moreover, the NVDCF algorithm can filter out the influence of special situations such as pedestrians suddenly crouching or jumping on the trajectory direction detection of the top midpoint of the bounding box.

[0052] For instructions on using the NVDCF algorithm, please refer to https: / / blog.csdn.net / weixin_41817841 / article / details / 133977715. It will not be described in detail here.

[0053] Preferably, the method further includes step 4: determining whether the relative positions of the left and right skeletal points of the pedestrian have changed; if a change occurs, the pedestrian is determined to turn around, and if the pedestrian is in the ROI area at this time, an alarm is issued; if there is no change, the detection continues.

[0054] In this embodiment, after determining the trajectory direction and the direction of the elevator, the accuracy of the reverse movement detection can be further enhanced by checking whether the pedestrian turns around.

[0055] The method for detecting pedestrian turning is as follows: It involves determining whether the relative positions of the pedestrian's left and right skeletal points have changed; in other words, whether the positions of the left and right skeletal points have been interchanged. Current pose estimation algorithms can detect the difference between the left and right skeletal points. Therefore, when a change in the relative positions of the pedestrian's left and right skeletal points is detected, it can be determined that the pedestrian is turning. If the pedestrian is still within the ROI area at this time, an alarm is issued. If the pedestrian has already left the escalator area, turning will not pose a danger, and no alarm is needed.

[0056] Preferably, the plurality of skeletal points include points corresponding to joints on the limbs, hip joints, shoulder joints, and head.

[0057] Preferably, the pedestrian bounding box, multiple skeletal points, and the confidence level of the skeletal points are obtained through a pose estimation algorithm.

[0058] Pose estimation algorithms can include YOLO-Pose, Kapao, LitePose, and lightweight OpenPose, all of which can derive skeletal points. In this embodiment, the YOLO-Pose algorithm is used to derive the pedestrian bounding box, multiple skeletal points, and the confidence scores of the skeletal points.

[0059] For instructions on using the YOLO-Pose algorithm, please refer to https: / / blog.csdn.net / wqthaha / article / details / 125249721. A detailed description will not be provided here.

[0060] The advantage of this method is that:

[0061] 1. By adding pedestrian occlusion detection, the accuracy of using the top midpoint of the bounding box as the pedestrian location point is improved, avoiding abnormal changes in the top midpoint of the bounding box from affecting the judgment of the pedestrian trajectory direction. Combining pedestrian occlusion detection and using the top midpoint of the bounding box as the pedestrian location point greatly improves the accuracy of reverse movement detection.

[0062] 2. By adding pedestrian turn detection, the reliability of reverse movement detection is further enhanced. Combining bbox and turn detection, the detection of bbox will not be able to detect situations where pedestrians are coming down the escalator with their backs to the escalator. Therefore, judging the situation by pedestrian turn can identify reverse movement. By combining bbox and turn detection, the accuracy of reverse movement detection is greatly improved.

[0063] Example 2

[0064] refer to Figure 2 A pedestrian reverse-movement detection system for escalators based on human skeletons includes a camera and a server; the server includes the following modules:

[0065] Pedestrian recognition module: used to create a pedestrian bounding box and multiple skeleton points after a pedestrian enters the escalator area;

[0066] Occlusion detection module: Used to determine whether a pedestrian's upper body is occluded by using skeletal points; if occluded, the pedestrian is filtered out.

[0067] Reverse movement detection module: Used to determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator. If they are inconsistent, it is determined to be reverse movement and an alarm is issued.

[0068] Preferably, the server further includes:

[0069] Turning detection module: Determines whether the relative positions of the left and right skeletal points of a pedestrian have changed; if a change occurs, it determines that the pedestrian has turned around, and issues an alarm when the pedestrian is in the escalator area.

[0070] In this embodiment, the specific workflow of the escalator pedestrian reverse movement detection system is as follows:

[0071] The camera captures video stream images of the escalator. Staff first determine the ROI (Region of Interest) and direction of travel of the escalator. This can be done manually or by incorporating image semantic segmentation and optical flow into the reverse-traffic detection system. When a pedestrian enters the escalator area, the pedestrian recognition module creates a bounding box (BBox) and multiple skeletal points for the pedestrian. These BBoxes and skeletal points are then sent to the occlusion detection module. The occlusion detection module uses the skeletal points to determine the occluded parts of the pedestrian. If the upper body of the pedestrian is occluded, the pedestrian is filtered out; if the upper body is not occluded, the detection is repeated. The occlusion detection module sends the bounding box of the pedestrian whose upper body is not occluded to the reverse movement detection module. The reverse movement detection module judges the trajectory direction of the top midpoint of the bounding box and the direction of the staircase. If the trajectory direction is consistent with the direction of the staircase, there is no reverse movement. If the trajectory direction is inconsistent with the direction of the staircase, the bounding box and skeleton points of the pedestrian are sent to the turning detection module. The turning detection module judges the relative position of the left and right skeleton points of the pedestrian. If the relative position of the left and right skeleton points changes, it judges that the pedestrian has turned around. If the pedestrian is within the ROI area, the pedestrian is reverse movement and an alarm is issued. If the relative position of the left and right skeleton points does not change, the pedestrian is not reverse movement.

[0072] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for detecting pedestrians going against the flow on escalators based on the human skeleton, characterized in that, Includes the following steps: Step 1: Acquire the video stream of the escalator through the camera. After the pedestrian enters the escalator area, create the pedestrian's bounding box and multiple skeleton points; Step 2: Determine whether the upper body of the pedestrian is obscured by using skeletal points; If a pedestrian is constantly obscured within the escalator area, then filter out that pedestrian; if a pedestrian is not constantly obscured within the escalator area, proceed to the next step. Step 3: Determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator; if they are inconsistent, it is determined to be reverse movement and an alarm is issued; if they are consistent, there is no reverse movement.

2. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 1, characterized in that, The multiple skeletal points include points corresponding to joints on the limbs, hip joints, shoulder joints, and the head.

3. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 1, characterized in that, Step 1 specifically involves: acquiring images of the escalator through a camera; determining the ROI region of the escalator through image semantic segmentation; determining the escalator's direction of travel using optical flow; and establishing the pedestrian's bounding box and human skeleton points after the pedestrian enters the ROI region.

4. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 3, characterized in that, It also includes step 4: determining whether the relative positions of the pedestrian's left and right skeletal points have changed; if a change occurs, it is determined that the pedestrian has turned around, and if the pedestrian is in the ROI area at this time, an alarm is issued; if there is no change, the detection continues.

5. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 1, characterized in that, In step 2, the confidence level of the pedestrian skeleton points is used to determine whether they are occluded; when the confidence level is less than 0.7, they are determined to be occluded.

6. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 1, characterized in that, In step 3, the trajectory direction of the top midpoint of the bbox is obtained through the NVDCF algorithm.

7. The method for detecting pedestrians going against the flow on escalators based on human skeletons according to claim 1, characterized in that, The pose estimation algorithm is used to obtain the pedestrian bounding box, multiple skeletal points, and the confidence level of the skeletal points.

8. A pedestrian reverse movement detection system for escalators based on human skeleton, characterized in that, Includes a camera and a server; the server includes the following modules: Pedestrian recognition module: used to create a pedestrian bounding box and multiple skeleton points after a pedestrian enters the escalator area; Occlusion detection module: used to determine whether a pedestrian's upper body is occluded by using skeletal points; If the pedestrian is obscured, then filter out that pedestrian; Reverse movement detection module: Used to determine whether the trajectory direction of the top midpoint of the bbox is consistent with the direction of the elevator. If they are inconsistent, it is determined to be reverse movement and an alarm is issued.

9. The escalator pedestrian reverse movement detection system based on human skeleton as described in claim 8, characterized in that, The server also includes: Turning detection module: Determines whether the relative positions of the left and right skeletal points of a pedestrian have changed; if a change occurs, it determines that the pedestrian has turned around, and issues an alarm when the pedestrian is in the escalator area.

Images