A method and system for 3D intrusion detection of escalators based on human skeleton and image semantic segmentation

Abstract

The application belongs to the field of Internet of Things, and discloses a three-dimensional area intrusion detection method for escalator based on human skeleton and image semantic segmentation, which comprises the following steps: step 1: obtaining the image of the escalator through a camera; step 2: drawing a boundary line and a human skeleton; step 3: obtaining intersection point A and intersection point B through a first connecting line; step 4: obtaining a first tripwire and a second tripwire; step 5: calculating the included angle of the first tripwire and the second tripwire, if the included angle is less than or equal to a first preset angle, using the second tripwire as a standard tripwire, if the included angle is greater than the first preset angle, using the first tripwire as a standard tripwire; step 6: judging whether a specific part of the human skeleton exceeds the standard tripwire, if yes, alarming. The method does not need to track the human body when the passenger enters the elevator, does not need to require the camera installation precision too much, and the alarm is more sensitive. Meanwhile, the application also discloses a system based on the method.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things (IoT) technology, specifically to a method and system for three-dimensional intrusion detection of escalators based on human skeleton and image semantic segmentation. Background Technology

[0002] The applicant hereby submits an invention patent application CN114708552A, which discloses a three-dimensional region intrusion detection method based on human skeleton, including the following steps: Step 1: When a human body enters the camera area, the human skeleton is identified to obtain skeleton information, in which multiple position points are marked; Step 2: When a human body enters an escalator, a first image is acquired, and the height relationship between two position points at different heights of the handrail and the human body is calculated; Step 3: During the operation of the escalator, a second image is acquired, and based on the height relationship and the two position points, a point on the handrail that maintains the same horizontal height as the human body is determined; Step 4: A vertical line is drawn on the handrail at the point determined in Step 3, and it is determined whether the human skeleton intersects with the vertical line. If they intersect, an alarm is issued.

[0003] It has the following problems:

[0004] 1. This method uses the hip bone to determine the connection line, but since everyone's height is different, it is prone to large errors;

[0005] 2. This solution requires drawing a connection line once when entering the elevator, which involves complex calculations;

[0006] 3. This solution requires drawing a connection line when entering the elevator, which necessitates human body tracking, and the algorithm is prone to losing track of the human body.

[0007] 4. The camera needs to be installed directly facing the center of the elevator, which requires high installation standards;

[0008] 5. Cameras using wide-angle lenses may capture distorted images. This can lead to intrusion detection errors, especially in escalator areas.

[0009] Based on this, the technical problem solved in this case is: how to achieve three-dimensional intrusion detection and alarm for escalators without human body tracking, eliminating distortion from wide-angle lenses, or requiring overly standard installation requirements. Summary of the Invention

[0010] The purpose of this invention is to provide a three-dimensional intrusion detection method for escalators based on human skeleton and image semantic segmentation. This method uses the tibia of passengers to draw a first line, finds intersection point A and intersection point B, and sets two tripwires. One of the two tripwires is selected as the standard tripwire, which can reduce the error when only the first tripwire is used as the standard tripwire if the camera is not installed accurately.

[0011] The method of the present invention does not require human body tracking when passengers enter the elevator, does not require overly stringent requirements for camera installation accuracy, and its alarm is relatively sensitive.

[0012] Meanwhile, the present invention also provides a three-dimensional intrusion detection system for escalators based on human skeleton and image semantic segmentation.

[0013] To achieve the above objectives, the present invention provides the following technical solution: a method for 3D region intrusion detection of escalators based on human skeleton and image semantic segmentation, comprising the following steps:

[0014] Step 1: Obtain images of the escalator using a camera;

[0015] Step 2: Draw the boundary lines between the floor slabs and walkways of the escalator in the image, as well as the boundary lines between the walkways and the side panels on both sides of the walkways, using an image semantic segmentation algorithm.

[0016] The human skeleton of the human body on the walkway in the image is drawn using a human skeleton algorithm. The slope of the human spine is calculated and the first line connecting the lower part or lower end of the two tibias of the human body is drawn.

[0017] Step 3: Extend the first line to the boundary line between the walkway and the barriers on both sides of the walkway to obtain intersection point A and intersection point B;

[0018] Step 4: Draw a perpendicular line along the dividing line between the floor slab and the walkway, and translate the perpendicular line to the intersection point A and intersection point B to obtain the first tripping line; draw the second tripping line from intersection point A and intersection point B according to the slope of the human spine.

[0019] Step 5: Calculate the angle between the first tripwire and the second tripwire. If the angle is less than or equal to the first preset angle, the second tripwire is used as the standard tripwire; if the angle is greater than the first preset angle, the first tripwire is used as the standard tripwire.

[0020] Step 6: Determine if a specific part of the human skeleton exceeds the standard trip line; if so, issue an alarm.

[0021] In the above-mentioned escalator 3D region intrusion detection method based on human skeleton and image semantic segmentation, the first preset angle is 20-40°.

[0022] In the above-mentioned method for 3D region intrusion detection of escalators based on human skeleton and image semantic segmentation, the specific location is one or more of the following: wrist, elbow, shoulder joint, cervical spine, spine, hip joint, knee joint, and lumbar spine.

[0023] Meanwhile, the present invention also discloses a three-dimensional intrusion detection system for escalators based on human skeleton and image semantic segmentation, including a camera whose image range covers the escalator and a server;

[0024] The server includes the following modules:

[0025] Communication module: used to capture the head image from the camera;

[0026] Image semantic segmentation calculation module: used to draw the boundary lines between the floor slabs and walkways of the escalator in the image, as well as the boundary lines between the walkways and the baffles on both sides of the walkways, using image semantic segmentation algorithms.

[0027] Human skeleton calculation module: used to draw the human skeleton of the human body on the walkway in the image using human skeleton algorithm, calculate the slope of the human spine, and draw the first line connecting the lower part or lower end of the two tibias of the human body.

[0028] Intersection Point Drawing Module: Used to extend the first line on the image to the boundary line between the walkway and the barriers on both sides of the walkway, to obtain intersection point A and intersection point B;

[0029] Tripwire drawing module: used to draw a perpendicular line on the image along the dividing line between the floor slab and the walkway, and translate the perpendicular line to the intersection point A and intersection point B to obtain the first tripwire; used to draw a second tripwire on the image according to the slope of the human spine from intersection point A and intersection point B.

[0030] Standard tripwire selection module: used to calculate the angle between the first tripwire and the second tripwire. If the angle is less than or equal to the first preset angle, the second tripwire is used as the standard tripwire; if the angle is greater than the first preset angle, the first tripwire is used as the standard tripwire.

[0031] Judgment module: Used to determine whether a specific part of the human skeleton exceeds the standard trip line; if so, an alarm is triggered.

[0032] In the aforementioned escalator 3D region intrusion detection system based on human skeleton and image semantic segmentation, the first preset angle is 20-40°.

[0033] In the aforementioned escalator 3D region intrusion detection system based on human skeleton and image semantic segmentation, the specific location is one or more of the following: wrist, elbow, shoulder joint, cervical spine, spine, hip joint, knee joint, and lumbar spine.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] This invention utilizes a semantic segmentation algorithm to obtain the boundary between floor slabs and walkways, regardless of the camera's mounting angle (horizontal or vertical), and uses this boundary as a horizontal line. The true vertical line is then derived from the horizontal line.

[0036] This method is extremely helpful for on-site camera installation. Because the actual installation angle cannot be completely parallel to the escalator. Or, if the camera is not installed in the center of the escalator, the resulting image will also have a horizontal deviation. Furthermore, camera lens distortion can be corrected using vertical lines.

[0037] The trip line is drawn by using the intersection of the human tibia with the left and right side panels. There's no need to consider each person's height, and therefore no tracking algorithm is needed to avoid losing height information due to the inability to track a particular individual.

[0038] In theory, intrusion detection can be achieved simply by using the first tripwire. However, with the widespread use of wide-angle lenses, using only the first tripwire for detection at a distance in the lens will result in significant errors. For example, with most commercially available wide-angle lenses, 50 pixels represent 1 meter at a distance in the image, and image distortion is more likely to occur at a distance. In this case, using the first tripwire will easily generate false alarms.

[0039] To solve this problem, we draw the second trip line using the tibia of the human leg. The tibia is close to the ground, so there is no need to introduce the variable of height.

[0040] Of course, there are a few unexpected situations that need to be ignored or considered when using the second tripwire:

[0041] 1. When a person stands with their legs staggered, the first line connecting them will be inaccurate. This is not a problem that this invention considers or solves. This invention only addresses the case where the tibias are side by side. Note: If this case needs to be considered, the skeletal judgment algorithm is still used to calculate whether the two tibias are side by side. If they are not side by side, the method of this invention is not used.

[0042] 2. When the human body is tilted, the second tripping line of the spine is inaccurate. Therefore, in this invention, it is necessary to determine the angle between the first tripping line and the second tripping line to determine whether to choose the first tripping line or the second tripping line. One of the purposes of this invention is: if the second tripping line can be used, the first tripping line should be used as much as possible. If the second tripping line is seriously inaccurate, the first tripping line should be used.

[0043] The above design reduces (but does not eliminate) false alarms generated by the first tripwire calculation, improves alarm accuracy, reduces computational load, and eliminates the need for human tracking. Attached Figure Description

[0044] Figure 1This is a schematic diagram of the image semantic segmentation after Example 1;

[0045] Figure 2 yes Figure 3 A simplified diagram;

[0046] Figure 3 This is a diagram showing passengers on the walkway;

[0047] Figure 4 This is a flowchart of Example 1;

[0048] Figure 5 This is a structural block diagram of Example 2;

[0049] Figure 6 This is the image display status when an alarm occurs in Example 2. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example 1

[0052] refer to Figure 2 , 3 4. A method for 3D intrusion detection of escalators based on human skeleton and image semantic segmentation, comprising the following steps:

[0053] Step 1: Acquire an image of the escalator using camera 1;

[0054] The camera 1 can be any of the commercially available cameras 1. In this embodiment, a wide-angle camera 1 is used. The wide-angle camera 1 has the advantage of a large shooting range, but the image deep inside its lens is prone to distortion.

[0055] Step 2: Draw the boundary line between the floor slab H and the walkway G of the escalator in the image, and the boundary line between the walkway G and the baffles C on both sides of the walkway G, using the image semantic segmentation algorithm.

[0056] Image semantic segmentation algorithms group / segment pixels according to their semantic meaning in an image. This algorithm differs from general object recognition algorithms in that, in addition to identifying specific objects, it also needs to identify the positions of all pixels representing such objects within the image. It is currently commonly used in autonomous driving functions in automobiles. This invention introduces this algorithm to automatically identify various areas of an escalator, such as floor slabs H (cyan area), left and right side panels C, and walkways G (red area). As follows... Figure 1 ;

[0057] It should be noted that the image semantic segmentation algorithm is a mature algorithm in this field and is not the innovation of this invention. Therefore, no further restrictions or emphasis will be placed on it.

[0058] The human skeleton of the human body on the walkway G in the image is drawn using the human skeleton algorithm. The slope of the human spine E is calculated and the first line D connecting the lower ends (i.e., the ankle position) of the two tibias F is drawn.

[0059] Classic algorithms for human skeleton detection can use human skeleton keypoint detection models such as Skeleton or AlphaPose, pytorch-openpose, etc.

[0060] Human pose estimation algorithms commonly include AlphaPose, OpenPose, MMPose, BlazePose, and OpenPifPAF. They can be broadly categorized into two types based on their principles: top-down and bottom-up. Regardless of the algorithm, the final output is the coordinates and confidence scores of the human skeleton points.

[0061] Image semantic segmentation algorithms commonly include FCN-HRNetW, PP-Lite-Seg, Segformer, and DeepLabv3+. Currently, the algorithm used for region segmentation is FCN-HRNetW18, with HRNet (High Resolution Networks) as its backbone and FCN (Fully Convolutional Networks) as its head. HRNet is a multi-branch, high-resolution network originally designed for pose estimation tasks. However, because the network maintains high resolution throughout, it preserves the spatial information of the input image, reducing spatial information loss due to downsampling, making it naturally suitable for pixel-level prediction tasks like semantic segmentation. In addition to preserving high-resolution spatial information, the network also performs multi-scale information fusion, enabling it to extract strong semantic information and further enhancing its expressive power.

[0062] Step 3: Extend the first line D to the boundary line between the walkway G and the baffles C on both sides of the walkway G to obtain the intersection point A and the intersection point B;

[0063] It should be noted that this step is based on the premise that the two tibias F are side by side when the person is standing. If the two tibias F are not side by side, the method of this invention will not be used. To determine whether the two tibias F are side by side, the skeletal diagram drawn by the human skeleton algorithm in step 2 can be used to determine whether the two tibias F are approximately equal in length. If the two tibias F are located in an anterior-posterior position, then the two tibias F are not equal in length. In this case, other methods need to be used to determine this, which is not within the scope of protection of this invention.

[0064] Step 4: Draw a perpendicular line I along the dividing line between the floor slab H and the walkway G. Translate the perpendicular line I to the intersection point A and the intersection point B to obtain the first tripping line J. Draw the second tripping line K from the intersection point A and the intersection point B according to the slope of the human spine E.

[0065] The first tripwire J and the second tripwire K can be referenced. Figure 2 ;

[0066] Step 5: Calculate the angle between the first tripwire J and the second tripwire K. If the angle is less than or equal to the first preset angle, such as 30°, then the second tripwire K is used as the standard tripwire; if the angle is greater than the first preset angle, then the first tripwire J is used as the standard tripwire.

[0067] In practical applications, the allowable included angle between the first tripwire J and the second tripwire K can be 20°, 25°, 30°, 35° or 40°;

[0068] If the angle between the first tripping line J and the second tripping line K is less than or equal to 30°, it means that the person is most likely not leaning or protruding, and the person's spine is used as the standard for verticality; if the angle between the first tripping line J and the second tripping line K is greater than 30°, it means that the person's spine is severely tilted, and the first tripping line J is used as the standard tripping line.

[0069] Step 6: Determine if a specific part of the human skeleton exceeds the standard trip line; if so, issue an alarm.

[0070] The specific location is one or more of the following: wrist, elbow, shoulder joint, cervical vertebrae, spine, hip joint, knee joint, and lumbar vertebrae. In this embodiment, it is the elbow.

[0071] The two standard trip lines form an area that is only used to determine whether the passenger has crossed the handrail area and is at risk of falling. If the risk of other passengers needs to be assessed, steps 1-6 should be repeated.

[0072] The above methods offer the following advantages over earlier applications:

[0073] 1. No human tracking required;

[0074] 2. There is no need to detect the human body when the passenger enters the walkway G and compare the result with the calculation at the time of entry in real time;

[0075] 3. The installation requirements for camera 1 have been lowered;

[0076] 4. Compatible with most wide-angle lenses;

[0077] 5. Small alarm error.

[0078] Example 2

[0079] refer to Figure 5 A three-dimensional intrusion detection system for escalators based on human skeleton and image semantic segmentation includes a camera 1 with an image range covering the escalator and a server 2.

[0080] The server 2 includes the following modules:

[0081] Communication module 3: Used to acquire a headshot from camera 1;

[0082] Image semantic segmentation calculation module 4: used to draw the boundary line between the floor slab H and the walkway G of the escalator in the image, and the boundary line between the walkway G and the baffles C on both sides of the walkway G, using the image semantic segmentation algorithm.

[0083] Human skeleton calculation module 5: used to draw the human skeleton of the human body on the walkway G in the image through human skeleton algorithm, calculate the slope of the human spine E, and draw the first line D connecting the lower part or lower end of the two tibias F of the human body.

[0084] Intersection Point Drawing Module 6: Used to extend the first connecting line D on the image to the boundary line between the walkway G and the baffles C on both sides of the walkway G, to obtain intersection point A and intersection point B;

[0085] Tripline drawing module 7: used to draw a perpendicular line I on the image along the dividing line between the floor slab H and the walkway G, and translate the perpendicular line I to the intersection point A and the intersection point B to obtain the first tripline J; used to draw a second tripline K on the image according to the slope of the human spine E from the intersection point A and the intersection point B.

[0086] Standard tripwire selection module 8: Used to calculate the angle between the first tripwire J and the second tripwire K. If the angle is less than or equal to the first preset angle, the second tripwire K is used as the standard tripwire; if the angle is greater than the first preset angle, the first tripwire J is used as the standard tripwire.

[0087] Judgment Module 9: Used to determine whether a specific part of the human skeleton exceeds the standard trip line; if so, an alarm is triggered.

[0088] During operation, camera 1 continuously acquires images and transmits them to communication module 3. Communication module 3 then sends the images to image semantic segmentation calculation module 4 and human skeleton calculation module 5. Image semantic segmentation calculation module 4 outputs semantic segmentation results, marking the boundary between floor slab H and walkway G, and the boundary between walkway G and the baffles C on both sides of walkway G on the image. Human skeleton calculation module 5 marks the skeletal node diagram of passengers on the elevator, and based on the skeletal node diagram, calculates the slope of the human spine E and draws the lower part of the two tibias F. The first connecting line D at the bottom; the intersection point drawing module 6 extends the first connecting line D on the image to the boundary line between the walkway G and the baffles C on both sides of the walkway G, obtaining intersection points A and B; the tripwire drawing module 7 draws the first tripwire J and the second tripwire K on the image; the standard tripwire selection module 8 selects which tripwire as the standard tripwire according to the set standard; the judgment module 9 judges whether a specific part of the human skeleton exceeds the standard tripwire, and if so, an alarm is triggered; the alarm can be displayed on the screen connected to server 2, either as a text or image alarm, see reference. Figure 6 , Figure 6 When a passenger on the central walkway extends their hand beyond the guardrail, a red circle will appear near their elbow to indicate an image alarm, or to control a buzzer connected to server 2 to generate an alarm; this buzzer can be placed near the escalator.

Claims

1. An escalator three-dimensional area intrusion detection method based on human skeleton and image semantic segmentation, characterized in that, The method comprises the following steps: Step 1: obtaining an image of the escalator through a camera; Step 2: drawing a boundary line between the floor plate and the stepway of the escalator in the image, and a boundary line of the stepway and the baffle on both sides of the stepway through an image semantic segmentation algorithm; drawing a human skeleton of a human on the stepway in the image through a human skeleton algorithm, calculating a slope of the human spine, and drawing a first connecting line of lower parts or lower ends of two shank bones of the human; Step 3: extending the first connecting line to the boundary line of the stepway and the baffle on both sides of the stepway to obtain intersection point A and intersection point B; Step 4: drawing a perpendicular line along the boundary line between the floor plate and the stepway, and translating the perpendicular line to intersection point A and intersection point B to obtain a first tripwire; and drawing a second tripwire from intersection point A and intersection point B according to the slope of the human spine; Step 5: calculating an included angle of the first tripwire and the second tripwire, if the included angle is less than or equal to a first preset angle, using the second tripwire as a standard tripwire; if the included angle is greater than the first preset angle, using the first tripwire as the standard tripwire; Step 6: judging whether a specific part of the human skeleton exceeds the standard tripwire, and if so, alarming.

2. The escalator 3D zone intrusion detection method based on human skeleton and image semantic segmentation according to claim 1, characterized in that, The first preset angle is 20-40°.

3. The escalator 3D zone intrusion detection method based on human skeleton and image semantic segmentation of claim 1, wherein, The specific part is one or more of a wrist, an elbow, a shoulder joint, a cervical vertebra, a spine, a hip joint, a knee joint, and a lumbar vertebra.

4. An escalator three-dimensional area intrusion detection system based on human skeleton and image semantic segmentation, characterized in that, The camera and the server cover the image range of the escalator; The server comprises the following modules: a communication module for obtaining a head portrait from the camera; an image semantic segmentation calculation module for drawing a boundary line between the floor plate and the stepway of the escalator in the image, and a boundary line of the stepway and the baffle on both sides of the stepway through an image semantic segmentation algorithm; a human skeleton calculation module for drawing a human skeleton of a human on the stepway in the image through a human skeleton algorithm, calculating a slope of the human spine, and drawing a first connecting line of lower parts or lower ends of two shank bones of the human; an intersection point drawing module for extending the first connecting line to the boundary line of the stepway and the baffle on both sides of the stepway to obtain intersection point A and intersection point B; a tripwire drawing module for drawing a perpendicular line along the boundary line between the floor plate and the stepway, and translating the perpendicular line to intersection point A and intersection point B to obtain a first tripwire; and for drawing a second tripwire from intersection point A and intersection point B according to the slope of the human spine; a standard tripwire selection module for calculating an included angle of the first tripwire and the second tripwire, if the included angle is less than or equal to a first preset angle, using the second tripwire as a standard tripwire; if the included angle is greater than the first preset angle, using the first tripwire as the standard tripwire; a judging module for judging whether a specific part of the human skeleton exceeds the standard tripwire, and if so, alarming.

5. The escalator 3D zone intrusion detection system based on human skeleton and image semantic segmentation of claim 4, wherein, The first preset angle is 20-40°.

6. The escalator 3D zone intrusion detection system based on human skeleton and image semantic segmentation of claim 4, wherein, The specific part is one or more of a wrist, an elbow, a shoulder joint, a cervical vertebra, a spine, a hip joint, a knee joint, and a lumbar vertebra.

Images