A method for detecting people trapped in elevators based on video analysis

Abstract

The present invention belongs to the technical field of elevator monitoring. The purpose is to provide an elevator entrapment detection method based on video analysis, which is a highly real-time and adaptive multi-scenario elevator entrapment detection method to reduce the workload of manual monitoring. The technical solution is: a method for detecting elevator entrapment based on video analysis; comprising the following steps: Step 1: Manually label the passengers, elevator doors and floor indications of each frame image in the elevator car monitoring video, and use this as a data set to train a deep convolutional neural network to obtain a multi-task detection model LiftDet for passenger target tracking, elevator door status judgment and floor indication recognition; Step 2: For the video to be detected, use the multi-task detection model LiftDet to obtain the detection set Ro; Step 3: Select the part of the elevator car monitoring video where no abnormal entrapment occurs, and evaluate the standard limit frame number of the passenger's normal stay.

Description

Technical Field

[0001] The present invention belongs to the technical field of elevator monitoring, and in particular is a method for detecting abnormalities of people being trapped in elevators based on video analysis. Background Art

[0002] Elevators are frequently used transportation equipment. Failures can occur due to inadequate maintenance, aging components, and improper use. Elevator entrapment is a common phenomenon caused by elevator malfunctions. This occurs when an elevator suddenly stops during normal operation due to disconnected safety or door lock circuits, control system failure or malfunction, power outages, or other human error, trapping passengers inside.

[0003] To accurately detect elevator entrapment anomalies and enable timely emergency response, a Chinese patent application (Application No. 202111412911.1) discloses an elevator entrapment detection system. This system uses infrared sensors combined with rails and signals from the elevator car's gravity sensor, sets a passenger time threshold to determine whether the elevator is entrapped, and uses a motor push rod to automatically trigger the emergency button. A Chinese patent application (Application No. 202011421436.X) discloses a computer vision-based elevator entrapment detection method. This method detects entrapment by determining whether the elevator is closed and whether the duration of the closure exceeds a set threshold. It also uses a convolutional neural network to identify information such as the number of people in the elevator and feeds it back to a monitoring center. Ma Zhenbao (Elevator Car Entrapping Intelligent Alarm System Based on Video Images [J]. China Elevator, 2022) proposed a system for intelligently alarming entrapped elevator cars using video images. Based on the elements of elevator entrapment, the system uses edge detection technology to obtain the movement status of the car door, optical character recognition technology to identify floor information, edge detection technology for background modeling, and background subtraction to determine whether the elevator is carrying passengers, thereby achieving intelligent detection and alarm. However, it uses a manually designed feature extraction and recognition method, which requires manual adjustment of the algorithm's hyperparameters in different scenarios.

[0004] In summary, the shortcomings of existing technical solutions are as follows: (1) In elevator occupant detection based on sensor signals, the sensor signals are accurate but not rich, and the robustness is poor in complex scenarios. (2) In the vision-based elevator occupant detection method, manual feature extraction and pattern recognition are mainly used, and deep learning algorithms are only used to assist in the judgment of information such as the number of people in the elevator. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned background technology and provide an elevator entrapment detection method based on video analysis, which is a real-time and adaptive multi-scenario elevator entrapment detection method to reduce the workload of manual monitoring.

[0006] The technical solution provided by the present invention is:

[0007] A method for detecting people trapped in an elevator based on video analysis, comprising the following steps:

[0008] Step 1: Manually label the passengers, elevator doors, and floor numbers in each frame of elevator car surveillance video. This data set is used to train a deep convolutional neural network to develop a multi-task detection model, LiftDet, that performs passenger tracking, elevator door status determination, and floor number recognition.

[0009] Step 2: For the video to be detected, use the multi-task detection model LiftDet to obtain the detection set R o ;

[0010] R o ={(m ij ,c ij ,d i ,l i )|i=1,2,…,K,j=1,2,3,…,M i};

[0011] Where: m ij Indicates the center coordinates, width and height of the BoundingBox of the jth passenger in the i-th frame;

[0012] c ij represents the ID number of the jth passenger in the i-th frame;

[0013] d i represents the state of the elevator door in the i-th frame, and the specific state is determined by formula (1);

[0014] l i Indicates the floor number of the elevator in the i-th frame;

[0015] K represents the number of video frames;

[0016] M i represents the total number of passengers in the i-th frame;

[0017]

[0018] Step 3: Select the part of the elevator car surveillance video where no abnormal entrapment occurs, and evaluate the standard limit frame number of the passenger's normal stay; specifically: record the frame number of the passenger with ID number c entering and leaving the elevator as and Calculate the number of frames the passenger stayed in this time according to formula (2) According to formula (3), the maximum number of stay frames Update; calculate the standard limit frame number t of normal stay according to formula (4);

[0019]

[0020]

[0021]

[0022] Among them, c represents the passenger's ID number, Indicates that the passenger enters the elevator key frame, Indicates the key frame that the passenger leaves the elevator. represents the number of frames the passenger stays, ξ represents the update coefficient of the maximum number of frames the passenger stays, L represents the maximum number of floors the elevator is located on, and t represents the standard limit number of frames the passenger stays.

[0023] Step 4: Starting from the current frame k, select a sequence of consecutive frames in which the elevator floor number remains unchanged and the elevator door is closed. and a continuous sequence of frames of the elevator door closing remember is the number of static frames of the elevator floor, is the number of elevator door closing frames; if the current frame k satisfies formula (5), k is the key frame for judging the abnormality of trapped people;

[0024]

[0025] Among them, M k Indicates the number of passengers in the elevator under the k-th frame image;

[0026] Step 5: Starting from the key frame k of the abnormal person entrapment, select the subsequent consecutive frames ZP = {i|i = k, k + 1, ..., k + t};

[0027] The abnormal coefficient p of elevator entrapment is calculated by formula (6) and (7): i ;

[0028]

[0029]

[0030] Where t represents the standard limit frame number of passengers staying, d i Indicates the closed state of the elevator door in the i-th frame, l i Indicates the floor number of the i-th frame;

[0031] Step 6: Calculate the elevator trapped person abnormal threshold α by formula (8); if p i >α, then it is judged that there is a problem of people being trapped in the current elevator;

[0032]

[0033] Among them, Mi represents the number of passengers in the i-th frame, W i Indicates the number of matching passenger IDs between the i-th frame and the i-1-th frame;

[0034] Step 7: Evaluate the situation of people being trapped in the elevator. Specifically, calculate the number of passengers currently trapped in the elevator n according to formula (9); calculate the time t of people being trapped in the elevator in real time according to formula (10);

[0035]

[0036]

[0037] where r s Indicates the frame rate of the video, f i Indicates the frame number of the current frame;

[0038] Compared with the existing technology, the beneficial effects of the present invention are: the present invention tracks passengers in real time, detects the closing status of elevator doors, and identifies floor indications, analyzes the conditions in the elevator car frame by frame based on the video, accurately determines whether there is a problem of people being trapped in the elevator and estimates the number of trapped people and the length of time they have been trapped. It has the characteristics of strong real-time performance and adaptability to multiple scenarios, can reduce the workload of manual monitoring, and thus provide a reference for rescue by maintenance personnel and monitoring centers. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 An image showing the multi-task detection results in elevator surveillance videos.

[0040] Figure 2 This is an image of the process of detecting trapped people in elevator surveillance video.

[0041] Figure 3 This is an image showing the calculation result of the trapped time in the elevator surveillance video. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the embodiments.

[0043] The method for detecting people trapped in an elevator based on video analysis of the present invention specifically comprises the following steps:

[0044] Step 1: Manually label the passengers, elevator doors, and floor numbers in each frame of elevator car surveillance video. This data set is used to train a deep convolutional neural network to obtain the multi-task detection model LiftDet, which performs passenger target tracking, elevator door status judgment, and floor number recognition.

[0045] Step 2: For the video to be detected, use the multi-task detection model LiftDet to obtain the detection set R o ;

[0046] Ro ={(m ij ,c ij ,d i ,l i )|i=1,2,…,K,j=1,2,3,…,M i}; where m ij Indicates the coordinates of the center point, width and height of the BoundingBox of the jth passenger in the i-th frame, c ij represents the ID number of the jth passenger in the i-th frame, d i represents the state of the elevator door in the i-th frame. The specific state is determined by formula (1). i Indicates the floor number of the elevator in the i-th frame, K represents the number of video frames, M i represents the total number of passengers in the i-th frame; in this embodiment, the results of the multi-task detection model are as follows Figure 1 As shown, the passenger ID number is 27, the floor number is 9, and the elevator door is closed;

[0047]

[0048] Step 3: Select the part of the elevator car surveillance video where no abnormal person is trapped, and evaluate the standard limit frame number of the normal passenger stay. Specifically, record the frame number of the passenger with ID number c entering and exiting the elevator as and Calculate the number of frames the passenger stayed in this time according to formulas (2) and (3) And the maximum number of frames to stay Update; calculate the standard limit frame number f for normal stay according to formula (4); in this embodiment, the standard limit frame number is 1776;

[0049]

[0050]

[0051]

[0052] Among them, c represents the passenger's ID number, Indicates that the passenger enters the elevator key frame, Indicates the key frame that the passenger leaves the elevator. represents the number of frames the passenger stays, ξ represents the update coefficient of the maximum number of frames the passenger stays, L represents the maximum number of floors the elevator is located on, and t represents the standard limit number of frames the passenger stays.

[0053] Step 4: Starting from the current frame k, select a sequence of consecutive frames in which the elevator floor number remains unchanged and the elevator door is closed. and remember is the number of static frames of the elevator floor, is the number of frames of elevator door closing; if the current frame k satisfies formula (5), k is the key frame for judging the abnormality of trapped people; the calculation results are as follows Figure 2 As shown, in this embodiment, the number of elevator floor static frames is 1777, and the number of elevator door closing frames is 2116;

[0054]

[0055] Among them, M k represents the number of passengers in the elevator at the kth frame;

[0056] Step 5: Starting from key frame k, select the subsequent continuous frames ZP = {i|i = k, k + 1, ..., k + t}; calculate the elevator entrapment abnormality coefficient p by equations (6) and (7) i In this embodiment, the abnormal coefficient of elevator entrapment is 0.973.

[0057]

[0058]

[0059] Where t represents the standard limit frame number of passengers staying, d i Indicates the closed state of the elevator door in the i-th frame, l i Indicates the floor number of the i-th frame;

[0060] Step 6: Calculate the elevator trapped person abnormal threshold α (i.e. confidence level) by formula (8); if p i >α, it is determined that there is a problem of people being trapped in the current elevator; in this embodiment, the confidence level α is 0.5;

[0061]

[0062] Among them, M i represents the number of passengers in the i-th frame, W i Indicates the number of matching passenger IDs between the i-th frame and the i-1-th frame;

[0063] Step 7: Evaluate the situation of people being trapped in the elevator. Specifically, calculate the number of passengers n currently trapped in the elevator according to formula (9); calculate the time t of people being trapped in the elevator in real time according to formula (10); in this embodiment, the number of passengers is 1 and the time t of people being trapped is 142s. Figure 3 As shown;

[0064]

[0065]

[0066] where r s Indicates the frame rate of the video, fi Indicates the frame number of the current frame.

Claims

1. A method for detecting people trapped in an elevator based on video analysis, comprising the following steps: Step 1: Manually label the passengers, elevator doors, and floor numbers in each frame of elevator car surveillance video. This data set is used to train a deep convolutional neural network to develop a multi-task detection model, LiftDet, that performs passenger tracking, elevator door status determination, and floor number recognition. Step 2: For the video to be detected, use the multi-task detection model LiftDet to obtain the detection set R o ; Step 3: Select the portion of the elevator car surveillance video where no abnormal passenger entrapment occurs, and evaluate the standard limit frame number for a normal passenger stay; Step 4: From the current frame k Start by selecting the continuous frame sequence LP={ i | i=k , k -1,…, } and a sequence of consecutive frames MP of the elevator door closing; Step 5: Determine the current frame of the abnormal person entrapment k As the starting point, select the subsequent continuous frames ZP={ i | i=k , k +1, …,k+t }; Calculate the abnormal coefficient of elevator entrapment p i ; Step 6: Calculate the elevator entrapment abnormality threshold ; like p i > , then it is judged that there is a problem of people being trapped in the current elevator; Step 7: Assess the situation of people trapped in the elevator; Indicates the number of static frames of the elevator floor. t Indicates the standard limit frame number for a passenger to stay normally.

2. The method for detecting people trapped in an elevator based on video analysis according to claim 1, characterized in that: The detection set in step 2 R o ={( m ij ,c ij , d i , l i )| i =1,2,…, K, j= 1,2,3,…, M i }; in: m ij Indicates the i The first j The coordinates of the center point, width and height of each passenger's BoundingBox; c ij Indicates the i The first j Passenger ID number; d i Indicates the i The elevator door state in the frame, the specific state is determined by formula (1); l i Indicates the i The floor number of the elevator in the frame; K Indicates the number of video frames; M i Indicates the i The total number of passengers in the frame; (1)。 3. The method for detecting people trapped in an elevator based on video analysis according to claim 2, characterized in that: The specific method of evaluation in step 3 is: The ID number is c The frame numbers of passengers entering and leaving the elevator are respectively recorded as and ; Calculate the number of frames the passenger stayed this time according to formula (2) , according to formula (3) for the maximum number of stay frames Update; calculate the standard limit frame number of normal passenger stay according to formula (4) t ; (2) (3) (4) in, c Indicates the passenger's ID number, Indicates that the passenger enters the elevator key frame, Indicates the key frame that the passenger leaves the elevator. Indicates the number of frames the passenger stays. The update coefficient representing the maximum number of frames to stay, L Indicates the maximum number of floors that the elevator can reach.

4. The method for detecting people trapped in an elevator based on video analysis according to claim 3, characterized in that: The expression of the continuous frame sequence MP of the elevator door closing in step 4 is: MP={ i | i=k , k -1,…, }, The number of frames for the elevator door to close; If the current frame k Satisfying formula (5), record k The current frame for determining the abnormality of human entrapment; (5) Among them, M k Indicates the k The number of passengers in the elevator in the frame image.

5. The method for detecting people trapped in an elevator based on video analysis according to claim 4, characterized in that: The abnormal coefficient of people trapped in the elevator in step 5 p i Calculated by formula (6) and (7): (6) (7) in, t Indicates the standard limit frame number for passengers to stay. d i Indicates the i Frame of the elevator door closed state, l i Indicates the i The floor number of the frame.

6. The method for detecting people trapped in an elevator based on video analysis according to claim 5, characterized in that: The abnormal threshold value of people trapped in the elevator in step 6 Calculated by formula (8): (8) in, M i Indicates the i The number of passengers in the frame, W i Indicates the i Frame and i -1 The number of matching passenger IDs between frames.

7. The method for detecting people trapped in an elevator based on video analysis according to claim 6, characterized in that: The method for evaluating the situation of people being trapped in the elevator in step 7 is: Calculate the number of passengers currently trapped in the elevator according to formula (9) n ; According to formula (10), the time it takes for people to be trapped in the elevator is calculated in real time t ; (9) (10) in r s Indicates the frame rate of the video. f i Indicates the frame number of the current frame.

Images