Escalator operation abnormality monitoring method and system

Abstract

The application is an escalator operation abnormality monitoring method and system, which can judge the operation state of the escalator and whether the escalator appears abnormality such as reverse rotation and emergency stop by detecting the markers on the handrail of the escalator in the video stream. The deep learning target detection method can effectively improve the detection accuracy and speed compared with the single optical flow method, and further improve the robustness by combining the optical flow method for auxiliary judgment when the markers are blocked.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular, it relates to a method and system for monitoring abnormal operation of escalators. Background Technology

[0002] In recent years, with the continuous improvement and development of urban rail transit, the number of escalator safety accidents has also been increasing. The safe operation of escalators is closely related to public safety and has always been highly valued by safety supervision departments from all sectors of society. Real-time monitoring of escalator operation status, and the immediate detection and alarm notification of staff to maintain escalators when abnormalities occur, are crucial to ensuring the normal operation of escalators and the safety of passengers.

[0003] With the development and application of computer vision technology, the advantages of high precision and portability of deep learning methods have become increasingly prominent. Chinese invention patent (application number 202111082065.1) discloses a method and system for detecting abnormal states of escalators. It proposes to remove obstacles other than escalators by using an improved U-Net model, and then use optical flow method to detect abnormal states of escalators. Although the U-Net model has good image processing effect as a model in the field of image segmentation, it has a large amount of computation and is slow. At the same time, although the optical flow method is a traditional motion analysis method, it is easy to implement but has poor real-time performance and low precision. Summary of the Invention

[0004] The technical problem that this invention aims to solve is that existing detection methods suffer from poor real-time performance and low accuracy.

[0005] This invention proposes a method and system for monitoring abnormal operation of escalators.

[0006] A method for monitoring escalator malfunctions includes the following steps:

[0007] Step 1: Add at least one marker to the movable part of the escalator and obtain a sample of the surveillance video image of the escalator.

[0008] Step 2: Apply VisionTransformer to Backbone to perform attention weighting on the channels, build a YOLOv5 and VisionTransformer fusion target detection model, learn and train on the markers in the samples, and obtain the trained target detection model.

[0009] Step 3: Use the trained target detection model for real-time monitoring video. Set the area of ​​the escalator going up or down as the target detection area. Obtain the position of the marker through the target detection model. When the position changes in the opposite direction of the running direction or the position remains unchanged, send an alarm message.

[0010] Existing methods employ a two-stage approach for escalator anomaly detection, consisting of an image processing stage and a motion state analysis stage. The first stage uses U-net image segmentation to remove obstacles other than the escalator, while the second stage uses a single optical flow method to analyze the escalator's motion state. This invention employs a single-stage approach for escalator anomaly detection. It directly analyzes the motion state of the escalator in the acquired video stream using target detection technology, reducing the significant computational power and time consumed in the image processing stage. Furthermore, it combines optical flow for auxiliary analysis in scenarios with dense pedestrian occlusion. This combination of target detection and optical flow effectively improves the robustness of escalator motion state detection in practical applications. By building a fusion model of YOLOv5 and an optimized VisionTransformer for marker detection, the detection accuracy of escalator anomalies is optimized while maintaining real-time performance, achieving higher accuracy and faster speed than existing single optical flow motion analysis methods.

[0011] Preferably, in step 2, after building the target detection model, the characteristics of pedestrians are also learned and trained. In step 3, if a pedestrian is detected and no landmark is detected within a preset time range, the optical flow method is used to obtain the running direction of the escalator. If the direction is opposite to the normal running direction, an alarm prompt is sent.

[0012] Preferably, in step 2, in the input part of VisionTransformer, the channels obtained by convolution sampling are passed through a linear layer to obtain absolute position information, and the LayerNorm layer and MLP in VisionTransformer are replaced with a linear layer. In the output part of VisionTransformer, the channel dimension is restored using the Reshape method.

[0013] Preferably, in step 3, obtaining the position of the marker includes detecting the ordinate of the center point of the marker target box. When there is only one marker target box in the detection result, the ordinate of the center point is directly used as the current position. When there are multiple marker target boxes in the detection result, the ordinate of the center point of the smallest bounding rectangle of the multiple target boxes is used as the current position of the marker.

[0014] Preferably, in step 3, the handrail tilting stage is taken as the target detection area. The target detection area can be a rectangle or a convex polygon. At most one marker can appear on one side of the handrail at the same time within the target detection area.

[0015] Preferably, when adding more than one marker, different markers are selected for different side handrails, and the same marker is selected for the same side handrail.

[0016] Preferably, in step 2, the markers in the sample include marker 1 and marker 2. In step 3, the method for judging the change in the position of the markers is as follows: record the ordinates y1 and y2 of the center points of marker 1 and marker 2 at time i, and after a time interval T, record the ordinates y3 and y4 of the center points of marker 1 and marker 2 at time i+T.

[0017] If y3-y1>L / 2 or y4-y2>L / 2, where L is the width of the marker, it indicates that the escalator is operating in the normal direction.

[0018] If 0≤y3-y1≤L / 2 or 0≤y4-y2≤L / 2, the operation will stop and an alarm message will be issued.

[0019] If y3-y1<0 or y4-y2<0, it indicates that the operation has reversed and an alarm message will be issued.

[0020] Preferably, when adding more than two markers to the handrail on the same side, the markers should be spaced evenly across the handrail.

[0021] Preferably, the Lucas-Kanade optical flow method is used to detect the escalator's operating direction. The detection process of the optical flow method is as follows:

[0022] (1) Obtain the images of the i-th frame and the (i+1)-th frame;

[0023] (2) Cropping the detection region in the image;

[0024] (3) Perform feature corner detection on the captured target area to obtain corner feature information;

[0025] (4) Find the corresponding feature points in the i-th frame in the (i+1)-th frame;

[0026] Calculate the average value of feature points in the i-th and i+1-th frames respectively, calculate the optical flow field, determine the running direction of the escalator, and issue an alarm if the direction is abnormal.

[0027] An escalator operation anomaly monitoring system, applying the aforementioned escalator operation anomaly monitoring method, includes a monitoring device and a marker. The monitoring device is used to photograph the escalator, and the marker is set on the escalator handrail.

[0028] Compared with the prior art, the present invention has the following advantages and effects:

[0029] By detecting the markers on the escalator handrail in the video stream, the operating status of the escalator can be determined, and whether the escalator has experienced abnormalities such as reversal or sudden stop. Compared with the single optical flow method, the deep learning object detection method can effectively improve the detection accuracy and speed. At the same time, when the marker is occluded, combining it with the optical flow method to assist in the judgment further improves the robustness. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 This is a schematic diagram illustrating the setting of markers and the delineation of detection areas in the escalator monitoring scenario of the present invention.

[0032] Figure 2 This is a network structure diagram of the target detection model of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] Example 1:

[0035] A method for monitoring abnormal operation of escalators includes the following steps:

[0036] Step 1: Deploy at least one marker on the escalator, such as on the steps or handrail, as follows: Figure 1 The diagram shows two markers placed on the handrail, or two markers placed on either side of a tread. The markers can be physical or non-physical, and can be placed on or to one side of the escalator handrail. Non-physical markers are recommended, such as adding symbols to the handrail in a different color than the escalator itself. The number of markers can be increased appropriately according to the length of the conveyor belt, and the markers should be evenly spaced along the handrail. Different markers can be selected for the left and right handrails, such as different colored symbols, and multiple markers on the same side of the handrail should be identical. For example, marker 1 can be placed on the left handrail, and marker 2 on the right handrail.

[0037] Step 2: Obtain video footage from the escalator monitoring system, collect image samples of landmarks, and label landmark 1, landmark 2, and pedestrians as classification targets.

[0038] Step 3: Build a YOLOv5 and optimized VisionTransformer fusion object detection model. The overall model structure is as follows: Figure 2As shown. Since VisionTransformer, as an image classification model, cannot be directly combined with the feature pyramid of YOLOv5, the VisionTransformer structure needs to be optimized before being applied to the Backbone for attention-weighted processing of channels. The main optimizations are as follows:

[0039] In the input part of the VisionTransformer, the channels obtained by convolutional sampling are processed through a LiNear linear layer to obtain absolute position information. Replacing the LiNearNorm and MLP layers in the VisionTransformer with linear layers accelerates computational efficiency without affecting model training stability. Since the VisionTransformer module only needs to perform attention weighting on the channels and does not need to obtain classification results like the original image classification task, the Reshape method is directly used in the output part to restore the channel dimensions, ensuring that the channels are correctly positioned for subsequent convolutional calculations in the feature pyramid.

[0040] Step 4: Use labeled data samples to train the model, and use the trained model for target detection in real-time monitoring video stream. Based on this, the detection results of landmark 1, landmark 2 and pedestrians are obtained.

[0041] Step 5, Detection area filtering: Set the detection area according to the following rules and conditions, such as... Figure 2 As shown: The detection area can be rectangular or convex polygonal. Select a portion of the escalator's upward or downward movement, specifically the handrail tilting phase, as the detection area. Within the target detection area, at most one marker can appear on one handrail at any given time, i.e., one marker on each of the left and right handrails. By setting the detection area, the target detection results from step 4 outside the detection area are filtered out, and the remaining detection results are sent to step 6 for anomaly detection.

[0042] Step 6, as follows Figure 1 As shown, the escalator is in normal upward motion, and the camera is located above the escalator entrance. Record the ordinates y1 and y2 of the center points of marker 1 and marker 2 at time i. After a time interval T, record the ordinates y3 and y4 of the center points of marker 1 and marker 2 at time i+T.

[0043] If y3-y1>L / 2 or y4-y2>L / 2, where L is the width of the marker, it indicates that the escalator is running in the normal direction; if 0≤y3-y1≤L / 2 or 0≤y4-y2≤L / 2, it indicates that the escalator has stopped and an alarm message is issued; if y3-y1<0 or y4-y2<0, it indicates that the escalator has reversed direction and an alarm message is issued; if marker 1 or marker 2 is not detected after time interval T, proceed to step 7 for anomaly judgment.

[0044] Step 7: Determine if there are pedestrians in the area.

[0045] If there are no pedestrians and the sign is not obstructed, the escalator will stop operating and an alarm will be triggered.

[0046] If there are pedestrians or the number of pedestrians exceeds the set threshold, it may be due to the marker being blocked, resulting in the marker not being detected. In this case, the abnormal judgment in step 8 will be initiated.

[0047] Step 8: The Lucas-Kanade optical flow method is used to detect the escalator's operating direction. The anomaly detection process of the optical flow method is as follows:

[0048] Obtain the images of the i-th frame and the (i+1)-th frame;

[0049] The detection region in the image is cropped;

[0050] Feature corner detection is performed on the captured target area to obtain corner feature information;

[0051] Find the corresponding feature points in the i+1th frame of the image;

[0052] Calculate the average value of feature points in the i-th and i+1-th frames respectively, calculate the optical flow field, determine the running direction of the escalator, and issue an alarm if the direction is abnormal.

[0053] An escalator operation anomaly monitoring system, applying the aforementioned escalator operation anomaly monitoring method, includes a monitoring device and markers. The monitoring device is used to photograph the escalator, for example, a high-definition camera. The markers are placed on the escalator handrail, for example, yellow markers.

[0054] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An escalator abnormality operation monitoring method characterized by, The method comprises the following steps: Step 1: adding at least one marker on the movable part of the escalator to obtain a monitoring video image sample of the escalator; Step 2: applying a Vision Transformer to the Backbone to perform attention weighting processing on the channels, obtaining absolute position information of the channels obtained by convolution sampling at the input part of the Vision Transformer, replacing the LayerNorm layer and the MLP in the Vision Transformer with a Linear layer, and adopting a Reshape method to restore the channel dimension at the output part of the Vision Transformer to build a target detection model fused with yolov5 and the Vision Transformer, learning and training the markers in the sample to obtain a trained target detection model; Step 3: using the trained target detection model for real-time monitoring of the video, setting the area where the escalator is running upward or downward as the target detection area, and obtaining the position of the marker through the target detection model, and sending an alarm information when the position changes and the running direction is opposite or the position is unchanged.

2. The escalator abnormality monitoring method according to claim 1, characterized by, In the step 2, after building the target detection model, the features of pedestrians are also learned and trained, and in the step 3, if a pedestrian is detected and no marker is detected within a preset time range, a flow method is used to obtain the running direction of the escalator, and an alarm prompt is sent when the direction is opposite to the normal running direction.

3. The escalator abnormality monitoring method according to claim 1, characterized by, In the step 3, obtaining the position of the marker includes detecting the center point vertical coordinate of the marker target box, when the detection result has only one marker target box, directly using the center point vertical coordinate as the current position, and when the detection result has multiple marker target boxes, using the center point vertical coordinate of the minimum circumscribed rectangle of the multiple target boxes as the current position of the marker.

4. The escalator abnormality operation monitoring method according to claim 1, characterized by In the step 1, the marker is arranged on the handrail, and in the step 3, the handrail inclination stage is set as the target detection area, and the target detection area can be a rectangle or a convex polygon, and at most one marker appears on the same side of the handrail at the same time in the target detection area.

5. The escalator abnormality monitoring method according to claim 1, characterized by In the step 1, the marker is arranged on the handrail, and when more than one marker is added, different markers are selected for different sides of the handrail, and the same marker is selected for the same side of the handrail.

6. The escalator abnormality operation monitoring method according to claim 1, characterized by In the step 2, the markers in the sample include marker 1 and marker 2, and in the step 3, the judgment method for the position change of the marker is as follows: recording the center point vertical coordinates y1 and y2 of the marker 1 and the marker 2 at the ith moment, and recording the center point vertical coordinates y3 and y4 of the marker 1 and the marker 2 at the (i+T)th moment after a time interval T; If y3-y1>L / 2 or y4-y2>L / 2, L is the width of the marker, indicating that the running direction of the escalator is normal; If 0≤y3-y1≤L / 2 or 0≤y4-y2≤L / 2, indicating that the running is stopped, and an alarm information is sent; If y3-y1<0 or y4-y2<0, indicating that the running is reversed, and an alarm information is sent.

7. The escalator abnormality operation monitoring method according to claim 2, characterized by The Lucas-Kanade optical flow method is used to detect the running direction of the escalator, and the detection process of the optical flow method is as follows: (1) obtaining the i-th frame and the i+1-th frame images; (2) intercepting the detection area in the image; (3) detecting the feature corner points of the intercepted target area to obtain the corner feature information; (4) finding the corresponding feature points in the i-th frame image in the i+1-th frame image; The average values of the feature points of the i-th frame and the i+1-th frame images are calculated, the optical flow field is calculated, the running direction of the escalator is judged, and if the direction is abnormal, an alarm information is sent.

8. The escalator abnormality operation monitoring method according to claim 5, characterized by When more than two markers are added to the same side handrail, the setting interval of the markers needs to be equal to the handrail.

9. An escalator abnormality monitoring system, which applies the escalator abnormality monitoring method according to any one of claims 1 to 8, characterized by The monitoring device is used to shoot the escalator, and the markers are arranged on the movable part of the escalator.

Images