CCTV image analysis-based escalator emergency stop and reverse running detection system and its method

Abstract

The present invention relates to a system and method for detecting sudden stops and reverse movement of an escalator based on CCTV video analysis, and more particularly to a system and method for detecting abnormal situations of an escalator that automatically detects abnormal behaviors such as sudden stops and reverse movement of an escalator using deep learning-based video analysis technology, and simultaneously analyzes the situation of passenger density on the escalator to detect accident risks in advance. The present invention comprises: a video acquisition unit that receives real-time video data from a CCTV camera; a video preprocessing unit that performs preprocessing on the video data regarding at least one of setting a region of interest, image stabilization, lighting correction, or noise removal; an abnormal situation detection unit that detects a sudden stop or reverse movement of an escalator by analyzing the dense optical flow between consecutive frames of the preprocessed video data and analyzing the movement patterns of passengers based on deep learning; and a determination and notification unit that determines whether an abnormal situation exists based on the analysis results of the abnormal situation detection unit and transmits a notification regarding the abnormal situation to a control system according to the determination result. A CCTV video analysis-based escalator sudden stop and reverse movement detection system and method are provided, comprising a control linkage unit that transmits a control signal including a deceleration or stop command in conjunction with an escalator control system according to the judgment result.

Description

Technology Field

[0001] The present invention relates to a system and method for detecting sudden stops and reverse movements of an escalator based on CCTV video analysis, and more specifically, to a system and method for detecting abnormal situations of an escalator that automatically detects abnormal movements such as sudden stops and reverse movements of an escalator using deep learning-based video analysis technology, and simultaneously analyzes the situation of escalator passenger density to detect the risk of accidents in advance. Background Technology

[0003] Escalators are representative automated transportation facilities widely used to efficiently support human movement in multi-use facilities such as large shopping malls, subway stations, airports, and large buildings. Since these escalators are designed to carry a continuous number of users, there is a possibility that mechanical malfunctions or errors in the direction of travel during operation could lead to major safety accidents.

[0004] In particular, types of accidents that can occur on escalators include sudden stopping of operation, moving in reverse, and falls caused by overcrowding of users. If these situations are not detected and responded to quickly, they can lead to a chain of casualties. Therefore, there is a continuously increasing need for technology that can monitor abnormal operation and usage conditions of escalators in real time and detect dangerous situations early.

[0005] In this regard, the prior Korean Registered Patent No. 10-1329526 relates to a control method for a real-time safe operation system of an escalator, disclosing CCTV video-based surveillance, detection of human movement by a video detection unit, escalator control in the event of an abnormal situation, monitoring, and DVR recording.

[0006] However, conventional technology is structured to detect dangerous situations based on human behavior and is merely a method of determining accident situations by detecting human actions occurring around the escalator using CCTV footage. It has a problem in that it cannot directly determine abnormal movements, such as sudden stops or reverse movement, by analyzing the actual operating state of the escalator, and thus cannot detect risks caused by mechanical defects. Prior art literature

[0008] Korean Registered Patent No. 10-1329526 The problem to be solved

[0009] The present invention was devised to solve the problems of the prior art described above, and aims to provide a system capable of analyzing the operating status of an escalator in real time by analyzing CCTV video data installed on the escalator and automatically detecting abnormal operations such as sudden stops and reverse movement.

[0010] In addition, the present invention aims to provide an escalator safety monitoring system that can detect accident risk situations in advance and strengthen safety management by analyzing the behavior and crowding status of escalator passengers using deep learning-based video analysis technology.

[0011] In addition, the present invention aims to provide an efficient surveillance system capable of detecting abnormal escalator conditions by utilizing existing CCTV infrastructure without the installation of additional mechanical sensors, compared to existing sensor-based safety systems. means of solving the problem

[0013] To achieve the above objective, the present invention provides a CCTV video analysis-based escalator sudden stop and reverse movement detection system for detecting abnormal situations of an escalator, comprising: an image acquisition unit that receives real-time video data from at least one CCTV camera installed on the escalator; an image preprocessing unit that performs preprocessing on the video data for at least one of setting a region of interest, image stabilization, lighting correction, or noise removal; an abnormal situation detection unit that detects a sudden stop or reverse movement situation of the escalator by analyzing dense optical flow between consecutive frames of the preprocessed video data and analyzing the movement pattern of a passenger based on deep learning; a determination and notification unit that determines whether an abnormal situation exists based on the analysis result of the abnormal situation detection unit and transmits a notification regarding the abnormal situation to a control system according to the determination result; and a control linkage unit that transmits a control signal including a deceleration or stop command in conjunction with an escalator control system according to the determination result.

[0014] Preferably, the image preprocessing unit includes an ROI setting unit that sets an ROI in at least one of the step area, handrail area, or passenger area of ​​the escalator, and the ROI setting unit can be set to an ROI that is pre-set by a user to detect at least one abnormal situation of the escalator.

[0015] Preferably, the image preprocessing unit further includes an image stabilization unit that corrects camera vibration or image shake.

[0016] Preferably, the image preprocessing unit may further include a lighting correction unit that performs adaptive histogram equalization processing to remove lighting changes or shadow effects.

[0017] Preferably, the image preprocessing unit may further include a noise removal unit that removes image noise using a Gaussian filter or a non-local mean filter.

[0018] Preferably, the system may further include an optical flow analysis module that detects changes in the movement speed or direction of an escalator step by analyzing dense optical flow between consecutive frames of the preprocessed image data; a step state classification module that classifies the operating state of the escalator using a deep learning-based image classification model; and a passenger behavior analysis module that analyzes the movement patterns of passengers through object detection and multi-object tracking.

[0019] According to an embodiment, the optical flow analysis module may calculate a pixel-unit motion vector between consecutive frames using the Farneback or Lucas-Kanade algorithm, generate a sudden stop candidate event when the motion vector decreases to a size below a preset threshold, and generate a reverse motion candidate event when the direction of the motion vector is reversed by more than a predetermined angle.

[0020] According to an embodiment, the step state classification module can classify the driving state into at least one driving state including normal uphill, normal downhill, sudden stop, reverse driving, or stop state using a CNN-based image classification model.

[0021] According to an embodiment, the occupant behavior analysis module can track the movement trajectory of the occupant using an object detection model or a multi-object tracking algorithm, and detect abnormal behavior patterns by analyzing at least one of the occupant density, movement speed, or change in movement direction of the occupant.

[0022] According to an embodiment, the judgment and notification unit fuses the results of the optical flow analysis module, the step state classification module, and the occupant behavior analysis module to calculate a final score (S) using a weighted voting method, wherein the mathematical formula of the weighted voting method is calculated by S = w1 × P(OF) + w2 × P(DL) + w3 × P(BA), and if the final score (S) exceeds a preset threshold, it can be finally determined as an abnormal situation of the corresponding grade.

[0023] Preferably, the control linkage unit can communicate with a PLC or inverter control system to transmit the control signal and record an event that automatically saves images before and after the occurrence of an abnormal situation.

[0024] In addition, the present invention provides a CCTV video analysis-based escalator stop and reverse movement detection method for detecting sudden stop and reverse movement situations of an escalator using CCTV video analysis, comprising: receiving real-time video data from at least one CCTV camera installed on the escalator; performing preprocessing on the video data for at least one of setting a region of interest, image stabilization, lighting correction, or noise removal; analyzing dense optical flow between consecutive frames of the preprocessed video data and analyzing the movement pattern of a passenger based on deep learning to detect a sudden stop or reverse movement situation of the escalator; determining whether an abnormal situation exists based on the results of the analysis and transmitting a notification regarding the abnormal situation to a control system according to the determination result; and transmitting a control signal including a deceleration or stop command in conjunction with an escalator control system according to the determination result.

[0025] Preferably, the step of detecting the sudden stop or reverse driving situation may further include: calculating a pixel-unit motion vector between consecutive frames of preprocessed image data using a Farneback or Lucas-Kanade algorithm, generating a sudden stop candidate event when the motion vector decreases to a size below a preset threshold, and generating a reverse driving candidate event when the direction of the motion vector reverses by more than a predetermined angle; classifying the driving state into at least one driving state including normal uphill, normal downhill, sudden stop, reverse driving, or stop state using a CNN-based image classification model; and tracking the movement trajectory of a passenger using an object detection model or a multi-object tracking algorithm, and detecting an abnormal behavior pattern by analyzing at least one of the passenger's density, movement speed, or change in movement direction.

[0026] In addition, the present invention may provide a computer-readable storage medium in which a program having the above-described method implemented is stored. Effects of the invention

[0028] According to the present invention having the configuration described above, unlike conventional mechanical sensor-based systems, abnormal operating conditions of an escalator can be detected without the installation of separate sensors by utilizing CCTV video analysis, thereby providing the advantage of reducing installation and maintenance costs.

[0029] In addition, by combining optical flow-based motion analysis and deep learning-based state classification technology, there is an advantage of being able to detect abnormal situations such as sudden stops and reverse movement of escalators with high accuracy.

[0030] In addition, by analyzing the movement patterns and density of escalator users, there is the advantage of being able to identify situations with a high probability of accidents in advance and take appropriate response measures.

[0031] In addition, there is the advantage of improving the efficiency of escalator safety management by rapidly detecting abnormal situations through real-time video analysis and transmitting alerts to managers. Brief explanation of the drawing

[0033] Figure 1 shows the overall system configuration diagram of the CCTV video analysis-based escalator sudden stop and reverse movement detection system of the present invention. FIG. 2 is an image screen showing the setting of a region of interest (ROI) for an escalator area during an image preprocessing process according to an embodiment of the present invention. FIG. 3 is a screen for calculating pixel-unit movement vectors in an optical flow analysis module according to one embodiment of the present invention. FIG. 4 is a screen for analyzing the movement direction and operation status of an escalator step in a step status classification module according to one embodiment of the present invention. FIG. 5 is a video screen illustrating an example of detecting abnormal behavior by analyzing the movement pattern, density, or crowd state of a passenger in a passenger behavior analysis module according to an embodiment of the present invention. FIG. 6 is a video screen showing the detection of luggage in a situation where a passenger is using an escalator while carrying luggage, according to an embodiment of the present invention. FIG. 7 is a block diagram illustrating a process of calculating a final risk level and determining an abnormal situation by weighting the risk values ​​calculated from a plurality of analysis modules according to an embodiment of the present invention. FIG. 8 is a video screen that displays the status of an escalator user as text at the top of a monitor by transmitting the content analyzed by the judgment and notification unit to an actual CCTV video according to one embodiment of the present invention. FIG. 9 is a flowchart illustrating the overall processing procedure of the CCTV video analysis-based escalator sudden stop and reverse movement detection method of the present invention. FIG. 10 is a flowchart illustrating a detailed analysis procedure of an abnormal situation detection step according to an embodiment of the present invention. Specific details for implementing the invention

[0034] The present invention will be described in detail below with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Identical reference numerals in each drawing indicate components that perform substantially the same function.

[0035] The purpose and effects of the present invention may be naturally understood or become clearer through the following description, and the purpose and effects of the present invention are not limited solely to the description below. Furthermore, in describing the present invention, if it is determined that a detailed description of known technology related to the present invention may unnecessarily obscure the essence of the present invention, such detailed description will be omitted.

[0036] Figure 1 shows the overall system configuration diagram of the CCTV video analysis-based escalator sudden stop and reverse movement detection system (1) of the present invention.

[0037] Referring to FIG. 1, the system (1) of the present invention may include an image acquisition unit (10), an image preprocessing unit (20), an abnormal situation detection unit (30), a judgment and notification unit (40), and a control linkage unit (50).

[0038] The image acquisition unit (10) can perform the function of receiving real-time video data from at least one CCTV camera installed around the escalator.

[0039] The image preprocessing unit (20) can perform preprocessing processes such as setting a region of interest, image stabilization, lighting correction, and noise removal to improve the quality of image data and clarify the area to be analyzed.

[0040] The abnormal situation detection unit (30) can detect sudden stop or reverse movement of the escalator by performing optical flow analysis, deep learning-based operation status classification, and passenger behavior analysis based on preprocessed image data.

[0041] The judgment and notification unit (40) calculates the final risk level by synthesizing the above analysis results and determines whether there is an abnormal situation, and can transmit a notification signal to the control system if an abnormal situation occurs. The final risk level in this specification is the same concept as the final score (S).

[0042] The control linkage unit (50) can transmit a control signal including a deceleration or stop command by linking with a PLC or inverter-based escalator control system according to the judgment result.

[0043] FIG. 2 is an image screen showing the setting of a region of interest (ROI) for an escalator area during an image preprocessing process according to an embodiment of the present invention.

[0044] Referring to FIG. 2, (A) shows an original video of an escalator, (B) shows an image with a region of interest set, (C) shows an image with the analysis results of the region of interest visualized, and (D) shows an example of an image with the escalator step area highlighted.

[0045] Referring to FIG. 2(A), the original escalator video obtained from a CCTV camera is shown. The video may include the escalator step area, the handrail area, and the passenger area.

[0046] In the heatmap, the area between the two handrails of the escalator begins to be recognized as a single connected pattern and is separated into a unique pattern area distinct from the surrounding floor or wall surface. The ROI setting unit (21) can set the area requiring analysis in the escalator image as a Region of Interest (ROI). For example, at least one of the escalator step area, handrail area, or passenger movement area can be set as a Region of Interest.

[0047] Referring to Fig. 2(B), an image with an ROI set is shown, and the characteristic shape of the escalator structure can be extracted as a single active area.

[0048] In the feature map, it is observed that the overall trapezoidal / triangular shape of the escalator converges into a single activation area. In particular, the diagonal activation pattern corresponding to the inclination angle of the escalator handrail is distinct. The ROI setting unit (21) can set an area of ​​interest, and the image stabilization unit (22) can perform image stabilization on the set area of ​​interest.

[0049] The image preprocessing unit (20) may include an image stabilization unit (22). The image stabilization unit (22) may perform a function to correct image shaking caused by minute vibrations, external shocks, or environmental changes that may occur in the installation environment of the CCTV camera.

[0050] According to an embodiment, the image stabilization unit (22) can perform image stabilization by tracking feature points between consecutive frames or calculating a transformation matrix between frames to estimate the movement of the image and correcting the estimated movement.

[0051] In addition, the image stabilization unit (22) can improve the accuracy of optical flow analysis and object tracking performed in subsequent steps by eliminating unnecessary image fluctuations caused by camera movement.

[0052] FIG. 2(C) shows an image in which the analysis results for the region of interest are visualized, and the lighting correction unit (23) can correct brightness deviations caused by lighting changes, shadows, or reflections. Referring to FIG. 2(C), even in an image (sample_1) with multiple people on board, activation is concentrated on the exposed handrail belt and comb plate area of ​​the entrance instead of the stair surface area obscured by the passengers. The lighting correction unit (23) can correct changes in image brightness caused by lighting changes, shadows, reflections, etc.

[0053] This demonstrates that the model has learned a part-based recognition strategy robust to occlusion. The analysis results for the region of interest are visualized, and lighting correction can be performed simultaneously.

[0054] The image preprocessing unit (20) may include an illumination correction unit (23). The illumination correction unit (23) may perform a function to correct changes in image brightness caused by changes in lighting, shadows, or reflections that may occur in an indoor environment.

[0055] According to an embodiment, the lighting correction unit (23) can improve the contrast of the image and correct the brightness deviation using an Adaptive Histogram Equalization or Contrast Limited Adaptive Histogram Equalization (CLAHE) technique.

[0056] This allows for clearer extraction of features of the escalator step area and passenger area, and improves the reliability of deep learning-based image classification and behavior analysis.

[0057] FIG. 2(D) shows an image in which the escalator step area is highlighted, and the noise removal unit (24) can remove sensor noise, compression noise, or environmental noise included in the image.

[0058] Referring to Fig. 2(D), the entire area where the escalator is located appears as a single hot zone in the heatmap, while the background area shows a low-activity state. It is confirmed that an activation pattern covering the entire range of the escalator is maintained even when multiple people are on board. The image is transformed into one with the escalator step area emphasized, and noise removal can be performed.

[0059] The image preprocessing unit (20) may include a noise removal unit (24). The noise removal unit (24) may perform the function of removing sensor noise, compression noise, or environmental noise that may occur during the image acquisition process.

[0060] According to an embodiment, the noise removal unit (24) may reduce noise in an image using an image filtering technique such as a Gaussian filter, a non-local means filter, or a median filter. However, it is not limited thereto and may include any configuration for removing noise from an image.

[0061] Through such noise removal processing, unnecessary pixel fluctuations within the image can be minimized, and false detections during optical flow analysis and object detection processes can be reduced.

[0062] In this way, the image preprocessing unit (20) may include an ROI setting unit (21), an image stabilization unit (22), an illumination correction unit (23), and a noise removal unit (24), and the preprocessing process can improve the accuracy of abnormal situation detection performed in subsequent steps by clearly extracting the area to be analyzed from the escalator image and improving image quality.

[0063] The abnormal situation detection unit (30) may include an optical flow analysis module (31), a step state classification module (32), and a passenger behavior analysis module (33), and through the learning process of the present invention, the results of each module are shown as in FIGS. 3 to 5.

[0064] The learning process of the abnormal situation detection unit (30) is explained below.

[0065] <Learning Process Example>

[0066] The exemplary model of the present invention can be trained by applying a multi-stage learning strategy to perform stable recognition in various CCTV environments. The following illustrates the multi-stage process of the exemplary learning of the present invention.

[0067] (1) Composition of training dataset

[0068] In an embodiment of the present invention, a training dataset including various occupant densities and various CCTV camera viewing angles was constructed as shown in [Table 1]. In particular, to enhance learning regarding occlusion situations that may occur in real environments, the frame ratio containing occupants was maintained at a high level.

[0069] division Number of images explanation Training dataset (Train) 4,258 sheets Training data including escalator frames extracted from CCTV footage Validation dataset (Validation) Chapter 68 Validation data separated independently from the training data Test dataset (Test) Chapter 41 Independent evaluation data not used in the learning process Number of classes 1 Escalator single class

[0070] Approximately 81% of the frames in the training data contain passengers (one or more), and this data composition can be used to learn to reliably recognize the escalator structure even in situations where it is obscured by passengers.

[0072] (2) Model training process

[0073] In an embodiment of the present invention, a three-step progressive training strategy was applied for model training as shown in [Table 2].

[0074] Learning stages Epochs Learning rate (LR) Data Augmentation note Step 1 (Initial Learning) 100 0.01 Basic Mosaic Augmentation COCO Pre-trained Weighted Transfer Learning Step 2 (Fine Tuning) 50 0.002 Copy-Paste 0.3, MixUp 0.15, Erasing 0.5 Step 1 result-based Step 3 (Extended Learning) 80 0.005 Copy-Paste 0.2, MixUp 0.1, Erasing 0.4 2-step result-based

[0075] The input video resolution was set to 640×1138 (height×horizontal) to maintain the 16:9 aspect ratio of the original CCTV video. In addition, mosaic data augmentation was disabled 10 epochs prior to the end of training (close_mosaic=10) to ensure the convergence stability of the model.

[0077] (3) Model performance evaluation

[0078] The model performance according to the embodiment of the present invention was quantitatively evaluated on a test dataset (Chapter 41) that was not used for training, as shown in [Table 3] to [Table 6]. As evaluation metrics, mIoU, Dice coefficient, pixel-level Precision / Recall, and COCO standard mAP, which are widely used in the field of segmentation, were used.

[0079] ① Segmentation performance

[0080] Evaluation Indicators Measured value Standard deviation explanation mIoU 0.8361 ±0.1297 Average IoU between the prediction mask and the correct answer mask Dice (F1-seg) 0.9044 ±0.0897 Segmentation F1-score Pixel Precision 0.8973 ±0.0977 Proportion of actual objects within the predicted mask area Pixel Recall 0.9222 ±0.1005 Proportion of correctly predicted actual object areas Pixel F1-score 0.9044 ±0.0897 Harmonic mean of Precision and Recall Pixel Accuracy 0.9914 ±0.0075 Proportion of correctly classified pixels out of the total

[0082] ② COCO Standard mAP Evaluation

[0083] Evaluation Indicators Measured value explanation Box mAP@50 0.9605 Average boundary box precision based on IoU 0.5 Box mAP@50-95 0.8139 IoU 0.5~0.95 Average precision Mask mAP@50 0.9616 Average mask precision based on IoU 0.5 Mask mAP@50-95 0.7926 IoU 0.5~0.95 Average mask precision

[0085] ③ Detection success rate by IoU threshold

[0086] IoU threshold Number of successful images success rate ≥ 0.50 40 / 41 97.6% ≥ 0.60 38 / 41 92.7% ≥ 0.70 35 / 41 85.4% ≥ 0.75 32 / 41 78.0% ≥ 0.80 31 / 41 75.6% ≥ 0.85 27 / 41 65.9% ≥ 0.90 19 / 41 46.3%

[0088] ④ Instance Detection Accuracy

[0089] Evaluation items Measured value Instance count matching accuracy 85.4% Average number of GT instances 1.85 Predicted average number of instances 1.85 0~2 occupant frame detection success rate 100% Inference speed 8.3 ms / frame

[0091] (4) Validation of ROI setting strategy

[0092] In an embodiment of the present invention, a strategy can be applied in which an escalator area is detected once in a CCTV image to set an ROI, and then the ROI is fixed and used.

[0093] To verify this, a total of 897 frames were extracted and evaluated at 10-second intervals from 2.5 hours of CCTV footage (1920×1080, 30fps).

[0094] Number of occupants in the frame Number of frames Escalator detection rate 0 people 1 100% 1~2 people 4 100% 3~5 people 102 89% 6~10 people 512 71% 11 or more people 278 68%

[0096] In particular, in low-density frames with 0 to 2 passengers, 2 escalators were detected with 100% accuracy. Accordingly, it was confirmed that a method of setting the ROI once in frames during times with few passengers after CCTV installation and then fixing it is practically effective.

[0097] In addition, the reliability of ROI setting can be further enhanced by applying a multi-frame consensus method that combines detection results from multiple frames.

[0098] FIG. 3 is a screen for calculating pixel-unit movement vectors in an optical flow analysis module according to one embodiment of the present invention.

[0099] Referring to FIG. 3, an example is illustrated of calculating a pixel-unit movement vector using optical flow to analyze the movement occurring in the step area of ​​an escalator.

[0100] Figure 3(A) shows the original video frame, Figure 3(B) shows the optical flow vector analysis result, Figure 3(C) shows the vector distribution in a normal operating state, and Figure 3(D) shows the vector change in a reverse driving or sudden stop state.

[0101] The original video frame of Fig. 3(A) illustrates the original video frame of the escalator obtained from a CCTV camera.

[0102] The above video may include the step area, handrail area, and passenger area of ​​the escalator, and such video data may be used as input data for performing motion analysis between consecutive frames by the optical flow analysis module (31).

[0103] The optical flow vector analysis result of Fig. 3(B) illustrates the optical flow vector calculated by analyzing pixel movement between consecutive image frames.

[0104] The optical flow analysis module (31) can calculate pixel-unit motion vectors between frames using the Farneback algorithm or the Lucas-Kanade algorithm, and can express the direction of movement and the magnitude of movement for each pixel or feature point in vector form.

[0105] This allows for the quantitative analysis of the movement direction and speed of the escalator steps.

[0106] The vector distribution in the normal operation state of Fig. 3(C) illustrates the optical flow vector distribution when the escalator is operating normally.

[0107] Under normal operating conditions, the optical flow vector is formed in a constant direction according to the direction of movement of the escalator step, and the magnitude of the vector can also maintain a relatively constant range.

[0108] For example, in the case of upward operation, an upward vector can be formed in a consistent pattern corresponding to the step movement direction.

[0109] The vector change in the reverse or sudden stop state of Fig. 3(D) illustrates the optical flow vector change occurring in the sudden stop or reverse movement situation of the escalator.

[0110] The optical flow analysis module (31) can determine abnormal situations by analyzing changes in the magnitude and direction of the vector, and if the magnitude of the vector decreases below a preset threshold, an emergency stop candidate event may be generated, and if the direction of the vector reverses by more than a predetermined angle relative to the normal driving direction, a reverse driving candidate event may be generated.

[0111] FIG. 4 is a screen for analyzing the movement direction and operation status of an escalator step in a step status classification module according to one embodiment of the present invention.

[0112] The step state classification module (32) can classify the escalator state according to the embodiment as normal up, normal down, sudden stop, reverse movement, and stop, using a CNN-based image classification model.

[0113] Referring to Fig. 4, ROI 1 DOWN can indicate a downward direction, ROI 2 UP, etc. Through this, the actual direction of the escalator and the direction of the person can be checked, and can be classified as normal upward, normal downward, sudden stop, reverse movement and stop, etc., as in the embodiment.

[0114] FIG. 5 is a video screen illustrating an example of detecting abnormal behavior by analyzing the movement pattern, density, or crowd state of a passenger in a passenger behavior analysis module according to an embodiment of the present invention.

[0115] The passenger behavior analysis module (33) can track the movement trajectory of the passenger using an object detection model and a multi-object tracking algorithm. Based on the analysis results obtained through the optical flow analysis module (31) and step state classification module (32) described above, it can detect passenger density, movement speed, and changes in movement direction.

[0116] Referring to FIG. 5(A), object bounding boxes are generated and displayed for each of the multiple passengers within the screen, and the passenger behavior analysis module (33) can calculate the number of passengers within the frame, the location coordinates of the passengers, the direction of movement of the passengers, and the speed of movement of the passengers based on this object detection.

[0117] In particular, as shown in Example Fig. 5(A), in situations where passengers are concentrated around the escalator entrance, density can also be calculated based on the distance between passengers. For example, crowd density can be calculated based on the number of passengers within a certain radius, and if the density exceeds a certain threshold, it can be determined as a situation of crowd concentration.

[0118] In addition, the passenger behavior analysis module (33) can track the movement trajectory of each passenger over time using a multi-object tracking algorithm, and through this, can detect abnormal behavior patterns such as stopping at the same location for a long time, moving repeatedly at the entrance of the escalator, or showing an abnormal movement path.

[0119] Through such analysis, accident risk situations that may occur around escalators can be detected in advance.

[0120] Referring to FIG. 5(B), another embodiment, an example is shown of analyzing a situation in which multiple passengers move simultaneously in a passageway around an escalator. Multiple pedestrians are detected in the image, and it can be seen that an object bounding box and a confidence value are displayed for each passenger.

[0121] In particular, as shown in Fig. 5(B), in situations where multiple passengers are moving at the same time near the entrance or exit of an escalator, pedestrian flow may become congested, and such situations may lead to safety accidents while using the escalator.

[0122] Accordingly, the passenger behavior analysis module (33) of the present invention can detect crowd concentration situations or abnormal behavior patterns by comprehensively analyzing features such as an increase in the number of passengers within a certain area, a decrease in movement speed, and a mixture of movement directions.

[0123] When analyzing as shown in Fig. 5, the passenger behavior analysis module (33) can detect potential dangerous situations that may occur around the escalator at an early stage by analyzing not only the passenger's movement patterns but also changes in crowd density and movement flow.

[0124] FIG. 6 is a video screen showing the detection of luggage in a situation where a passenger is using an escalator while carrying luggage, according to an embodiment of the present invention. It is a video screen showing an example in which a passenger behavior analysis module (33) detects luggage or portable items of a passenger using an escalator.

[0125] Referring to FIG. 6, an object within the image can be detected, and an object bounding box and a confidence score can be displayed for the detected object. The "0.85" displayed in FIG. 6 may represent a confidence score indicating the probability that the detected object corresponds to a load or moving item.

[0126] For example, if the above confidence value is displayed as 0.85, it may mean that there is about an 85% or higher probability that the object corresponds to a previously learned object class (e.g., suitcase or portable goods).

[0127] The passenger behavior analysis module (33) can additionally analyze information such as the type of item carried by the passenger, the size or movement trajectory of the item, and whether the item is located on the escalator step, based on the object detection results, similar to the analysis in FIG. 5.

[0128] In particular, when large items or suitcases move together while using an escalator, there is a possibility that the items may get stuck between the steps or lose balance during movement, which may increase the risk of safety accidents. Therefore, the passenger behavior analysis module (33) of the present invention can detect potential dangerous situations in advance by comprehensively analyzing the passenger's movement status, the movement path of the items, and the positional relationship between the items and the passenger.

[0129] FIG. 7 is a block diagram illustrating the process of a judgment and notification unit according to an embodiment of the present invention calculating a final risk level by weighted summing risk values ​​calculated from a plurality of analysis modules and determining an abnormal situation.

[0130] Referring to FIG. 7, it can be seen that the risk of an abnormal situation is calculated by combining the analysis result of the optical flow analysis module (31), the state classification result of the step state classification module (32), and the behavior analysis result of the occupant behavior analysis module (33) in a weighted voting manner.

[0131] The optical flow analysis module (31) can calculate the probability of an abnormal situation occurring P(OF) by analyzing the direction and magnitude of the movement vector occurring in the escalator step area. For example, if the magnitude of the escalator's movement vector decreases rapidly or the direction of movement reverses, the likelihood of it being determined as a sudden stop or reverse movement situation increases, and the result of this analysis can be calculated in the form of a probability value.

[0132] The step state classification module (32) can analyze the operating state of the escalator using a CNN-based image classification model. The step state classification module can classify operating states including normal up, normal down, sudden stop, reverse movement, or stop state, and calculate the probability P(DL) of an abnormal situation.

[0133] The passenger behavior analysis module (33) can analyze the passenger's movement trajectory, density, movement speed, and changes in movement direction using object detection and multi-object tracking algorithms. Through this, if an abnormal behavior pattern of the passenger is detected, the probability of an abnormal situation occurring P(BA) can be calculated.

[0134] The judgment and notification unit (40) can calculate a final risk score, i.e., a final score (S), by combining the abnormal situation probabilities calculated in each module using a weighted voting-based risk calculation method.

[0135] According to an embodiment, the final score (S) can be calculated by the following [formula].

[0136] [formula]

[0137]

[0138] Here, P(OF) represents the anomaly probability of the optical flow analysis module, P(DL) represents the anomaly probability of the deep learning-based state classification module, and P(BA) represents the anomaly probability of the occupant behavior analysis module.

[0139] w1, w2, and w3 are weights assigned to each module, and can satisfy w1 + w2 + w3 = 1. The judgment and notification unit (40) can determine whether there is an abnormal situation by comparing the calculated final score (S) with a preset threshold T.

[0140] For example, if S > T, it can be determined as an abnormal situation, and if S ≤ T, it can be determined as a normal situation.

[0141] When an abnormal situation is determined, the determination and notification unit (40) can transmit a notification signal to the control system and can transmit a control signal including a deceleration or stop command to the escalator control system through the control linkage unit (50).

[0142] FIG. 8 is a video screen that displays the status of an escalator user as text at the top of a monitor by transmitting the content analyzed by the judgment and notification unit (40) according to one embodiment of the present invention to an actual CCTV video.

[0143] Referring to FIG. 8, according to an embodiment, frame images are displayed over time in the order from the top left to the right, and then from the bottom left to the right, and the content of the screen, which changes moment by moment, can be analyzed and displayed as text on the monitoring screen. According to an embodiment, the situation detection results and notifications (text) are as follows in order.

[0144] On screen 1, it reads, “A person wearing black clothes moves around near the elevator multiple times, and is sometimes seen getting on or off the elevator while holding a child. They are active near the elevator safety bar and are sometimes seen holding objects in their hands.”

[0145] On screen 2, “A person wearing black clothes boards the escalator at the bottom and holds an item in their hand while moving upward; they ride the escalator up, reach the top at 194 seconds, and then disappear.”

[0146] On screen 3, “a person wearing black clothes is seen standing on the stairs in front of the elevator holding the handrail, or moving up and down the stairs.”

[0147] On screen 4, “You can see a person wearing black clothes and white shoes moving up on an escalator while holding onto the handrail.”

[0148] On screen 5, “A person wearing black clothes uses the elevator multiple times, repeating the action of going down from the top to the bottom or up from the bottom to the top. They are also seen holding an object in their hand and are observed moving while holding the elevator safety handle.”

[0149] On screen 6, “A person wearing black clothes stands at the top of the elevator holding a child, then rides the elevator down, arrives at the bottom of the elevator at 03:12, and continues moving.”

[0150] As described above, the judgment and notification unit (40) can transmit detected content over time, and if the judgment result is determined to be dangerous, the control linkage unit (50) can transmit a control signal containing a deceleration or stop command to an escalator control system (not shown, including any system linked to the escalator that performs a deceleration or stop command). At the same time, a message notifying of danger can be transmitted and displayed on a monitor.

[0151] Below, a method for detecting sudden stops and reverse movement of an escalator based on CCTV video analysis using the system described above is explained.

[0152] FIG. 9 is a flowchart illustrating the overall processing procedure of the CCTV video analysis-based escalator sudden stop and reverse movement detection method of the present invention.

[0153] Referring to FIG. 9, the escalator abnormal situation detection method of the present invention may include a video data reception step (S10), a video preprocessing step (S20), an abnormal situation detection step (S30), a notification transmission step (S40), and a control signal transmission step (S50).

[0154] The video data reception step (S10) can receive real-time video data from a CCTV camera installed around the escalator.

[0155] Specifically, the system can be connected to a CCTV device via a network to receive a continuous video stream, and the video data can be input at a constant frame rate. Additionally, the video data may be video data capturing an escalator usage area including the escalator landing section, the landing entrance, and the landing exit.

[0156] In the image preprocessing step (S20), a region of interest (ROI) can be set for the received image data, and preprocessing work can be performed to improve image quality.

[0157] Specifically, the above preprocessing step can perform region of interest setting, video stabilization, illumination correction, and noise reduction.

[0158] Setting the region of interest may be a process of limiting the area to be analyzed by extracting escalator step areas or passenger movement areas within the video.

[0159] Image stabilization may be a processing step to correct image shake caused by camera vibration or the external environment, and a correction method using feature points between frames may be applied.

[0160] Lighting correction is a process to correct differences in image brightness caused by changes in lighting, and techniques such as histogram correction or gamma correction may be applied.

[0161] Noise removal is a process for eliminating image noise caused by image sensors or environmental factors, and filtering techniques may be applied.

[0162] The abnormal situation detection step (S30) is a process of detecting a sudden stop or reverse movement of the escalator by analyzing the movement pattern of the occupant.

[0163] Specifically, the system can determine the operating state of the escalator by analyzing motion vectors or movement patterns between video frames. For example, if the magnitude of a passenger's movement vector decreases rapidly, it can be determined as a possibility of an emergency stop, and if the direction of movement is suddenly reversed, it can be determined as a possibility of moving in reverse.

[0164] In addition, the above analysis process can determine abnormal situations by comprehensively considering the occupant's movement trajectory, movement speed, and changes in movement direction.

[0165] In the notification transmission step (S40), information regarding abnormal situations can be transmitted to the control system. For example, if an escalator sudden stop or reverse movement is detected, the system can generate notification information including the location of the abnormal situation, the time of occurrence, and the type of detected event, and transmit it to the control server or management system. Through this, the administrator can check the escalator abnormal situation in real time in a remote control environment.

[0166] In the control signal transmission step (S50), a control signal including a deceleration or stop command can be transmitted in conjunction with the escalator control system. For example, if a dangerous situation involving a sudden stop is detected, a deceleration command for the escalator can be issued, or if a situation involving reverse movement is detected, a control signal including an emergency stop command can be generated and transmitted to the escalator control system. This enables immediate safety response to abnormal operation situations of the escalator.

[0167] FIG. 10 is a flowchart illustrating a detailed analysis procedure of an abnormal situation detection step according to an embodiment of the present invention.

[0168] Referring to FIG. 10, the abnormal situation detection step (S30) may include a motion vector analysis step (S301), an operation status classification step (S302), and an abnormal behavior pattern detection step (S303).

[0169] In the motion vector analysis step (S301), motion vectors representing the direction and speed of movement of the occupant can be analyzed using information on changes between video frames.

[0170] Specifically, the system can calculate optical flow-based motion vectors by analyzing pixel movement between consecutive video frames. At this time, the following events can be generated based on changes in the magnitude or direction of the motion vectors.

[0171] If the magnitude of the motion vector decreases rapidly compared to a set threshold, a sudden stop candidate event can be generated, and if the direction of the motion vector is reversed, a reverse driving candidate event can be generated.

[0172] In the operation status classification step (S302), the operation status of the escalator can be classified based on the analysis results. For example, the system can determine the state of the escalator as a normal upward state, a normal downward state, an emergency stop state, a reverse state, or a stop state. These operation statuses can be determined by comprehensively analyzing the direction of the motion vector, changes in speed, and movement patterns.

[0173] In the abnormal behavior pattern detection step (S303), abnormal behavior patterns can be detected by analyzing the movement trajectory, density, movement speed, or change in the direction of movement of the occupants. For example, situations such as when an occupant suddenly loses balance, when multiple occupants move in a dense manner, or when the direction of movement changes suddenly can be detected.

[0174] Additionally, according to another embodiment, the method may further include a step of analyzing the relative vector between the passenger's movement vector and the escalator step's movement direction.

[0175] Specifically, the system can calculate a passenger motion vector representing the direction and speed of movement of the occupant through video analysis. The motion vector can be calculated using position change information between consecutive video frames, for example, through optical flow or object tracking-based motion vector analysis.

[0176] Additionally, the system can set the step motion direction corresponding to the escalator's direction of travel as the reference direction. For example, when the escalator is moving upward, the step motion direction can be defined as an upward vector, and when the escalator is moving downward, the step motion direction can be defined as a downward vector. In this case, the system can calculate a relative motion vector between the passenger's motion vector and the step motion direction vector.

[0177] According to an embodiment, a normal movement state is when the movement vector of a passenger has a direction similar to the movement direction of the escalator step. For example, if the step movement direction is upward and the passenger movement direction is upward, it may be a normal escalator usage state.

[0178] When an escalator comes to a sudden stop, the passenger's movement speed may suddenly decrease or stop. Therefore, if the magnitude of the step movement direction vector decreases rapidly or the magnitude of the passenger's movement vector decreases below a threshold, the system may generate a candidate event for a sudden stop situation.

[0179] In the event of escalator reverse movement, the directional relationship between the rider's movement vector and the step's movement direction vector may be reversed. For example, a relative reversal of direction may occur, such as the step's movement direction being upward and the rider's movement direction being downward. In this case, the system may generate a reverse movement candidate event.

[0180] In addition, the system can detect the following dangerous situations by utilizing the change patterns of relative vectors. When a passenger loses balance and the direction of movement changes abruptly, when the direction of movement of multiple passengers changes irregularly simultaneously, or when the movement vector differs significantly from the step direction, such situations can be identified as abnormal behavior patterns with a high probability of causing an escalator accident.

[0181] Such behavioral patterns can be identified as dangerous situations with a high probability of escalator accidents. Therefore, the system can analyze these behavioral patterns to more accurately determine whether there is an abnormal situation in the escalator.

[0182] In addition, the present invention may be provided by implementing the entire processing procedure of the method for detecting sudden stop and reverse movement of an escalator based on CCTV video analysis as software, and the method invention described above may be stored in the software.

[0183] In addition, the present invention may provide a computer-readable storage medium in which a program in which the method invention is implemented is stored.

[0184] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications can be made to the above-described embodiments within the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the claims set forth below as well as all modifications or variations derived from the claims and equivalent concepts. Explanation of the symbols

[0186] 1: CCTV Video Analysis-Based Escalator Emergency Stop and Reverse Movement Detection System 10: Image acquisition unit 20: Image Preprocessing Unit 21: ROI Setting Unit 22: Image Stabilization Unit 23: Lighting Correction Unit 24: Noise Removal Section 30: Abnormal situation detection unit 31: Optical flow analysis module 32: Step Status Classification Module 33: Passenger Behavior Analysis Module 40: Judgment and Notification Section 50: Control Interlock Unit

Claims

Claim 1 A CCTV video analysis-based escalator sudden stop and reverse movement detection system for detecting abnormal situations of an escalator comprises: an image acquisition unit that receives real-time video data from at least one CCTV camera installed on the escalator; an image preprocessing unit that performs preprocessing on the video data for at least one of setting a region of interest, image stabilization, lighting correction, or noise removal; an abnormal situation detection unit that detects a sudden stop or reverse movement situation of the escalator by analyzing dense optical flow between consecutive frames of the preprocessed video data and analyzing the movement pattern of a passenger based on deep learning; a determination and notification unit that determines whether an abnormal situation exists based on the analysis result of the abnormal situation detection unit and transmits a notification regarding the abnormal situation to a control system according to the determination result; and a control linkage unit that transmits a control signal including a deceleration or stop command in conjunction with an escalator control system according to the determination result, and includes an optical flow analysis module that detects a change in the movement speed or direction of an escalator step by analyzing dense optical flow between consecutive frames of the preprocessed video data; and a step state classification module that classifies the operating state of the escalator using a deep learning-based video classification model. It further includes a passenger behavior analysis module that analyzes the passenger's movement patterns through object detection and multi-object tracking, and the judgment and notification unit fuses the results of the optical flow analysis module, the step state classification module, and the passenger behavior analysis module to calculate a final score (S) using a weighted voting method, and the mathematical formula of the weighted voting method is A CCTV video analysis-based escalator sudden stop and reverse movement detection system characterized by being calculated by (where, P(OF): optical flow module abnormality probability, P(DL): deep learning classification module abnormality probability, P(BA): behavior analysis module abnormality probability, S: final score (Score), w1 + w2 + w3 = 1), and determining as an abnormal situation of the corresponding grade if the final score (S) exceeds a preset threshold. Claim 2 A CCTV video analysis-based escalator sudden stop and reverse movement detection system, wherein, in claim 1, the image preprocessing unit includes an ROI setting unit that sets an ROI of interest in at least one of the step area, handrail area, or passenger area of ​​the escalator, and the ROI setting unit can be set to an ROI of interest pre-set by a user to detect at least one abnormal situation of the escalator. Claim 3 A CCTV video analysis-based escalator sudden stop and reverse movement detection system, wherein, in claim 1, the image preprocessing unit further includes an image stabilization unit that corrects camera vibration or image shaking. Claim 4 A CCTV video analysis-based escalator sudden stop and reverse movement detection system, wherein, in claim 1, the image preprocessing unit further includes a lighting correction unit that performs adaptive histogram equalization processing to remove lighting changes or shadow effects. Claim 5 A CCTV image analysis-based escalator sudden stop and reverse movement detection system, wherein, in claim 1, the image preprocessing unit further comprises a noise removal unit that removes image noise using a Gaussian filter or a non-local mean filter. Claim 6 delete Claim 7 A CCTV video analysis-based escalator sudden stop and reverse movement detection system according to claim 1, wherein the optical flow analysis module calculates a pixel-unit motion vector between consecutive frames using the Farneback or Lucas-Kanade algorithm, generates a sudden stop candidate event when the motion vector decreases below a preset threshold size, and generates a reverse movement candidate event when the direction of the motion vector is reversed by more than a predetermined angle. Claim 8 A CCTV video analysis-based escalator sudden stop and reverse movement detection system according to claim 1, wherein the step state classification module classifies the operation state into at least one operation state including normal up, normal down, sudden stop, reverse movement, or stop state using a CNN-based image classification model. Claim 9 A CCTV video analysis-based escalator sudden stop and reverse movement detection system according to claim 1, wherein the passenger behavior analysis module tracks the movement trajectory of a passenger using an object detection model or a multi-object tracking algorithm, and detects an abnormal behavior pattern by analyzing at least one of the passenger density, movement speed, or change in movement direction of the passenger. Claim 10 delete Claim 11 A CCTV video analysis-based escalator sudden stop and reverse movement detection system according to claim 1, wherein the control interlocking unit communicates with a PLC or inverter control system to transmit the control signal and automatically saves video footage before and after the occurrence of an abnormal situation. Claim 12 A method for detecting sudden stop and reverse movement of an escalator based on CCTV video analysis, which detects sudden stop and reverse movement situations of an escalator using CCTV video analysis, comprising: receiving real-time video data from at least one CCTV camera installed on the escalator; performing preprocessing on the video data for at least one of setting a region of interest, image stabilization, lighting correction, or noise removal; analyzing dense optical flow between consecutive frames of the preprocessed video data and analyzing the movement pattern of a passenger based on deep learning to detect a sudden stop or reverse movement situation of the escalator; determining whether an abnormal situation exists based on the results of the analysis and transmitting a notification regarding the abnormal situation to a control system according to the determination result. The method includes the step of transmitting a control signal containing a deceleration or stop command in conjunction with an escalator control system according to the above determination result, and the step of detecting the sudden stop or reverse movement situation comprises: calculating a pixel-unit motion vector between consecutive frames of preprocessed image data using a Farneback or Lucas-Kanade algorithm, generating a sudden stop candidate event when the motion vector decreases below a preset threshold size, and generating a reverse movement candidate event when the direction of the motion vector reverses by more than a predetermined angle; and classifying the operation state into at least one operation state including normal up, normal down, sudden stop, reverse movement, or stop state using a CNN-based image classification model.The method further includes the step of tracking the movement trajectory of a passenger using an object detection model or a multi-object tracking algorithm, and detecting an abnormal behavior pattern by analyzing at least one of the passenger's density, movement speed, or change in movement direction; detecting a change in the movement speed or direction of an escalator step by analyzing the dense optical flow between consecutive frames of the preprocessed image data; classifying the operating state of the escalator using a deep learning-based image classification model; analyzing the passenger's movement pattern through object detection and multi-object tracking; and calculating a final score (S) using a weighted voting method to detect whether an abnormal situation exists, wherein the mathematical formula of the weighted voting method is; A method for detecting sudden stop and reverse movement of an escalator based on CCTV video analysis, characterized by being calculated by (wherein, P(OF): optical flow module anomaly probability, P(DL): deep learning classification module anomaly probability, P(BA): behavior analysis module anomaly probability, S: final score, w1 + w2 + w3 = 1), and if the final score (S) exceeds a preset threshold, it is finally determined as an anomaly situation of the corresponding grade. Claim 13 delete Claim 14 delete

Images