Escalator abnormal state detection method and electronic equipment

Abstract

The invention discloses an escalator abnormal state detection method and electronic equipment. The escalator abnormal state detection method comprises the following steps that S1, a video stream for monitoring escalator operation is obtained; s2, preprocessing image frames in a video stream to obtain an input image; S3, if a potential abnormal frame is detected, performing a step S4, otherwise, continuing to detect; s4, the prediction scores of the continuous c frames after the potential abnormal frames are stored, and meanwhile an optical flow method is adopted for judging whether the running direction of the escalator is in an abnormal state or not; s5, the average score scoreavg and the highest score scoremax of the escalator abnormal state scores of the continuous c frames are obtained, and the number n of times of abnormal state judgment through an optical flow method is obtained; and S6, performing logic judgment and alarming. According to the method, the accuracy and robustness of the model are improved, whether the abnormal state exists or not can be detected in the running process of the escalator, and the method is suitable for various conditions such as high-speed or low-speed running of the escalator and retrograde running of passengers.

Description

Technical Field

[0001] The present invention relates to the field of image recognition, and in particular to an escalator abnormal state detection method and electronic equipment. Background Art

[0002] Escalators are core facilities in important transportation hubs such as high-speed railway stations and subway stations. Their safe operation is of vital importance to ensuring the normal passage of personnel. However, traditional escalator status detection methods mainly rely on manual inspections, sensors and traditional machine learning methods. Due to limited human resources, manual inspections are prone to problems such as inadequate monitoring and untimely response. The sensor method requires the installation of sensors at key locations on the escalator and regular maintenance and replacement. This not only requires a lot of material and financial resources, but may also affect the normal operation of the escalator. Traditional machine learning methods usually require manual feature engineering, and features need to be redesigned and selected for different scenarios. They are sensitive to noise and have weak model generalization capabilities, making them difficult to adapt to complex scenarios. For example, when a pedestrian is walking in the opposite direction on the escalator, the existing method easily identifies the direction of the pedestrian's movement as the direction of the escalator's movement. In addition, the existing detection model cannot distinguish between the slow and stationary states of the escalator, and is prone to false alarms. Summary of the invention

[0003] In order to solve the above technical problems, the present invention proposes an escalator abnormal state detection method and electronic equipment. The purpose of the present invention is achieved through the following technical solutions:

[0004] A method for detecting abnormal state of an escalator comprises the following steps:

[0005] S1. Obtain the video stream for monitoring the operation of the escalator;

[0006] S2. Preprocess the image frames in the video stream to obtain the input image:

[0007] S3, input the input image into the trained deep learning model to obtain the probability score a of the normal operation of the escalator. If a < the preset threshold b, the corresponding image frame is marked as a potential abnormal frame; if a potential abnormal frame appears, proceed to step S4, otherwise continue detection;

[0008] S4. The trained deep learning model stores the prediction scores of the consecutive c frames after the potential abnormal frame, and uses the optical flow method to judge the running direction of the escalator. If the judgment result is different from the setting result of the escalator control system, it is recorded as an abnormal state;

[0009] S5, obtaining the average score score_avg and the maximum score score_max of the escalator abnormal state scores of consecutive c frames, and the number of times n when the optical flow method determines that the escalator is in an abnormal state;

[0010] S6. Logical judgment and alarm: If score_avg≥first threshold thres_avg, score_max≥second threshold thres_max, and n≥third threshold d are satisfied at the same time, an alarm is issued; otherwise, the process returns to step S3 to continue detection.

[0011] For further improvement, the specific steps of step S2 are as follows:

[0012] S21, randomly expanding the escalator area in the image frame to the detection area;

[0013] S22. Use the cv2.cvtColor function to convert the detection area from the original BGR color gamut to the HSV color gamut to obtain HSV

[0014] Color gamut image: Binarize the HSV color gamut image to obtain a color gamut conversion image, in which the yellow area in the HSV color gamut image becomes white and the non-yellow area becomes black;

[0015] S23, performing a bitwise AND operation on the color gamut conversion image and the mask to obtain a composite image;

[0016] S24, dilating the yellow line area in the synthesized image to obtain an input image.

[0017] Further improvements, b∈[0.5, 0.8], c∈[5,10], th res_avg∈[0.5,0.7], th res_max∈[0.7,0.9], d∈[3,6] and d<c.

[0018] As a further improvement, the optical flow method is an LK pyramid optical flow method.

[0019] As a further improvement, in step S2, image frames in the video stream are extracted every 3-5 frames for preprocessing to obtain input images.

[0020] As a further improvement, in step S4, the method for obtaining the trained deep learning model is as follows:

[0021] S41, constructing short video training samples of escalator operation, and dividing the short video training samples into five categories according to the escalator operation direction and speed: fast up, fast down, slow up, slow down and still; the short video training samples contain the same number of image frames;

[0022] S42, randomly expanding the sample range: defining the area where the monitored escalator in the short video training sample is located as the current escalator area, and defining the areas where other escalators are located as non-expandable areas; then randomly expanding the current escalator area, and the range of the random expansion does not include the non-expanded area to obtain the final training data set;

[0023] S43. Input the final training data set into the deep learning model to obtain a trained deep learning model.

[0024] As a further improvement, in step S41, when performing detection, one frame is extracted for detection every e frame interval, and for the videos of the escalator going up quickly and the escalator going down quickly, a set of frames is extracted every ef frame interval as short video training samples of slow ascent and slow descent.

[0025] Further improvement, e=4, f=1-3.

[0026] As a further improvement, the deep learning model is a TSN or TDN video classification model.

[0027] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method as claimed in any one of claims 1 to 9.

[0028] The beneficial effects of the present invention are:

[0029] The present invention improves the accuracy and robustness of the model by enriching the diversity of video classification model training samples, improving data enhancement methods, optimizing alarm logic and other methods. The method can detect whether there is an abnormal state during the operation of the escalator. The specific method is: after classifying the video samples in the model training stage, different frame extraction processes are performed to make the samples of the slow escalator richer and improve the detection readiness rate of the slow escalator. By expanding the escalator area, the deep learning model can learn more different background information according to the escalator expansion area, reducing the interference of the running direction of other external objects or people on the escalator detection. Adding the optical flow method can focus more on the recognition of the running direction of the escalator yellow line, eliminate the interference of passengers on the escalator to the greatest extent, and overcome the problem of easily misjudging the reverse direction of passengers as the direction of travel of the escalator. By performing different preprocessing on the image frame, the deep learning model and the optical flow method are used for detection respectively, and the judgment results of the two are combined, the accuracy of the escalator state judgment is effectively improved, so that when the escalator is in an abnormal running state, an alarm is issued in time to remind nearby security guards or monitoring personnel to pay attention, effectively ensuring the safe operation of the escalator, and reducing accidents caused by abnormal escalator states. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention is further described with reference to the accompanying drawings, but the contents in the accompanying drawings do not constitute any limitation to the present invention.

[0031] Figure 1 It is a schematic diagram of the process of the present invention;

[0032] Figure 2a It is a schematic diagram of the escalator area and the non-expandable area;

[0033] Figure 2b This is a schematic diagram before the escalator area is expanded;

[0034] Figure 2c This is a schematic diagram of the expanded escalator area;

[0035] Figure 3 Schematic diagram of the image processing steps of the optical flow method;

[0036] Figure 4a is an escalator area image cut out from the frame-sampling image;

[0037] Figure 4b The extracted yellow line result image;

[0038] Figure 4c This is the effect diagram after the yellow line is expanded;

[0039] Figure 5 This is the optical flow method to judge the escalator operation effect diagram;

[0040] Figure 6 It is the logic judgment of escalator abnormality. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0043] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first" and "second" are used only for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more features. In the description of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0044] In this application, the word "exemplary" is used to mean "serving as an example, illustration, or description". Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0045] The overall process of the present invention is as follows Figure 1 As shown, it mainly includes the following steps:

[0046] First, the framed image is acquired according to the set frame extraction interval. Considering the running speed of the escalator and the real-time requirements of the alarm, the frame extraction interval of the escalator monitoring video stream is recommended to be set to 3-5 frames. This ensures that while capturing sufficient motion information, it will not cause excessive image data processing and affect real-time performance.

[0047] For the deep learning method and the optical flow method, different preprocessing is performed on the demarcated escalator area. These two methods have different requirements for image data, so different preprocessing steps are required to make the input data meet the requirements of each method and improve the subsequent detection efficiency and accuracy.

[0048] "Optical flow (optical flow or optic flow) is a concept in the detection of object motion in the field of view. It is used to describe the motion of the observed target, surface or edge caused by the motion relative to the observer. The optical flow method is a method that uses the change of pixels in the time domain in the image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. There are two premise assumptions when using the optical flow method: First: the tracked pixel target should remain basically unchanged between consecutive frames. Second: the tracked pixel target should have a similar motion trend between consecutive frames. The movement of the escalator obviously meets the above assumptions. The optical flow method is very useful in pattern recognition, computer vision and other image processing fields. It can be used for motion detection, object cutting, calculation of collision time and object expansion, motion compensation coding, or three-dimensional measurement of object surfaces and edges, etc.

[0049] The state of the preprocessed image is detected by the deep learning model, and the detection score of the model is output. At the same time, the optical flow method is used to track the feature points in the image and determine the movement direction of the feature points. Finally, a comprehensive logical judgment is made based on the model detection score and the direction of the feature points determined by the optical flow method. According to the judgment result, if there is an abnormal state, the corresponding alarm information is output to notify the relevant personnel to deal with it in time. The model used in the present invention is a TSN or TDN video classification model.

[0050] The TSN video classification model is a classic 2D-CNN-based solution in the field of video classification. This method mainly solves the problem of long-term behavior judgment in videos. By sparsely sampling video frames instead of dense sampling, it can not only capture the global information of the video, but also remove redundancy and reduce the amount of calculation. Finally, the overall features of the video are obtained by averaging the features of each frame and used for classification. The model implemented in this code is a TSN network structure based on a single-channel RGB image, and Backbone uses the ResNet-50 structure.

[0051] The TDN video classification model is a two-level structure, which mainly includes two modules: Short-term temporal difference module (S-TDM) and Long-term temporal difference module (L-TDM). Each video V is divided into T segments with equal length and no overlap. A frame is randomly selected from each segment to obtain T frames. These frames are input into a 2D CNN to extract frame-level features.

[0052] This process design can ensure that we achieve a good balance between accuracy and real-time performance, so that the present invention has efficient and accurate performance in escalator abnormal state detection.

[0053] 1. Fast / slow processing of training samples and its optimization

[0054] Since most of the samples collected are escalators running at normal speed (i.e., fast running), there are fewer and extremely single samples of slow-running escalators, usually without pedestrians on them and with different escalator speeds at different points. If the original samples are directly used for training, the model cannot accurately distinguish between slow escalators and stationary escalators. Therefore, the present invention performs fast / slow processing on video samples before model training, artificially constructs samples of various speeds, enriches the types and quantities of samples, and improves the accuracy of the model.

[0055] In actual scenarios, the speed of escalators is a key factor. In conventional monitoring and data collection, most samples are escalators running at normal speeds, that is, running fast. This results in relatively few samples of slow-running escalators with relatively simple characteristics. Usually, there are no pedestrians on the escalators, and the escalator speeds at different points vary. If the model is trained only on these raw samples, it will be difficult for the model to accurately distinguish between slow escalators and completely stationary escalators.

[0056] Therefore, in order to improve the recognition accuracy of the model, the present invention performs fast / slow processing on the video samples during model training. Such a processing strategy aims to artificially construct and simulate samples of various speeds, thereby enriching the types and quantities of samples, thereby improving the performance of the model in practical applications.

[0057] Specific method:

[0058] ① Sample segmentation and classification processing

[0059] The collected long videos are divided into 4-second short video samples according to the model training requirements, and all training samples are carefully classified into five categories according to the running status and direction of the escalator: fast up, fast down, slow up, slow down and stationary.

[0060] ② Escalator sample frame extraction strategy

[0061] Assuming that the frame extraction interval during algorithm operation is set to 4 frames, then for the samples of the two categories of fast up and fast down, the frame extraction interval is randomly selected to be 2-4 frames when extracting the training video, ensuring that a total of 15 frames of images are extracted from a 4-second video for model training. Among them: when 2-3 frames are used, the interval between the previous and next images is short and the escalator movement distance is reduced, so the extracted 15 frames of images effectively simulate the visual characteristics of the escalator in a slow running state and increase the diversity of the scene. When a 4-frame interval is used, it is the effect of the escalator running quickly.

[0062] For the two categories of samples, slow up and slow down, the frame extraction interval is randomly selected as 3-5 frames when extracting frames from the training video, and a total of 15 frames of images are extracted from the 4s video and input into the model. Among them, the interval of 3 frames is selected to simulate video samples that are slower than the original video, and the interval of 5 frames is selected to simulate the acquisition of video samples that are faster than the original video. This strategy aims to compensate for the sample deviation caused by the difference in the slow speed setting of escalators at different monitoring points, thereby improving the recognition ability of the model.

[0063] 2. Randomly expand the sample range

[0064] In the original training method, the escalator area is cut out separately and sent to the model for training. However, due to the small amount of feature information of the escalator itself, the detection effect of the model is easily affected by pedestrians or other objects on the escalator, especially when passengers go against the flow or staff inspect the escalator, the model prediction result is extremely unstable. Therefore, this method proposes to randomly expand the sample range to improve the robustness of the model. The principle is as follows: random expansion is to expand each video to different ranges. Even if multiple videos are collected at the same point, the expansion range will be different, so the surrounding information will be different, and more can be learned, so the robustness can be improved, and the detection will not be unstable due to a slight change in the surrounding environment. The expansion method is used because the input of the algorithm front end is the area of ​​the escalator (because this area is used in the optical flow method), so the model uses the expansion method to expand the area. If it is set to the surrounding area of ​​the escalator at the beginning, the area of ​​the escalator cannot be obtained from this area.

[0065] In the data preprocessing stage of model training, the current escalator area and the non-expandable area (the area of ​​the escalator when other escalators exist but are not included in the training) are set according to each point, such as Figure 2a As shown. Randomly select the expansion ratios of width and height, ratio_w and ratio_h. Usually, ratio_w is 0.1 to 2, and ratio_h is 0.1 to 0.25. Then the expanded width and height are:

[0066] w ex =w+w*2*ratio_w

[0067] h ex =h+h*2*ratio_h

[0068] Among them, w ex 、h esrepresents the width and height of the sample after expansion, w and h represent the width and height of the sample before expansion, and *2 means that both the top and bottom or the left and right sides should be expanded. When expanding, it should be noted that the expansion range of the current escalator cannot include other escalator areas. This design is mainly to prevent the expansion range from being too large, causing the sample to contain irrelevant information. The sample image before expansion is as follows Figure 2b As shown, after expansion Figure 2c shown.

[0069] 3. LK pyramid optical flow method to determine the direction of escalator operation

[0070] Since the escalator itself has few features, directly using the optical flow method on the image frame will cause the feature points to be selected on other pedestrians and objects. When they are inconsistent with the running direction of the escalator, the accuracy of the optical flow method will be greatly reduced. Therefore, the present invention will perform a series of image preprocessing based on the escalator area, so that the optical flow method will focus on the movement direction of the yellow line on the escalator when extracting feature points, thereby improving the judgment accuracy. The overall processing flow is as follows: Figure 3 .

[0071] 1) Use cv2.cvtColor(image,cv2.COLOR_BGR2HSV) to transform the color gamut of the escalator area image, and use cv2.inRange(hsv_img,yellow_lower,yellow_upper) to get the mask. Among them, image is the escalator area image extracted, hsv_img is the hsv image after color gamut transformation, yellow_lower and yellow_upper are the color ranges of the extracted yellow line, usually with the values:

[0072] yellow_lower=np.array([20,40,40])

[0073] yellow_upper=np.array([80,255,255])

[0074] 2) Use cv2.bitwise_and(image,image,mask=mask_yellow) to perform a bitwise AND operation on the image and the mask, where mask_yellow is the mask obtained in the previous step.

[0075] 3) Considering that there may be pedestrians, objects or light changes on the escalator, the extracted yellow line pixels may be affected. Therefore, cv2.dilate() is further used to dilate the image to highlight the yellow line part of the image.

[0076] cv2.cvtColor: color space conversion function.

[0077] HSV image: refers to a subset of visible light in the H, S, V three-dimensional color space, which contains all colors in a certain color domain.

[0078] cv2.inRange: That is about the cv2.inRange function in python.

[0079] cv2.bitwise_and: It is one of the bitwise operation functions in OpenCV, which is used to AND two images bitwise. This function performs a logical & operation on the binary values ​​of two corresponding pixels.

[0080] The result of optical flow image processing is as follows Figure 4a-4c As shown. Figure 4a is the escalator area image cut out from the frame image. Figure 4b is the yellow line result extracted in step ② above. Figure 4c This is the result after the expansion process in step ③ above. The optical flow method is used to judge the running effect of the escalator. Figure 5 As shown in the figure, after appropriate image processing, the optical flow method can extract feature points on the yellow line of the escalator. Compared with using the original image, it can eliminate the interference of other features and accurately determine the running direction of the escalator.

[0081] 4. Alarm judgment mechanism combining deep learning and optical flow method

[0082] In order to improve the alarm accuracy of low-probability events, the present invention designs a composite judgment mechanism of the deep learning model detection results and the optical flow method judgment results. The overall idea is as follows Figure 6 This mechanism aims to make full use of the advantages of both methods to ensure that the accuracy of the alarm is improved when an escalator abnormality is detected. The specific implementation steps are as follows:

[0083] Step 1: Model prediction stage. The trained deep learning model is used to predict the real-time video frames. When the probability score of the escalator's normal operation predicted by the model is less than 0.5, the frame is marked as a potential abnormal frame and then enters the alarm logic judgment.

[0084] Step 2: Continuous monitoring stage: The deep learning model prediction scores of the five consecutive frames marked as potential abnormal frames are stored, and the judgment results of the escalator running direction by the optical flow method are recorded.

[0085] Step 3: Data statistics stage. After 5 frames, calculate the average score score_avg and the maximum score score_max of the escalator abnormal state, as well as the number of times n that the optical flow method determines it as an abnormal state.

[0086] Step 4: Logical judgment and alarm stage. When the following three conditions are met: score_avg≥th res_avg, score_max≥th res_max, n≥3, an alarm will be generated. Among them, rh res_avg is the average score alarm threshold, usually set to 0.5~0.7, and rh res_max is the maximum score alarm threshold, usually set to 0.7~0.9. When the above alarm conditions are met, the relevant alarm pictures and abnormal status information are output to the front-end interface. If not met, the alarm logic judgment is terminated and the algorithm continues to monitor the next frame of video.

[0087] The above method combines the judgment results of the optical flow method with the preset results of the trained deep learning model, and then makes a comprehensive judgment on whether the escalator is operating normally. The obtained result is more accurate, which effectively overcomes the defect of the existing method that is prone to false alarms.

[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting abnormal state of an escalator, characterized in that: The steps include: S1. Obtain the video stream for monitoring the operation of the escalator; S2. Preprocess the image frames in the video stream to obtain the input image: S3, input the input image into the trained deep learning model to obtain the probability score a of the normal operation of the escalator. If a<preset threshold b, the corresponding image frame is marked as a potential abnormal frame; If a potential abnormal frame appears, proceed to step S4, otherwise continue to detect; S4. The trained deep learning model stores the prediction scores of the consecutive c frames after the potential abnormal frame, and uses the optical flow method to judge the running direction of the escalator. If the judgment result is different from the setting result of the escalator control system, it is recorded as an abnormal state; S5, obtaining the average score score_avg and the maximum score score_max of the escalator abnormal state scores of consecutive c frames, and the number of times n when the optical flow method determines that the escalator is in an abnormal state; S6. Logical judgment and alarm: If score_avg≥first threshold thres_avg, score_max≥second threshold thres_max, and n≥third threshold d are satisfied at the same time, an alarm is issued; otherwise, the process returns to step S3 to continue detection.

2. The escalator abnormal state detection method according to claim 1, characterized in that: The specific steps of step S2 are as follows: S21, randomly expanding the escalator area in the image frame to the detection area; S22. Use cv2.cvtColor function to convert the detection area from the original BGR color gamut to the HSV color gamut to obtain an HSV color gamut image: binarize the HSV color gamut image to obtain a color gamut conversion image, in which the yellow area in the HSV color gamut image becomes white, and the non-yellow area becomes black; S23, performing a bitwise AND operation on the color gamut conversion image and the mask to obtain a composite image; S24, dilating the yellow line area in the synthesized image to obtain an input image.

3. The escalator abnormal state detection method according to claim 1, characterized in that: b∈[0.5,0.8],c∈[5,10],thres_avg∈[0.5,0.7],thres_max∈[0.7,0.9],d∈[3,6] and d<c.

4. The escalator abnormal state detection method according to claim 1, characterized in that: The light flow method is the LK pyramid light flow method.

5. The escalator abnormal state detection method according to claim 1, characterized in that: In step S2, image frames are extracted from the video stream every 3-5 frames and preprocessed to obtain input images.

6. The escalator abnormal state detection method according to claim 5, characterized in that: In step S4, the method for obtaining the trained deep learning model is as follows: S41, constructing short video training samples of escalator operation, and dividing the short video training samples into five categories according to the escalator operation direction and speed: fast up, fast down, slow up, slow down and still; the short video training samples contain the same number of image frames; S42, randomly expanding the sample range: defining the area where the monitored escalator in the short video training sample is located as the current escalator area, and defining the areas where other escalators are located as non-expandable areas; then randomly expanding the current escalator area, and the range of the random expansion does not include the non-expanded area to obtain the final training data set; S43. Input the final training data set into the deep learning model to obtain a trained deep learning model.

7. The escalator abnormal state detection method according to claim 6, characterized in that: In step S41, when performing detection, one frame is extracted for detection at every interval of e frames, and for the videos of the escalator going up quickly and the escalator going down quickly, a set of frames is extracted at every interval of ef frames as short video training samples of slow ascent and slow descent.

8. The escalator abnormal state detection method according to claim 7, characterized in that: e=4, f=1-3.

9. The escalator abnormal state detection method according to claim 1, characterized in that: The deep learning model is a TSN or TDN video classification model.

10. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

Images