Escalator intelligent monitoring method and system based on multi-source perception and machine vision

Abstract

The invention discloses an escalator intelligent monitoring method and system based on multi-source perception and machine vision, and belongs to the technical field of escalator safety detection.The escalator intelligent monitoring method comprises the following steps that the real-time vibration acceleration, temperature and noise spectrum of mechanical parts in a machine room truss of an escalator are obtained; the operation fault state of the escalator is monitored and obtained in real time, and pedestrian dangerous behaviors of pedestrians on the escalator are monitored and obtained in real time; a PHM-based fault prediction and health management method is adopted, and the vibration acceleration, the temperature, the noise spectrum, the operation fault state and the pedestrian dangerous behavior are analyzed by using the Internet of Things technology and the deep learning algorithm so as to judge the fault early warning trend of the escalator; and a three-dimensional structure model based on the escalator is established, and the real-time vibration acceleration, temperature, noise spectrum and fault early warning trend are positioned and reflected on the three-dimensional structure model in real time to be visually displayed.

Description

Technical Field

[0001] The present invention belongs to the technical field of escalator safety detection, and in particular relates to an escalator intelligent monitoring method and a system thereof based on multi-source perception and machine vision. Background Art

[0002] As a means of transportation for gathering and distributing passengers in railway stations, escalators mainly solve the problem of rapid evacuation of passengers. Since the ticket hall, waiting hall and railway departure hall of the station are often not on the same floor, passengers mainly rely on stairs to go up and down. Escalators provide a kind of automatic transportation capability for passengers. It can send passengers to or from the station safely, quickly and comfortably, and is a very important part of the architectural design of railway stations.

[0003] Due to the increasing passenger flow at stations and railways, the carrying load of escalators continues to increase. Although load requirements are set when purchasing escalators, escalators have been running at full load for a long time. Bearing wear has increased the failure rate and safety hazards of escalators. Some escalators have been running with problems before overhaul and are not easy to be discovered.

[0004] At the same time, pedestrians may encounter accidents due to walking against the flow, falling, climbing over the escalator railings, or carrying illegal items such as baby strollers. The escalator will not brake urgently, which will cause continuous and serious injuries to people. Traditional video surveillance systems cannot analyze videos in real time. The recognition rate and response speed of manual monitoring are low, which cannot meet the needs of large-scale applications. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide an escalator intelligent monitoring method and system based on multi-source perception and machine vision, which can effectively reduce the maintenance costs and downtime losses caused by sudden failures and reduce the overall operating costs.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0007] In a first aspect, the present invention provides an escalator intelligent monitoring method based on multi-source perception and machine vision, comprising the following steps:

[0008] Obtain the real-time vibration acceleration, temperature, and noise spectrum of the mechanical parts in the truss of the escalator's machine room, monitor and obtain the operating fault status of the escalator in real time, and monitor and obtain the dangerous behavior of pedestrians on the escalator in real time;

[0009] Based on PHM fault prediction and health management method, the Internet of Things technology and deep learning algorithm are used to analyze vibration acceleration, temperature, noise spectrum, operating fault status, and pedestrian dangerous behavior to determine the fault warning trend of escalators;

[0010] A three-dimensional structural model based on the escalator is established, and the real-time vibration acceleration, temperature, noise spectrum and fault warning trend are reflected in the three-dimensional structural model for intuitive visualization.

[0011] Furthermore, most of the mechanical parts of the escalator are located in the machine room truss, and the main driving load moving parts are the motor, the reducer, the main driving wheel, and the step chain tensioner.

[0012] Furthermore, under normal circumstances, escalator monitoring uses vibration sensors, temperature sensors, and noise sensors;

[0013] The sensor setup includes:

[0014] Install one monitoring point on each gearbox housing, polish and remove paint from the housing surface, and install a vertical vibration measuring point by bonding or threading;

[0015] Install a vibration measuring point at the left and right bearings of the main drive wheel and the step chain tensioner respectively by gluing or threaded interface;

[0016] One non-contact temperature monitoring point is installed on each of the left and right armrests, and another temperature sensor is installed to measure the ambient temperature for comparative analysis.

[0017] Furthermore, under normal circumstances, data is collected once every 10 seconds. The collection frequency of each sensor can be adjusted in the software according to the actual situation. When abnormal data occurs, the collection frequency of the corresponding sensor is increased to accurately predict the fault;

[0018] The collected vibration signals, noise and temperature signals are converted into A / D signals and sent to the fault diagnosis device for intelligent analysis. The analysis results and important spectrum data are uploaded to the maintenance center server through the backbone network for diagnostic verification and intelligent decision-making.

[0019] Furthermore, during the operation monitoring process of the escalator, the monitored vibration, temperature and noise signals mostly show periodic non-stationary characteristics. Wigner-Ville (WVD) is used to extract features and perform fault diagnosis on the escalator vibration signal containing cross-term interference.

[0020] Furthermore, the method for real-time monitoring and obtaining dangerous behaviors of pedestrians on escalators includes:

[0021] The human body structure is acquired based on the Openpose key point monitoring algorithm, and then the SVM classifier is used to classify the human body posture, so as to realize the monitoring and alarm of pedestrian falls in the picture;

[0022] The YOLO target detection model is used to obtain pedestrians and large luggage targets in the picture; then the KCF tracking algorithm is called on the pedestrian targets to monitor and alarm for possible dangerous behaviors in the surveillance video.

[0023] Furthermore, the multi-target tracking technology is integrated to track and identify moving objects on the escalator in real time, including:

[0024] The escalator area is segmented based on the FCN-HRNetW18 model algorithm;

[0025] The fusion algorithm of the pyramid network is used to recognize the human posture of pedestrians on the escalator, and the 3D coordinate calculation of the spatial transformation is combined to determine the pedestrian's body distortion and vertical state;

[0026] Use the YOLOv5 deep learning model to perform object detection on escalators for baby strollers, suitcases, and shopping carts.

[0027] In the second aspect, the present invention provides an escalator intelligent monitoring system based on multi-source perception and machine vision, comprising the following layers:

[0028] The data collection layer is equipped with data collection terminals and video collection terminals. The data collection terminals are used to collect vibration acceleration, noise, and temperature data information transmitted by various sensors, and the video collection terminals are used to collect the behavior data information of pedestrians on the escalator.

[0029] The data aggregation layer is configured with wired or wireless means to transmit data to the data application layer;

[0030] The data application layer receives the data information transmitted by the data aggregation layer and is equipped with platform monitoring servers, switches, and various terminal devices. It adopts the PHM-based fault prediction and health management method, uses the Internet of Things technology and deep learning algorithm, and analyzes the vibration acceleration, temperature, noise spectrum, operating fault status, and pedestrian dangerous behavior to determine the fault warning trend of the escalator; and establishes a three-dimensional structural model based on the escalator, and locates the real-time vibration acceleration, temperature, noise spectrum and fault warning trend in the three-dimensional structural model in real time, and transmits it to various terminal devices for intuitive visualization.

[0031] In a third aspect, the present invention provides a monitoring system, comprising:

[0032] processor;

[0033] a memory for storing instructions executable by the processor;

[0034] The processor is configured to execute the instructions to implement the escalator intelligent monitoring method based on multi-source perception and machine vision as described in any one of the first aspects.

[0035] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: the intelligent monitoring method and system for escalators based on multi-source perception and machine vision provided by the present invention can monitor the operating status of the escalator in real time through a variety of sensors (such as vibration sensors, temperature sensors and noise sensors), and can promptly discover potential hidden faults; by using machine vision technology, it can accurately identify and track pedestrians and their behaviors, and monitor pedestrians' dangerous behaviors (such as falls, retrograde, etc.); by using PHM (fault prediction and health management) technology, combined with deep learning algorithms, the collected data is analyzed, and early warning of fault trends can be given, thereby reducing the occurrence of accidents and ensuring the safety and reliability of the escalator; by establishing a three-dimensional structural model of the escalator and visually displaying the monitoring data in real time, managers can intuitively understand the operating status and potential problems of the escalator, which can effectively reduce the maintenance costs and downtime losses caused by sudden failures, reduce overall operating costs, and reduce waste of human resources and improve work efficiency through optimized scheduling and maintenance strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The present invention provides a flowchart of an escalator intelligent monitoring method based on multi-source perception and machine vision.

[0037] Figure 2 A schematic diagram of an escalator intelligent monitoring system based on multi-source perception and machine vision provided by the present invention.

[0038] Figure 3 A schematic diagram of a pyramid network fusion algorithm provided by the present invention.

[0039] Figures 4 to 11 A schematic diagram of calculating the distortion and verticality of pedestrians by spatially transforming 3D coordinates provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0041] Example

[0042] like Figure 1 As shown, an escalator intelligent monitoring method based on multi-source perception and machine vision is provided in an embodiment of the present invention, and the specific steps are as follows:

[0043] Obtain the real-time vibration acceleration, temperature, and noise spectrum of the mechanical parts in the truss of the escalator's machine room, monitor and obtain the operating fault status of the escalator in real time, and monitor and obtain the dangerous behavior of pedestrians on the escalator in real time;

[0044] Based on PHM fault prediction and health management method, the Internet of Things technology and deep learning algorithm are used to analyze vibration acceleration, temperature, noise spectrum, operating fault status, and pedestrian dangerous behavior to determine the fault warning trend of escalators;

[0045] A three-dimensional structural model based on the escalator is established, and the real-time vibration acceleration, temperature, noise spectrum and fault warning trend are reflected in the three-dimensional structural model for intuitive visualization.

[0046] like Figure 2 As shown in the figure, the monitoring system consists of three parts: data collection layer, data aggregation layer and data application layer.

[0047] The data collection layer is where the data collection terminal collects sensor data, and the video analysis terminal obtains real-time video streams for analysis and identification. The data is aggregated and transmitted to the application server via wireless / wired means, where calculations are performed. Finally, the on-site data can be viewed through the platform.

[0048] Based on multi-source perception fault diagnosis and machine vision pedestrian behavior recognition technology, the faulty parts of the escalator are located and dangerous behaviors are identified. During the operation of the escalator, the vibration, temperature, and noise spectrum are analyzed through the data acquisition terminal; and the pedestrian behavior is analyzed in real time through the video acquisition terminal.

[0049] Vibration sensors, temperature sensors and noise sensors are used to acquire data for escalator monitoring.

[0050] The monitoring points of the escalator are shown in Table 1 below.

[0051] Table 1

[0052]

[0053] The data collector in the escalator sub-unit performs A / D conversion on the collected vibration signals, noise and temperature signals, and sends them to the fault diagnosis device for intelligent analysis. The analysis results and important spectrum data are uploaded to the maintenance center server through the backbone network for diagnostic verification and intelligent decision-making, and the data is applied and stored.

[0054] The data collection frequency of the sensor can be adjusted intelligently. Under normal circumstances, data is collected once every 10 seconds (the collection frequency of each sensor can be adjusted in the software according to actual conditions). When the data is abnormal, the collection frequency of the corresponding sensor is increased to accurately predict the fault.

[0055] One monitoring point is installed on each gearbox housing, the housing surface is polished and paint removed, and a vertical vibration measuring point is installed by bonding or threading.

[0056] A vibration measuring point is installed at the left and right bearings of the main drive wheel and the step chain tensioner by gluing or threading. A non-contact temperature monitoring point is installed on each of the left and right handrails. The installation form is shown in the figure above. A new mounting bracket needs to be designed according to the on-site truss structure of the escalator. A temperature sensor is also installed to measure the ambient temperature for comparative analysis.

[0057] The escalator needs to install the data acquisition terminal and the junction box on site. The size of the junction box is determined according to the user's needs. It is arranged in the machine room on the escalator and is installed in a pull-out manner to save space as much as possible. The video acquisition terminal is installed on the upper and lower foundation pit ceilings, as shown in the figure below. The specific installation location on site must be determined according to the actual situation of each escalator.

[0058] In this embodiment, the monitoring faults of the escalator are shown in Table 2 below.

[0059] Table 2

[0060]

[0061] The data convergence layer (Transport Layer) is an important part of the system data transmission. It is mainly responsible for providing reliable data transmission services between two communicating network nodes. It is located above the perception layer and below the application layer.

[0062] The data transmission layer uses the HTTP (Hypertext Transfer Protocol) transmission protocol to ensure the reliability and integrity of the data. It ensures the correct transmission of data by establishing connections, flow control, error checking, and retransmission mechanisms. The data transmission layer is also responsible for segmenting and reassembling the data. At the sending end, the data from the application layer is divided into smaller data segments, and sequence numbers, checksums, and other information are added to each data segment. At the receiving end, the data is reassembled and checked according to the sequence number and checksum information to ensure the integrity of the data. In addition, the data transmission layer is also responsible for multiplexing and demultiplexing. Multiplexing refers to integrating data from different application layers into a data stream for transmission over the network, while demultiplexing decomposes the received data stream into different data and sends it to the corresponding application layer.

[0063] The data collection layer uses 4G wireless network to transmit monitoring data to the application layer. After being processed by the application server, the monitoring information can be uploaded to the system platform and APP through the transmission layer.

[0064] The data application layer is composed of data analysis modules and application service modules. The data analysis module is composed of intelligent analysis terminals, which use artificial intelligence, machine learning, pattern recognition and other technologies to conduct in-depth analysis of the data obtained by the front end; the application service module provides users with a convenient and fast human-computer interaction system. The system has functions such as fault alarm, status monitoring, fault prediction, health ranking management, escalator operation status management, fault library and feature management, maintenance management, etc., to realize the intelligentization, automation and health assessment of escalator monitoring;

[0065] In addition, relevant maintenance personnel can remotely view on-site working conditions through the system, check whether there are any problems in the construction at any time and record them, shorten the time required for maintenance and overhaul, improve the efficiency of maintenance operations, effectively improve the safety factor of railway operations, and reduce manual maintenance costs.

[0066] The system platform records the monitoring data of the entire project monitoring period, and establishes a structural three-dimensional model based on the escalator. The data measured by the front-end equipment is reflected on the three-dimensional model, and the escalator is displayed intuitively to meet the risk warning monitoring needs.

[0067] For all kinds of front-end data collected on the platform, the platform provides multiple protocols to support data collection from various data sources. The platform integrates Apache NiFi middleware to support data collection through files, HTTP, FTP, Kafka, databases, etc. It supports active round-robin query and passive data acquisition. For data types, it supports structured data sets, forms, static JSON data, semi-structured dynamic JSON data, and unstructured binary data such as files and pictures. The platform provides distributed massive storage space for collected data. For structured data storage, the platform provides a MySQL database cluster solution based on MHA high availability, which provides large-volume relational data storage through the strategy of sub-library and sub-table; for semi-structured data, the platform provides an HBase storage solution built on Hadoop, which supports efficient storage and retrieval of massive data; for unstructured binary data such as files and pictures, the platform provides a distributed storage solution based on MinIO. By building a multi-node MinIO cluster, it not only improves the efficiency of data storage, but also avoids single point failures and ensures data security.

[0068] The system of this application is to improve the safety and management level of escalators. It adopts PHM (fault prediction and health management) theory, combines Internet of Things technology and deep learning algorithm, and establishes an escalator monitoring technology service management system. Through the spectrum analysis of vibration acceleration, noise, temperature and other escalator components and real-time monitoring of escalator operation status and fault status, the fault warning trend is realized, thereby changing the traditional regular maintenance method and realizing the maintenance strategy of "maintenance on demand", ultimately reducing accidents and reducing the maintenance of escalators throughout the life cycle.

[0069] The system of the present application obtains the human body structure based on the Openpose key point monitoring algorithm, and then uses the SVM classifier to classify the human body posture, so as to realize the monitoring and alarm of pedestrian falls in the picture; and uses the YOLO target detection model to obtain pedestrians, large luggage and other targets in the picture, and then calls the KCF tracking algorithm for the pedestrian target to realize the monitoring and alarm of dangerous behaviors that may exist in the surveillance video.

[0070] The algorithm used for escalator area segmentation is FCN-HRNetW18, whose backbone is HRNet (High Resolution Networks) and head is FCN (Fully Convolutional Networks). HRNet is a multi-branch high-resolution network designed for posture estimation tasks. However, since the network maintains high resolution throughout the process, it can maintain the spatial information of the input image and reduce the loss of spatial information caused by downsampling, which is naturally suitable for pixel-level prediction tasks such as semantic segmentation. In addition to maintaining high-resolution spatial information, its network structure also performs multi-scale information fusion, allowing the network to extract strong semantic information, further enhancing the network's expressive power.

[0071] FCN is a fully convolutional network and a classic semantic segmentation algorithm. Its characteristic is that all layers of the network are composed of convolutional layers and there is no fully connected layer. Therefore, it can adapt to input images of different spatial sizes. Then, deconvolution is used to upsample the feature map of the last convolutional layer to the input image to restore it to the same size as the input image. Then, each pixel is classified to obtain the final prediction output.

[0072] FCN-HRNet is a popular semantic segmentation algorithm. It uses HRNet as the backbone to extract feature maps containing rich semantic and spatial information, and then sends them to the prediction head composed of FCN to predict the category of each pixel. It is characterized by a simple network structure and high accuracy. HRNet maintains high resolution throughout the process and has multiple multi-scale feature fusions, so the network overhead is large. Therefore, HRNet has several different network configurations to adapt to different application scenarios. HRNetW18 is a lightweight configuration that takes into account both accuracy and speed. The input of FCN-HRNetW18 is a picture of size H×W×3, and the output is a prediction of H×W×C, where C represents the number of categories and the value of each category channel represents the probability that the pixel belongs to that category.

[0073] In this embodiment, the fusion algorithm of the pyramid network is used for posture recognition. The fusion algorithm principle of the pyramid network is similar to that of YOLO. The main difference is that a key point detection branch is added on the basis of the detection head of YOLO, and an OKS loss is designed for training to improve the accuracy of the posture estimation results. Its network structure is relatively simple, consisting of backbone, neck, and head. The backbone is CSP-DarkNet, the neck is the pyramid network PAN, and the head is composed of box head and keypointhead (the main difference from YOLO). Each head is composed of several convolutional layers, and the box head couples box prediction and class prediction.

[0074] The algorithm output size can be calculated as follows:

[0075] Calculation formula

[0076]

[0077] in:

[0078] B: batch size

[0079] A: Number of anchors, fixed to 3 in yolo-pose

[0080] H:height, enter the image height

[0081] W:width, enter the image width

[0082] According to the symbols defined in the above calculation formula, the algorithm outputs the size calculation process, such as Figure 3The output size of each layer with different strides is downsampled by stride times the width and height of the input image. For example, the size of the head output at stride 8 is reduced by 8 times the width and height of the entire image. The outputs of heads with different strides are concatenated after inference decoding to obtain the final output.

[0083] In this embodiment, object detection using a deep learning model is used. The YOLO series is the most widely used model in the industry for target detection tasks due to its detection speed and accuracy. Each generation has improved performance and speed over the previous one. Therefore, the YOLOv5 algorithm is used for the baby stroller, suitcase, and shopping cart detection algorithm.

[0084] Like v4 and v3, the YOLOv5 model architecture is divided into three parts: backbone, neck, and multi-scale prediction, and the modules and tricks used in them are different. YOLOv5 provides four models of different depths and widths, namely YOLOv5s, YOLOv5m, YOLOv5l, and YOLOv5x.

[0085] The modules used in the four models of YOLOv5 are basically the same. The main difference lies in the depth and width of the stacking of each module. Taking YOLOv5s as an example, the overall structure of the model and the specific construction of each module are mainly different in the following points:

[0086] (1) Input: Mosaic data enhancement, adaptive anchor box calculation, and adaptive image scaling;

[0087] (2) Backbone: Focus structure, CSP structure;

[0088] (3) Neck: FPN+PAN structure;

[0089] (4)Prediction: GIOU_Loss.

[0090] Specifically, (1) the input end uses the tricks of Mosaic data enhancement, adaptive image scaling, and adaptive anchor box calculation.

[0091] Mosaic data enhancement, compared to CutMix's method of cropping part of an image and then filling it with part of another image, Mosaic data enhancement first randomly scales, crops, and flips the four images, then places the four images at the four corners of the new image, and finally splices them into a larger image. The target bounding box of the new image is corrected by the real box of the original four images, and the occluded target and bounding box are removed. Therefore, the Mosaic data enhancement method allows one image to contain the information of four images, and the background of the image is more complex, so the model can learn to acquire the ability to detect targets in complex backgrounds, so that the trained model has better generalization performance.

[0092] In the YOLO algorithm, for different data sets, there will be an anchor box with an initial set length and width. During network training, the network outputs a predicted box based on the initial anchor box, and then compares it with the groundtruth box to calculate the difference between the two, and then reversely updates and iterates the network parameters. Therefore, the initial anchor box is also a relatively important part. In Yolov3 and Yolov4, when training different data sets, the value of the initial anchor box is calculated by running a separate program. In Yolov5, this function is embedded in the code, and each time training is performed, the best anchor box value in different training sets is adaptively calculated. The calculation of the initial anchor box of v3, v4, and v5 all uses the k-means algorithm.

[0093] In the target detection algorithm, since the input image sizes are often different, the images are generally scaled to the same size and then input into the network for training. However, direct scaling will change the aspect ratio of the image, causing image distortion and reducing the model training effect. The YOLO series of algorithms use padding scaling, which scales the image while maintaining the aspect ratio, and then fills the image with 0 pixels on both sides to make the image consistent with the input size required by the model.

[0094] In actual project use, many images have different aspect ratios, so after scaling and padding, the black edges at both ends are different in size. If there is too much padding, there will be information redundancy, which will affect the inference speed. Therefore, Yolov5 has made some modifications to this, adaptively padding the least pixels of the original image.

[0095] (2) Backbone

[0096] Drawing on the idea of ​​CSPNet, two different CSP structures are designed.

[0097] The Focus operation slices an image into four small images and then concats them, reducing the image size by two times and the channel by four times.

[0098] (3) Neck

[0099] The Neck part uses FPN and PANnet structures for feature fusion.

[0100] (4)Prediction

[0101] Multi-scale prediction, GIOU_Loss is used as the loss function of Boundingbox, and weighted nms is adopted.

[0102] In this embodiment, the 3D coordinate calculation combined with the spatial transformation

[0103] (1) Setting initialization

[0104] (2) Determine the horizontal direction: Find the ladder path location based on the image segmentation results and obtain the red line slope k (i.e., the horizontal slope)

[0105] (3) Determine the baffle position

[0106] The left baffle position C and the right baffle position C are obtained according to the image segmentation algorithm; Figure 4 shown.

[0107] (4) Determine whether the body is distorted

[0108] The body distortion vector is determined by the midpoint of nose and left_hip & right_hip, and the body distortion is determined according to the direction of the vector (such as Figure 5 The red arrow, if the slope exceeds the set threshold range, is considered to be distorted). If the head cannot be detected, it is not distorted by default.

[0109] (5) Determine the vertical direction

[0110] If the body is not distorted, the distortion vector is taken as the vertical direction of the person by default; if the body is distorted, the vertical line in the horizontal direction is taken as the vertical direction of the person. Figure 6 .

[0111] (6) Three-dimensional intrusion algorithm process:

[0112] From the previous step, we can know: the left baffle position C, the right baffle position C, the horizontal direction k, the human body posture αi of the i-th person, and the vertical direction βi of the position.

[0113] Determine whether it belongs to the target object to be identified based on the ankle position (judgment logic: if one of the ankles is on the floor, it does not belong to the target object to be identified; otherwise, if both ankles are in the elevator ladder, it belongs to the target object to be identified; otherwise, it does not belong to the target object to be identified). If not, it is assumed to be legal.

[0114] Three-dimensional invasion algorithm steps:

[0115] ① If the human body posture αi in the target object has complete joint points, execute ②, otherwise execute ④;

[0116] ② Take the point with the larger Y coordinate value of left_ankle or right_ankle, and make a straight line with a slope of k, which intersects the left baffle position C at point a and the right baffle position C at point b. Make straight lines L1 and L2 with slopes of βi through points a and b respectively. L1 and L2 determine the area M, such as Figure 7 shown.

[0117] ③If the entire human body αi is within region M, then it is legal and steps ④ to ⑤ are skipped;

[0118] ④ If the complete posture cannot be estimated, perform out-of-bounds judgment according to the following steps:

[0119] If all observable joint points are outside the elevator outline, it is out of bounds;

[0120] If some joints are out of bounds, the hands are inside the elevator outline (or invisible) and the body is distorted, then it is out of bounds (such as Figure 8 ), the hands are inside the outline, but the head is outside the outline.

[0121] If the hand is not inside the elevator contour, the direction of the line vector between the wrist and elbow and the direction of the line vector between the elbow and shoulder determine whether the hand is in an extended state (e.g. Fig. 9 ), stretching it would cross the boundary.

[0122] ⑤ For a complete posture, follow the steps below to determine if the boundary has been crossed:

[0123] For the posture point, set a tolerance threshold (15% of the escalator width). Assume that the coordinates of point a are (xa, ya), the coordinates of point b are (xb, yb), and the coordinates of each posture point are (xi, yi). If |xa-xi|+|xb-xi|-|xa–xb|>σ, it is considered out of bounds, otherwise it is legal. Fig.10 .

[0124] In machine learning, there are many tracking algorithms, including CSRT / MOSSE / KCF / BOOSTING / Goturn, etc. In deep learning, there are SORT, DeepSort, MOTDT, etc. Multiple Object Tracking (MOT) has the main task of finding moving objects in a given image sequence and identifying moving objects in different frames, that is, giving a definite and accurate ID. Fig.11 .

[0125] The currently adopted tracker algorithm uses a discriminative correlation filter (DCF) based visual tracker to learn a correlation filter specific to an object and use the learned correlation filter to localize the same object in the next frame. This correlation filter learning and localization is usually done on a per-object basis in a typical MOT implementation, which may create a large number of small CUDA kernel launches when processed on a GPU. This inherently poses a challenge to maximize GPU utilization, especially when a large number of objects from multiple video streams need to be tracked on a single GPU.

[0126] For each tracked object, the DCF tracker defines a search region around its predicted position in the next frame that is large enough so that the same object is detected in the search region. The position of the object on the new video frame is predicted using the state estimator module. The searchRegionPaddingScale attribute determines the size of the search region as a multiple of the diagonal of the object's bounding box.

[0127] For each cropped image patch, visual appearance features such as color names or histograms of oriented gradients (HOG) are extracted. The relevant filters are generated by applying an attention window (using a sliding window) at the center of the target bbox. The user is allowed to move the center of the attention window in the vertical direction. For example, it will cause the center of the attention window to be located in the feature map with a height considering a typical surveillance or CCTV camera installation, which is located at a medium height to monitor a large range of environments, such as a retail store or a traffic intersection.

[0128] Once a correlation filter is generated for a target, DCF-based trackers typically employ an exponential moving average to achieve temporal consistency when creating and updating the best correlation filter over consecutive frames.

[0129] Association of object IDs across frames, for robust tracking, usually requires similarity matching based on visual appearance, for which visual appearance features are extracted at each candidate location. This is usually a computationally expensive process and often becomes a performance bottleneck in object tracking. Unlike existing methods that extract visual features from all candidate locations and perform feature matching between all candidate objects, the DCF tracker utilizes the correlation response (already obtained at the target detection stage) as the tracking confidence map for each tracker, searches the region and simply looks up the confidence value of each candidate location (i.e., the location bbox of each detector object) to obtain visual similarity. By comparing the confidence between trackers, we can determine which tracker has a higher visual similarity with a specific detector object and use it as part of the data association matching score. Therefore, visual similarity matching during the data association process can be performed very efficiently by performing a simple lookup table (LUT) operation on the existing association responses.

[0130] In this embodiment, during the operation monitoring process of the escalator, the monitored vibration, temperature, and noise signals mostly present periodic non-stationary characteristics, especially the fault signals of gears, bearings, etc. are more typical. At this time, the traditional signal analysis method based on Fourier transform is not suitable, and it is difficult to extract effective fault features. Therefore, it is necessary to use a non-stationary signal analysis method. The time-frequency analysis method maps the one-dimensional time domain signal to the two-dimensional time-frequency surface, comprehensively reflects the time-frequency distribution characteristics of the signal change caused by time change, and is an effective tool for analyzing non-stationary signals. Among these time-frequency analysis methods, Wigner-Ville distribution (WVD) has become an important tool for analyzing non-stationary time-varying signals due to its excellent time-frequency characteristics, and can be used for feature extraction and fault diagnosis of vibration faults. However, WVD has a defect, that is, it contains interference of cross terms, which affects the extraction of vibration signal features and the analysis of fault diagnosis. Although the cross terms cannot be completely removed in theory, they can be effectively suppressed by some methods to achieve satisfactory results. In view of the periodic non-stationary characteristics of the escalator vibration signal, WVD is used to extract features and diagnose faults of the escalator vibration signal.

[0131] The main defect of WVD is the existence of cross-interference terms. Cross-interference terms refer to the fact that when a signal contains multiple components, the WVD of the signal will have oscillation components without any physical meaning at the midpoint of the time-frequency center coordinates between the two components. They provide false energy distribution and affect the physical interpretation of WVD. From a mathematical point of view, the existence of cross-terms is due to the interference of time and frequency caused by nonlinear transformation.

[0132] The WVD of the sum of two signal components is not the sum of their individual WVDs. In addition to the two self-term components, its WVD also produces a mutual term, namely the cross term, which affects the intuitive expression of WVD and makes the process of extracting useful information from the WVD distribution more complicated. There are many combinations of self-terms and cross-terms, and cross-terms may also appear in the position of self-terms, which makes the self-term distribution more strongly disturbed. These are all situations that need to be avoided in the actual application of WVD. For example, in the instantaneous frequency estimation based on time-frequency analysis, its principle is to estimate the peak value where the energy is concentrated to determine the frequency change. If there is a cross term, it will affect our ability to obtain the correct peak estimate.

[0133] It is calculated that when a signal contains N components, the number of cross terms is N(N-1) / 2. For actual signals, cross terms may be mixed with self terms, making the interference behavior more complicated. Cross terms are usually oscillating, and the amplitude can be twice that of the self terms. When performing time-frequency analysis on multi-component signals, a large number of cross terms will drown out or seriously interfere with the self terms of the signal, which limits the application of WVD.

[0134] In addition, an embodiment of the present invention further provides a monitoring system, including:

[0135] processor;

[0136] a memory for storing instructions executable by the processor;

[0137] Wherein, the processor is configured to execute the instructions to implement the above-mentioned escalator intelligent monitoring method based on multi-source perception and machine vision.

[0138] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An intelligent monitoring method for escalators based on multi-source perception and machine vision, characterized in that: The steps include: Obtain the real-time vibration acceleration, temperature, and noise spectrum of the mechanical parts in the truss of the escalator's machine room, monitor and obtain the operating fault status of the escalator in real time, and monitor and obtain the dangerous behavior of pedestrians on the escalator in real time; Based on PHM fault prediction and health management method, the Internet of Things technology and deep learning algorithm are used to analyze vibration acceleration, temperature, noise spectrum, operating fault status, and pedestrian dangerous behavior to determine the fault warning trend of escalators; A three-dimensional structural model based on the escalator is established, and the real-time vibration acceleration, temperature, noise spectrum and fault warning trend are reflected in the three-dimensional structural model for intuitive visualization.

2. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 1 is characterized in that: Most of the mechanical parts of the escalator are located in the machine room truss, and the main driving load moving parts are the motor, reducer, main drive wheel, and step chain tensioner.

3. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 2 is characterized in that: Normally, escalator monitoring uses vibration sensors, temperature sensors, and noise sensors; The sensor setup includes: Install one monitoring point on each gearbox housing, polish and remove paint from the housing surface, and install a vertical vibration measuring point by bonding or threading; Install a vibration measuring point at the left and right bearings of the main drive wheel and the step chain tensioner respectively by gluing or threaded interface; One non-contact temperature monitoring point is installed on each of the left and right armrests, and another temperature sensor is installed to measure the ambient temperature for comparative analysis.

4. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 3 is characterized in that: Under normal circumstances, data is collected once every 10 seconds. The collection frequency of each sensor can be adjusted in the software according to the actual situation. When the data is abnormal, the collection frequency of the corresponding sensor is increased to accurately predict the fault. The collected vibration signals, noise and temperature signals are converted into A / D signals and sent to the fault diagnosis device for intelligent analysis. The analysis results and important spectrum data are uploaded to the maintenance center server through the backbone network for diagnostic verification and intelligent decision-making.

5. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 1 is characterized in that: During the operation monitoring of escalators, the monitored vibration, temperature and noise signals mostly show periodic non-stationary characteristics. Wigner-Ville (WVD) is used to extract features of escalator vibration signals containing cross-interference and perform fault diagnosis.

6. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 1 is characterized in that: Methods for real-time monitoring and obtaining dangerous behaviors of pedestrians on escalators include: The human body structure is acquired based on the Openpose key point monitoring algorithm, and then the SVM classifier is used to classify the human body posture, so as to realize the monitoring and alarm of pedestrian falls in the picture; The YOLO target detection model is used to obtain pedestrians and large luggage targets in the picture; then the KCF tracking algorithm is called on the pedestrian targets to monitor and alarm for possible dangerous behaviors in the surveillance video.

7. The escalator intelligent monitoring method based on multi-source perception and machine vision according to claim 6 is characterized in that: Integrate multi-target tracking technology to track and identify moving objects on escalators in real time, including: The escalator area is segmented based on the FCN-HRNetW18 model algorithm; The fusion algorithm of the pyramid network is used to recognize the human posture of pedestrians on the escalator, and the 3D coordinate calculation of the spatial transformation is combined to determine the pedestrian's body distortion and vertical state; Use the YOLOv5 deep learning model to perform object detection on escalators for baby strollers, suitcases, and shopping carts.

8. The escalator intelligent monitoring system based on multi-source perception and machine vision is characterized by: These include: The data collection layer is equipped with data collection terminals and video collection terminals. The data collection terminals are used to collect vibration acceleration, noise, and temperature data information transmitted by various sensors, and the video collection terminals are used to collect the behavior data information of pedestrians on the escalator. The data aggregation layer is configured with wired or wireless means to transmit data to the data application layer; The data application layer receives data information transmitted by the data aggregation layer and is equipped with platform monitoring servers, switches, and various terminal devices. It adopts the PHM-based fault prediction and health management method, uses the Internet of Things technology and deep learning algorithms, and analyzes vibration acceleration, temperature, noise spectrum, operating fault status, and pedestrian dangerous behavior to determine the fault warning trend of the escalator; A three-dimensional structural model based on the escalator is established, and the real-time vibration acceleration, temperature, noise spectrum and fault warning trend are reflected in the three-dimensional structural model in real time, and transmitted to various terminal devices for intuitive visualization.

9. A monitoring system comprising: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the escalator intelligent monitoring method based on multi-source perception and machine vision as described in any one of claims 1 to 7.

Images