Escalator health state monitoring method and system fusing behavior vision and electromechanical signals

Abstract

This invention discloses a method and system for monitoring the health status of escalators by integrating behavioral vision and electromechanical signals, belonging to the field of intelligent escalator maintenance technology. Key technical points include: simultaneously acquiring binocular video image streams, motor current signals, and vibration signals from multiple locations; calculating the real-time differential speed between the handrail and treads based on binocular vision and optical flow 3D mapping, extracting key skeletal points of passengers to determine their standing, walking, or leaning behavior, and extracting current and vibration spectrum features; dynamically calling a matching evaluation benchmark model based on the behavior status, and performing multimodal cross-validation of the differential speed, current features, and vibration spectrum to determine the real-time health status; and outputting maintenance strategies including status records, maintenance warnings, or emergency shutdown based on the health status. This invention can filter out interference from human behavior, reduce false alarm rates, achieve non-contact accurate speed measurement and complex fault location, and improve the level of intelligent escalator maintenance.

Description

Technical Field

[0001] This invention belongs to the field of intelligent maintenance technology for escalators, and particularly relates to a method and system for monitoring the health status of escalators that integrates behavioral vision and electromechanical signals. Background Technology

[0002] Escalators are indispensable special transportation equipment in public places such as rail transit and large shopping malls, and their safe and stable operation is directly related to the safety of public life and property. With the continuous growth of the number of escalators in operation, how to efficiently and accurately monitor their health status and carry out intelligent maintenance has become a core technical problem facing the industry.

[0003] Existing structural health monitoring schemes for escalator drive systems objectively determine whether the structural health of the drive chain has been damaged by monitoring the transfer function using sensors on the first and second drive components. However, this scheme is limited to physical monitoring of a single mechanical transmission component inside the escalator, employing a fixed alarm threshold, and fails to consider the highly dynamic nature of the load state during actual escalator operation. Different behaviors of passengers, such as standing, walking, and leaning, will generate different transient impacts on the equipment. The fixed threshold scheme cannot distinguish between transient physical feedback fluctuations caused by normal passenger movement and the early characteristics of actual mechanical failures. In practical applications, this easily leads to false alarms, increasing unnecessary on-site inspections and escalator shutdowns for troubleshooting, and seriously affecting operational efficiency.

[0004] Existing escalator status monitoring systems and methods collect multi-dimensional signals such as vibration, temperature, sound, and power to extract real-time status feature vectors. These are then combined with historical fault data and expert knowledge bases, employing algorithms such as neural networks for intelligent diagnosis and fault prediction. However, in real-world scenarios, these systems cannot detect the behavioral states of passengers, such as standing, walking, or leaning, and lack a dynamic threshold adjustment mechanism based on changes in external load. This results in the system's inability to adaptively relax thresholds or shield against specific high-frequency vibration interference generated by passenger movement under high-dynamic load conditions, leading to persistent false alarms. Furthermore, in emergency situations such as passengers leaning, the system cannot directly trigger the highest-priority protection response beyond conventional data thresholds, posing a potential risk of delayed safety procedures.

[0005] Furthermore, traditional contact-type speed sensors, due to prolonged frictional contact with handrails or pedals, suffer from wear and tear, high maintenance costs, and increasing measurement errors over time, making them unsuitable for long-term, maintenance-free, and accurate monitoring. Simultaneously, relying solely on single-dimensional electromechanical signals for fault diagnosis makes it difficult to accurately locate complex and hidden mechanical faults such as foreign object obstruction, loose pedals in the reversing zone of the ladder guide rail, or broken clips, easily leading to missed or false diagnoses and failing to provide a reliable basis for intelligent maintenance decisions. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a method and system for monitoring the health status of escalators that integrates behavioral vision and electromechanical signals, thereby resolving the issues present in the prior art.

[0007] In a first aspect, to achieve the above objectives, the present invention provides a method for monitoring the health status of escalators that integrates behavioral vision and electromechanical signals, comprising the following steps: Simultaneously collect binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation; The speed of the pedal and the speed of the handrail are calculated from the binocular video image stream to obtain the real-time differential speed. The key skeletal points of the passengers are identified to determine the passenger behavior status. At the same time, real-time current features and vibration spectrum features are extracted from the motor current signal and vibration signal. The preset evaluation benchmark model is dynamically invoked based on the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The real-time differential speed, real-time current characteristics, and vibration spectrum characteristics are input into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. Maintenance intervention strategies are generated and output based on the real-time health status.

[0008] Optionally, the process of synchronously collecting data includes: A binocular wide-angle camera positioned directly above the escalator is used to acquire a binocular video image stream containing depth information from a parallel overhead angle. The current signal of the drive motor is acquired by a non-invasive current sensor at a sampling frequency no less than the frame rate of binocular video. The vibration signal is acquired by a vibration sensor, and the sampling frequency of the vibration sensor is set to above 2kHz.

[0009] Optionally, the process of calculating the pedal speed and the handrail belt speed includes: The edge warning line of the pedal is identified as a feature anchor point. The Euclidean spatial displacement distance of the feature anchor point within adjacent frames is calculated. The ratio of the Euclidean spatial displacement distance to the inter-frame time difference is used as the pedal running speed. Microscopic textures on the surface of the handrail are extracted, and optical flow is used to calculate the two-dimensional motion vectors of the textures in consecutive image frames. The two-dimensional motion vectors are then mapped to three-dimensional physical space using depth information to calculate the actual physical displacement. The ratio of the actual physical displacement to the time difference is used as the running speed of the handrail.

[0010] Optionally, the process of identifying skeletal key points of elevator passengers to determine their elevator riding behavior includes: Extract key spatial positions of the head, hands, and legs of elevator passengers; Based on the relative spatial topological relationship between the key points and the dynamic displacement trajectory over time, the elevator posture of passengers is evaluated, and the elevator behavior state is divided into standing state, walking state and tilting state.

[0011] Optionally, the process of extracting real-time current features and vibration spectrum features from motor current signals and vibration signals includes: Extract the effective value, average value, peak factor, and short-time volatility characteristics of the motor current signal; The spectral distribution of the vibration signal is extracted by fast Fourier transform to obtain the energy proportion, peak frequency and its amplitude, and kurtosis characteristics of the preset frequency band.

[0012] Optionally, the process of dynamically invoking a preset evaluation benchmark model based on elevator riding behavior includes: When the elevator riding behavior status is standing, the basic load assessment benchmark is invoked. When the elevator riding behavior is in the moving state, the transient fluctuation threshold of the allowable differential speed threshold range in the evaluation benchmark model is relaxed, the peak value judgment benchmark in the current benchmark feature is adjusted upward, and the high-frequency vibration signal caused by the movement of people is shielded. When the elevator is in a tilting state, an emergency stop command and a rescue alarm command are triggered.

[0013] Optionally, the process of obtaining the real-time health status of escalators includes: When the real-time differential speed continuously leads or lags, the motor current signal slowly deviates from the steady-state current reference, and the vibration signal is within the low-frequency tolerance range, it is determined that there is a foreign object blocking the escalator or an abnormality in the transmission system. When the real-time differential speed experiences a momentary jump, the motor current signal shows an abnormal peak, and the vibration signal at the main drive sprocket bearing housing shows abnormal excitation energy exceeding the tolerance range, it is determined that the escalator drive chain is at risk of jamming or breakage. When the speed difference between the handrail and the tread is within the allowable range, and an abnormal high-frequency vibration signal appears in the guide rail reversing zone, it is determined that the tread in the escalator reversing zone is loose or the clip is broken.

[0014] Optionally, the process of generating and outputting maintenance intervention strategies based on real-time health status includes: When all multimodal real-time monitoring data are within the allowable range of the evaluation benchmark model, a normal status record is generated; When a single dimension in the multimodal real-time monitoring data shows a slow deterioration trend or multiple dimensions show slight and occasional exceedances, a maintenance recommendation work order is generated for the abnormal component. When the multimodal real-time monitoring data shows multiple dimensions crossing limits, a linkage control signal is output to the escalator control cabinet to execute a shutdown operation.

[0015] Secondly, the present invention also provides an escalator health status monitoring system that integrates behavioral vision and electromechanical signals, for implementing an escalator health status monitoring method that integrates behavioral vision and electromechanical signals, the system comprising: The multimodal data acquisition module is used to synchronously acquire binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation. The vision processing and state recognition module is used to calculate the running speed of the pedal and the running speed of the handrail from the binocular video image stream to obtain the real-time differential speed, and to identify the skeletal key points of the passengers to determine the riding behavior state. At the same time, it extracts real-time current features and vibration spectrum features from the motor current signal and vibration signal. The dynamic benchmark scheduling module is used to dynamically call a preset evaluation benchmark model according to the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The multimodal fusion evaluation module is used to input the real-time differential speed, real-time current characteristics and vibration spectrum characteristics into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. The maintenance strategy output module is used to generate and output maintenance intervention strategies based on the real-time health status.

[0016] Compared with the prior art, the present invention has the following advantages and technical effects: This invention provides a method and system for monitoring the health status of escalators by integrating behavioral vision and electromechanical signals. It treats the standing, walking, and leaning behaviors of escalators as environmental variables, and intelligently distinguishes between normal human activity and actual mechanical faults through a dynamic threshold adjustment mechanism. When people are walking, the system automatically relaxes the judgment threshold and shields high-frequency vibration interference from normal physical feedback, significantly reducing the system's false alarm rate. When people lean, it ignores conventional thresholds and directly triggers the highest level of protection response, significantly improving the equipment's safety sensitivity and emergency response speed. Through a non-contact binocular vision and optical flow three-dimensional mapping differential speed measurement scheme, it overcomes the problems of easy wear and high maintenance costs associated with traditional contact-based speed measurement, while also solving the technical challenge of easily losing targets when visually tracking the homogeneous surface of the handrail. Deep cross-validation of multimodal data can accurately distinguish complex and hidden faults such as foreign object blockage, transmission chain jamming and breakage, and loosening of the reversing zone pedals, achieving high-precision monitoring and assessment of escalator mechanical faults. This promotes the upgrade of maintenance from reactive repair to proactive early warning, effectively saving maintenance costs and optimizing resource allocation efficiency. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating an escalator health status monitoring method that integrates behavioral vision and electromechanical signals according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the positions of various types of sensors pre-arranged during the acquisition of multimodal fact monitoring data in an embodiment of the present invention. Among them, 1-non-invasive current sensor, 21, 22-first vibration sensor, 31, 32-third vibration sensor, 4-second vibration sensor, 5-binocular vision acquisition module. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] Example 1 like Figure 1 As shown, this embodiment provides a method for monitoring the health status of escalators by integrating behavioral vision and electromechanical signals, including: Simultaneously collect binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation; The speed of the pedal and the speed of the handrail are calculated from the binocular video image stream to obtain the real-time differential speed. The key skeletal points of the passengers are identified to determine the passenger behavior status. At the same time, real-time current features and vibration spectrum features are extracted from the motor current signal and vibration signal. The preset evaluation benchmark model is dynamically invoked based on the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The real-time differential speed, real-time current characteristics, and vibration spectrum characteristics are input into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. Maintenance intervention strategies are generated and output based on the real-time health status.

[0021] Furthermore, the process of synchronously collecting data includes: A binocular wide-angle camera positioned directly above the escalator is used to acquire a binocular video image stream containing depth information from a parallel overhead angle. The current signal of the drive motor is acquired by a non-invasive current sensor at a sampling frequency no less than the frame rate of binocular video. The vibration signal is acquired by a vibration sensor, and the sampling frequency of the vibration sensor is set to above 2kHz.

[0022] Specifically, the implementation process of this embodiment includes: S100. Acquire multimodal real-time monitoring data. In this step, multimodal real-time monitoring data during the escalator's operation is simultaneously collected using pre-deployed multiple types of sensors. The data to be collected includes at least binocular video image streams, motor current signals, and vibration signals from key components. The specific acquisition method is as follows: like Figure 2 As shown, a binocular vision acquisition module 5 is installed directly above the escalator, specifically on the ceiling of the building or on a specially erected gantry. The binocular vision acquisition module 5 includes two rigorously calibrated wide-angle industrial cameras with parallel optical axes, covering the entire operating area of ​​the escalator from a top-down angle. It can synchronously acquire binocular video image streams containing depth information at a fixed frame rate (e.g., 25 frames / second). The baseline distance between the two cameras is set according to the installation height and field of view requirements. For example, when the installation height is 3–4 meters, the baseline distance can be selected between 120 and 200 millimeters to ensure sufficient depth measurement accuracy in the tread and handrail areas.

[0023] A non-invasive current sensor 1 is installed on the power supply line or inverter output of the escalator drive unit to collect the current signal of the drive motor in real time at a sampling frequency no less than the frame rate of binocular video. This current signal can accurately reflect the transient changes of the drive load. Simultaneously, piezoelectric or MEMS vibration sensors are installed on multiple core mechanical components, specifically: first vibration sensors 21 and 22 are installed on the bearing housing of the escalator's main drive sprocket to sense the impact and vibration of the drive chain's operation; a second vibration sensor 4 is installed on the handrail friction wheel bracket to monitor the friction and slippage characteristics of the handrail drive system; and third vibration sensors 31 and 32 are installed on the guide rails in the step reversing zone, i.e., at the escalator's up and down bends, to capture the vibration impact when the steps enter or leave the reversing zone. The sampling frequency of the vibration sensors is preferably set to 2kHz or higher to meet the requirements of mechanical vibration spectrum analysis. Data collected by various sensors are all imported into the same edge computing node or industrial control computer via a network or analog / digital interface, and a unified timestamp is added to the data frame header to ensure strict time synchronization of multimodal data.

[0024] Furthermore, the process of calculating the pedal speed and the handrail belt speed includes: The edge warning line of the pedal is identified as a feature anchor point. The Euclidean spatial displacement distance of the feature anchor point within adjacent frames is calculated. The ratio of the Euclidean spatial displacement distance to the inter-frame time difference is used as the pedal running speed. Microscopic textures on the surface of the handrail are extracted, and optical flow is used to calculate the two-dimensional motion vectors of the textures in consecutive image frames. The two-dimensional motion vectors are then mapped to three-dimensional physical space using depth information to calculate the actual physical displacement. The ratio of the actual physical displacement to the time difference is used as the running speed of the handrail.

[0025] Specifically, the implementation process of this embodiment includes: S200. Visual and Sensor Feature Extraction. This step extracts key features that characterize the escalator's operating status and human behavior from the binocular video image stream, motor current signal, and vibration signal obtained in step S100.

[0026] The specific implementation method for calculating the operating speed and real-time differential speed of the handrail belt and pedals is as follows: First, stereo correction and depth calculation are performed on the binocular video image stream to obtain a dense or semi-dense depth map corresponding to each frame. For the real-time running speed of the pedal, existing yellow warning lines or pedal edge lines on the pedal surface are used as feature anchor points. An image processing algorithm combining Canny edge detection and Hough transform is used to accurately extract the pixel coordinates of the anchor points from the image, and the 3D spatial coordinates of the anchor points are obtained using the depth map. Between two adjacent frames or several frames in between, the displacement distance of the same anchor point in Euclidean space is calculated. Divide by the elapsed time To obtain the real-time running speed of the pedal To reduce measurement noise, a moving average filter is applied to the velocity values ​​at multiple anchor points or across multiple consecutive frames.

[0027] To address the real-time operating speed of handrails, which are typically made of black or dark-colored homogeneous rubber with a lack of distinct corners, traditional feature point tracking methods are prone to target loss. This embodiment introduces a three-dimensional spatial mapping strategy combining optical flow and depth information: first, subtle material textures, molded markings, or deliberately applied laser speckle are selected or identified on the handrail surface; then, the pyramid LK optical flow algorithm is used to calculate the two-dimensional pixel motion vectors of these texture features in consecutive video image frames. By combining the depth information generated synchronously by the binocular cameras, the start and end points of the two-dimensional motion vectors are mapped to the three-dimensional world coordinate system, and the actual physical displacement is calculated. Remove the physical bit at intervals. This gives the real-time operating speed of the handrail. Then, the real-time speed difference between the two can be calculated. This differential value directly reflects the synchronicity between the handrail belt and the pedal, and is one of the core characteristics for judging faults such as slippage and drive failure.

[0028] Furthermore, the process of identifying key skeletal features of elevator passengers to determine their elevator-riding behavior includes: Extract key spatial positions of the head, hands, and legs of elevator passengers; Based on the relative spatial topological relationship between the key points and the dynamic displacement trajectory over time, the elevator posture of passengers is evaluated, and the elevator behavior state is divided into standing state, walking state and tilting state.

[0029] Furthermore, the process of extracting real-time current features and vibration spectrum features from motor current signals and vibration signals includes: Extract the effective value, average value, peak factor, and short-time volatility characteristics of the motor current signal; The spectral distribution of the vibration signal is extracted by fast Fourier transform to obtain the energy proportion, peak frequency and its amplitude, and kurtosis characteristics of the preset frequency band.

[0030] Specifically, the implementation process of this embodiment includes: In S200, the identification of elevator riding behavior status utilizes a human skeletal keypoint detection algorithm. In each frame or keyframe of the binocular video image stream, a pre-trained deep neural network model extracts the human skeletal keypoints of the passenger, including at least the spatial position information of the head, neck, shoulders, elbows, wrists, hips, knees, and ankles. Using the binocular depth information, the two-dimensional keypoint coordinates are converted into three-dimensional spatial coordinates, obtaining the skeletal topology of each passenger. By analyzing the changes in the spatial coordinates of keypoints within consecutive frames, the elevator riding behavior status can be determined. A typical determination logic is as follows: If the horizontal displacement velocity of the occupant's hip center point remains below the first set threshold for a period of time, and the relative position changes of each joint are minimal, the occupant is judged to be in a "standing" state. If the horizontal displacement velocity of the occupant's hip exceeds the above threshold, the key points of both legs exhibit alternating swinging characteristics, and the vertical fluctuation is within the normal gait range, the occupant is judged to be in a "walking" state. If the occupant's torso tilt angle exceeds the safety limit, or the height of the head / hip key points drops sharply in a short period of time, accompanied by a rapid loss of horizontal displacement, the occupant is judged to be in a "falling" state.

[0031] When there are multiple occupants in the image, the behavior state with the highest degree of danger is used as the global state. That is, if even one person is in a tilted state, the global state is determined to be tilted; if multiple people are moving and no one is tilted, it is determined to be moving; the rest are standing.

[0032] Simultaneously, this step analyzes the collected motor current and vibration signals to extract corresponding real-time current and vibration spectrum features. For the motor current signal, features such as RMS value, average value, peak factor, and short-time volatility are extracted. For the vibration signal, the spectral distribution of each vibration sensor is extracted using Fast Fourier Transform, including the energy proportion of a specific frequency band, peak frequency and its amplitude, kurtosis, and other statistical features. These features will serve as inputs for the health assessment of mechanical components.

[0033] S300, Human-Machine State Coupling Determination. This step categorizes and confirms the elevator behavior states obtained in step S200, ultimately outputting three distinct posture labels: standing, walking, and leaning. These posture labels will serve as the "human-machine state" variable for the entire monitoring system to perceive dynamic changes in environmental load, and will be used throughout subsequent evaluation benchmark scheduling and cross-validation. The fundamental purpose of introducing this dimension is to enable the system to understand whether fluctuations in electromechanical signals originate from normal passenger activity or equipment malfunctions, thereby providing a basis for dynamically adjusting judgment thresholds and filtering strategies.

[0034] Furthermore, the process of dynamically invoking a preset evaluation benchmark model based on elevator riding behavior includes: When the elevator riding behavior status is standing, the basic load assessment benchmark is invoked. When the elevator riding behavior is in the moving state, the transient fluctuation threshold of the allowable differential speed threshold range in the evaluation benchmark model is relaxed, the peak value judgment benchmark in the current benchmark feature is adjusted upward, and the high-frequency vibration signal caused by the movement of people is shielded. When the elevator is in a tilting state, an emergency stop command and a rescue alarm command are triggered.

[0035] Specifically, the implementation process of this embodiment includes: S400, Dynamic Scheduling Evaluation Benchmark. Based on the escalator riding behavior status output in step S300, a preset evaluation benchmark model matching the current load condition is dynamically invoked. This evaluation benchmark model is a set of parameters containing multi-dimensional allowable ranges. This set is pre-set in the system's knowledge base and can be obtained through theoretical calculations, no-load / load calibration experiments, or learning from historical data. Specifically, after confirming that the escalator is in a normal and healthy state, different combinations of load conditions and riding behavior states are simulated sequentially, and differential speed, current, and vibration data are collected simultaneously. For each combination, confidence intervals or quantiles of the feature distribution are extracted to determine the boundaries of the allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range.

[0036] The evaluation benchmark model specifically adopts a multi-dimensional benchmark parameter table structure indexed by "elevator riding behavior status". Elevator riding behavior status includes standing, walking, and leaning (leaning status directly triggers emergency protection and is not involved in parameter matching). Each group of "behavior status" in the table stores the following three types of benchmark parameters: Allowable differential speed threshold range: This includes the allowable steady-state differential speed range (such as the upper and lower limits of the percentage difference in steady-state speed between the handrail and the pedal in a standing state), and the allowable transient fluctuation threshold (such as the maximum allowable instantaneous differential speed jump amplitude and duration). The transient fluctuation threshold can be dynamically relaxed according to the behavior state.

[0037] Current reference feature set: This includes the reference curve of the steady-state RMS value of the motor current, the upper limit envelope of the peak value during startup and constant speed, and the upper limit value of the odd harmonic components of the current under normal load. Under different load conditions, this reference curve and the upper limit of the peak value will be scaled according to the calibration factor.

[0038] Vibration spectrum tolerance range: According to the installation position of the vibration sensor, it is divided into three independent tolerance bands: (1) Main drive sprocket bearing seat tolerance band (corresponding to the first vibration sensor 21, 22): used to sense the impact and vibration of the drive chain running state, and set the upper limit of energy and kurtosis threshold of chain meshing frequency and its harmonic frequency; (2) Handrail friction wheel bracket tolerance band (corresponding to the second vibration sensor 4): used to monitor the friction and slip characteristics of the handrail drive system, and set the upper limit of energy ratio of friction frequency band and the threshold of intermittent impact amplitude; (3) Step guide rail reversing area tolerance band (corresponding to the third vibration sensor 31, 32): used to capture the vibration impact when the step enters or leaves the reversing area, and set the peak value and energy tolerance of the characteristic frequency of the reversing area. Each tolerance band is clearly marked as the suppressable frequency band of ground coupling step frequency caused by personnel walking. In the walking state, the vibration component of this frequency band of the corresponding sensor channel is weighted and suppressed or band-stop filtered, and only the continuous fault characteristic frequency band is included in the cross-validation.

[0039] The specific dynamic scheduling logic is as follows: When the escalator riding behavior status is determined to be "standing," it indicates that the passengers on the escalator are relatively stationary and the load is stable. At this time, the basic load assessment benchmark is invoked. This benchmark sets a relatively strict allowable differential speed threshold range, the current RMS value benchmark revolves around the normal operation curve, and the vibration spectrum tolerance range is referenced to the background mechanical vibration frequency band energy.

[0040] When the elevator riding behavior status is determined to be "moving," the current condition is considered to be a high dynamic load condition. To reduce false alarms caused by normal physical feedback from passenger movement, the system adaptively makes the following adjustments: the allowable threshold for transient fluctuations in real-time differential speed is relaxed; for example, no alarm is triggered when the duration of a short-term over-limit is within 0.5 seconds and the amplitude does not exceed 150% of the basic threshold; the peak value judgment benchmark for the motor current signal is increased, raising the instantaneous allowable peak value to the corresponding adjustment coefficient of the nominal value; at the same time, specific vibration signals caused by personnel movement are automatically shielded, for example, by suppressing ground-coupled step frequency vibrations of about 1-3Hz through band-stop filtering, or by weighting and suppressing the intermittent impact characteristics transmitted by passengers walking through the handrail friction wheel support, focusing only on observing the continuous fault characteristic frequency band.

[0041] When the elevator riding behavior status is determined to be "tipping", it indicates that a dangerous elevator riding event has occurred. At this time, regardless of whether other multimodal monitoring data exceed the normal fault threshold, the highest priority protection response is immediately triggered, and the emergency shutdown intervention process is directly entered without waiting for the subsequent multimodal cross-validation process.

[0042] Furthermore, the process of obtaining the real-time health status of escalators includes: When the real-time differential speed continuously leads or lags, the motor current signal slowly deviates from the steady-state current reference, and the vibration signal is within the low-frequency tolerance range, it is determined that there is a foreign object blocking the escalator or an abnormality in the transmission system. When the real-time differential speed experiences a momentary jump, the motor current signal shows an abnormal peak, and the vibration signal at the main drive sprocket bearing housing shows abnormal excitation energy exceeding the tolerance range, it is determined that the escalator drive chain is at risk of jamming or breakage. When the speed difference between the handrail and the tread is within the allowable range, and an abnormal high-frequency vibration signal appears in the guide rail reversing zone, it is determined that the tread in the escalator reversing zone is loose or the clip is broken.

[0043] Specifically, the implementation process of this embodiment includes: S500, Data Cascading Fusion and Cross-validation. Within the evaluation benchmark model framework dynamically scheduled in step S400, the real-time differential speed, real-time current characteristics, and vibration spectrum characteristics extracted in step S200 are synchronously input for multimodal data feature cascading and cross-validation. This comprehensive calculation determines the real-time health status of the escalator. This cross-validation process is implemented through a rule-based inference engine or fuzzy logic decision tree, correlating evidence from different sensor dimensions to distinguish between normal behavioral interference and genuine mechanical faults. The diagnostic logic for several typical operating conditions is listed below: Condition 1: Obstruction by foreign objects or chronic failure of the drive / transmission system. When the real-time differential speed exhibits a continuous positive or negative deviation, i.e., the handrail belt continuously leads or lags behind the steps; simultaneously, the drive motor current signal shows a slow deviation from the steady-state current reference, while the frequency domain characteristics of all vibration sensors remain within the corresponding low-frequency tolerance range, and no sudden impact energy has occurred. This combination of multi-dimensional characteristics points to the possibility of foreign object obstruction in the escalator guide rail, increased running resistance due to wear of the step wheel bearings, or insufficient frictional torque in the handrail belt drive.

[0044] Operating Condition 2: Risk of main drive chain jamming or breakage. When a sudden, large jump occurs in the real-time differential speed, a spike-like abnormal peak is detected in the motor current signal, and the signal collected by the first vibration sensor installed in the main drive sprocket bearing housing shows abnormal wideband excitation energy exceeding the tolerance range, especially with a significant energy increase at the chain passing frequency and its harmonics. This multi-dimensional transient concurrent characteristic clearly points to an immediate risk of tooth skipping, jamming, or even breakage of the transmission chain.

[0045] Condition 3: Loose or broken steps in the reversing zone. When the speed difference between the handrail and the pedals remains within a reasonable range, and the current signal shows no obvious abnormalities, only the third vibration sensor installed in the reversing zone of the step guide rail exhibits continuous or abnormal high-frequency impact vibrations, and this vibration is periodic in the time domain, consistent with the step's passing rhythm. This characteristic can be identified as a localized fault in the reversing zone of the step guide rail, such as peeling of step rollers, broken pedal clips, or elongated and loosened step chains.

[0046] Health status can be calculated using a deduction model, with a base score of 100. Each diagnostic rule deducts points based on the severity and duration of the violation, resulting in a real-time health index ranging from 0 to 100. When the health status falls below a preset threshold, a corresponding warning or alarm level is triggered.

[0047] Furthermore, the process of generating and outputting maintenance intervention strategies based on real-time health status includes: When all multimodal real-time monitoring data are within the allowable range of the evaluation benchmark model, a normal status record is generated; When a single dimension in the multimodal real-time monitoring data shows a slow deterioration trend or multiple dimensions show slight and occasional exceedances, a maintenance recommendation work order is generated for the abnormal component. When the multimodal real-time monitoring data shows multiple dimensions crossing limits, a linkage control signal is output to the escalator control cabinet to execute a shutdown operation.

[0048] Specifically, the implementation process of this embodiment includes: S600, Output Maintenance Intervention Strategy. Based on the real-time health status and fault determination results obtained in step S500, generate and output the corresponding equipment operation rating and specific maintenance intervention strategy. The maintenance intervention strategy is specifically divided into three levels: Level 1, Normal Status: All multimodal real-time monitoring data are within the allowable range of the evaluation benchmark model, and the health index remains at a high level. Only the feature data and timestamp of this monitoring period are stored in the operation log database, without triggering any alarms.

[0049] Level 2, Planned Maintenance Early Warning: A single dimension in the multimodal data shows a slow deterioration trend, or multiple dimensions exhibit minor, intermittent exceedances that can recover spontaneously, or early potential hazards are detected through cross-validation in step S500 but have not yet affected safe operation. In this case, a routine maintenance recommendation work order for the specific component is automatically generated, such as "It is recommended to check the handrail belt tension and friction wheel condition this month" or "It is recommended to inspect the guide rail reversing area step rollers," and pushed to the maintenance management system.

[0050] Level 3 Emergency Shutdown Intervention: When multimodal data shows severe multi-dimensional cross-limit violations, such as when the risk assessment for working conditions 2 or 3 in step S500 or other immediate mechanical faults is established, and the health status drops sharply below the danger threshold, immediately output a linkage control signal to the escalator control cabinet via dry contact signal or fieldbus to execute a smooth deceleration and shutdown operation, allowing the escalator to safely stop operation without human intervention. Simultaneously trigger the audible and visual alarm to issue a distress alarm, and push the fault location information and real-time monitoring screen to the monitoring center's large screen.

[0051] By using the aforementioned technical means of deeply integrating visual analysis of passenger behavior with electromechanical signals and dynamic scheduling judgment criteria, this embodiment can effectively filter out interference such as normal movement of people, and accurately identify hidden and dangerous faults such as foreign object blockage, chain failure, and step detachment. This enables an upgrade from post-fault emergency repair to intelligent pre-fault early warning and proactive protection maintenance mode, significantly improving the operational safety and maintenance efficiency of escalators.

[0052] Example 2 Based on the same general inventive concept, this invention also provides an escalator health status monitoring system that integrates behavioral vision and electromechanical signals. The escalator health status monitoring system integrating behavioral vision and electromechanical signals provided by this invention is described below. The escalator health status monitoring system integrating behavioral vision and electromechanical signals described below can be referred to in correspondence with the escalator health status monitoring method integrating behavioral vision and electromechanical signals described above. The system includes: The multimodal data acquisition module is used to synchronously acquire binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation. The vision processing and state recognition module is used to calculate the running speed of the pedal and the running speed of the handrail from the binocular video image stream to obtain the real-time differential speed, and to identify the skeletal key points of the passengers to determine the riding behavior state. At the same time, it extracts real-time current features and vibration spectrum features from the motor current signal and vibration signal. The dynamic benchmark scheduling module is used to dynamically call a preset evaluation benchmark model according to the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The multimodal fusion evaluation module is used to input the real-time differential speed, real-time current characteristics and vibration spectrum characteristics into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. The maintenance strategy output module is used to generate and output maintenance intervention strategies based on the real-time health status.

[0053] This invention provides a method and system for monitoring the health status of escalators by integrating behavioral vision and electromechanical signals. It treats the standing, walking, and leaning behaviors of escalators as environmental variables, and intelligently distinguishes between normal human activity and actual mechanical faults through a dynamic threshold adjustment mechanism. When people are walking, the system automatically relaxes the judgment threshold and shields high-frequency vibration interference from normal physical feedback, significantly reducing the system's false alarm rate. When people lean, it ignores conventional thresholds and directly triggers the highest level of protection response, significantly improving the equipment's safety sensitivity and emergency response speed. Through a non-contact binocular vision and optical flow three-dimensional mapping differential speed measurement scheme, it overcomes the problems of easy wear and high maintenance costs associated with traditional contact-based speed measurement, while also solving the technical challenge of easily losing targets when visually tracking the homogeneous surface of the handrail. Deep cross-validation of multimodal data can accurately distinguish complex and hidden faults such as foreign object blockage, transmission chain jamming and breakage, and loosening of the reversing zone pedals, achieving high-precision monitoring and assessment of escalator mechanical faults. This promotes the upgrade of maintenance from reactive repair to proactive early warning, effectively saving maintenance costs and optimizing resource allocation efficiency.

[0054] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for monitoring the health status of escalators by integrating behavioral vision and electromechanical signals, characterized in that, Includes the following steps: Simultaneously collect binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation; The speed of the pedal and the speed of the handrail are calculated from the binocular video image stream to obtain the real-time differential speed. The key skeletal points of the passengers are identified to determine the passenger behavior status. At the same time, real-time current features and vibration spectrum features are extracted from the motor current signal and vibration signal. The preset evaluation benchmark model is dynamically invoked based on the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The real-time differential speed, real-time current characteristics, and vibration spectrum characteristics are input into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. Maintenance intervention strategies are generated and output based on the real-time health status.

2. The method according to claim 1, characterized in that, The process of synchronously collecting data includes: A binocular wide-angle camera positioned directly above the escalator is used to acquire a binocular video image stream containing depth information from a parallel overhead angle. The current signal of the drive motor is acquired by a non-invasive current sensor at a sampling frequency no less than the frame rate of binocular video. The vibration signal is acquired by a vibration sensor, and the sampling frequency of the vibration sensor is set to above 2kHz.

3. The method according to claim 1, characterized in that, The process of calculating the pedal speed and the handrail speed includes: The edge warning line of the pedal is identified as a feature anchor point. The Euclidean spatial displacement distance of the feature anchor point within adjacent frames is calculated. The ratio of the Euclidean spatial displacement distance to the inter-frame time difference is used as the pedal running speed. Microscopic textures on the surface of the handrail are extracted, and optical flow is used to calculate the two-dimensional motion vectors of the textures in consecutive image frames. The two-dimensional motion vectors are then mapped to three-dimensional physical space using depth information to calculate the actual physical displacement. The ratio of the actual physical displacement to the time difference is used as the running speed of the handrail.

4. The method according to claim 1, characterized in that, The process of identifying key skeletal features of elevator passengers to determine their elevator-riding behavior includes: Extract key spatial positions of the head, hands, and legs of elevator passengers; Based on the relative spatial topological relationship between the key points and the dynamic displacement trajectory over time, the elevator posture of passengers is evaluated, and the elevator behavior state is divided into standing state, walking state and tilting state.

5. The method according to claim 1, characterized in that, The process of extracting real-time current features and vibration spectrum features from motor current signals and vibration signals includes: Extract the effective value, average value, peak factor, and short-time volatility characteristics of the motor current signal; The spectral distribution of the vibration signal is extracted by fast Fourier transform to obtain the energy proportion, peak frequency and its amplitude, and kurtosis characteristics of the preset frequency band.

6. The method according to claim 1, characterized in that, The process of dynamically invoking a preset evaluation benchmark model based on elevator riding behavior includes: When the elevator riding behavior status is standing, the basic load assessment benchmark is invoked. When the elevator riding behavior is in the moving state, the transient fluctuation threshold of the allowable differential speed threshold range in the evaluation benchmark model is relaxed, the peak value judgment benchmark in the current benchmark feature is adjusted upward, and the high-frequency vibration signal caused by the movement of people is shielded. When the elevator is in a tilting state, an emergency stop command and a rescue alarm command are triggered.

7. The method according to claim 1, characterized in that, The process of obtaining the real-time health status of escalators includes: When the real-time differential speed continuously leads or lags, the motor current signal slowly deviates from the steady-state current reference, and the vibration signal is within the low-frequency tolerance range, it is determined that there is a foreign object blocking the escalator or an abnormality in the transmission system. When the real-time differential speed experiences a momentary jump, the motor current signal shows an abnormal peak, and the vibration signal at the main drive sprocket bearing housing shows abnormal excitation energy exceeding the tolerance range, it is determined that the escalator drive chain is at risk of jamming or breakage. When the speed difference between the handrail and the tread is within the allowable range, and an abnormal high-frequency vibration signal appears in the guide rail reversing zone, it is determined that the tread in the escalator reversing zone is loose or the clip is broken.

8. The method according to claim 1, characterized in that, The process of generating and outputting maintenance intervention strategies based on real-time health status includes: When all multimodal real-time monitoring data are within the allowable range of the evaluation benchmark model, a normal status record is generated; When a single dimension in the multimodal real-time monitoring data shows a slow deterioration trend or multiple dimensions show slight and occasional exceedances, a maintenance recommendation work order is generated for the abnormal component. When the multimodal real-time monitoring data shows multiple dimensions crossing limits, a linkage control signal is output to the escalator control cabinet to execute a shutdown operation.

9. An escalator health status monitoring system integrating behavioral vision and electromechanical signals, characterized in that, The system for implementing the method of any one of claims 1-8 comprises: The multimodal data acquisition module is used to synchronously acquire binocular video image streams, drive motor current signals, and vibration signals at the main drive sprocket bearing housing, handrail friction wheel bracket, and step guide rail reversing zone during escalator operation. The vision processing and state recognition module is used to calculate the running speed of the pedal and the running speed of the handrail from the binocular video image stream to obtain the real-time differential speed, and to identify the skeletal key points of the passengers to determine the riding behavior state. At the same time, it extracts real-time current features and vibration spectrum features from the motor current signal and vibration signal. The dynamic benchmark scheduling module is used to dynamically call a preset evaluation benchmark model according to the elevator riding behavior state. The evaluation benchmark model has an allowable differential speed threshold range, current benchmark characteristics, and vibration spectrum tolerance range that match the current load conditions. The multimodal fusion evaluation module is used to input the real-time differential speed, real-time current characteristics and vibration spectrum characteristics into the evaluation benchmark model to perform multimodal data feature cascading and cross-validation to obtain the real-time health status of the escalator. The maintenance strategy output module is used to generate and output maintenance intervention strategies based on the real-time health status.

Images