Escalator intelligent perception and safety early warning method and system based on multi-sensor fusion

Abstract

The present application relates to the technical field of sensing detection, in particular to a kind of escalator intelligent perception and safety early warning method and system of multi-sensor fusion;System through perception layer, fusion decision layer and intervention execution layer, has built the cooperation perception structure of radar main detection, visual auxiliary verification, environment self-adaption, hierarchical intervention;The present application adopts radar active detection and visual passive wake-up linkage mechanism, normal only radar low-power consumption work, wake up vision after risk trigger, give consideration to low power consumption and behavior recognition accuracy;Before fusion, realize sensor confidence dynamic evaluation through environment perception, adaptively adjust radar and visual fusion weight, solve the problem of invalid fusion under complex environment;And through the three progressive hierarchical intervention strategies of prompt level, warning level, protection level, replace traditional binary shutdown mode, under the premise of guaranteeing safety, maximum limit escalator operation efficiency, reduce false alarm shutdown probability, significantly optimize passenger boarding experience.

Description

Technical Field

[0001] This invention relates to the field of sensing and detection technology, and more specifically, to an intelligent sensing and safety early warning system based on the fusion of millimeter-wave radar and visual sensors, applicable to the monitoring of human behavior in public transportation facilities such as escalators. Background Technology

[0002] Sensor detection technology is the core foundation for intelligent equipment monitoring and safety early warning, and it is widely used in rail transit, special equipment, and public facilities. Escalators, as key special equipment in densely populated areas, are directly related to the safety of public life and property. Statistics show that over 70% of escalator accidents are related to abnormal passenger behavior, such as falls, going against the flow of traffic, and leaning against the skirt panel. Therefore, real-time sensing and accurate early warning of escalator behavior are crucial for improving escalator safety.

[0003] However, the unique characteristics of escalator scenarios pose serious challenges to behavior perception technology: First, environmental conditions are extremely variable, ranging from indoors to semi-outdoors, with alternating periods of bright daylight, darkness at night, and rainy / foggy weather; second, behavioral characteristics are complex and ambiguous, with normal behaviors (such as bending over to tie shoelaces) and dangerous behaviors (such as falling forward) being visually highly similar and easily confused; and third, false alarms are costly, as frequent escalator shutdowns not only affect operational efficiency but may even trigger passenger panic.

[0004] In the prior art, Chinese patent application CN117533911A discloses a "multi-sensor fusion method, system, and electronic device for elevator passenger hazard identification." This scheme collects data through sensors (accelerometer, torque, smoke, distance) and cameras. After data preprocessing, it uses the R3Det algorithm for target detection, and then sends the detection results and sensor data into an LSTM to identify dangerous behaviors, finally triggering warning alarms and other measures. Although this scheme achieves multi-sensor fusion, it still has the following shortcomings: First, all sensors operate continuously, resulting in high power consumption and heavy data processing pressure; second, it adopts decision-level fusion, which has a shallow fusion level and does not fully utilize the complementary advantages of multimodal data; third, it does not consider the impact of environmental changes on sensor performance, resulting in poor fusion performance in complex environments; fourth, the intervention method is singular, only outputting a binary "risky / no risk" signal, which triggers an alarm upon false alarm, easily leading to unnecessary elevator shutdowns.

[0005] Another Chinese patent application, CN117422971A, discloses a "Dual-modal target detection method and system based on cross-modal attention mechanism fusion." This scheme uses point cloud Transformer and CSPDarkNet to extract radar features and image features respectively, and performs feature fusion through a cross-modal attention module. This scheme is innovative in feature-level fusion, but it also suffers from problems such as continuous operation and high power consumption. Furthermore, it does not design a graded intervention mechanism for escalator scenarios, making it difficult to directly apply to escalator safety monitoring.

[0006] Furthermore, FAW Jiefang's patent application CN121384056A discloses a "method and system for autonomous driving environmental perception based on multispectral fusion." This method constructs a sensor confidence estimation network to output the real-time confidence scores of each sensor, which are then converted into dynamic weight vectors for weighted fusion. While this scheme achieves dynamic weight allocation based on confidence, the confidence score originates from the sensor's own feature data rather than external environmental parameters, and its application scenario is autonomous driving, fundamentally different from escalator behavior perception.

[0007] In summary, existing technical solutions still suffer from core problems in escalator scenarios, such as low detection accuracy, poor environmental adaptability, high power consumption, and limited intervention modes, making it difficult to meet practical application needs. Summary of the Invention

[0008] The technical problem to be solved by this invention is to provide a method and system for intelligent sensing and safety early warning of escalators based on multi-sensor fusion. With sensing and detection technology as the core, combined with millimeter-wave radar and visual heterogeneous sensing, it solves the long-standing technical problems in the field that are mutually restrictive: all-weather monitoring, low power consumption operation, high-precision identification, and low false alarm intervention.

[0009] The technical solution adopted by this invention to solve its technical problem is: a multi-sensor fusion intelligent sensing and safety early warning method for escalators, comprising the following steps:

[0010] S1: The escalator area is monitored in real time by a millimeter-wave radar sensor to generate radar point cloud data and detect the presence and movement of targets; the vision sensor is in a low-power deep standby mode and does not collect images.

[0011] S2: When the millimeter-wave radar sensor detects that a target has entered the preset warning area or that the target's movement trajectory is abnormal, a wake-up signal is generated, which triggers the vision sensor to switch from deep standby mode to working mode and acquire images of the local area.

[0012] S3: Perform time synchronization and spatial calibration on radar and visual data to acquire spatiotemporally aligned multimodal data;

[0013] S4: Real-time collection of environmental parameters through the environmental sensing unit, dynamic evaluation of the confidence level of each sensor based on the environmental parameters, and generation of dynamic fusion weights;

[0014] S5: Extract the spatial features of radar point cloud data and the semantic features of visual image data, perform multimodal feature fusion based on the dynamic fusion weight, and identify the elevator riding behavior type and determine the risk level through a deep neural network;

[0015] S6: Implement differentiated intervention strategies based on the risk level.

[0016] Furthermore, the dynamic evaluation of the confidence level of each sensor in step S4 of the present invention includes: calculating the confidence level of the visual sensor based on at least one of light intensity, ambient humidity, and vibration amplitude; calculating the confidence level of the radar sensor based on at least one of electromagnetic interference intensity and target echo signal-to-noise ratio; and normalizing to obtain dynamic fusion weights.

[0017] Furthermore, in step S3 of this invention, the time synchronization is achieved at the microsecond level using the PTP precise time protocol, and the spatial calibration is achieved by obtaining the transformation matrix between the radar coordinate system and the image coordinate system through joint calibration, thereby realizing the projection mapping from radar point cloud to image pixel coordinates.

[0018] Furthermore, the deep neural network described in step S5 of this invention includes:

[0019] The radar feature extraction branch is used to extract target spatial location, velocity, radar cross section, and time-series trajectory features from radar point cloud data.

[0020] The visual feature extraction branch is used to extract target bounding boxes, human key points, pose estimation, and behavioral temporal features from image data.

[0021] The feature alignment layer is used to align radar features with visual features in the feature space according to spatial calibration parameters;

[0022] The attention fusion layer employs an attention mechanism to weightedly fuse features from different modalities based on the dynamic fusion weights.

[0023] The classification output layer is used to output the probability of behavior categories and map them to risk levels.

[0024] Furthermore, the risk levels described in this invention include a warning level, an alert level, and a protection level; the differentiated intervention strategy in step S6 includes:

[0025] When the risk level is warning level, the voice broadcast unit is triggered to play a general safety prompt, and the information display unit displays the direction of travel, without interfering with the escalator operation;

[0026] When the risk level is warning level, the voice broadcast unit is triggered to play a targeted warning message, the information display unit flashes the danger sign, and a deceleration command is sent through the escalator control interface;

[0027] When the risk level is protection level, the voice broadcast unit is triggered to play an emergency stop prompt, an emergency stop command is sent through the escalator control interface, and on-site images and risk information are uploaded to the monitoring center through the remote alarm unit.

[0028] Furthermore, the present invention also includes the following steps:

[0029] S7: Record historical risk events and intervention results, including sensor data, fusion decision results, and manual confirmation tags; perform incremental training on the fusion decision model and optimize model parameters based on false alarms and missed alarms reported by humans; and automatically adjust the warning area division and risk threshold setting based on actual passenger flow data.

[0030] Meanwhile, the present invention also provides a multi-sensor fusion intelligent sensing and safety early warning system for escalators using the above-mentioned method, comprising:

[0031] The perception layer includes at least one millimeter-wave radar sensor, at least one visual sensor, and an environmental perception unit.

[0032] The fusion decision layer includes a data preprocessing module, a confidence assessment module, and a heterogeneous fusion module, which are used to execute steps S3 to S5 of the method.

[0033] The intervention execution layer includes a voice broadcast unit, an information display unit, and an escalator control interface, which are used to execute step S6 of the method.

[0034] Furthermore, the confidence assessment module of the present invention is configured to calculate the dynamic fusion weight of each sensor according to a multi-factor confidence function. The environmental parameters on which the multi-factor confidence function is based include at least one of light intensity, ambient humidity, vibration amplitude, electromagnetic interference intensity, and target echo signal-to-noise ratio.

[0035] Furthermore, the data preprocessing module of the present invention includes:

[0036] The time synchronization unit is used to achieve microsecond-level time synchronization between the radar sensor and the vision sensor;

[0037] The spatial calibration unit is used to obtain the transformation matrix between the radar coordinate system and the image coordinate system;

[0038] The dynamic calibration unit is used to periodically check the calibration accuracy and trigger recalibration when the accuracy decreases.

[0039] Furthermore, the heterogeneous fusion module of the present invention includes: a radar feature extraction branch, a visual feature extraction branch, a feature alignment layer, an attention fusion layer, and a classification output layer.

[0040] The beneficial effect of this invention is that it solves the defects existing in the prior art.

[0041] 1. A radar-triggered vision linkage mechanism is adopted. Normally, only the radar is working, while the vision sensor is in a low-power standby mode. The vision sensor is only awakened when the radar detects an anomaly. This achieves real-time monitoring around the clock and avoids the high power consumption caused by continuous operation of the vision sensor. This is a significant improvement in terms of technical performance.

[0042] 2. Through a dynamic confidence evaluation mechanism, the system will adaptively adjust the sensor fusion weights according to environmental conditions, such as light, humidity, vibration, electromagnetic interference and other environmental parameters. Under low light conditions, the visual weights will be reduced and the radar weights will be increased, so that the system can maintain high reliability in complex environments (such as night, backlight, rain, fog and strong electromagnetic interference), avoiding the problem of blind fusion and random fusion in complex environments of traditional fusion algorithms.

[0043] 3. A three-level early warning and intervention model was established, abandoning the traditional binary response mode. Instead of simply alarming or stopping, different measures were taken according to the risk level. A three-level intervention model of prompting, warning and protection was established, and each level corresponds to a different combination of sound, light and control actions. This minimizes the emergency stop of escalators caused by minor risks (such as passengers bending over to tie their shoelaces) or minor false alarms. Under the premise of ensuring absolute safety, the operating efficiency of escalators and passenger experience are significantly improved.

[0044] 4. The visual sensor only acquires local images briefly after the radar is triggered, without uploading the original video stream, effectively reducing the risk of privacy leakage; and it can continuously optimize the model by recording historical events and human feedback, making the system smarter the more it is used and adapting to the differentiated needs of different scenarios. Attached Figure Description

[0045] Figure 1 This is a flowchart of the overall method of the present invention;

[0046] Figure 2 This is a schematic diagram of the hierarchical intervention logic of the present invention. Detailed Implementation

[0047] The present invention will now be described in further detail with reference to the accompanying drawings and preferred embodiments. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0048] Example 1:

[0049] This embodiment provides a multi-sensor fusion-based intelligent sensing and safety early warning method for escalators, applied to an escalator at a rail transit station. For example... Figure 1 As shown, the method includes the following steps:

[0050] S1: Routine Monitoring

[0051] A 77GHz millimeter-wave radar deployed at the upper and lower entrances of the escalator and in the central turning area operates continuously, generating point cloud data. Radar parameters are set as follows: detection range 0.5-30m, distance accuracy 0.1m, speed accuracy 0.1m / s, and frame rate 20fps. The vision sensor (low-power CMOS camera, 1920×1080 resolution) is in standby mode with a power consumption of 0.3W.

[0052] S2: Visual Arousal

[0053] The radar will activate the visual sensor if it detects any of the following events:

[0054] The target enters the entrance warning area (within 3m of the comb plate) or the exit warning area (within 1.5m of the comb plate).

[0055] The target's movement speed is abnormal (>1.2 times the normal operating speed or <0.1 m / s and the duration is >3 s);

[0056] The target is moving in the opposite direction and its displacement is >0.5m;

[0057] If the number of targets detected in a single frame is greater than 15, it is considered a peak passenger flow period.

[0058] The visual sensor wake-up time is ≤80ms, and it can acquire 3 consecutive frames of images at a frame rate of 30fps.

[0059] S3: Data Preprocessing

[0060] Time synchronization: The PTP (IEEE 1588) precise time protocol is used to achieve microsecond-level synchronization between the radar and the vision sensor, with a synchronization accuracy of ≤80μs. Each frame of radar data and image data carries a timestamp. The preprocessing module aligns the data according to the timestamp and selects the radar frame closest to the image acquisition time (time difference <10ms) as the matching frame.

[0061] Spatial calibration: The projection matrix H (3×4) from the radar coordinate system to the image coordinate system is obtained through joint calibration. Calibration process: Six corner reflectors are arranged in the escalator area to collect no less than 12 sets of corresponding point pairs. The Direct Linear Transform (DLT) algorithm is used to solve for H, and the reprojection error is ≤3 pixels.

[0062] Dynamic calibration: Calibration verification is automatically triggered every 500 hours of operation or when the cumulative number of impacts detected by the vibration sensor exceeds 1000. If the reprojection error exceeds 5 pixels, maintenance is prompted.

[0063] S4: Environmental Perception and Confidence Assessment

[0064] The environmental sensing unit includes a light sensor, a temperature and humidity sensor, and a vibration sensor, which collect data in real time. The confidence assessment module uses a multi-factor function for calculation.

[0065] Visual sensor confidence level: Cv=f(L,H,V), where L is the light intensity, H is the ambient humidity, and V is the vibration amplitude;

[0066] Radar sensor confidence level: Cr=g(E,T), where E is the electromagnetic interference intensity and T is the target echo signal-to-noise ratio;

[0067] Normalization yields the dynamic fusion weights: Wv=Cv / (Cv+Cr), Wr=Cr / (Cv+Cr).

[0068] When the confidence level of the visual sensor is lower than a preset lower limit (e.g., 0.3), it is marked as "low confidence" and the fusion weight Wv is set to 0, relying solely on radar decision-making; when the confidence level of the radar sensor is lower than 0.3, it is marked as "low confidence" and the fusion weight Wr is set to 0, relying solely on visual decision-making; if both are lower than the threshold, a sensor fault alarm is triggered.

[0069] S5: Feature Extraction and Fusion Decision

[0070] Radar feature extraction branch: DBSCAN clustering is performed on each frame of radar point cloud (ε=0.5m, MinPts=3), and a feature vector F_r∈R^64 is extracted for each target, including the target center coordinates (x,y,z), radial velocity v_r, RCS value, point cloud distribution range, existence duration, and trajectory sequence of the past 5 frames. A 3-layer fully connected network (64→128→64) is used to extract deep features.

[0071] Visual feature extraction branch: The improved YOLOv8 network is used to detect the target bounding box and 17 human body key points. Behavioral features such as human body center height, torso tilt angle, and hip joint angle are calculated to obtain the feature vector F_v∈R^256. The dimensionality is reduced to 64 dimensions through a 2-layer fully connected network (256→128→64).

[0072] Feature alignment layer: Each radar target is projected onto the image plane according to the projection matrix H, and local features centered on the projection point are extracted from the image feature map and associated with the corresponding radar features.

[0073] Attention Fusion Layer: The radar features and visual features are concatenated into a 128-dimensional vector. SENet-style channel attention is introduced, and dynamic fusion weights Wv and Wr are used as attention biases to weight and fuse the features.

[0074] Temporal modeling layer: LSTM (128-dimensional hidden layer) is used to perform temporal modeling on the fused features of 5 consecutive frames, and the hidden state at the last moment is output.

[0075] Classification output layer: The fully connected layer outputs the probabilities of 6 types of behaviors (normal, falling, going against the flow, leaning, congestion, and others), and is activated by Softmax.

[0076] The fusion network outputs the behavior class probability P_class ∈ R^6. Based on the preset risk mapping table, the base risk value Risk_base[i] is obtained. The final risk value R is calculated as follows:

[0077]

[0078] Risk level determination: If R < 0.3: no risk; if 0.3 ≤ R < 0.6: warning level risk; if 0.6 ≤ R < 0.85: alert level risk; if R ≥ 0.85: protective level risk.

[0079] S6: Tiered Intervention

[0080] Implement according to risk level Figure 2 The intervention strategy shown is as follows:

[0081] Prompt Level: The voice announces "Please hold the handrail tightly and watch your step," at a volume of 65dB, repeated twice; the display shows a green direction arrow and scrolling prompts; the escalator is operating normally.

[0082] Warning level: Targeted voice warning (e.g., "Do not lean against the skirt board"), volume 75dB, repeated 3 times; yellow arrow flashes on the display screen, showing relevant markings; the escalator control interface sends a deceleration command (decelerate to 0.3m / s, original speed 0.5m / s), continues for 5 seconds, and if there is no risk of escalation, the original speed is restored; an event notification is sent to the monitoring center.

[0083] Protection level: Emergency voice prompt "Emergency situation, escalator is about to stop, please hold the handrail tightly", volume 85dB; sends emergency stop command; uploads on-site images (3 seconds of video before and after triggering) to the monitoring center via 4G remote alarm unit; notifies on-site management personnel.

[0084] S7: Self-learning optimization

[0085] Record radar data, image data, fusion decision results, and manually confirmed labels for each risk event. Incrementally train the fusion network weekly using data from the most recent four weeks: freeze the first two layers of the feature extraction layer, train the last two layers and the classification layer, with a learning rate of 0.0001 and 5 epochs. If the new model reduces the false alarm rate by ≥5% on the test set, it passes safety verification and is then released for updates. Simultaneously, automatically adjust the warning area threshold based on passenger flow data, for example, dynamically adjust the "congestion" judgment threshold to ensure an average of ≤5 alarms per day.

[0086] Example 2:

[0087] This embodiment provides a preferred implementation of confidence assessment. The confidence assessment module uses a continuous function to calculate the confidence level of each sensor and incorporates historical false alarm statistics for dynamic correction.

[0088] Define the confidence level Cv of a vision sensor:

[0089]

[0090] Where Lopt=500 lux, Hopt=50%RH, Hrange=80%RH, Vmax=2.0 m / s², and α, β, and γ are adjustment coefficients, which are 0.6, 0.2, and 0.2, respectively.

[0091] Define the confidence level Cr of a radar sensor:

[0092]

[0093] Where Eopt = -90dBm, Emax = 30dB, Topt = 20dB, and δ and ε are 0.5 and 0.5 respectively.

[0094] The lower confidence threshold of the visual sensor is dynamically adjusted based on the false alarm rate (FAR) over the past 7 days.

[0095] Tv_min=0.3×(1+0.5×(FAR-0.05))

[0096] When Cv < T_v_min, the visual sensor fusion weight Wv is set to 0.

[0097] Example 3:

[0098] This embodiment illustrates the training process of the fusion network. The training dataset contains 100,000 frames of annotated escalator monitoring data, including behaviors such as normal movement, falls, wrong-way movement, leaning, and congestion. Each frame includes synchronized radar point cloud and image. Data augmentation includes: random image cropping, rotation, and brightness adjustment; random shaking of the radar point cloud and dropping some points. The loss function is cross-entropy loss, the optimizer is Adam, the initial learning rate is 0.001, the batch size is 32, and the training lasts for 50 epochs. After training, the accuracy on the validation set is 98.2%, the recall is 97.5%, and the false positive rate is 1.8%.

[0099] Example 4: Performance Test Data

[0100] After 30 days of actual deployment at a subway station, the test results are as follows:

[0101] index This method Pure visual pure radar Simple parallel Fall detection accuracy 98.7% 92.3% 78.5% 94.1% Accuracy of reverse detection 97.2% 94.5% 81.2% 95.3% False alarm rate (times / day) 1.2 4.7 8.3 3.5 Nighttime detection accuracy 96.3% 82.1% 95.8% 89.2% Daily average power consumption (Wh) 3.8 7.2 2.5 8.5

[0102] The above description is merely a specific embodiment of the present invention. Various examples and illustrations do not constitute a limitation on the substantive content of the present invention. Those skilled in the art can make modifications or variations to the above-described specific embodiments after reading the specification without departing from the essence and scope of the invention.

Claims

1. A multi-sensor fusion method for intelligent sensing and safety early warning of escalators, characterized in that, Includes the following steps: S1: The escalator area is monitored in real time by a millimeter-wave radar sensor to generate radar point cloud data and detect the presence and movement of targets; the vision sensor is in a low-power deep standby mode and does not collect images. S2: When the millimeter-wave radar sensor detects that a target has entered the preset warning area or that the target's movement trajectory is abnormal, a wake-up signal is generated, which triggers the vision sensor to switch from deep standby mode to working mode and acquire images of the local area. S3: Perform time synchronization and spatial calibration on radar and visual data to acquire spatiotemporally aligned multimodal data; S4: Real-time collection of environmental parameters through the environmental sensing unit, dynamic evaluation of the confidence level of each sensor based on the environmental parameters, and generation of dynamic fusion weights; S5: Extract the spatial features of radar point cloud data and the semantic features of visual image data, perform multimodal feature fusion based on the dynamic fusion weight, and identify the elevator riding behavior type and determine the risk level through a deep neural network; S6: Implement differentiated intervention strategies based on the risk level.

2. The multi-sensor fusion intelligent sensing and safety early warning method for escalators according to claim 1, characterized in that, The dynamic evaluation of the confidence of each sensor in step S4 includes: calculating the confidence of the visual sensor based on at least one of light intensity, ambient humidity, and vibration amplitude; calculating the confidence of the radar sensor based on at least one of electromagnetic interference intensity and target echo signal-to-noise ratio; and normalizing the results to obtain the dynamic fusion weights.

3. The multi-sensor fusion intelligent sensing and safety early warning method for escalators according to claim 1, characterized in that, The time synchronization in step S3 uses the PTP precise time protocol to achieve microsecond-level synchronization. The spatial calibration obtains the transformation matrix between the radar coordinate system and the image coordinate system through joint calibration, thereby realizing the projection mapping from radar point cloud to image pixel coordinates.

4. The multi-sensor fusion intelligent sensing and safety early warning method for escalators according to claim 1, characterized in that, The deep neural network mentioned in step S5 includes: The radar feature extraction branch is used to extract target spatial location, velocity, radar cross section, and time-series trajectory features from radar point cloud data. The visual feature extraction branch is used to extract target bounding boxes, human key points, pose estimation, and behavioral temporal features from image data. The feature alignment layer is used to align radar features with visual features in the feature space according to spatial calibration parameters; The attention fusion layer employs an attention mechanism to weightedly fuse features from different modalities based on the dynamic fusion weights. The classification output layer is used to output the probability of behavior categories and map them to risk levels.

5. The multi-sensor fusion intelligent sensing and safety early warning method for escalators according to claim 1, characterized in that, The risk levels include alert level, warning level, and protection level; the differentiated intervention strategies in step S6 include: When the risk level is warning level, the voice broadcast unit is triggered to play a general safety prompt, and the information display unit displays the direction of travel, without interfering with the escalator operation; When the risk level is warning level, the voice broadcast unit is triggered to play a targeted warning message, the information display unit flashes the danger sign, and a deceleration command is sent through the escalator control interface; When the risk level is protection level, the voice broadcast unit is triggered to play an emergency stop prompt, an emergency stop command is sent through the escalator control interface, and on-site images and risk information are uploaded to the monitoring center through the remote alarm unit.

6. The multi-sensor fusion intelligent sensing and safety early warning method for escalators according to claim 1, characterized in that, It also includes steps, S7: Record historical risk events and intervention results, including sensor data, fusion decision results, and manual confirmation tags; perform incremental training on the fusion decision model and optimize model parameters based on false alarms and missed alarms reported by humans; and automatically adjust the warning area division and risk threshold setting based on actual passenger flow data.

7. A multi-sensor fusion intelligent sensing and safety early warning system for escalators for implementing the method of any one of claims 1-6, characterized in that, include: The perception layer includes at least one millimeter-wave radar sensor, at least one visual sensor, and an environmental perception unit. The millimeter-wave radar sensor operates continuously, while the vision sensor is in deep standby mode by default and is triggered to wake up when the millimeter-wave radar detects an anomaly. The fusion decision layer includes a data preprocessing module, a confidence assessment module, and a heterogeneous fusion module, used to execute steps S3 to S5 of the method described in claim 1; wherein, the confidence assessment module dynamically calculates the fusion weights of each sensor based on environmental parameters; The intervention execution layer includes a voice broadcast unit, an information display unit, and an escalator control interface, which are used to execute step S6 of the method described in claim 1 to achieve a three-level progressive hierarchical intervention.

8. The multi-sensor fusion intelligent sensing and safety early warning system for escalators according to claim 7, characterized in that, The confidence assessment module calculates the dynamic fusion weight of each sensor based on a multi-factor confidence function. The environmental parameters on which the multi-factor confidence function is based include at least one of light intensity, ambient humidity, vibration amplitude, electromagnetic interference intensity, and target echo signal-to-noise ratio. When the confidence of any sensor is lower than a preset threshold, the fusion weight of that sensor is reset to zero, and the system switches to single-sensor decision mode.

9. The multi-sensor fusion intelligent sensing and safety early warning system for escalators according to claim 7, characterized in that, The data preprocessing module includes: The time synchronization unit is used to achieve microsecond-level time synchronization between the radar sensor and the vision sensor; The spatial calibration unit is used to obtain the transformation matrix between the radar coordinate system and the image coordinate system; The dynamic calibration unit is used to periodically check the calibration accuracy and trigger recalibration when the accuracy decreases.

10. The multi-sensor fusion intelligent sensing and safety early warning system for escalators according to claim 7, characterized in that, The heterogeneous fusion module includes: The radar feature extraction branch is used to extract spatial features from radar point cloud data. The visual feature extraction branch is used to extract semantic features from image data; The feature alignment layer is used to align radar features with visual features in the feature space. An attention fusion layer is used to weight and fuse features of different modalities according to the dynamic fusion weights; The classification output layer is used to output the probability of behavior categories and map them to risk levels.

Images