An image recognition-based elevator operation state monitoring and emergency rescue system

Abstract

The application discloses an elevator operation state monitoring and emergency rescue system based on image recognition, relates to the technical field of image recognition and intelligent monitoring, and comprises an elevator global image acquisition, image preprocessing and multi-scale feature extraction, operation state intelligent identification, operation trend prediction, abnormal event grading detection and early warning, cross-system emergency rescue collaborative scheduling, elevator control instruction linkage execution, edge local intelligent reasoning and emergency disposal, monitoring effect closed-loop verification and model iteration optimization, full-process data encryption storage and tracing units, and each unit cooperates to complete elevator full-process monitoring and emergency disposal. The application realizes full-area non-blind-area monitoring of the elevator, improves recognition accuracy by fusing multi-modal data, guarantees emergency capability by independent operation of the edge, can predict potential faults, optimizes emergency rescue scheduling, forms a closed-loop optimization mechanism, and provides complete data support for accident analysis.

Description

Technical Field

[0001] This invention relates to the field of image recognition and intelligent monitoring technology, and in particular to an elevator operation status monitoring and emergency rescue system based on image recognition. Background Technology

[0002] As a core vertical transportation device in high-rise buildings, the safe operation of elevators is directly related to the safety of people's lives. Traditional elevator monitoring systems mainly rely on manual operation and post-event video playback. Monitoring personnel need to view multiple video feeds simultaneously, which can easily lead to visual fatigue and delayed detection of abnormal events due to prolonged work. A large amount of monitoring data is only used for post-event tracing, failing to achieve real-time anomaly detection and early warning. When an elevator experiences emergencies such as entrapment, overshooting, or bottoming out, trapped passengers often need to manually call for help, resulting in a significant delay in rescue response. Early elevator monitoring systems mostly used single-sensor detection methods, acquiring operating parameters through acceleration, vibration, and load sensors installed on the elevator. These could only detect the basic operating status of the elevator and could not identify abnormal events that require visual information, such as people trapped in doors, doors not closing, foreign objects in the shaft, or people falling. The sensors themselves are susceptible to environmental interference and malfunction, and since each sensor operates independently, the data lacks correlation and cannot comprehensively reflect the true operating status of the elevator.

[0003] Existing image recognition-based elevator monitoring systems mostly only monitor the interior of the elevator car, neglecting critical areas such as landing doors, hoistway, and door operators, making them prone to missing anomalies. Landing door malfunctions, foreign objects in the hoistway, and door operator failures are significant causes of elevator accidents, and these anomalies cannot be detected through in-car monitoring. Current systems often employ a centralized cloud processing architecture, requiring all image data to be uploaded to the cloud for analysis. When network fluctuations or interruptions occur, the system completely loses its monitoring and early warning capabilities, failing to respond to emergencies. Furthermore, cloud processing suffers from transmission delays, making it difficult to meet the real-time response requirements for elevator anomalies. In addition, most existing systems can only detect and alarm on already occurring anomalies, lacking the ability to predict elevator operating trends, failing to identify potential equipment failures in advance, and hindering preventative maintenance.

[0004] Existing elevator emergency rescue and monitoring systems operate independently. When an emergency occurs, manual information transmission and rescue dispatch are required, resulting in a cumbersome process and slow response. The system cannot automatically assess the condition of trapped individuals, and rescue resource allocation lacks specificity, making it difficult to adjust rescue priorities based on the actual situation of those trapped. Furthermore, the existing system lacks a complete end-to-end data traceability mechanism; the occurrence, detection, early warning, dispatch, and handling of emergency events are not fully recorded, making comprehensive analysis and liability determination difficult after an accident. In addition, the existing system's recognition models often use fixed parameter designs, failing to continuously iterate and optimize based on actual operating data. As elevator equipment ages and the operating environment changes, the model's recognition accuracy gradually decreases, affecting the system's long-term operational performance. Summary of the Invention

[0005] This invention proposes an elevator operation status monitoring and emergency rescue system based on image recognition to solve the problems mentioned in the prior art.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an elevator operation status monitoring and emergency rescue system based on image recognition, comprising: The elevator full-domain image acquisition unit completes real-time image and video data acquisition of the elevator car, landing doors, shaft, and door operator areas. The image preprocessing and multi-scale feature extraction unit completes the preprocessing of the acquired images and multi-dimensional feature extraction. The elevator operation status intelligent recognition unit completes the classification and recognition of the normal operation status of the elevator. The elevator operation trend prediction unit completes the early prediction of potential faults and operation trend prediction based on continuous operation status time series data. The abnormal event graded detection and early warning unit completes the detection and graded early warning of abnormal elevator events. The cross-system emergency rescue collaborative scheduling unit completes the cross-departmental emergency rescue scheduling after an abnormal event occurs. The elevator control command linkage execution unit completes the command interaction and control execution with the original elevator control system. The edge-end local intelligent reasoning and emergency response unit is deployed in the elevator machine room to complete local data preprocessing, lightweight reasoning, and rapid handling of emergency abnormal events.

[0007] Furthermore, it also includes a multimodal data fusion enhancement module. This module has bidirectional communication connections with the elevator's original sensor system, image preprocessing and multi-scale feature extraction unit, and edge-end local intelligent reasoning and emergency response unit. It collects sensor data on the elevator's acceleration, vibration, load, door operator position, and running speed. After converting the sensor data into feature vectors, it performs weighted fusion with image features to generate multimodal comprehensive features, which are then input to the elevator operating status intelligent recognition unit, the elevator operating trend prediction unit, and the abnormal event classification detection and early warning unit, respectively.

[0008] Furthermore, it also includes an intelligent assessment module for the status of trapped personnel. This module is bidirectionally connected to the elevator's full-domain image acquisition unit, the edge-end local intelligent reasoning and emergency response unit, and the cross-system emergency rescue collaborative scheduling unit. Through real-time images collected inside the elevator car, it identifies the number, age, posture, actions, and emotional state of trapped personnel, generating a comprehensive assessment result for their status. The edge-end local intelligent reasoning and emergency response unit can independently complete a lightweight assessment of the trapped personnel's status when the network is interrupted. The cross-system emergency rescue collaborative scheduling unit adjusts the rescue priority and resource allocation based on the assessment results.

[0009] Furthermore, the elevator's overall image acquisition unit includes a car monitoring subunit, a landing door monitoring subunit, a hoistway monitoring subunit, and a door operator monitoring subunit. The car monitoring subunit is deployed at a diagonal position on the top of the car and is equipped with wide dynamic range and infrared supplementary lighting functions to collect images of personnel activities and the overall environment inside the car. The landing door monitoring subunit is deployed above each landing door to collect images of the landing door's opening and closing status, personnel entering and exiting, and foreign objects in the landing door. The hoistway monitoring subunit is deployed at the top and bottom of the hoistway to collect images of the car's running position in the hoistway, the guide rail status, and foreign objects in the hoistway. The door operator monitoring subunit is deployed on the side of the car door operator to collect images of the door operator's operating status, door clearance, and the door opening and closing process.

[0010] Furthermore, the image preprocessing and multi-scale feature extraction unit incorporates an image preprocessing submodule and a multi-scale feature extraction submodule. The image preprocessing submodule sequentially performs Gaussian filtering for noise reduction, camera distortion correction, adaptive histogram equalization for illumination compensation, and motion blur restoration of the acquired image. The multi-scale feature extraction submodule employs a three-layer convolutional structure and fuses features at different scales through a multi-scale feature adaptive fusion model. The calculation formula is as follows: ;in, This represents the fused multi-scale integrated feature map, where S is the total number of feature extraction scales. Let be the adaptive weight coefficient of the s-th scale feature, and σ be the nonlinear activation function. Let be the convolution weight matrix for the s-th scale feature. This is the original feature map at the s-th scale. This is the bias term for the s-th scale feature.

[0011] Furthermore, the elevator operation status intelligent recognition unit is pre-trained based on the labeled elevator operation status dataset. It has a built-in state classification submodule and a state tracking submodule. The state classification submodule completes the classification and recognition of the elevator's normal operation status, including going up, going down, stopping, opening the door, closing the door, empty, full load, and overload. The state tracking submodule completes the continuous tracking of the elevator operation status and the time sequence analysis of state transitions based on feature matching of continuous frame images. The recognition results are simultaneously input to the elevator operation trend prediction unit and the abnormal event classification detection and early warning unit.

[0012] Furthermore, the abnormal event classification detection and early warning unit incorporates an abnormal feature library, a classification detection submodule, and an early warning release submodule. The abnormal feature library includes image features and multimodal features of abnormal events such as entrapment, overshooting, bottoming out, door trapping, operation with doors not closed, foreign objects in the hoistway, and car overload. The classification detection submodule, based on the comparison between multimodal comprehensive features and the abnormal feature library, completes the detection and classification of abnormal events. An abnormal event confidence score is generated through an abnormal event confidence calculation model, with the calculation formula as follows: ;in, For the confidence level of abnormal events, α is the spatial feature weighting coefficient, β is the temporal feature weighting coefficient, γ is the sensor data weighting coefficient, and δ is the trend prediction weighting coefficient. For spatial anomaly scoring based on a single frame image, For temporal anomaly scoring based on consecutive frame sequences, For anomaly scoring based on elevator sensor data, Anomaly scores are based on predictions of operational trends.

[0013] Furthermore, the cross-system emergency rescue collaborative dispatch unit incorporates a rescue resource management submodule, a dispatch instruction generation submodule, and an on-site guidance submodule. The rescue resource management submodule uniformly manages the location and status information of property duty personnel, elevator maintenance personnel, fire rescue personnel, and medical rescue personnel. The dispatch instruction generation submodule generates the optimal rescue dispatch plan based on the level of abnormal events, the status of trapped personnel, and the distribution of rescue resources, and sends rescue instructions and on-site information to the corresponding rescue personnel. The on-site guidance submodule plays reassurance information and self-rescue guidance to trapped personnel through in-car voice and display devices, and simultaneously provides on-site rescue personnel with real-time elevator status and trapped personnel information.

[0014] Furthermore, the monitoring effect closed-loop verification and model iteration optimization unit has built-in effect verification submodule and model iteration submodule. The effect verification submodule, based on preset monitoring indicators, completes real-time statistics and comparison of indicators such as operation status recognition accuracy, abnormal event detection accuracy, trend prediction accuracy, false alarm rate, false alarm rate, and emergency response time, to verify the operation effect of the system. The model iteration submodule, based on newly added labeled data and verification results, completes incremental training and parameter updates of the elevator operation status recognition model, trend prediction model, and abnormal event detection model.

[0015] Furthermore, the end-to-end data encryption storage and traceability unit incorporates a data encryption submodule, a time-series storage submodule, and an end-to-end traceability submodule. The data encryption submodule employs an end-to-end encryption algorithm to complete encryption protection during data transmission and storage. The time-series storage submodule completes the time-series full storage of image data, sensor data, recognition results, trend prediction results, early warning information, scheduling instructions, and execution feedback data. Based on the stored end-to-end data, the end-to-end traceability submodule completes the end-to-end traceability and review of abnormal events from occurrence, detection, early warning, scheduling to handling.

[0016] Compared with existing technologies, the beneficial effects of this invention are: This invention utilizes a full-area elevator image acquisition unit to capture images of the entire elevator car, landing doors, shaft, and door operator area, covering all critical aspects of elevator operation, eliminating blind spots, and providing comprehensive image data support for elevator operation status identification and abnormal event detection. The image preprocessing and multi-scale feature extraction unit eliminates image interference caused by changes in lighting, vibration, and motion during elevator operation, while extracting image features at different scales, taking into account both global state features and local detail features, thus improving the comprehensiveness and accuracy of feature representation.

[0017] This invention utilizes a multimodal data fusion enhancement module to weightedly fuse image features with existing elevator sensor data, generating multimodal comprehensive features that more comprehensively reflect the elevator's operating status, improving the accuracy of operating status recognition, trend prediction, and abnormal event detection. An edge-based local intelligent inference and emergency response unit, deployed in the elevator machine room, can perform local data preprocessing, lightweight inference, and rapid handling of emergency abnormal events. It operates independently even during cloud network outages, enhancing system reliability and emergency response speed.

[0018] This invention utilizes an elevator operation trend prediction unit to perform early prediction of potential faults and operational trends based on continuous operational status time-series data. This enables the early identification of abnormal elevator operation trends, transforming post-accident alarms into pre-accident prevention and reducing elevator accidents. The abnormal event classification and early warning unit combines multimodal comprehensive features with operational trend prediction results to detect and classify abnormal events, effectively reducing false alarm and false negative rates and improving the accuracy of early warnings.

[0019] This invention utilizes an intelligent assessment module for the state of trapped individuals to identify their number, age, posture, actions, and emotional state, providing a basis for rescue dispatch and making the allocation of rescue resources more targeted. The cross-system emergency rescue collaborative dispatch unit can uniformly manage various rescue resources, generate optimal rescue dispatch plans, and simultaneously play reassurance messages and self-rescue guidance to trapped individuals through in-car voice and display devices, improving the efficiency and effectiveness of emergency rescue.

[0020] This invention, through a closed-loop verification and model iteration optimization unit for monitoring effects, can verify the system's operational performance in real time and perform incremental training and parameter updates based on newly added labeled data, continuously improving the model's recognition accuracy and generalization ability. The end-to-end data encryption storage and traceability unit enables encrypted storage and end-to-end traceability of all system data, providing complete and reliable data support for accident analysis and liability determination. Attached Figure Description

[0021] Figure 1 This is a schematic block diagram of the overall process flow for elevator monitoring and emergency rescue proposed in this invention; Figure 2 This is a schematic block diagram of the multimodal data preprocessing and feature fusion flowchart proposed in this invention; Figure 3 This is the logic diagram for operational status identification and trapped personnel assessment proposed in this invention; Figure 4 This is a flowchart of the abnormal event hierarchical detection and early warning judgment proposed in this invention; Figure 5 This is a flowchart of the cross-system rescue dispatch and monitoring closed-loop process proposed in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Reference Figures 1 to 5An elevator operation status monitoring and emergency rescue system based on image recognition, comprising: The elevator full-domain image acquisition unit acquires real-time image and video data of the elevator car, landing doors, shaft, and door operator areas. The image preprocessing and multi-scale feature extraction unit preprocesses the acquired images and extracts multi-dimensional features. The elevator operation status intelligent recognition unit classifies and recognizes the normal operation status of the elevator. The elevator operation trend prediction unit predicts potential faults and operation trends based on continuous operation status time-series data. The abnormal event graded detection and early warning unit detects and provides graded early warnings for abnormal elevator events. The cross-system emergency rescue collaborative scheduling unit coordinates cross-departmental emergency rescue after an abnormal event occurs. The elevator control command linkage execution unit interacts with and executes commands with the original elevator control system. The edge-end local intelligent reasoning and emergency response unit is deployed in the elevator machine room to perform local data preprocessing, lightweight reasoning, and rapid handling of emergency abnormal events. It operates independently when the cloud network is interrupted. The monitoring effect closed-loop verification and model iteration optimization unit verifies the monitoring effect in real time and continuously iterates and optimizes the recognition model. The full-process data encryption storage and traceability unit encrypts and stores the system's full-process data and performs full-link traceability.

[0024] This invention also includes a multimodal data fusion enhancement module. This module is bidirectionally connected to the elevator's original sensor system, image preprocessing and multi-scale feature extraction unit, and edge-end local intelligent reasoning and emergency response unit. It collects sensor data on the elevator's acceleration, vibration, load, door operator position, and running speed. After converting the sensor data into feature vectors, it performs weighted fusion with image features to generate multimodal comprehensive features. These features are then input to the elevator operating status intelligent recognition unit, the elevator operating trend prediction unit, and the abnormal event classification detection and early warning unit, respectively, thereby improving the accuracy of operating status recognition, trend prediction, and abnormal event detection.

[0025] This invention also includes an intelligent assessment module for the status of trapped personnel. This module is bidirectionally connected to the elevator full-domain image acquisition unit, the edge-end local intelligent reasoning and emergency response unit, and the cross-system emergency rescue collaborative scheduling unit. Through real-time images acquired inside the elevator car, it identifies the number, age, posture, actions, and emotional state of the trapped personnel and generates a comprehensive assessment result for their status. The edge-end local intelligent reasoning and emergency response unit can independently complete a lightweight assessment of the trapped personnel's status when the network is interrupted. The cross-system emergency rescue collaborative scheduling unit adjusts the rescue priority and rescue resource allocation based on the assessment results.

[0026] In this invention, the elevator global image acquisition unit includes a car interior monitoring subunit, a landing door monitoring subunit, a shaft monitoring subunit, and a door operator monitoring subunit. The car interior monitoring subunit is deployed at a diagonal position on the top of the car and is equipped with wide dynamic range and infrared supplementary lighting functions to acquire images of personnel activities and the overall environment inside the car. The landing door monitoring subunit is deployed above each landing door to acquire images of the landing door opening and closing status, personnel entering and exiting, and foreign objects in the landing door. The shaft monitoring subunit is deployed at the top and bottom of the shaft to acquire images of the car's running position in the shaft, the guide rail status, and foreign objects in the shaft. The door operator monitoring subunit is deployed on the side of the car door operator to acquire images of the door operator's operating status, door gap, and door opening and closing process. All acquisition units acquire data synchronously at a preset frequency to achieve time synchronization of images in each area.

[0027] In this invention, the image preprocessing and multi-scale feature extraction unit incorporates an image preprocessing submodule and a multi-scale feature extraction submodule. The image preprocessing submodule sequentially performs Gaussian filtering for noise reduction, camera distortion correction, adaptive histogram equalization for illumination compensation, and motion blur restoration of the acquired image, eliminating image interference caused by changes in illumination, vibration, and motion during elevator operation. The multi-scale feature extraction submodule employs a three-layer convolutional structure to extract edge, texture, shape, and semantic features at scales of 3×3, 5×5, and 7×7, respectively. The fusion of features at different scales is achieved through a multi-scale feature adaptive fusion model, calculated using the following formula: ;in, The fused multi-scale comprehensive feature map is represented by a dimension of 1, and S represents the total number of feature extraction scales, also with a dimension of 1. Let be the adaptive weight coefficient of the s-th scale feature, with a dimension of 1, automatically learned during network training, and all The sum is 1, σ is a nonlinear activation function with a dimension of 1. Let be the convolution weight matrix for the s-th scale feature, with a dimension of 1. Let be the original feature map at the s-th scale, with a dimension of 1. The bias term of the s-th scale feature has a dimension of 1. Through adaptive weighted fusion of features at different scales, the global state features and local detail features of elevator operation are taken into account, thereby improving the comprehensiveness and accuracy of feature representation.

[0028] In this invention, the elevator operation status intelligent recognition unit is pre-trained based on a labeled elevator operation status dataset. It has a built-in state classification submodule and a state tracking submodule. The state classification submodule classifies and recognizes the normal operation status of the elevator, including going up, going down, stopping, opening the door, closing the door, empty, full load, and overload. The state tracking submodule performs continuous tracking and state transition time sequence analysis of the elevator operation status based on feature matching of continuous frame images, generates elevator operation status time sequence curves, and simultaneously inputs the recognition results into the elevator operation trend prediction unit and the abnormal event classification detection and early warning unit, providing a normal state benchmark for trend prediction and abnormal event detection.

[0029] In this invention, the abnormal event classification detection and early warning unit incorporates an abnormal feature library, a classification detection submodule, and an early warning release submodule. The abnormal feature library includes image features and multimodal features of abnormal events such as entrapment, overshooting, bottoming out, door trapping, door not closing, foreign objects in the shaft, and car overload. The classification detection submodule, based on the comparison between multimodal comprehensive features and the abnormal feature library, combined with the elevator operation trend prediction results, completes the detection and classification of abnormal events. An abnormal event confidence score is generated through an abnormal event confidence score calculation model, with the calculation formula as follows: ;in, The confidence level for anomalies is represented by a dimension of 1; a larger value indicates a higher probability of the anomaly occurring. α is the spatial feature weighting coefficient (dimension 1), β is the temporal feature weighting coefficient (dimension 1), γ is the sensor data weighting coefficient (dimension 1), and δ is the trend prediction weighting coefficient (dimension 1). The sum of α, β, γ, and δ is 1. Spatial anomaly scoring based on a single frame image, with a dimension of 1. This is a temporal anomaly score based on a continuous frame sequence, with a dimension of 1. The anomaly score is based on elevator sensor data and has a dimension of 1. The anomaly score is based on the prediction of the operation trend and has a dimension of 1. By integrating the anomaly features of spatial, temporal, and multimodal sensors with the trend prediction, the system can accurately detect elevator anomalies. The early warning release submodule releases graded early warning information to the corresponding personnel and system based on the anomaly level.

[0030] In this invention, the cross-system emergency rescue collaborative dispatch unit incorporates a rescue resource management submodule, a dispatch instruction generation submodule, and an on-site guidance submodule. The rescue resource management submodule uniformly manages the location and status information of property duty personnel, elevator maintenance personnel, fire rescue personnel, and medical rescue personnel. The dispatch instruction generation submodule generates the optimal rescue dispatch plan based on the abnormal event level, the status of trapped personnel, and the distribution of rescue resources, and sends rescue instructions and on-site information to the corresponding rescue personnel. The on-site guidance submodule plays reassurance information and self-rescue guidance to trapped personnel through in-car voice and display devices, and simultaneously provides on-site rescue personnel with real-time elevator status and trapped personnel information.

[0031] In this invention, the monitoring effect closed-loop verification and model iteration optimization unit has built-in effect verification submodule and model iteration submodule. The effect verification submodule, based on preset monitoring indicators, completes real-time statistics and comparison of indicators such as operation status recognition accuracy, abnormal event detection accuracy, trend prediction accuracy, false alarm rate, false alarm rate, and emergency response time, to verify the operation effect of the system. The model iteration submodule, based on newly added labeled data and verification results, completes incremental training and parameter updates of the elevator operation status recognition model, trend prediction model, and abnormal event detection model, continuously improving the recognition accuracy and generalization ability of the model.

[0032] In this invention, the end-to-end data encryption storage and traceability unit incorporates a data encryption submodule, a time-series storage submodule, and an end-to-end traceability submodule. The data encryption submodule employs an end-to-end encryption algorithm to complete encryption protection during data transmission and storage. The time-series storage submodule completes the time-series full storage of image data, sensor data, recognition results, trend prediction results, early warning information, scheduling instructions, and execution feedback data, with a storage period of no less than 90 days. Based on the stored end-to-end data, the end-to-end traceability submodule completes the end-to-end traceability and review of abnormal events from occurrence, detection, early warning, scheduling to handling, providing complete and reliable data support for accident analysis and responsibility determination.

[0033] The following two examples further illustrate the specific implementation of this system: Example 1: Implementation Method for Monitoring Elevator Operation Status and Emergency Rescue in High-Rise Residential Buildings This embodiment is applied to a 32-story high-rise residential building in the core urban area. The building is equipped with two passenger elevators, each serving 128 households and operating an average of 1,200 times per day. The users cover residents of all ages, and the building is characterized by concentrated passenger flow during morning and evening peak hours, a high proportion of elderly and children using the elevators, and continuous demand for operation at night. The safety of elevator operation is directly related to the daily life and life safety of residents. It is necessary to realize full-area monitoring of elevator operation status, real-time early warning of abnormal events, and rapid emergency rescue to ensure the long-term stable operation of the elevators.

[0034] A comprehensive elevator image acquisition unit is deployed to capture real-time images of all critical areas of the elevator. The car monitoring subunit is positioned at two diagonal points on the top of the car, equipped with wide dynamic range and infrared illumination, enabling clear image acquisition even in low-light or backlit conditions, covering the entire car space and capturing images of personnel activity, the car environment, and equipment status. The landing door monitoring subunit is positioned centrally above each landing door, with 32 points, capturing the opening and closing status of each landing door, personnel entry and exit, and any foreign objects in the landing area. The hoistway monitoring subunits are deployed at the top and bottom of the hoistway, capturing the car's position within the hoistway, the guide rail status, and any foreign objects in the hoistway. The door operator monitoring subunit is deployed on the side of the car door operator, capturing the operator's operating status, door clearance, and the entire door opening and closing process. All acquisition units synchronously acquire data at a frequency of 25 frames per second to ensure temporal consistency of images across all areas.

[0035] An image preprocessing and multi-scale feature extraction unit is deployed to complete the preprocessing and feature extraction of the acquired images. The image preprocessing submodule sequentially performs Gaussian filtering for noise reduction, camera distortion correction, adaptive histogram equalization for illumination compensation, and motion blur restoration on the acquired images to eliminate image interference caused by changes in illumination, mechanical vibration, and car movement during elevator operation, thereby improving image quality. The multi-scale feature extraction submodule adopts a three-layer convolutional structure to extract edge, texture, shape, and semantic features at different scales. A multi-scale feature adaptive fusion model is used to fuse features at different scales, generating a multi-scale comprehensive feature map. A multi-modal data fusion enhancement module is deployed simultaneously, connecting to the elevator's existing sensor system to collect sensor data on elevator acceleration, vibration, load, door operator position, and operating speed. The sensor data is converted into feature vectors and then weighted and fused with image features to generate multi-modal comprehensive features.

[0036] The system deploys an intelligent elevator operation status recognition unit and an elevator operation trend prediction unit. The intelligent elevator operation status recognition unit is pre-trained based on a labeled dataset of hundreds of thousands of elevator operation statuses. It includes a built-in state classification submodule and a state tracking submodule. The state classification submodule classifies and identifies the elevator's normal operating states: upward, downward, stopped, door open, door closed, empty, full load, and overloaded. The state tracking submodule uses feature matching from consecutive frame images to continuously track the elevator's operation status and perform time-series analysis of state transitions, generating a time-series curve of the elevator's operation status. The elevator operation trend prediction unit, based on continuous operation status time-series data and multimodal comprehensive features, performs early prediction of potential elevator faults and operation trends, identifying early fault characteristics such as door operator wear, guide rail noise, and wire rope slack.

[0037] An abnormal event classification and early warning unit is deployed to detect and classify elevator abnormal events. This unit has a built-in abnormal feature library containing image features and multimodal features of various abnormal events, including entrapment, overshooting, bottoming out, door trapping, door not closing, foreign objects in the shaft, and car overload. The classification detection submodule compares the multimodal comprehensive features with the abnormal feature library, combined with elevator operation trend prediction results, to detect and classify abnormal events into three levels: general warning, important warning, and emergency warning. The early warning dissemination submodule issues classified warning information to corresponding personnel based on the abnormality level. General warnings are pushed to property management staff, important warnings are pushed to both property management and elevator maintenance personnel, and emergency warnings are pushed to property management, maintenance, fire, and medical rescue personnel simultaneously.

[0038] An edge-based local intelligent inference and emergency response unit is deployed inside the elevator machine room, equipped with independent computing and storage resources. This unit performs local data preprocessing, lightweight model inference, and rapid handling of emergency events, and can operate independently even when the cloud network is interrupted. When an emergency event such as a person being trapped in a door or entrapped in the elevator car is detected, the edge unit can directly send control commands to the elevator control command execution unit, while simultaneously playing reassuring messages to the trapped individuals via the in-car voice system. The elevator control command execution unit communicates bidirectionally with the existing elevator control system, receiving dispatch commands and converting them into control signals recognizable by the elevator to execute door opening, deceleration, and floor stopping control actions.

[0039] A cross-system emergency rescue collaborative dispatch unit is deployed to uniformly manage the location and status information of property duty personnel, elevator maintenance personnel, fire rescue personnel, and medical rescue personnel. When an abnormal event occurs, the dispatch instruction generation submodule generates the optimal rescue dispatch plan based on the abnormal event level, the status of trapped personnel, and the distribution of rescue resources, and sends rescue instructions and on-site information to the corresponding rescue personnel. The intelligent assessment module for the status of trapped personnel identifies the number, age, posture, actions, and emotional state of trapped personnel through real-time images inside the elevator car, generating a comprehensive status assessment result. The cross-system emergency rescue collaborative dispatch unit adjusts rescue priorities and rescue resource allocation based on the assessment result. The on-site guidance submodule plays reassuring information and self-rescue guidance to trapped personnel through in-car voice and display devices, simultaneously providing on-site rescue personnel with real-time elevator status and trapped personnel information.

[0040] The system deploys a closed-loop verification and model iteration optimization unit for monitoring effectiveness, and a full-process data encryption storage and traceability unit. The closed-loop verification and model iteration optimization unit, based on preset monitoring indicators, performs real-time statistics and comparisons of indicators such as operational status identification accuracy, abnormal event detection accuracy, trend prediction accuracy, false alarm rate, missed alarm rate, and emergency response time, verifying the system's operational effectiveness. The model iteration submodule, based on newly added labeled data and verification results, completes incremental training and parameter updates for each identification model. The full-process data encryption storage and traceability unit employs end-to-end encryption algorithms to achieve encryption protection during data transmission and storage. The time-series full-volume storage period for all data is 90 days. The full-link traceability submodule, based on the stored full-process data, completes full-link traceability and review of abnormal events from occurrence, detection, early warning, scheduling to handling.

[0041] Table 1. Operational Performance of Elevator Monitoring and Rescue System in High-Rise Residential Buildings

[0042] Table 1 provides a comprehensive picture of the system's actual operational performance in high-rise residential elevator scenarios. The accuracy rates for elevator operation status identification and abnormal event detection demonstrate the effectiveness of the multimodal data fusion and multi-scale feature extraction mechanisms, accurately identifying various elevator operation states and abnormal events. The accuracy rate for early prediction of potential faults reflects the role of the operation trend prediction unit, enabling early identification of elevator fault characteristics. The average response time for abnormal events and the independent runtime at the edge demonstrate the advantages of the edge-side local intelligent inference architecture, maintaining the system's emergency response capabilities even during network outages. The average arrival time for emergency rescue reflects the effectiveness of the cross-system collaborative scheduling mechanism, effectively shortening rescue response time and ensuring the safety of trapped personnel.

[0043] Example 2: Implementation Method for Elevator Operation Status Monitoring and Emergency Rescue in Commercial Complexes This embodiment is applied to a 5-story commercial complex in the core business district of the city. The building is equipped with 4 passenger elevators and 2 freight elevators, with an average daily passenger flow of 35,000 people. During holidays, the peak passenger flow can reach 120,000 people. The elevators run an average of 2,800 times per day. The building is characterized by concentrated peak passenger flow, high personnel mobility, frequent freight transportation, and high elevator operation intensity. The elevators are prone to abnormal events such as overloading, people being trapped in the doors, and foreign objects in the shaft. Therefore, there are high requirements for elevator operation safety and emergency rescue response speed.

[0044] A comprehensive elevator image acquisition unit is deployed to capture real-time images of all critical areas of the elevator. The car monitoring subunit is positioned at two diagonal points on the top of the car, equipped with wide dynamic range and strong light suppression capabilities to adapt to the complex lighting environment of commercial complexes, capturing images of personnel activity, goods transport, and the overall environment within the car. The landing door monitoring subunit is positioned at the center above each landing door, with 30 points, capturing the opening and closing status of each landing door, personnel entry and exit, and any foreign objects in the landing area. The shaft monitoring subunits are deployed at the top and bottom of each elevator shaft, capturing the car's position within the shaft, the guide rail status, and any foreign objects within the shaft. The door operator monitoring subunit is deployed on the side of each elevator car door operator, capturing the door operator's operating status, door clearance, and images of the entire door opening and closing process. All acquisition units synchronously acquire data at a frequency of 30 frames per second to ensure temporal consistency of images across all areas.

[0045] An image preprocessing and multi-scale feature extraction unit is deployed to complete the preprocessing and feature extraction of the acquired images. The image preprocessing submodule sequentially performs Gaussian filtering for noise reduction, camera distortion correction, adaptive histogram equalization for illumination compensation, and motion blur restoration on the acquired images to eliminate image interference caused by complex lighting in commercial complexes and high-frequency elevator operation, thereby improving image quality. The multi-scale feature extraction submodule adopts a three-layer convolutional structure to extract edge, texture, shape, and semantic features at different scales. A multi-scale feature adaptive fusion model is used to fuse features at different scales, generating a multi-scale comprehensive feature map. A multi-modal data fusion enhancement module is deployed simultaneously, connecting to the existing elevator sensor system to collect sensor data on elevator acceleration, vibration, load, door operator position, and operating speed. The sensor data is converted into feature vectors and then weighted and fused with image features to generate multi-modal comprehensive features.

[0046] The system deploys an intelligent elevator operation status recognition unit and an elevator operation trend prediction unit. The intelligent elevator operation status recognition unit is pre-trained based on a labeled dataset of 200,000 commercial elevator operation statuses. It includes a built-in state classification submodule and a state tracking submodule. The state classification submodule classifies and identifies the elevator's normal operating states: upward, downward, stopped, door open, door closed, empty, full load, and overloaded. The state tracking submodule uses feature matching of consecutive frame images to continuously track the elevator's operation status and perform time-series analysis of state transitions, generating a time-series curve of the elevator's operation status. The elevator operation trend prediction unit, based on continuous operation status time-series data and multimodal comprehensive features, performs early prediction of potential elevator faults and operation trends, identifying early fault characteristics such as guide rail wear, brake aging, and door operator malfunctions.

[0047] An abnormal event classification and early warning unit is deployed to detect and classify elevator abnormal events. This unit incorporates an anomaly feature library optimized for commercial elevators, including image features and multimodal features of various abnormal events such as entrapment, overshooting, bottoming out, door trapping, door not closing, foreign objects in the shaft, car overload, and falls. The classification detection submodule compares the multimodal comprehensive features with the anomaly feature library, combined with elevator operation trend prediction results, to detect and classify abnormal events into three levels: general warning, important warning, and emergency warning. The early warning dissemination submodule issues classified warning information to corresponding personnel based on the anomaly level. General warnings are pushed to mall property management staff on duty, important warnings are simultaneously pushed to property management and elevator maintenance personnel, and emergency warnings are simultaneously pushed to property management, maintenance, fire, and medical rescue personnel.

[0048] An edge-based local intelligent inference and emergency response unit is deployed, with one edge device deployed for each elevator. These devices are located inside the elevator machine room and equipped with independent computing and storage resources. The edge-based local intelligent inference and emergency response unit performs local data preprocessing, lightweight model inference, and rapid handling of emergency events, and can operate independently even when the cloud network is interrupted. When an emergency event such as a person being trapped in a door, entrapped in the elevator car, or a person falling is detected, the edge unit can directly send control commands to the elevator control command execution unit, and simultaneously play reassuring messages to the trapped or injured person through the in-car voice device. The elevator control command execution unit communicates bidirectionally with the elevator's existing control system, receiving dispatch commands and converting them into control signals recognizable by the elevator to execute door opening, deceleration, and floor stopping control actions.

[0049] A cross-system emergency rescue collaborative dispatch unit is deployed to uniformly manage the location and status information of mall property management personnel, elevator maintenance personnel, fire rescue personnel, and medical rescue personnel. When an abnormal event occurs, the dispatch instruction generation submodule generates the optimal rescue dispatch plan based on the abnormal event level, the status of trapped personnel, and the distribution of rescue resources, and sends rescue instructions and on-site information to the corresponding rescue personnel. The intelligent assessment module for trapped personnel's status identifies the number, age, posture, actions, and emotional state of trapped personnel through real-time images inside the elevator car, generating a comprehensive assessment result. The cross-system emergency rescue collaborative dispatch unit adjusts rescue priorities and rescue resource allocation based on the assessment result. The on-site guidance submodule plays reassuring information and self-rescue guidance to trapped personnel through in-car voice and display devices, simultaneously providing on-site rescue personnel with real-time elevator status and trapped personnel information.

[0050] The system deploys a closed-loop verification and model iteration optimization unit for monitoring effectiveness, and a full-process data encryption storage and traceability unit. The closed-loop verification and model iteration optimization unit, based on preset monitoring indicators, performs real-time statistics and comparisons of indicators such as operational status identification accuracy, abnormal event detection accuracy, trend prediction accuracy, false alarm rate, missed alarm rate, and emergency response time, verifying the system's operational effectiveness. The model iteration submodule, based on newly added labeled data and verification results, completes incremental training and parameter updates for each identification model. The full-process data encryption storage and traceability unit employs end-to-end encryption algorithms to achieve encryption protection during data transmission and storage. The time-series full-volume storage period for all data is 180 days. The full-link traceability submodule, based on the stored full-process data, completes full-link traceability and review of abnormal events from occurrence, detection, early warning, scheduling to handling.

[0051] Table 2. Operational Performance of Elevator Monitoring and Rescue System in Commercial Complexes

[0052] Table 2 comprehensively reflects the actual operational performance of this system in the elevator scenario of a commercial complex. The accuracy rates of elevator operation status identification and abnormal event detection demonstrate the system's adaptability to the high-frequency operation and complex scenarios of commercial elevators, accurately identifying various operating states and abnormal events. The accuracy rate of early prediction of potential faults reflects the adaptability of the operation trend prediction unit to the high-intensity operating conditions of commercial elevators, enabling early identification of equipment faults. The average response time for abnormal events and the independent operation time of the edge terminal reflect the reliability of the edge terminal architecture, enabling rapid response to various emergency events. The average arrival time for emergency rescue reflects the effectiveness of the cross-system collaborative scheduling mechanism in commercial scenarios, fully utilizing the rescue resources within the mall, shortening rescue response time, and ensuring the safety of mall customers and staff.

[0053] Reference Figure 2 This diagram details the mechanism by which the system processes complex, multi-source data and extracts effective features at the front end. The system employs a dual-thread parallel processing architecture. The left-hand visual stream receives image data from the elevator car, landing doors, hoistway, and door operator, sequentially performing standardized operations such as Gaussian denoising, distortion correction, illumination compensation, and motion blur restoration. Subsequently, a three-layer convolutional network is used to extract visual features at different scales, such as edges and textures. The right-hand sensor stream simultaneously receives signals from the elevator's physical sensors, including acceleration, vibration, and load, and converts them into feature vectors. The multi-scale visual features and the physical features from the sensors are spatially and temporally aligned in the fusion module. An adaptive weighting mechanism generates multi-modal comprehensive features containing multi-dimensional information, thus providing a solid data foundation for subsequent high-precision state assessment.

[0054] Reference Figure 3This diagram highlights the logic behind elevator daily status recognition and personnel status assessment based on fused features. Upon receiving multimodal integrated features, the model performs two core tasks in parallel. The status classification submodule identifies the elevator's current mechanical operation status, such as up / down movement, door opening / closing, and load conditions, and passes this information to the status tracking submodule to generate a continuous running trajectory curve. This curve is then output to the trend prediction engine to detect potential early signs of malfunctions. Simultaneously, the personnel assessment module performs in-depth analysis of the elevator car's interior features, accurately counting passengers and identifying their posture and emotional state. If the system determines that a passenger is in a state of panic or injury, it automatically generates a high-priority personnel status report, which directly affects the allocation of subsequent rescue resources.

[0055] Reference Figure 4 This diagram illustrates the comprehensive judgment logic of the anomaly detection engine when faced with complex features. The engine has a rich built-in anomaly feature library, breaking down real-time multidimensional data into four independent scores: spatial anomalies, time-series anomalies, physical sensor anomalies, and trend prediction anomalies. The system fuses these four scores using a weighted calculation model to derive a quantified anomaly event confidence level. Based on this, the system sets strict judgment branches: if the confidence level is below a preset threshold, it is considered a single fluctuation or false alarm and filtered out; if the confidence level exceeds the threshold, the system will identify the specific fault type based on feature comparison and then classify the warning level. Different warning levels will trigger differentiated release strategies, accurately delivering warnings to the corresponding management and maintenance personnel.

[0056] Reference Figure 5 This diagram illustrates the entire process of cross-departmental collaboration and closed-loop optimization, from anomaly confirmation to rescue execution and subsequent model iteration and evolution. Upon receiving an anomaly warning and the status of trapped personnel, the scheduling engine assesses available rescue resources from property management, maintenance, fire services, and medical facilities, calculates the optimal scheduling plan, and automatically generates execution instructions. These instructions are simultaneously sent to on-site personnel and the existing control system at the elevator's base for coordinated intervention. During the rescue, the system inside the elevator car provides voice and screen guidance to reassure passengers. After the incident subsides, the verification submodule extracts real business data such as response time and false negative rate for performance review. Finally, all data is archived and encrypted, and key samples are extracted for incremental training of the recognition algorithm, giving the system the ability to continuously evolve.

[0057] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An elevator operation status monitoring and emergency rescue system based on image recognition, characterized in that, include: Elevator full-area image acquisition unit: Completes real-time image and video data acquisition of the elevator car, landing doors, shaft, and door operator area; Image preprocessing and multi-scale feature extraction unit: performs preprocessing and multi-dimensional feature extraction of the acquired images; Elevator operating status intelligent recognition unit: completes the classification and recognition of the normal operating status of the elevator; Elevator operation trend prediction unit: Based on continuous operation status time series data, it completes early prediction of potential faults and prediction of operation trends; Abnormal Event Classification Detection and Early Warning Unit: Completes the detection and classification of abnormal elevator events; Cross-system emergency rescue coordination and dispatch unit: Completes cross-departmental emergency rescue dispatch after an abnormal event occurs; Elevator control command linkage execution unit: completes command interaction and control execution with the existing elevator control system; Edge-based local intelligent inference and emergency response unit: Deployed in the elevator machine room, it completes local data preprocessing, lightweight inference, and rapid handling of emergency anomalies.

2. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, It also includes a multimodal data fusion enhancement module, which is bidirectionally connected to the elevator's original sensor system, image preprocessing and multi-scale feature extraction unit, and edge-end local intelligent reasoning and emergency response unit. It collects sensor data on the elevator's acceleration, vibration, load, door operator position, and running speed, converts the sensor data into feature vectors, and then performs weighted fusion with image features to generate multimodal comprehensive features, which are then input to the elevator operating status intelligent recognition unit, elevator operating trend prediction unit, and abnormal event graded detection and early warning unit, respectively.

3. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, It also includes an intelligent assessment module for the status of trapped personnel. This module is bidirectionally connected to the elevator's full-domain image acquisition unit, the edge-end local intelligent reasoning and emergency response unit, and the cross-system emergency rescue collaborative scheduling unit. Through real-time images collected inside the elevator car, it identifies the number, age, posture, actions, and emotional state of trapped personnel and generates a comprehensive assessment result for their status. The edge-end local intelligent reasoning and emergency response unit can independently complete a lightweight assessment of the trapped personnel's status when the network is interrupted. The cross-system emergency rescue collaborative scheduling unit adjusts the rescue priority and resource allocation based on the assessment results.

4. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The elevator's full-area image acquisition unit includes a car monitoring subunit, a landing door monitoring subunit, a hoistway monitoring subunit, and a door operator monitoring subunit. The car monitoring subunit is deployed at a diagonal position on the top of the car and is equipped with wide dynamic range and infrared supplementary lighting functions to collect images of personnel activities and the overall environment inside the car. The landing door monitoring subunit is deployed above each landing door to collect images of the landing door's opening and closing status, personnel entering and exiting, and foreign objects in the landing door. The hoistway monitoring subunit is deployed at the top and bottom of the hoistway to collect images of the car's running position in the hoistway, the guide rail status, and foreign objects in the hoistway. The door operator monitoring subunit is deployed on the side of the car door operator to collect images of the door operator's operating status, door clearance, and the door opening and closing process.

5. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The image preprocessing and multi-scale feature extraction unit includes an image preprocessing submodule and a multi-scale feature extraction submodule. The image preprocessing submodule sequentially performs Gaussian filtering for noise reduction, camera distortion correction, adaptive histogram equalization for illumination compensation, and motion blur restoration of the acquired image. The multi-scale feature extraction submodule adopts a three-layer convolutional structure and uses a multi-scale feature adaptive fusion model to fuse features at different scales. The calculation formula is as follows: ;in, This represents the fused multi-scale integrated feature map, where S is the total number of feature extraction scales. Let be the adaptive weight coefficient of the s-th scale feature, and σ be the nonlinear activation function. Let be the convolution weight matrix for the s-th scale feature. This is the original feature map at the s-th scale. This is the bias term for the s-th scale feature.

6. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The elevator operation status intelligent recognition unit is pre-trained based on a labeled elevator operation status dataset. It has a built-in state classification submodule and a state tracking submodule. The state classification submodule classifies and recognizes the normal operation status of the elevator, including going up, going down, stopping, opening the door, closing the door, empty, full load, and overload. The state tracking submodule uses feature matching of continuous frame images to continuously track the elevator operation status and perform time-series analysis of state transitions. The recognition results are simultaneously input into the elevator operation trend prediction unit and the abnormal event classification detection and early warning unit.

7. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The abnormal event classification detection and early warning unit incorporates an abnormal feature library, a classification detection submodule, and an early warning release submodule. The abnormal feature library includes image features and multimodal features of abnormal events such as entrapment, overshooting, bottoming out, door trapping, operation with doors not closed, foreign objects in the hoistway, and car overload. The classification detection submodule, based on the comparison between multimodal comprehensive features and the abnormal feature library, completes the detection and classification of abnormal events. An abnormal event confidence score is generated through an abnormal event confidence calculation model; the calculation formula is as follows: ;in, For the confidence level of abnormal events, α is the spatial feature weighting coefficient, β is the temporal feature weighting coefficient, γ is the sensor data weighting coefficient, and δ is the trend prediction weighting coefficient. For spatial anomaly scoring based on a single frame image, For temporal anomaly scoring based on consecutive frame sequences, For anomaly scoring based on elevator sensor data, Anomaly scores are based on predictions of operational trends.

8. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The cross-system emergency rescue collaborative dispatch unit has built-in rescue resource management submodule, dispatch instruction generation submodule, and on-site guidance submodule. The rescue resource management submodule uniformly manages the location and status information of property duty personnel, elevator maintenance personnel, fire rescue personnel, and medical rescue personnel. The dispatch instruction generation submodule generates the optimal rescue dispatch plan based on the abnormal event level, the status of trapped personnel, and the distribution of rescue resources, and sends rescue instructions and on-site information to the corresponding rescue personnel. The on-site guidance submodule plays reassurance information and self-rescue guidance to trapped personnel through in-car voice and display equipment, and simultaneously provides on-site rescue personnel with real-time elevator status and trapped personnel information.

9. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The monitoring effect closed-loop verification and model iteration optimization unit has built-in effect verification submodule and model iteration submodule. The effect verification submodule, based on preset monitoring indicators, completes real-time statistics and comparison of indicators such as operation status recognition accuracy, abnormal event detection accuracy, trend prediction accuracy, false alarm rate, false alarm rate, and emergency response time, to verify the operation effect of the system. The model iteration submodule, based on newly added labeled data and verification results, completes incremental training and parameter updates of the elevator operation status recognition model, trend prediction model, and abnormal event detection model.

10. The elevator operation status monitoring and emergency rescue system based on image recognition according to claim 1, characterized in that, The end-to-end data encryption storage and traceability unit has built-in data encryption submodule, time-series storage submodule, and end-to-end traceability submodule. The data encryption submodule adopts an end-to-end encryption algorithm to complete the encryption protection during data transmission and storage. The time-series storage submodule completes the time-series full storage of image data, sensor data, recognition results, trend prediction results, early warning information, scheduling instructions, and execution feedback data. The end-to-end traceability submodule, based on the stored end-to-end data, completes the end-to-end traceability and review of abnormal events from occurrence, detection, early warning, scheduling to handling.

Images