An elevator safety detection system based on vision and multi-modal data fusion

Abstract

The application relates to the technical field of elevator safety monitoring, and discloses an elevator safety detection system based on visual and multi-modal data fusion, which comprises the following modules: a data preprocessing module, which collects original multi-modal sensor data of elevator operation and performs preprocessing; a feature extraction module, which performs operation condition recognition and extracts multi-scale features for different operation conditions; a multi-modal fusion module, which calculates the reliability coefficients of various modes and performs adaptive fusion; an abnormality detection module, which extracts transient features of operation condition switching and performs abnormality judgment and abnormality evolution trend mining; a sample enhancement module, which extracts normal samples and rare fault samples, generates synthetic samples and performs quality screening; a model construction module, which trains a multi-level fault recognizer and performs optimized learning by fusing a multi-modal feature library; and an intelligent early warning module, which calculates a safety level and reasons fault causes and disposal measures; the application realizes accurate dynamic safety monitoring of an elevator in an ultra-high-rise building under an extreme load switching operation condition.

Description

Technical Field

[0001] This invention relates to the field of elevator safety monitoring technology, and more specifically, to an elevator safety detection system based on the fusion of visual and multimodal data. Background Technology

[0002] With the acceleration of urbanization and the emergence of numerous super high-rise buildings, the safety and reliability of elevators, as core equipment in vertical transportation, are receiving increasing attention. Elevator systems in super high-rise buildings face complex operating environments characterized by high speeds, frequent load changes, and drastic shifts in operating conditions. Traditional elevator safety testing methods are clearly insufficient in addressing these challenges.

[0003] Existing elevator safety inspection technologies primarily rely on single sensors or fixed data fusion strategies. Single-sensor-based methods can only acquire partial information about the elevator's operating status, making it difficult to comprehensively reflect its safety condition and prone to missed detections and false alarms. For example, vibration sensors alone cannot identify abnormalities inside the car, and visual monitoring alone cannot detect internal faults in the mechanical system. While fixed fusion strategies integrate data from multiple sensors, they fail to consider the dynamic changes in the quality of data from each sensor under different operating conditions and environmental environments, resulting in insufficient reliability of the fusion results.

[0004] Furthermore, existing technologies have significant limitations in handling extreme load switching conditions. When elevators rapidly switch between no-load, half-load, and full-load states, as well as undergo drastic changes in operating conditions such as acceleration, constant speed, and deceleration, the stress states on mechanical components become complex and variable, easily generating short-lived but crucial transient abnormal signals containing important fault information. Traditional steady-state monitoring methods struggle to effectively capture these transient signals, leading to the omission of early fault symptoms. Simultaneously, existing technologies also face difficulties in acquiring fault samples; the limited number of samples for rare fault types results in insufficient ability of fault identification models to recognize these rare but serious fault types.

[0005] Therefore, there is an urgent need for an intelligent detection system that can adapt to the complex operating environment of elevators in super high-rise buildings, integrate multimodal data, adaptively adjust fusion strategies, effectively capture transient abnormal signals, and accurately identify various faults, so as to improve the comprehensiveness, accuracy, and predictability of elevator safety monitoring. Summary of the Invention

[0006] This invention provides an elevator safety detection system based on the fusion of vision and multimodal data, which solves the technical problems in related technologies that are difficult to adapt to the highly dynamic and complex scenarios of elevators in super high-rise buildings under extreme load switching conditions, and are prone to missing transient abnormal signals and progressive fault signs.

[0007] This invention provides an elevator safety inspection system based on vision and multimodal data fusion, comprising:

[0008] The data preprocessing module collects raw multimodal sensor data from elevator operation and preprocesses it to obtain a multimodal preprocessed dataset.

[0009] The feature extraction module identifies operating conditions based on a multimodal preprocessed dataset, extracts multi-scale features for different operating conditions, and obtains a multimodal feature library and operating condition identification sequences.

[0010] The multimodal fusion module calculates the reliability coefficients of each mode in the multimodal feature library and performs adaptive fusion to obtain the elevator safety status judgment result;

[0011] The anomaly detection module, based on the multimodal preprocessing dataset and operating condition identification sequence, extracts transient features of operating condition switching and makes anomaly judgments, mines the anomaly evolution trend, and obtains progressive fault identification results.

[0012] The sample augmentation module extracts normal samples and rare fault samples from the multimodal feature library, combines them with the elevator safety status judgment results to generate synthetic samples and performs quality screening to obtain an augmented training dataset.

[0013] The model building module trains a multi-level fault identifyer based on the enhanced training dataset, and performs optimization learning by integrating a multi-modal feature library to obtain a fault identification model;

[0014] The intelligent early warning module calculates the safety level and infers the cause of the fault and the handling measures based on the safety status judgment results, progressive fault identification results and fault identification model, and obtains graded early warning information and maintenance suggestions.

[0015] In a preferred embodiment, the data preprocessing module includes:

[0016] Install cameras to collect visual image sequences, install accelerometers to collect vibration signals, deploy microphone arrays to collect sound signals, and obtain operating parameter data through the elevator control system data interface;

[0017] Deploy a clock synchronization module and use a network time protocol for clock synchronization. Sensors that cannot directly obtain a unified timestamp will perform time alignment through hardware trigger signals.

[0018] Resampling is performed on sensor data with different sampling frequencies. For data with sampling frequencies lower than the unified time resolution, an interpolation method is used for upsampling, and for data with sampling frequencies higher than the unified time resolution, a sliding window averaging method is used for downsampling.

[0019] The visual image data is normalized for illumination, the vibration signal data is de-electrified and de-electrified for low-frequency drift and high-frequency noise, the sound signal data is pre-emphasized and then framed, and the operating parameter data is de-electrified and de-electrified for abrupt noise and missing values ​​are filled.

[0020] In a preferred embodiment, the feature extraction module includes:

[0021] A multi-level operating condition identification strategy is adopted. The first level is based on speed and acceleration for coarse-grained classification, the second level is based on the speed change trend within a time window for medium-grained classification, and the third level is based on the car load parameters within a time window for fine-grained classification. The elevator operating status is divided into nine operating conditions: no-load acceleration, no-load constant speed, no-load deceleration, half-load acceleration, half-load constant speed, half-load deceleration, full-load acceleration, full-load constant speed, and full-load deceleration.

[0022] For visual image sequences, a multi-branch convolutional neural network is used to extract multi-scale spatial features, and a bidirectional long short-term memory network is used to extract temporal features.

[0023] For vibration signals, wavelet packet decomposition is used to extract multi-band features, empirical mode decomposition is used to extract intrinsic mode functions, and statistical features of instantaneous frequency and instantaneous amplitude are calculated.

[0024] For sound signals, Mel frequency cepstral coefficients and differential features are extracted, and short-time energy, zero-crossing rate, and spectral entropy are calculated.

[0025] In a preferred embodiment, the multimodal fusion module includes:

[0026] Deep neural network anomaly detectors are constructed for the three modalities of vision, vibration, and sound, and the probability distribution of the output safe state is used as the basic probability allocation function.

[0027] Calculate the quality assessment index and historical accuracy of each modality data, and calculate the reliability coefficient of each modality by weighted summation;

[0028] The basic probability assignment function is discounted using a reliability coefficient.

[0029] An improved Dempster combination rule is used for evidence synthesis. When the conflict coefficient exceeds the preset conflict threshold, the abnormal evidence sources are discounted twice and the evidence is recombined.

[0030] In a preferred embodiment, the anomaly detection module includes:

[0031] Based on the data before and after the switching of operating conditions, the peak change rate, spectral centroid shift and energy mutation index of the vibration signal are calculated, the loudness change gradient of the sound signal is calculated, and they are organized into transient feature vectors.

[0032] Gaussian mixture model is used to model the probability distribution of transient characteristics for each type of working condition transition, and a normal mode library of working condition transition is constructed.

[0033] The transient feature vector is input into the corresponding model to calculate the log-likelihood probability. When it is lower than the anomaly detection threshold, it is judged as a transient anomaly.

[0034] Establish a time-series correlation database for abnormal signals, and use time-series pattern mining algorithms to analyze the evolution trend of abnormalities and identify progressive and acute faults.

[0035] In a preferred embodiment, the sample enhancement module includes:

[0036] A conditional generative adversarial network is constructed. The generator takes random noise vector, fault type code and working condition type code as input, and outputs fault sample feature vector. The discriminator outputs the true and false judgment.

[0037] Add a physical constraint layer after the generator output layer to impose range and logical constraints on the generated feature values;

[0038] For rare fault types, synthetic samples are generated under different operating conditions. Quality screening is performed based on feature similarity, and synthetic samples with similarity higher than a preset quality threshold are retained. Clustering methods are used to select representative samples.

[0039] In a preferred embodiment, the model building module includes:

[0040] Modal-specific recognizers were constructed for the three modalities of vision, vibration, and sound, and training employed a focus loss function and a class-balanced sampling strategy.

[0041] A modality fusion recognizer is constructed, and the output probability vectors of each modality-specific recognizer are concatenated and then used with a multilayer perceptron for feature fusion and classification.

[0042] An integrated decision-maker is constructed using the gradient boosting decision tree algorithm, with the output of the modality fusion recognizer and the original multimodal features as inputs for optimization.

[0043] The model is then quantized and compressed before being deployed to edge computing devices.

[0044] In a preferred embodiment, the intelligent early warning module includes:

[0045] Establish an elevator safety knowledge graph; build a multi-level safety status assessment model based on the fuzzy comprehensive evaluation method, calculate the comprehensive score and classify the safety level;

[0046] For detected anomalies and faults, reasoning queries are performed in the knowledge graph to obtain the causes of the faults and corresponding solutions.

[0047] Based on the safety level, a graded early warning strategy is generated and the elevator control system is activated to initiate protective measures.

[0048] In a preferred embodiment, the intelligent early warning module further includes:

[0049] Establish a health status file for each elevator and record data throughout its entire life cycle;

[0050] Based on health record data, a degradation model is used to predict the remaining service life of components. A Weibull distribution model is used to describe the failure time distribution of components. A Bayesian update method is used to correct the distribution parameters and calculate the probability of component failure and the expected time for performance degradation to the failure threshold.

[0051] Predictive maintenance plans are generated based on the remaining useful life prediction results.

[0052] In a preferred embodiment, the security knowledge graph includes device entity, fault type entity, fault cause entity, fault phenomenon entity, and handling measure entity, and the relationships include causal relationship, subordinate relationship, manifestation relationship, and handling relationship;

[0053] The multi-level security status assessment model includes a detection dimension layer, a subsystem layer, and an overall security layer.

[0054] The tiered early warning strategy includes logging in the normal state, sending prompt messages in the alert state, triggering audible and visual alarms in the warning state, and activating protective measures in the linkage control system in the dangerous state.

[0055] The beneficial effects of this invention are as follows:

[0056] By integrating multi-dimensional data sources such as visual images, vibration signals, sound signals, and operating parameters, an adaptive multimodal data fusion mechanism was constructed, enabling precise dynamic safety monitoring of elevators in super high-rise buildings under extreme load switching conditions.

[0057] By establishing a unified time reference system and resampling mechanism, the time synchronization problem of sensor data with different sampling frequencies was solved, ensuring accurate correlation of multimodal data. A multi-level operating condition recognition strategy was designed to achieve rapid and accurate classification of elevator operating states, laying the foundation for adaptive feature extraction. Advanced methods such as wavelet packet decomposition, empirical mode decomposition, and multi-scale convolutional neural networks were employed to achieve deep feature extraction from different modalities, fully mining the safety information contained within each modality. An adaptive fusion method based on improved evidence theory dynamically adjusts the fusion weights according to the real-time reliability of each modality, effectively addressing the challenge of data quality fluctuations under complex operating conditions and improving the accuracy and robustness of fusion judgments.

[0058] By employing a cross-condition sliding time window and time-series correlation analysis, the system effectively captures transient abnormal signals and identifies progressive faults early during condition switching, overcoming the shortcomings of traditional steady-state monitoring methods. Rare fault samples are generated using conditional generative adversarial networks, resolving the problem of imbalanced fault sample categories and improving the model's ability to identify rare faults. By constructing a multi-level fault identification model and a knowledge graph-based intelligent early warning system, the system achieves intelligent management throughout the entire process from data acquisition to fault diagnosis and maintenance decision-making, providing strong support for the safe operation and predictive maintenance of elevators. Attached Figure Description

[0059] Figure 1 This is a block diagram of the present invention;

[0060] Figure 2 This is a detailed flowchart of the present invention. Detailed Implementation

[0061] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0062] At least one embodiment of the present invention discloses an elevator safety detection system based on visual and multimodal data fusion, such as... Figures 1 to 2 As shown, it includes:

[0063] The data preprocessing module collects raw multimodal sensor data from elevator operation and preprocesses it to obtain a multimodal preprocessed dataset.

[0064] Specifically, it includes the following:

[0065] Step 1.1: Deploy multiple types of sensors at key locations in the elevator system to collect raw data;

[0066] A high-definition wide-angle camera with a resolution of 1920×1080 pixels is installed at the center of the elevator car's top, covering the entire interior space. The camera captures images at 30 frames per second, recording visual information such as passenger behavior, the status of items inside the car, and the door area environment. Three-axis accelerometers are installed at three locations: the traction machine base, the guide rail support, and the bottom of the car. These sensors measure from -16g to +16g and are set to a sampling frequency of 1000 times per second, collecting vibration signals generated during elevator operation. The sensors at these three locations reflect the vibration characteristics of the traction system, the guiding system, and the car system, respectively. Microphone arrays are positioned at the four corners of the car's top, with a sampling frequency of 16,000 times per second, to collect sound signals generated during elevator operation, including traction machine noise, guide rail friction noise, and door opening and closing sounds. The elevator's operating parameters, including current floor, direction of travel, real-time speed, real-time acceleration, car load weight, and door status, are acquired in real-time through the elevator control system's data interface, with a data update frequency of 10 times per second. All raw data collected by the sensors are transmitted via wired or wireless means to edge computing devices installed in the elevator shaft for processing.

[0067] Step 1.2: Establish a unified time reference system to achieve high-precision time synchronization of multimodal data;

[0068] Because different types of sensors have different sampling frequencies (30 frames per second for visual data, 1000 times per second for vibration data, 16000 times per second for sound data, and 10 times per second for operational parameter data), directly using the timestamps of each sensor would lead to inconsistent time bases, affecting the correlation analysis of multimodal data. To solve this problem, a high-precision clock synchronization module is deployed in the edge computing device, using a network time protocol to synchronize with an external standard time server. The synchronization period is set to once every 10 minutes to ensure the long-term stability of the local clock. Each sensor node reads a unified timestamp provided by the edge computing device while acquiring data; the timestamp accuracy is at the microsecond level. For sensors that cannot directly obtain a unified timestamp, time alignment is achieved through hardware trigger signals. The edge computing device sends a trigger pulse signal to each sensor at fixed time intervals. When the sensor receives the trigger signal, it records its current local time. Time synchronization between different sensors is achieved through the timestamp of the trigger signal.

[0069] Step 1.3: Resample sensor data at different sampling frequencies to achieve a unified time resolution;

[0070] To facilitate the fusion and processing of multimodal data, data with different sampling frequencies need to be converted to a uniform time resolution. A uniform time resolution of 100 sampling points per second is chosen, which can capture the dynamic changes in elevator operation while balancing computational resource consumption.

[0071] For data with a sampling frequency below 100 times per second, including visual data at 30 frames per second and operational parameter data at 10 frames per second, cubic spline interpolation is used for upsampling. Cubic spline interpolation constructs a piecewise cubic polynomial function to ensure the continuity of the function values, first derivative, and second derivative at the nodes of the interpolation curve, thus achieving smooth data interpolation. Specifically, for visual data, an intermediate frame is generated by interpolation between two adjacent frames. The interpolation process considers the spatial location and temporal variation trend of pixels, uses optical flow to estimate the motion trajectory of pixels, and calculates the interpolated pixel values ​​based on the motion trajectory. For operational parameter data, cubic spline interpolation is directly performed on numerical parameters, while for discrete parameters such as floor and door status, a forward fill method is used, that is, the value of the previous sample is kept unchanged at the time point between two samplings.

[0072] For data with a sampling frequency higher than 100 times per second, including vibration data at 1000 times per second and sound data at 16000 times per second, a sliding window averaging method is used for downsampling. Specifically, for vibration data, a sliding window with a length of 10 sampling points is used, and the sampling points within the window are arithmetically averaged to obtain the output of one sampling point. The window slides by 10 sampling points each time, thereby reducing the sampling frequency from 1000 times per second to 100 times per second. For sound data, a sliding window with a length of 160 sampling points is used, and the sampling points within the window are arithmetically averaged. The window slides by 160 sampling points each time, reducing the sampling frequency from 16000 times per second to 100 times per second.

[0073] Step 1.4: Perform targeted adaptive preprocessing on multimodal data to improve data quality;

[0074] For visual image data, since the lighting conditions inside the car vary significantly with time and location, illumination normalization is required to improve image consistency. An adaptive histogram equalization method is employed, dividing the image into 8×8 sub-blocks. Histogram equalization is performed on each sub-block separately, and the equalization results of each sub-block are smoothly blended using bilinear interpolation to obtain the overall equalized image. This method enhances local image contrast and improves the visibility of dark areas.

[0075] Furthermore, a multi-scale Retinex algorithm is employed for illumination normalization. Based on an object reflection model, the Retinex algorithm decomposes the image into reflection and illumination components, removing the illumination component while retaining the reflection component to achieve illumination invariance. Specifically, the image is convolved with Gaussian kernels of three different scales to obtain illumination estimates at three different scales. The original image is then divided by the weighted average of the illumination estimates to obtain the illumination-normalized image.

[0076] For vibration signal data, the raw signal contains low-frequency drift and high-frequency noise components, requiring filtering. A fourth-order Butterworth bandpass filter is used, with a passband frequency range of 0.5 Hz to 500 Hz. Signal components below 0.5 Hz mainly originate from sensor temperature drift and circuit noise, while signal components above 500 Hz mainly originate from electromagnetic interference and sampling aliasing noise. The bandpass filter removes these irrelevant components, retaining the effective signal related to elevator mechanical vibration. The Butterworth filter has a flat amplitude-frequency response and good phase-frequency response within the passband, enabling it to remove noise while preserving the signal's waveform characteristics.

[0077] For audio signal data, a pre-emphasis filter is used to enhance high-frequency components and facilitate subsequent feature extraction. The transfer function of the pre-emphasis filter is 1 minus 0.97 times the unit delay. This filter performs a first-order high-pass filter on the signal, increasing the energy of high-frequency components and compensating for high-frequency attenuation during audio signal propagation. The pre-emphasized audio signal is then framed using a Hamming window with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. The Hamming window reduces discontinuities at frame boundaries and minimizes spectral leakage.

[0078] For the operating parameter data, since the sensor occasionally exhibits abnormal fluctuations, median filtering is used to remove abrupt noise. The median filter window length is set to 5 sampling points. The values ​​within the window are sorted, and the median value is taken as the output. This method has a good suppression effect on impulse noise. For missing values ​​in the operating parameter data, Lagrange interpolation is used for imputation. A Lagrange interpolation polynomial is constructed based on the three known sampling points before and after the missing value, and the parameter value at the missing time is calculated.

[0079] After the above preprocessing, a multimodal preprocessed dataset is obtained. The visual image sequences, vibration signal time series data, sound signal frame data, and running parameter vectors in this dataset all have a uniform time resolution.

[0080] In some embodiments, since sensor data may be lost or damaged when the elevator operates in extreme environments, a deep learning-based data repair method can be adopted to improve the robustness of the system in the event of sensor failure. Specifically, a spatiotemporal correlation model of multimodal data is constructed. Using data from other normal sensors and historical data, the spatiotemporal correlation between different modal data is learned through a long short-term memory network. When a continuous loss or abnormal jump in data from a certain sensor is detected, the trained correlation model is used to infer based on the current data from other sensors and the historical normal data of that sensor to generate estimated data for the missing period, thereby achieving intelligent data repair and ensuring the integrity and continuity of the multimodal dataset.

[0081] The feature extraction module identifies operating conditions based on a multimodal preprocessed dataset, extracts multi-scale features for different operating conditions, and obtains a multimodal feature library and operating condition identification sequences.

[0082] Specifically, it includes the following:

[0083] Step 2.1: Design a multi-level operating condition recognition strategy to achieve fast and accurate elevator operating status classification;

[0084] The operating conditions of an elevator directly affect the feature distribution and abnormal patterns of various modal data. Accurate operating condition identification is a prerequisite for achieving adaptive feature extraction and fusion. A three-level operating condition identification strategy is designed, with a progressive classification from coarse to fine between levels, ensuring both real-time identification and classification accuracy.

[0085] The first level is a coarse-grained rapid classification, which judges based on the instantaneous values ​​of real-time speed and acceleration in the elevator's operating parameters. When the speed is zero, it is judged as stationary; when the speed is greater than zero and the direction is upward, it is judged as ascending; and when the speed is greater than zero and the direction is downward, it is judged as descending. The judgment delay of this level is only 0.1 seconds, which can quickly respond to the switching of elevator status.

[0086] The second level is a medium-granularity trend classification. Based on the judgment of the first level, it further classifies the speed change trend within a 3-second time window. It calculates the first derivative of the speed within the window. When the speed derivative is greater than the first threshold, it is determined to be an acceleration state. When the absolute value of the speed derivative is less than the second threshold, it is determined to be a constant speed state. When the speed derivative is less than the negative first threshold, it is determined to be a deceleration state. This level can distinguish the dynamic change stages of the elevator during operation.

[0087] The third level is fine-grained load classification, which analyzes the car load parameters within a 5-second time window. Based on the elevator's rated load capacity, the load is divided into three levels: when the load is less than 30% of the rated load capacity, it is judged as unloaded; when the load is between 30% and 70% of the rated load capacity, it is judged as half-loaded; and when the load is greater than 70% of the rated load capacity, it is judged as fully loaded. At the same time, the running distance calculated by speed integral is combined to correct the load judgment and avoid misjudgment caused by short-term load fluctuations.

[0088] Through a three-tiered progressive assessment, the elevator's operating status is precisely categorized into nine typical conditions: no-load acceleration, no-load constant speed, no-load deceleration, half-load acceleration, half-load constant speed, half-load deceleration, full-load acceleration, full-load constant speed, and full-load deceleration. A sliding window and incremental update mechanism are employed, updating the statistics within the window and recalculating the condition assessment results each time new data arrives, enabling continuous tracking of operating conditions. The identified operating condition types are used as labels, linked to timestamps, and organized chronologically to form an operating condition identification sequence for subsequent feature extraction and fusion analysis. This operating condition identification sequence records the trajectory of operating condition types over time during elevator operation, providing foundational data for cross-condition time-series correlation analysis.

[0089] Step 2.2: Design a multi-scale convolutional neural network to extract spatial and temporal features for visual image sequences;

[0090] Visual images contain rich information about the interior environment of the car, including the number of passengers, passenger behavior, the accumulation of items, obstacles in the door area, etc. Features at different scales reflect different levels of semantic information.

[0091] A multi-branch convolutional neural network is constructed for multi-scale spatial feature extraction. The network contains three parallel convolutional branches. The first branch uses a 7×7 convolutional layer with a large receptive field to extract global scene features, including macroscopic information such as overall illumination distribution and personnel density. The second branch uses a 3×3 convolutional layer with a medium receptive field to extract local target features, including the position, pose, and movement trends of individual passengers. The third branch uses a 1×1 convolutional layer to extract fine-grained texture features, including clothing texture and surface features of objects. Each of the three branches has a convolutional layer depth of 5 layers, followed by a batch normalization layer and a ReLU activation function to improve the network's training stability and feature representation capability. The features extracted by the three branches are concatenated along the channel dimension to obtain a fused multi-scale spatial feature vector.

[0092] To capture the temporal dependencies of visual information, multi-scale spatial features extracted from 10 consecutive frames of images were organized into a sequence in chronological order and input into a bidirectional long short-term memory (LSTM) network for temporal feature extraction. The LSTM network contains forward and backward information transmission paths. The forward path transmits information from the beginning to the end of the time series, capturing the influence of the past on the present; the backward path transmits information from the end to the beginning of the time series, capturing the influence of the future on the present. The hidden states from both directions are concatenated to obtain a feature representation containing bidirectional temporal context. The hidden layer dimension of the LSTM network is set to 128, and a gating mechanism is used to selectively retain and forget historical information, effectively mitigating the gradient vanishing problem in long sequence modeling. The fused multi-scale spatial features and bidirectional temporal features are combined to obtain a visual spatiotemporal feature vector, which comprehensively expresses the spatial layout and temporal evolution information of the car's interior environment.

[0093] Step 2.3: Wavelet packet decomposition and empirical mode decomposition are used to extract multi-band and modal features from the vibration signal time series data;

[0094] Vibration signals reflect the operating status of elevator mechanical systems. Vibration energy in different frequency bands corresponds to the vibration characteristics of different components, and different modal components reflect the inherent oscillation modes of vibration signals.

[0095] Wavelet packet decomposition was employed for multi-band analysis of the vibration signal. The Daubechies wavelet was selected as the wavelet basis function due to its good time-frequency localization and orthogonality. A five-level wavelet packet decomposition was performed, dividing the original vibration signal into 32 sub-signals in different frequency bands, each with a bandwidth of approximately 15 Hz, covering an effective frequency range from 0.5 Hz to 500 Hz. Energy and frequency characteristics were calculated for each sub-signal. The energy characteristic was obtained by calculating the logarithm of the sum of squares of the sub-signals, reflecting the vibration intensity of that band. The frequency characteristic was calculated using Fast Fourier Transform to determine the power spectrum of the sub-signals, identifying the peak frequency, which reflects the dominant vibration frequency of that band. The energy and peak frequency characteristics of the 32 frequency bands were organized into a 64-dimensional wavelet packet feature vector.

[0096] An empirical mode decomposition (EMD) method is used for adaptive time-frequency analysis of vibration signals. EMD decomposes the signal into a series of intrinsic mode functions (EMFs) from high to low frequencies and a residual component through an iterative screening process. Each EMF represents an intrinsic oscillation mode of the signal. Specifically, all local maxima and minima of the signal are identified. Maximum and minimum envelopes are fitted using cubic spline interpolation, and the mean of the two envelopes is calculated as the local mean. The original signal is subtracted from the local mean to obtain a candidate EMF. The candidate EMF is then evaluated to determine if it meets the conditions for an EMF: the number of extrema differs from the number of zero-crossings by at most one, and the mean of the local maximum and minimum envelopes is close to zero at any given time. If these conditions are met, the candidate EMF is considered an EMF; otherwise, the candidate EMF is used as a new original signal, and the screening process is repeated until the conditions are met. Subtract the extracted intrinsic mode functions from the original signal, and repeat the above decomposition process on the remaining signal to extract multiple intrinsic mode functions in turn, until the remaining signal becomes a monotonic function or has fewer than two extreme points.

[0097] The first eight intrinsic mode functions (EMFs) are extracted. For each EMF, the instantaneous frequency and instantaneous amplitude are calculated. The instantaneous frequency is then transformed using Hilbert transform to obtain the analytic signal of the EMF. The time derivative of the phase of the analytic signal is the instantaneous frequency, and the instantaneous amplitude is the magnitude of the analytic signal. The mean instantaneous frequency and mean instantaneous amplitude, as well as the standard deviation of the instantaneous frequency and standard deviation of the instantaneous amplitude, are calculated for each EMF, resulting in a 32-dimensional empirical mode feature vector. The wavelet packet feature vector and the empirical mode feature vector are concatenated to obtain a 96-dimensional comprehensive feature of the vibration signal.

[0098] Step 2.4: Extract Mel frequency cepstral coefficients and time-domain statistical features from the framed data of the audio signal;

[0099] The sound signal contains the acoustic characteristics generated by various components during elevator operation, which is of great value for identifying mechanical abnormalities and operating status.

[0100] For each frame of the audio signal, a Fast Fourier Transform (FFT) is performed to obtain the spectrum. Mel frequency cepstral coefficients are then extracted using a Mel filter bank. The Mel filter bank simulates the auditory characteristics of the human ear, with narrower bandwidth in the low-frequency region and wider bandwidth in the high-frequency region, consistent with the human ear's ability to distinguish different frequencies. The Mel filter bank contains 26 triangular filters, covering a frequency range from 0 Hz to 8000 Hz. The signal spectrum is weighted and summed through each Mel filter to obtain energy values ​​for the 26 Mel frequency bands. The logarithm of these energy values ​​is then followed by a Discrete Cosine Transform (DCT) to obtain the Mel frequency cepstral coefficients. The first 13 coefficients are retained; these 13 coefficients reflect the spectral envelope characteristics of the audio signal.

[0101] Further calculations were performed on the first and second-order differences of the Mel-frequency cepstral coefficients. The first-order difference reflects the rate of change of the spectral envelope over time, while the second-order difference reflects the acceleration of this rate of change. Each of the first and second-order differences contains 13 coefficients, and together they constitute a 39-dimensional Mel-frequency cepstral coefficient feature. For each frame of the audio signal, time-domain statistical features were also calculated, including short-time energy, zero-crossing rate, and spectral entropy. Short-time energy is obtained by calculating the sum of squares of all sampling points within the frame, reflecting the intensity of the audio signal. The zero-crossing rate is obtained by counting the number of times the signal waveform crosses zero points within the frame, reflecting the frequency characteristics of the audio signal; high-frequency signals have a higher zero-crossing rate. Spectral entropy is obtained by calculating the information entropy of the normalized power spectrum, reflecting the spectral complexity of the audio signal; signals with a single frequency component have a lower spectral entropy, while signals with multiple frequency components have a higher spectral entropy. The Mel-frequency cepstral coefficient features, short-time energy, zero-crossing rate, and spectral entropy were organized into a 42-dimensional audio feature vector.

[0102] Step 2.5: Establish a multimodal feature library related to operating conditions to achieve structured storage of features;

[0103] The operating condition labels identified in step 2.1, the visual spatiotemporal feature vectors extracted in step 2.2, the comprehensive vibration signal features extracted in step 2.3, and the sound feature vectors extracted in step 2.4 are associated in a time-aligned manner to construct feature data records. Each feature data record contains a timestamp, an operating condition label, and feature vectors for each modality. Feature data records are categorized and stored according to the operating condition labels, establishing feature sub-libraries for nine operating condition categories. For each operating condition sub-library, feature data is further organized chronologically to support time-series queries and statistical analysis. A feature index structure is established to support rapid retrieval based on time range, operating condition type, and feature similarity. The feature library is maintained regularly, deleting expired historical feature data and retaining feature data from the most recent period for real-time detection and analysis.

[0104] After the above processing, a multimodal feature library and a working condition recognition sequence were obtained, which were classified by working condition and organized by time. The multimodal feature library not only contains rich information on multiple modes such as vision, vibration, and sound, but also reflects the differences in feature distribution under different working conditions. The working condition recognition sequence records the trajectory of the elevator's working condition changes. Together, they provide high-quality data input for subsequent adaptive fusion and anomaly detection.

[0105] In some embodiments, due to differences in the mechanical characteristics and installation environment of different elevator equipment, directly using uniform feature extraction parameters may result in inaccurate feature representation. Therefore, an adaptive feature extraction method can be adopted to improve the adaptability of feature extraction to different equipment. Specifically, when the system is first deployed to a new elevator equipment, a 7-day self-learning phase is implemented. During this phase, the system collects multimodal data of the elevator under various operating conditions during normal operation, analyzes the statistical characteristics of each modal data under different operating conditions, including the dominant frequency range of vibration signals, the energy distribution range of sound signals, and the illumination conditions of visual images. Based on these statistical characteristics, the feature extraction parameters are adaptively adjusted. For example, the number of decomposition layers of wavelet packet decomposition is adjusted to match the dominant frequency range, the frequency distribution of the Mel filter bank is adjusted to match the sound energy distribution, and the enhancement parameters of image preprocessing are adjusted to match the illumination conditions, thereby achieving personalized optimization of the feature extraction strategy for specific equipment.

[0106] The multimodal fusion module calculates the reliability coefficients of each mode in the multimodal feature library and performs adaptive fusion to obtain the elevator safety status judgment result;

[0107] Specifically, it includes the following:

[0108] Step 3.1: Construct a deep neural network anomaly detector for each modality to generate a basic probability assignment function;

[0109] Evidence theory represents uncertainty as a basic probability assignment function, which distributes probability mass across subsets of the proposition set. Compared to traditional probability representations, evidence theory can distinguish between uncertainty and unknowns, making it more suitable for handling multi-source information fusion problems. Dedicated deep neural network anomaly detectors are constructed for visual, vibration, and sound modalities.

[0110] The visual modality detector takes the visual spatiotemporal feature vector extracted in step 2 as input. Its network structure includes three fully connected layers with 256, 128, and 64 neurons per layer, respectively. The ReLU activation function is used. The output layer contains four neurons, corresponding to four safe states: normal, mildly abnormal, moderately abnormal, and severely abnormal, representing four severity levels: no abnormality, slight deviation, significant deviation, and significant fault, respectively. The softmax activation function is used to convert the output into a probability distribution. The vibration modality detector takes the comprehensive vibration signal features extracted in step 2 as input, and its network structure is the same as the visual modality detector. The sound modality detector takes the sound feature vector extracted in step 2 as input, and its network structure is consistent with other modality detectors.

[0111] The training data for the three detectors comes from historical normal operation data and labeled abnormal data. The cross-entropy loss function is used for training, and the Adam algorithm is employed as the optimizer. The learning rate is set to 0.001, and the batch size is set to 32. After training, for the input feature vector, the detector outputs the probability distribution of four safe states. This probability distribution is the basic probability assignment function for that modality.

[0112] Step 3.2: Calculate the reliability coefficient of each modality based on data quality indicators and historical accuracy.

[0113] The reliability of different modal data varies under different operating conditions and environmental conditions, and it is necessary to quantitatively evaluate the reliability of each modality.

[0114] For visual modalities, image quality assessment metrics are calculated, including sharpness, contrast, and information entropy. Sharpness is obtained by calculating the variance of the Laplacian operator response of the image; higher sharpness indicates a sharper image. Contrast is obtained by calculating the standard deviation of the image's gray-level histogram; higher contrast indicates richer tonal gradations in the image. Information entropy is obtained by calculating the Shannon entropy of the image's gray-level histogram; higher information entropy indicates a greater amount of information contained in the image. Sharpness, contrast, and information entropy are normalized and then weighted to obtain a quality score for the visual data.

[0115] For vibration modes, the signal-to-noise ratio (SNR) and integrity of the signal are calculated. The SNR is obtained by calculating the ratio of the effective power of the signal to the noise power. A spectral analysis method is used, treating frequency bands with concentrated energy as effective components and frequency bands with dispersed energy as noise. The power ratio of the two is calculated and converted to a decibel value. Integrity is obtained by comparing the number of signal sampling points per unit time with the theoretical number of sampling points. When the sensor malfunctions or communication is abnormal, sampling points will be missing. The integrity index reflects the reliability of the data. After normalizing the SNR and integrity, a weighted average is taken to obtain the quality score of the vibration data.

[0116] For sound modes, the calculation method is similar to that for vibration modes, yielding a quality score for the sound data. The detection results of each modality detector and manually labeled ground truth data from the feature library over a recent period are extracted. The historical accuracy of each modality detector is calculated, defined as the ratio of the number of correctly identified samples to the total number of samples. The comprehensive reliability coefficient for each modality is calculated by weighted summing of the data quality score and the historical accuracy. The weight of the quality score is set to 0.6, and the weight of the historical accuracy is set to 0.4. The reliability coefficient ranges from 0 to 1, with a higher value indicating a more reliable modality.

[0117] Step 3.3: Apply a reliability coefficient to the basic probability assignment function for discounting.

[0118] In evidence theory, the discounting operation is used to adjust the credibility of evidence sources. For evidence sources with low reliability, the discounting operation reduces their impact on the fusion result and increases the allocation of uncertainty.

[0119] The specific method of discounting is as follows: For each specific security state proposition, its original probability assignment value is multiplied by the reliability coefficient of that modality to obtain the discounted probability assignment value. For propositions representing complete uncertainty across the entire set, the discounted probability assignment value is equal to the original probability assignment value multiplied by the reliability coefficient, plus the complement of the reliability coefficient, i.e., one minus the reliability coefficient. The physical meaning of the discount rule is that the probability quality corresponding to the complement of the reliability coefficient is transferred from the specific proposition to the entire set, indicating an increase in uncertainty caused by distrust of the source of evidence. The discount operation is applied to the basic probability assignment functions of the three modalities respectively to obtain the discounted basic probability assignment functions.

[0120] Step 3.4: Use the improved Dempster combination rule to synthesize evidence;

[0121] Dempster's combination rule is a core fusion method in evidence theory. It achieves evidence synthesis by calculating the joint support of different evidence sources for each proposition. The basic principle of the combination rule is as follows: For any safe state proposition, iterate through all proposition pairs between the first and second evidence sources. When the intersection of the two propositions equals the target proposition, multiply and sum the probability assignments of the two propositions to obtain the joint support, and then normalize it using a normalization factor. The normalization factor is obtained by subtracting the conflict coefficient from one. The conflict coefficient is calculated as follows: iterate through all proposition pairs between the two evidence sources. When the intersection of the two propositions is empty, multiply and sum the probability assignments of the two propositions. The conflict coefficient reflects the degree of contradiction between different evidence sources. When the conflict coefficient is close to 1, it indicates a serious conflict between the evidence sources. The normalization factor of the traditional Dempster rule approaches zero, leading to unstable or even invalid fusion results.

[0122] To address this issue, a conflict adaptive adjustment mechanism is introduced. Specifically, a conflict threshold of 0.8 is set. When the calculated conflict coefficient exceeds 0.8, conflict analysis is performed on the evidence sources participating in the fusion. The similarity between any two evidence sources is calculated, and the similarity is obtained by calculating the cosine distance between two basic probability assignment functions. The smaller the cosine distance, the higher the similarity. Abnormal evidence sources with low similarity to other evidence sources are identified. The reliability coefficient of abnormal evidence sources is discounted twice, reducing it to half of its original value. The discount operation is then reapplied to reduce the impact of abnormal evidence. After the second discount, the conflict coefficient is recalculated. If the conflict coefficient still exceeds the conflict threshold, the most abnormal evidence source is excluded, and only the remaining evidence sources are fused.

[0123] Using the improved combination rule described above, the discounted evidence from the visual and vibration modes is combined pairwise. The combined results are then combined with the discounted evidence from the sound mode to obtain the final fusion basic probability assignment function. The proposition with the highest probability assignment from the fusion basic probability assignment function is selected as the fusion judgment result, and the confidence level of this judgment is calculated. The confidence level is equal to the probability assignment value of the proposition.

[0124] After the above processing, the elevator safety status judgment result based on comprehensive multimodal information was obtained. This result fully considers the reliability differences of each modal data and the conflict between modalities, and has high accuracy and robustness.

[0125] In some embodiments, due to the complexity of the elevator operating environment, the data quality of all modes may be unsatisfactory at certain times. A judgment correction method based on temporal consistency can be adopted. The purpose is to improve the reliability of the current judgment by utilizing historical judgment results. Specifically, a judgment history queue of length 10 is maintained to record the safety status judgment results of the most recent 10 moments. When the judgment confidence of the current moment is lower than the third threshold, the frequency of each state in the judgment history queue is analyzed. If the frequency of a certain state in the queue exceeds 70%, and the state is different from the current judgment result, it is considered that the current judgment may be affected by instantaneous data fluctuations. The judgment result is then corrected to the high-frequency state in the queue, thereby improving the stability of the judgment by utilizing temporal smoothing.

[0126] The anomaly detection module, based on the multimodal preprocessing dataset and operating condition identification sequence, extracts transient features of operating condition switching and makes anomaly judgments, mines the anomaly evolution trend, and obtains progressive fault identification results.

[0127] Specifically, it includes the following:

[0128] Step 4.1: Design a sliding time window across operating conditions to extract transient features of the operating condition switching process;

[0129] During elevator operation switching, the stress state of mechanical components changes abruptly, which can easily generate transient abnormal signals. These signals are short in duration but contain important fault information.

[0130] A 10-second sliding time window is used, with the center of the window aligned with the moment of operation condition switching, covering data from 5 seconds before and 5 seconds after the switching. The window slides in 1-second increments to achieve continuous transient monitoring. For the vibration signal within the window, signal segments are extracted within 1 second before and after the operation condition switching moment. The peak value of each signal segment is calculated, and the peak value after the switching is subtracted from the peak value before the switching and then divided by the peak value before the switching to obtain the vibration peak value change rate.

[0131] Perform a short-time Fourier transform on the vibration signal within the window to obtain the time spectrum. Calculate the average spectrum 1 second before and 1 second after the switch. Calculate the spectral centroid of the two time periods using the weighted centroid formula. The spectral centroid is defined as the sum of the products of each frequency and its power divided by the total power. Subtract the spectral centroid before the switch from the spectral centroid after the switch to obtain the spectral centroid drift.

[0132] The frequency range of the vibration signal is divided into three frequency bands: low frequency, medium frequency, and high frequency, corresponding to 0.5 to 50 Hz, 50 to 200 Hz, and 200 to 500 Hz, respectively. The energy of each frequency band before and after switching is calculated. The energy is obtained by integrating the power spectrum within the frequency band. The energy of each frequency band after switching is subtracted from the energy of the corresponding frequency band before switching and then divided by the energy before switching to obtain the energy mutation index of each frequency band.

[0133] For the sound signal within the window, calculate the sound loudness for 1 second before and after the switching condition. The loudness is obtained by calculating the average value of the short-time energy. Subtract the loudness before the switching from the loudness after the switching to obtain the loudness change. Then divide by the switching time interval to obtain the sound loudness change gradient.

[0134] The peak vibration rate of change, the spectral centroid shift, the energy mutation index of the three frequency bands, and the sound loudness gradient are organized into a 7-dimensional transient feature vector.

[0135] Step 4.2: Establish a normal mode library for operating condition conversion based on normal operating data;

[0136] To determine whether the extracted transient features are abnormal, it is necessary to establish a feature distribution model for normal operating condition transitions.

[0137] Data from historical elevator operation periods was selected. Based on the operating condition identification sequence, all operating condition transition events were extracted. Potential operating condition transitions during elevator operation include changes from standstill to acceleration, acceleration to constant speed, constant speed to deceleration, and deceleration to standstill, as well as changes in load status. For each type of operating condition transition, transient feature vectors were extracted to form a feature sample set for that type. A Gaussian mixture model (GMM) was used to model the probability distribution of the feature sample set for each operating condition transition type. The GMM assumes that the feature vectors follow a linear combination of multiple Gaussian components, allowing for flexible fitting of complex distribution patterns. The number of components in the GMM was selected through cross-validation, ranging from 2 to 10, with the number of components maximizing the log-likelihood chosen. The GMM was trained using the expectation-maximization algorithm, iteratively updating the mean vector, covariance matrix, and mixing coefficients of each Gaussian component until the model converged. The trained GMM was then stored as the normal mode for the corresponding operating condition transition type in a normal mode library.

[0138] Step 4.3: Calculate the likelihood probability of transient features in real time to determine anomalies;

[0139] For operating condition transition events detected during real-time monitoring, the corresponding Gaussian mixture model is retrieved from the normal operating condition transition model library based on the transition type. The extracted transient feature vector is then input into this model, and the log-likelihood probability of the feature vector is calculated. The formula for calculating the log-likelihood probability is: the log-likelihood equals the sum of the mixing coefficients of each Gaussian component multiplied by the logarithm of the probability density function values ​​of that component at the feature vector, where the probability density function is the density function of a multivariate Gaussian distribution. The higher the log-likelihood probability, the more the current transient feature conforms to the normal pattern.

[0140] A dynamic anomaly detection threshold is set, which is adaptively adjusted based on the current operating condition type and historical false alarm rate. Specifically, historical anomaly detection results and manual review results for each operating condition transition type are maintained, and the false alarm rate is calculated. The false alarm rate is defined as the number of false alarms divided by the total number of alarms. When the false alarm rate for a certain type is high, the detection threshold for that type is appropriately lowered to reduce false alarms; when the false alarm rate is low, the threshold is appropriately raised to increase detection sensitivity.

[0141] The calculated log-likelihood probability is compared with the current threshold of the corresponding working condition transition type. If the log-likelihood probability is lower than the threshold, it is determined that there is a transient anomaly in the current working condition transition process. The anomaly information is recorded, including time, working condition transition type, transient characteristic value, and anomaly degree. The anomaly degree is obtained by normalizing the difference between the threshold and the log-likelihood probability.

[0142] Step 4.4: Establish a time-series correlation database for progressive failure mode mining;

[0143] A single transient anomaly may be caused by accidental factors and does not necessarily represent a real fault. However, if similar anomalies occur repeatedly and gradually become more severe, they are likely a sign of a progressive fault.

[0144] An abnormal signal time-series correlation database is established to record detailed information on all detected transient abnormal events. A time-series pattern mining algorithm is used to analyze the temporal distribution patterns of these abnormal events. Specifically, a long-term observation window of 30 days and a short-term observation window of 3 days are set. For each type of transient anomaly, the frequency of its occurrence within the long-term observation window is counted. If the frequency exceeds 10 times, the temporal evolution trend of the anomaly severity is further analyzed. Linear regression is used to fit the relationship between the anomaly severity and time, and the slope of the regression line is calculated. If the slope is positive and the significance test passes, it indicates an upward trend in the anomaly severity, and this is identified as a progressive fault warning. For each type of transient anomaly, the frequency of its consecutive occurrences within the short-term observation window is counted. If the frequency exceeds 3 times, this is identified as an acute fault warning. Information such as the type, confidence level, and predicted development trend of the identified progressive and acute faults is output to provide a basis for maintenance decisions.

[0145] After the above processing, the progressive fault identification results are output, which realizes the effective capture of transient abnormal signals and early identification of progressive faults in elevators under extreme load switching conditions, making up for the shortcomings of traditional steady-state monitoring methods.

[0146] In some embodiments, since the manifestation of different fault types during the switching of operating conditions may involve the combined changes of multiple transient features, a multi-feature abnormal pattern recognition method based on association rule mining can be adopted. The purpose is to discover complex fault feature combination patterns. Specifically, whether each transient feature is abnormal is treated as a binary attribute, with 0 for normal and 1 for abnormal. Historical fault data is analyzed, and the Apriori algorithm is used to mine frequent abnormal feature combination patterns. Minimum support and minimum confidence are set, and association rules that meet the conditions are extracted. For example, a certain type of fault is always accompanied by abnormal vibration peak change rate and abnormal spectral centroid drift. The mined association rules are stored as a fault feature pattern library. In real-time monitoring, when multiple transient features are detected to be abnormal at the same time, the fault feature pattern library is queried to determine whether it matches a known fault pattern, thereby improving the accuracy of fault identification.

[0147] The sample augmentation module extracts normal samples and rare fault samples from the multimodal feature library, combines them with the elevator safety status judgment results to generate synthetic samples and performs quality screening to obtain an augmented training dataset.

[0148] Specifically, it includes the following:

[0149] Step 5.1: Construct the generator and discriminator architecture of the conditional generative adversarial network;

[0150] Conditional generative adversarial networks (GANs) introduce conditional information into standard GANs, making the generation process controllable.

[0151] The generator network takes a random noise vector, a one-hot encoded vector of the fault type, and a one-hot encoded vector of the operating condition label as input, and outputs a feature vector of the generated fault sample. The random noise vector has a dimension of 100 and follows a standard normal distribution to provide randomness and diversity in the generation. The dimension of the one-hot encoded vector of the fault type is equal to the number of fault categories, used to specify the fault type to be generated. The dimension of the one-hot encoded vector of the operating condition label is equal to the number of operating condition categories, used to specify the operating condition corresponding to the generated sample. The generator network contains four fully connected layers, with 256, 512, and 512 neurons in each layer, and a feature dimension of [missing information]. The first three layers use LeakyReLU activation function, and the output layer uses tanh activation function, mapping the output value to the range of -1 to +1.

[0152] The discriminator network takes sample feature vectors, fault type information, and operating condition labels as input, and outputs a probability judgment of whether the sample is a real sample or a generated sample. The discriminator network consists of three fully connected layers with 512, 256, and 1 neurons in each layer, respectively. The first two layers use LeakyReLU activation function, and the output layer uses sigmoid activation function, mapping the output to a probability range of 0 to 1.

[0153] Step 5.2: Use Wasserstein distance to train the generative adversarial network to improve stability;

[0154] Feature data is extracted from a multimodal feature library and combined with elevator safety status judgment results and annotation information from historical fault records to construct a training sample set. Specifically, feature data corresponding to normal safety status judgment results in the feature library are extracted as normal samples, and feature data corresponding to slightly abnormal, moderately abnormal, and severely abnormal safety status judgment results, as well as fault feature data already annotated in historical fault records, are extracted as fault samples. Fault samples are classified and statistically analyzed according to fault type. Fault types with a sample number less than 5% of the total number of fault samples are defined as rare faults, and these rare fault samples are the key learning objects of the conditional generative adversarial network.

[0155] Traditional generative adversarial networks (GANs) use cross-entropy loss, which is prone to mode collapse and gradient vanishing problems during training. This paper proposes using Wasserstein distance as the loss function for GANs. Wasserstein distance measures the optimal transmission cost between two probability distributions and has better mathematical properties than cross-entropy.

[0156] The discriminator's loss function is the mean of the discriminator's output on real samples minus the mean of the discriminator's output on generated samples; the training objective is to maximize this loss. The generator's loss function is the negative of the mean of the discriminator's output on generated samples; the training objective is to maximize the discriminator's output on generated samples. To satisfy the Lipschitz continuity constraint of the Wasserstein distance, gradient penalties are applied to the discriminator's parameters. Specifically, linear interpolation is performed between real and generated samples to obtain interpolated samples. The gradient norm of the discriminator at the interpolated samples is calculated, and the square of the difference between the gradient norm and 1 is added as a penalty term to the discriminator's loss function. The Adam optimizer is used to optimize the generator and discriminator separately, with a learning rate of 0.0002 and a momentum parameter of 0.5. During training, the generator is updated once per iteration, and the discriminator is updated five times to ensure that the discriminator maintains sufficient discriminative power.

[0157] Step 5.3: Embed a physical constraint layer in the generator to ensure the rationality of the generated samples;

[0158] To ensure that the generated fault samples conform to the mechanical characteristics and fault mechanisms of the elevator, a physical constraint layer is added after the generator's output layer.

[0159] The physical constraint layer applies range and logical constraints to the generated feature values ​​based on the physical laws of the elevator equipment. Range constraints include ensuring the vibration amplitude does not exceed the sensor's measurement limit, frequency components remain within the effective frequency range, and energy values ​​are non-negative. Logical constraints include ensuring the energy distribution across different frequency bands conforms to mechanical resonance laws and maintaining consistency in causal relationships between certain features. In practice, for range constraints, a truncation function is used to cut feature values ​​outside the range to boundary values. For logical constraints, a constraint violation metric is defined, calculating the degree to which generated samples violate logical constraints. This violation metric is added as a penalty to the generator's loss function, prompting the generator to learn a sample generation strategy that satisfies the logical constraints.

[0160] Step 5.4: Generate rare fault samples under multiple operating conditions and perform quality screening;

[0161] After training, synthetic samples were generated for each rare fault type under nine different operating conditions.

[0162] For each fault type and operating condition combination, one-hot encodings for the fault type and operating condition labels are set. 100 noise vectors are randomly sampled and input into the generator to obtain 100 synthetic sample feature vectors. To ensure the quality of the generated samples, a quality assessment method based on real fault samples is used for screening. For each synthetic sample, its feature similarity with all real fault samples is calculated. The similarity is measured using cosine similarity, which is defined as the inner product of two feature vectors divided by the product of their respective norms. The real sample with the highest similarity to the synthetic sample is selected, and this similarity score is used as the quality score of the synthetic sample.

[0163] A quality threshold of 0.7 was set, retaining synthetic samples with a quality score higher than 0.7 and discarding samples with a quality score lower than 0.7. For the retained synthetic samples, the diversity among samples was further examined, and the average similarity within the sample set was calculated. If the average similarity was too high, it indicated that the generated samples lacked diversity. In this case, a clustering method was used to divide the samples into multiple clusters, and representative samples were selected from each cluster to ensure sample diversity.

[0164] The filtered synthetic rare fault samples are merged with normal and fault samples to construct an enhanced training dataset, which increases the number of rare fault types and improves the sample class imbalance problem.

[0165] The model building module trains a multi-level fault identifyer based on the enhanced training dataset, and performs optimization learning by integrating a multi-modal feature library to obtain a fault identification model;

[0166] Specifically, it includes the following:

[0167] Step 6.1: Construct a modal-specific recognizer to achieve single-modal fault identification;

[0168] For visual, vibration, and sound modalities, dedicated deep neural network classifiers were constructed. The visual modal recognizer takes visual spatiotemporal features as input. The network consists of four fully connected layers with 512, 256, and 128 neurons in each layer, representing the number of fault categories, respectively. The first three layers use ReLU activation functions and add dropout layers to prevent overfitting. The dropout ratio is set to 0.3. The output layer uses softmax activation functions to output the probability distribution of each fault category. The vibration and sound modal recognizers have the same network structure as the visual modal recognizer, taking vibration features and sound features as input, respectively.

[0169] The training of the three modality-specific recognizers employs a focus loss function, which introduces a modulation factor on top of the cross-entropy loss. The modulation factor is 1 minus the power of the predicted probability, with the power parameter set to 2. This design allows the model to assign higher loss weights to hard-to-classify samples and lower weights to easy-to-classify samples, effectively mitigating class imbalance and increasing attention to rare faults. A class-balanced sampling strategy is adopted, where the sampling probability in each training batch is determined by the reciprocal of the number of samples in each class, ensuring a relatively balanced number of samples in each class within each batch. The optimizer uses the Adam algorithm, with a learning rate of 0.001, a batch size of 64, and 100 training epochs.

[0170] Step 6.2: Construct a modality fusion recognizer to achieve multimodal feature fusion classification;

[0171] The modality fusion recognizer takes the output probability vectors of three modality-specific recognizers as input for higher-level feature fusion and classification. The probability vectors output by the three recognizers are concatenated to obtain a fused feature vector with a dimension three times the number of fault categories. The modality fusion recognizer employs a multilayer perceptron architecture, containing three fully connected layers with 256, 128, and the number of fault categories in each layer, respectively. The first two layers use ReLU activation, and the output layer uses softmax activation. The training of the modality fusion recognizer also uses the focal loss function and a class-balanced sampling strategy, with the optimizer and training parameters consistent with those of the modality-specific recognizers.

[0172] Step 6.3: Construct an integrated decision-maker to achieve multi-model integrated optimization;

[0173] To further improve model performance and robustness, an ensemble decision-maker is built upon the modality fusion recognizer. The ensemble decision-maker employs a gradient boosting decision tree algorithm, taking the output probability vector of the modality fusion recognizer, statistical features extracted from the multimodal feature library for each operating condition (including the mean, variance, extreme values, and other statistics of each modality feature), and the original multimodal features as input for further learning and optimization. By incorporating operating condition-related statistical information from the feature library, the ensemble decision-maker can learn the differences in fault modes under different operating conditions, improving its cross-operating condition recognition generalization ability. The gradient boosting decision tree iteratively trains multiple decision trees, with each tree fitting the residuals of all preceding trees, and finally, the prediction results of all trees are weighted and summed. The number of decision trees is set to 100, the maximum depth of each tree is set to 6, the learning rate is set to 0.1, and cross-entropy is used as the loss function. The gradient boosting decision tree can automatically perform feature selection and nonlinear modeling, exhibiting strong fitting capabilities for complex patterns in the data.

[0174] Step 6.4: Use cross-validation to evaluate model performance and optimize hyperparameters;

[0175] Five-fold cross-validation was used to evaluate the model's generalization performance. The augmented training dataset was randomly divided into five subsets. In each iteration, four subsets were selected as the training set and one subset as the validation set, repeating this process five times so that each subset served as the validation set once. For each partition, a complete multi-level fault recognition model was trained, and the model's precision, recall, and F1 score were evaluated on the validation set. The average performance metric across the five validations was calculated as the final performance evaluation of the model.

[0176] The hyperparameters of the model are optimized using a grid search method. These hyperparameters include the number of layers in the neural network, the number of neurons, the learning rate, the Dropout ratio, and the number and depth of the gradient boosting decision tree. Within a preset range of hyperparameter values, all possible combinations of hyperparameters are explored. Cross-validation is performed on each hyperparameter combination, and the combination that yields the highest F1 score on the validation set is selected as the optimal configuration.

[0177] Step 6.5: Quantize and compress the model and deploy it to an edge computing device;

[0178] To meet the requirements of real-time detection, the trained model needs to be deployed to the local edge computing device in the elevator. Since the edge computing device has limited computing resources, the model needs to be quantized and compressed to reduce its size and inference latency. An 8-bit integer quantization method is used to convert model parameters and activation values ​​from 32-bit floating-point numbers to 8-bit integers. The quantization process establishes a linear mapping relationship by calculating the maximum and minimum values ​​of parameters and activation values ​​at each layer, mapping floating-point numbers to the integer range. The quantized model size is reduced to one-quarter of its original size, inference speed is increased by 2 to 3 times, and accuracy is only slightly reduced. The quantized model is then converted to a format supported by the edge computing device, deployed to the device, and its input / output interfaces are configured to interface with the data acquisition module and the early warning decision module.

[0179] After the above processing, a fault identification model integrating multi-level learning and multi-modal fusion was constructed, and optimization for edge deployment was completed, enabling accurate and fast real-time fault detection on resource-constrained edge devices.

[0180] The intelligent early warning module calculates the safety level and infers the cause of the fault and the handling measures based on the safety status judgment results, progressive fault identification results and fault identification model, and obtains graded early warning information and maintenance suggestions.

[0181] Specifically, the following steps are included:

[0182] Step 7.1: Establish an elevator safety knowledge graph to integrate domain knowledge;

[0183] An elevator safety knowledge graph is a structured representation of knowledge in the elevator field, comprising entities, attributes, and relationships. Entities include the elevator equipment itself, such as traction machines, guide rails, cars, and door systems; fault types, such as bearing wear, guide rail deformation, and brake failure; fault causes, such as poor lubrication, installation misalignment, and aging damage; fault phenomena, such as abnormal vibration, abnormal noise, and unstable operation; and remedial measures, such as lubrication maintenance, adjustment and correction, and component replacement. Attributes include the characteristic descriptions of entities, such as equipment attributes like component model and specifications, installation location, service life, and maintenance cycle; fault attributes like the severity, frequency, and scope of impact of the fault; and remedial measures attributes such as execution time, cost, and technical difficulty. Relationships include causal relationships, such as poor lubrication leading to bearing wear; subordinate relationships, such as the bearing belonging to the traction machine; manifestation relationships, such as bearing wear manifesting as abnormal vibration; and remedial relationships, such as bearing wear requiring replacement.

[0184] The knowledge graph is constructed based on elevator equipment specifications, maintenance manuals, and a fault case database. Information extraction techniques are used to extract entities and relationships from the text, and manual review ensures the accuracy of the knowledge. The knowledge graph is stored in a graph database, supporting efficient graph query and reasoning operations.

[0185] Step 7.2: Establish a multi-level safety status assessment model based on fuzzy comprehensive evaluation;

[0186] Elevator safety status assessment is a complex decision-making problem involving multiple factors. A fuzzy comprehensive evaluation method is used for quantitative assessment. The system obtains the safety status judgment result from step 3, the progressive fault identification result from step 4, and the real-time detection result of the fault identification model from step 6.

[0187] A three-tiered assessment system is established. The first tier is the detection dimension layer, which includes three dimensions: multimodal fusion detection based on safety status judgment results, transient anomaly detection based on progressive fault identification results, and fault type identification based on fault identification models. Each dimension assesses its risk level based on the corresponding detection results. The second tier is the subsystem layer, which includes three subsystems: car interior safety, mechanical system status, and abnormal operating parameters. The assessment results of each subsystem are integrated from the relevant detection dimensions. The third tier is the overall safety layer, which combines the assessment results of all subsystems to obtain the overall safety level of the elevator.

[0188] For each assessment factor, four evaluation levels are defined: Normal, Attention, Warning, and Danger, assigned scores of 1, 2, 3, and 4 respectively. Fuzzy membership functions are used to calculate the membership degree of each factor to each evaluation level. The membership function is determined based on the quantitative index value of the factor and the threshold range of the evaluation level. The analytic hierarchy process (AHP) is used to determine the weights of each assessment factor, and the rationality of the weights is ensured through expert scoring and consistency checks. The weights of the first-level detection dimensions are determined based on the accuracy and coverage of each detection method, while the weights of the second-level subsystems are determined based on the degree of impact of each subsystem on overall safety.

[0189] A weighted average method is used to calculate the comprehensive score layer by layer. The first layer calculates the score of each subsystem, and the second layer calculates the overall safety score. The safety level is divided according to the range of the overall safety score: a score of 1 to 1.5 corresponds to normal, 1.5 to 2.5 corresponds to caution, 2.5 to 3.5 corresponds to warning, and 3.5 to 4 corresponds to danger.

[0190] Step 7.3: Query the knowledge graph to analyze the causes of the fault and generate maintenance suggestions;

[0191] For abnormal states identified in the safety status assessment results, transient anomalies found in the progressive fault identification results, and specific fault types output by the fault identification model, reasoning queries are performed in the knowledge graph to obtain deeper information about the faults.

[0192] Based on the fault type, the system searches for the corresponding fault entity in the knowledge graph. It then traverses backwards along the causal relationship edges to find possible fault cause entities and obtain descriptive information about the causes. Next, it traverses forwards along the manifestation relationship edges to find other manifestations of the fault, comparing them with the currently detected phenomena to improve the reliability of the fault diagnosis. Finally, it searches along the handling relationship edges to find recommended handling measure entities and obtain detailed steps and precautions for the handling measures. For complex faults that may involve the interconnected effects of multiple components, a graph reasoning algorithm is used for multi-hop reasoning to discover implicit relationships. The obtained fault causes, development trend predictions, and maintenance handling suggestions are compiled into a structured report to support maintenance decisions.

[0193] Step 7.4: Generate a graded early warning strategy based on the security level and link it to the control system;

[0194] Based on the safety status assessment results in step 7.2, the progressive fault identification results, and the safety level assessed after the fault identification model output, different levels of early warning strategies are generated.

[0195] In normal operation, the system only records operational logs, including time, operating conditions, and test results, without triggering warnings, maintaining routine monitoring. In a "watch out" state, the system sends a notification message to the property management platform's monitoring interface, including precautions and relevant test data, suggesting property management personnel increase inspection frequency and monitor elevator operation. In a "warning" state, the system triggers audible and visual alarms, issuing alarm signals in the elevator machine room and monitoring center, and simultaneously pushing alarm information to maintenance personnel's mobile terminals via SMS and mobile applications. The alarm information includes the elevator number, fault type, severity, and handling suggestions, requiring maintenance personnel to promptly conduct on-site repairs. In a "dangerous" state, the system triggers the highest level warning. In addition to audible and visual alarms and information pushes, it also activates protective measures in conjunction with the elevator control system, implementing corresponding control strategies based on the fault type. For mechanical component faults, it limits elevator speed, reduces load requirements, and prohibits full-load operation. For severe faults, it instructs the elevator to stop at the nearest safe location and prohibits restarting, ensuring personnel safety. Simultaneously, it reports the fault information to the local elevator safety regulatory department, fulfilling its legal reporting obligations.

[0196] Step 7.5: Establish elevator health records to predict remaining lifespan and generate predictive maintenance plans;

[0197] A health status file is established for each elevator. The health file integrates the historical trend of safety status judgment results, the abnormal evolution trajectory of progressive fault identification results, and the fault type statistics of fault identification model to form a comprehensive health status profile. It records the elevator's basic information, historical operating data, test results, fault records, maintenance records and other full life cycle data.

[0198] Based on health record data, a degradation model is used to predict the remaining service life of elevator components. The degradation model assumes that the performance of the components gradually degrades over time, and this degradation process can be described by a mathematical model. A Weibull distribution model is used to describe the failure time distribution of the components, and the shape and scale parameters of the Weibull distribution are estimated based on historical failure data. For currently operating elevators, the Weibull distribution parameters are corrected using a Bayesian update method based on the operating time and current performance status. The probability of component failure in the future time period and the expected time for performance degradation to reach the failure threshold are calculated; this expected time is the remaining service life.

[0199] Based on the predicted remaining useful life, a predictive maintenance plan is generated to schedule maintenance before components reach their failure threshold, thus avoiding unexpected failures. The maintenance plan includes information such as maintenance time, maintenance content, required spare parts, and estimated costs, supporting the advance preparation and rational scheduling of maintenance resources.

[0200] Through the above processing, a multi-level comprehensive assessment of elevator safety status was achieved, generating graded early warning information and intelligent maintenance suggestions for different risk levels, effectively improving the scientific nature and predictability of elevator safety management.

[0201] In some embodiments, since high-rise buildings are typically equipped with multiple elevators forming elevator groups, there may be correlations in operating modes and the propagation of faults among the elevators. An elevator group collaborative monitoring method can be adopted to improve monitoring accuracy by utilizing the correlation information of elevators within the group. Specifically, an elevator group operating status database is established to record the operating conditions, detection results, and early warning information of each elevator in the group in real time. Cluster analysis is used to identify the similarity of operating modes of elevators within the group. For elevators with similar operating modes, if one elevator issues a certain type of fault warning, other similar elevators are monitored in a focused manner, and the detection threshold for that type of fault is appropriately lowered to improve the sensitivity of the early warning. In the case of multiple elevators in the group simultaneously experiencing similar faults, it is analyzed whether there are common external causes such as environmental factors or improper maintenance, providing clues for the investigation of systemic problems.

[0202] In one embodiment of the present invention, a specific example is provided:

[0203] This invention focuses on the application of intelligent safety monitoring for elevators in super high-rise buildings. Taking a 600-meter-high office building in a certain city as the application scenario, this building is equipped with 30 ultra-high-speed elevators, operating at speeds up to 10 meters per second, handling a large number of passenger transport tasks during morning and evening rush hours. The system was deployed and tested on elevator No. 5 of this building, which serves floors from the 2nd basement level to the 80th floor above ground, with a rated load capacity of 1600 kg and a rated capacity of 21 passengers.

[0204] Table 1 shows some of the operational data collected by the system during the morning rush hour on a certain weekday:

[0205] Table 1: Partial operational data during the morning rush hour on a certain weekday;

[0206]

[0207] The transient feature data extracted by the system during the operating condition switching process are shown in Table 2:

[0208] Table 2: Transient feature data extracted by the system during the switching of operating conditions;

[0209]

[0210] Through continuous monitoring, the system can perform real-time analysis of the entire elevator operation process, accurately identify different load and speed conditions, effectively extract multimodal features and perform fusion judgment, providing reliable protection for the safe operation of elevators.

[0211] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. An elevator safety inspection system based on visual and multimodal data fusion, characterized in that, include: The data preprocessing module collects raw multimodal sensor data from elevator operation and preprocesses it to obtain a multimodal preprocessed dataset. The feature extraction module identifies operating conditions based on a multimodal preprocessed dataset, extracts multi-scale features for different operating conditions, and obtains a multimodal feature library and operating condition identification sequences. The multimodal fusion module calculates the reliability coefficients of each mode in the multimodal feature library and performs adaptive fusion to obtain the elevator safety status judgment result; The anomaly detection module, based on the multimodal preprocessing dataset and operating condition identification sequence, extracts transient features of operating condition switching and makes anomaly judgments, mines the anomaly evolution trend, and obtains progressive fault identification results. The sample augmentation module extracts normal samples and rare fault samples from the multimodal feature library, combines them with the elevator safety status judgment results to generate synthetic samples and performs quality screening to obtain an augmented training dataset. The model building module trains a multi-level fault identifyer based on the enhanced training dataset, and performs optimization learning by integrating a multi-modal feature library to obtain a fault identification model; The intelligent early warning module calculates the safety level and infers the cause of the fault and the handling measures based on the safety status judgment results, progressive fault identification results and fault identification model, and obtains graded early warning information and maintenance suggestions. The anomaly detection module includes: Based on the data before and after the switching of operating conditions, the peak change rate, spectral centroid shift and energy mutation index of the vibration signal are calculated, the loudness change gradient of the sound signal is calculated, and they are organized into transient feature vectors. Gaussian mixture model is used to model the probability distribution of transient characteristics for each type of working condition transition, and a normal mode library of working condition transition is constructed. The transient feature vector is input into the corresponding model to calculate the log-likelihood probability. When it is lower than the anomaly detection threshold, it is judged as a transient anomaly. Establish a time-series correlation database for abnormal signals, and use time-series pattern mining algorithms to analyze the evolution trend of abnormalities and identify progressive and acute faults. The sample enhancement module includes: A conditional generative adversarial network is constructed. The generator takes random noise vector, fault type code and working condition type code as input, and outputs fault sample feature vector. The discriminator outputs the true and false judgment. Add a physical constraint layer after the generator output layer to impose range and logical constraints on the generated feature values; For rare fault types, synthetic samples are generated under different operating conditions. Quality screening is performed based on feature similarity, and synthetic samples with similarity higher than a preset quality threshold are retained. Clustering methods are used to select representative samples.

2. The elevator safety inspection system based on vision and multimodal data fusion according to claim 1, characterized in that, The data preprocessing module includes: Install cameras to collect visual image sequences, install accelerometers to collect vibration signals, deploy microphone arrays to collect sound signals, and obtain operating parameter data through the elevator control system data interface; Deploy a clock synchronization module and use a network time protocol for clock synchronization. Sensors that cannot directly obtain a unified timestamp will perform time alignment through hardware trigger signals. Resampling is performed on sensor data with different sampling frequencies. For data with sampling frequencies lower than the unified time resolution, an interpolation method is used for upsampling, and for data with sampling frequencies higher than the unified time resolution, a sliding window averaging method is used for downsampling. The visual image data is normalized for illumination, the vibration signal data is de-electrified and de-electrified for low-frequency drift and high-frequency noise, the sound signal data is pre-emphasized and then framed, and the operating parameter data is de-electrified and de-electrified for abrupt noise and missing values ​​are filled.

3. The elevator safety inspection system based on vision and multimodal data fusion according to claim 1, characterized in that, The feature extraction module includes: A multi-level operating condition identification strategy is adopted. The first level performs coarse-grained classification based on speed and acceleration; the second level performs medium-grained classification based on the speed change trend within a time window; and the third level performs fine-grained classification based on the car load parameters within the time window. The elevator operating state is divided into nine operating conditions: no-load acceleration, no-load constant speed, no-load deceleration, half-load acceleration, half-load constant speed, half-load deceleration, full-load acceleration, full-load constant speed, and full-load deceleration. For visual image sequences, a multi-branch convolutional neural network is used to extract multi-scale spatial features, and a bidirectional long short-term memory network is used to extract temporal features. For vibration signals, wavelet packet decomposition is used to extract multi-band features, empirical mode decomposition is used to extract intrinsic mode functions, and statistical features of instantaneous frequency and instantaneous amplitude are calculated. For sound signals, Mel frequency cepstral coefficients and differential features are extracted, and short-time energy, zero-crossing rate, and spectral entropy are calculated.

4. The elevator safety inspection system based on vision and multimodal data fusion according to claim 1, characterized in that, The multimodal fusion module includes: Deep neural network anomaly detectors are constructed for the three modalities of vision, vibration, and sound, and the probability distribution of the output safe state is used as the basic probability allocation function. Calculate the quality assessment index and historical accuracy of each modality data, and calculate the reliability coefficient of each modality by weighted summation; The basic probability assignment function is discounted using a reliability coefficient. An improved Dempster combination rule is used for evidence synthesis. When the conflict coefficient exceeds the preset conflict threshold, the abnormal evidence sources are discounted twice and the evidence is recombined.

5. The elevator safety inspection system based on vision and multimodal data fusion according to claim 1, characterized in that, The model building module includes: Modal-specific recognizers were constructed for the three modalities of vision, vibration, and sound, and training employed a focus loss function and a class-balanced sampling strategy. A modality fusion recognizer is constructed, and the output probability vectors of each modality-specific recognizer are concatenated and then used with a multilayer perceptron for feature fusion and classification. An integrated decision-maker is constructed using the gradient boosting decision tree algorithm, with the output of the modality fusion recognizer and the original multimodal features as inputs for optimization. The model is then quantized and compressed before being deployed to edge computing devices.

6. The elevator safety inspection system based on vision and multimodal data fusion according to claim 1, characterized in that, The intelligent early warning module includes: Establish an elevator safety knowledge graph; build a multi-level safety status assessment model based on the fuzzy comprehensive evaluation method, calculate the comprehensive score and classify the safety level; For detected anomalies and faults, reasoning queries are performed in the knowledge graph to obtain the causes of the faults and corresponding solutions. Based on the safety level, a graded early warning strategy is generated and the elevator control system is activated to initiate protective measures.

7. The elevator safety inspection system based on vision and multimodal data fusion according to claim 6, characterized in that, The intelligent early warning module also includes: Establish a health status file for each elevator and record data throughout its entire life cycle; Based on health record data, a degradation model is used to predict the remaining service life of components. A Weibull distribution model is used to describe the failure time distribution of components. A Bayesian update method is used to correct the distribution parameters and calculate the probability of component failure and the expected time for performance degradation to the failure threshold. Predictive maintenance plans are generated based on the remaining useful life prediction results.

8. The elevator safety inspection system based on vision and multimodal data fusion according to claim 6, characterized in that, The security knowledge graph includes entities such as equipment body, fault type, fault cause, fault phenomenon, and handling measures. The relationships include causal relationships, subordinate relationships, manifestation relationships, and handling relationships. The multi-level security status assessment model includes a detection dimension layer, a subsystem layer, and an overall security layer. The tiered early warning strategy includes logging in the normal state, sending prompt messages in the alert state, triggering audible and visual alarms in the warning state, and activating protective measures in the linkage control system in the dangerous state.

Images