Elevator door system anomaly detection method based on multi-source spatio-temporal information fusion

Abstract

The invention discloses an elevator door system anomaly detection method based on multi-source spatio-temporal information fusion. The method comprises the steps that elevator door opening and closing video signals and vibration signals within a period of time are collected; the collected signals are preprocessed to serve as training set data; establishing a time-space fusion fault detection model; training the time-space fusion fault detection model by using the training set data; and elevator door opening and closing video signals and vibration signals are collected in real time, and abnormal state detection is conducted through the trained time-space fusion fault detection model. The method comprises the following steps of: constructing time series data into a graph structure so as to represent a relative position relationship between sequences; a graph convolutional network is adopted to extract spatial dependency features, and a bidirectional long-short-term memory network is combined to capture an evolution law of signals in a time dimension, so that a spatio-temporal information fusion framework is constructed. Furthermore, through a multi-source feature fusion mechanism, information of different modes is effectively integrated, accurate recognition of the abnormal state of the elevator door is achieved, and the detection accuracy and generalization ability of the system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of elevator anomaly detection technology, and in particular to an elevator door system anomaly detection method based on multi-source spatiotemporal information fusion. Background Technology

[0002] As a crucial device ensuring the smooth entry and exit of passengers from the elevator car, the elevator door system plays a key role in the overall operation of the elevator. Its malfunctions often directly threaten passenger safety. Therefore, timely and accurate anomaly detection of the elevator door system not only helps prevent potential risks and protect passenger lives, but also serves as an important indicator of the overall health of the elevator, significantly contributing to improved elevator maintenance efficiency and operational reliability.

[0003] Existing elevator door anomaly detection methods can be mainly divided into two categories: vision-based detection methods and sensor signal-based detection methods. Vision-based methods can intuitively acquire the door's movement state and physical position, clearly distinguishing structural anomalies such as an open door. However, they are sensitive to ambient lighting conditions and are prone to false detections due to external obstructions, reflections, and other factors. Sensor signal-based detection methods are less affected by external environmental factors, exhibiting stronger stability and clearly distinguishing trend-based anomalies such as slower door opening and closing speeds. However, relying solely on a single sensor signal places a strong dependence on feature engineering and prior knowledge, limiting their generalization ability. Both methods have their advantages, but their individual use has significant limitations. Therefore, combining these two methods can effectively utilize the strengths of different signals, improving detection accuracy.

[0004] Chinese patent "CN116664552A Elevator Door Anomaly Detection Method Based on Video Image Analysis" proposes a scheme for anomaly detection by identifying the location of door edges in elevator door images. This scheme primarily relies on grayscale image processing methods for line identification and angle extraction. However, grayscale images are easily affected by external factors such as ambient lighting changes and image noise in practical applications, leading to unstable edge detection results. Furthermore, during the stage when the door is about to close, the edge lines of the landing door and car door often appear simultaneously in the image, and their edge features have structural similarities, causing misidentification or even overlapping identification during line fitting, reducing detection accuracy. Moreover, the region growing algorithm used in this scheme is highly sensitive to parameters such as initial seed point selection and grayscale threshold, making it difficult to adapt to various elevator models and complex environmental conditions in actual deployment. It exhibits poor generalization ability and requires frequent manual parameter tuning to adapt to different scenarios.

[0005] Chinese patent CN115043291A, "Method and Apparatus for Anomaly Detection of Elevator Doors," describes an invention that determines the door's state by analyzing the magnetic field changes of a magnetic field sensor. While this method improves the ability to detect changes in elevator door status to some extent, it remains susceptible to electromagnetic interference, particularly in environments with high-frequency electromagnetic noise, where sensor readings may fluctuate, affecting accuracy. Furthermore, this method relies solely on magnetic field data for anomaly identification, lacking detailed modeling of the door's movement state, making it difficult to distinguish subtle anomalies such as door jamming or incomplete door closure. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide an elevator door system anomaly detection method based on multi-source spatiotemporal information fusion. By fusing video signals and triaxial vibration sensor data, multi-source time-series signals during elevator door operation are obtained. Combining time dimension information and spatial structural features, the elevator door operation status is modeled and detected.

[0007] This invention provides an anomaly detection method for elevator door systems based on multi-source spatiotemporal information fusion, comprising:

[0008] Step 1: Collect video and vibration signals of the elevator doors opening and closing over a period of time. Randomly select 60% of the data to build a training set, 20% as a validation set, and 20% as a test set.

[0009] Step 2: Preprocess the acquired signals;

[0010] Step 3: Establish a spatiotemporal fusion fault detection model;

[0011] Step 4: Train the spatiotemporal fusion fault detection model using the training set;

[0012] Step 5: Input the real-time collected elevator door opening and closing video signals and vibration signals into the trained spatiotemporal fusion fault detection model to detect abnormal states.

[0013] The present invention provides an elevator door system anomaly detection method based on multi-source spatiotemporal information fusion, which has the following beneficial effects:

[0014] (1) The anomaly detection method proposed in this invention integrates multiple information from different sensors, which can not only identify the position and movement of the elevator door, but also sense the speed changes and vibration characteristics during operation, making full use of data from both time-varying and spatial structure aspects. This method, through the combination of models such as graph convolutional networks, convolutional neural networks, bidirectional long short-term memory networks, and attention mechanisms, can not only accurately capture sudden anomalies in the opening and closing process of the elevator door, but also discover potential problems that are not easily detected, such as speed changes.

[0015] (2) By introducing information fusion from both video and vibration sensors, the system can cope with complex environments such as occlusion and changes in lighting, while maintaining a high recognition rate for different types of faults. This overcomes the shortcomings of traditional methods, such as poor stability and narrow applicability in complex scenarios. The solution of this invention performs excellently in terms of detection accuracy, anti-interference ability, and practical applicability, and can effectively improve the recognition level of elevator door operation status. Attached Figure Description

[0016] Figure 1 This is a flowchart of an elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to the present invention. Detailed Implementation

[0017] like Figure 1 As shown, the present invention provides an elevator door system anomaly detection method based on multi-source spatiotemporal information fusion, comprising:

[0018] Step 1: Collect video and vibration signals of the elevator doors opening and closing over a period of time;

[0019] In practice, vibration signals are collected by installing attitude measurement sensors in the gap between the landing door and the car door on one side of the car door, and video signals of door opening and closing are collected by deploying cameras in the corners of the car. Historical signals collected over a period of time are used for subsequent model training.

[0020] The collected vibration and video dataset contains 802 samples, covering four operating states: normal, slower door opening / closing speed, jamming malfunction, and inability to open / close normally. 60% of the data was randomly selected to construct the training set, 20% as the validation set, and 20% as the test set.

[0021] Step 2: Preprocess the acquired signals, specifically as follows:

[0022] Step 2.1: Perform stereo correction on the video signal: Extract the edge lines of the elevator door in the video using image processing methods, statistically analyze the distribution characteristics of straight lines in the horizontal and vertical directions, and determine the coordinates of the four vertices of the elevator door frame accordingly.

[0023] Step 2.2: Based on the correspondence between the coordinates of the four vertices of the detected original image and the coordinates of the four ideal vertices in the corrected target image, calculate the perspective transformation matrix M, and perform geometric correction on each frame of the video to eliminate distortion errors caused by shooting angle or tilt. The specific formula for calculating the perspective transformation matrix is ​​as follows:

[0024]

[0025] Among them, P src To extract the coordinates of the four vertices of the initial elevator door frame, P dstThese are the corrected coordinates of the four ideal vertices in the target image.

[0026] Step 2.3: Based on the geometrically corrected video, determine the position of the gate edge in each frame using template matching and inter-frame difference method.

[0027] Step 2.4: Calculate the time difference between two frames and the door opening / closing curve using the video capture frame rate, and then calculate the running speed.

[0028] Step 2.5: Construct a graph structure using the KNN method, transforming the time-series signals of velocity and vibration into graph structure data suitable for graph convolutional network modeling. Specifically:

[0029] Step 2.5.1: Perform global standardization on the time series signal to eliminate numerical scale differences between different samples. The standardization formula is as follows:

[0030]

[0031] in, x is the standardized sampled value at time t. t Let be the sampled value at time t in the time series signal, and μ and σ be the global mean and standard deviation of the time series signal at all time points, respectively.

[0032] Step 2.5.2: Extract graph node features using the sliding window method: For time t, a sliding window is formed using itself and the subsequent w-1 time points as the feature of the t-th graph node in the graph structure, represented as:

[0033]

[0034] Among them, v t Let be the feature of the t-th graph node.

[0035] Step 2.5.3: To establish structural connections between nodes, the KNN method based on Euclidean distance is used to construct an edge set. For each graph node i, the k nearest non-self nodes in the feature space of the graph node are selected to form a neighbor set N. k (i), which takes the following form:

[0036]

[0037] in, and The standardized node features; ||·||2 represents the Euclidean distance; Top-k represents selecting the k neighboring nodes with the smallest distance.

[0038] Step 3: Establish a spatiotemporal fusion fault detection model, which specifically includes a time feature extraction module, a spatial feature extraction module, and a feature fusion module. The time series signals of velocity and vibration are input into the time feature extraction module to extract multi-source time features; the graph structure data converted from the time series signals of velocity and vibration are input into the spatial feature extraction module to obtain multi-source spatial features; the multi-source time features and multi-source spatial features are concatenated and spliced ​​through the feature fusion module to form spatiotemporal features; the constructed spatiotemporal feature set is input into a fully connected layer to build a mapping relationship between the fused features and the state pattern, and the elevator door operation state classification result is output.

[0039] In specific implementation, the temporal feature extraction module consists of CNN, BiLSTM, and multi-head attention pooling. The feature extraction process is as follows:

[0040] By leveraging the sliding window characteristic of the local receptive field in convolutional neural networks, time-series signals of velocity and vibration are captured, enhancing the ability to extract local features by capturing abrupt changes, texture variations, or instantaneous pattern changes within a short period of time.

[0041] The convolutional time series features are input into a BiLSTM network composed of a bidirectional LSTM network to integrate contextual information and build long-term temporal dependencies. BiLSTM effectively fuses the information before and after the current moment by performing parallel computations of two LSTM units, forward and backward, and captures the global temporal structure of the signal more comprehensively.

[0042] Its calculation process can be formally represented as follows:

[0043]

[0044] Among them, X t y represents the input vector at time step t. t This represents the output vector at time step t. The hidden layer state at time step t during forward propagation. For the hidden state at time step t during backpropagation, w k Let f(·) be the weight, f(·) be the activation function, and g(·) be the output transformation.

[0045] A multi-head attention pooling mechanism is introduced, which learns different attention weights and selects the time step information that is most discriminative for the classification task.

[0046] In specific implementation, the spatial feature extraction module adopts a GCN network, and the feature extraction process is as follows:

[0047] First, linear mapping is used to increase the dimensionality of the original low-dimensional graph structure data, thereby improving information representation.

[0048] Then, the upgraded features are input into the GCN network to construct the structural relationships between nodes, and the adjacency matrix of the graph is... Where A ij =1 indicates that node i is connected to node j. The graph propagation matrix is ​​constructed using the symmetric normalization method and is expressed as follows:

[0049]

[0050] Where i is the identity matrix. This is the corresponding degree matrix. The graph propagation matrix.

[0051] The propagation and update process for the l-th layer graph convolution is expressed as follows:

[0052]

[0053] in, Represents the node features of the l-th layer. Let σ be the learnable parameter of this layer, and σ(·) be the nonlinear activation function.

[0054] In specific implementation, the structure of the feature fusion module is represented as follows:

[0055] z fused =ReLU(W1[z time ;z graph ]+b1)

[0056] Among them, z fused For the fused spatiotemporal feature set, z time For a multi-source temporal feature set, z graph is the multi-source spatial feature set, W1 is the linear transformation weight of the fusion layer, and b1 is the bias of the fusion layer.

[0057]

[0058] in, The output represents the probability distribution for each state category, including four operating states: normal, slower door opening / closing speed, jamming fault, and inability to open / close the door normally; W2 is the classifier weight, and b2 is the classifier bias.

[0059] Step 4: Train the spatiotemporal fusion fault detection model using the training set data.

[0060] The system trains and validates the model. It saves the best-performing model from the validation dataset. After reaching the maximum number of epochs, the optimal model is used to identify anomalies on the test set.

[0061] In practice, accuracy and F1 score are used as the core metrics for evaluating model performance. Accuracy represents the proportion of samples correctly predicted by the model out of the total samples, reflecting the overall classification precision; the F1 score combines precision and recall, and is suitable for performance evaluation in imbalanced data or scenarios with misclassification. The F1 score ranges from 0 to 1, with a higher value indicating a better trade-off between precision and recall. The formulas for calculating each evaluation metric are as follows:

[0062]

[0063] TP, FP, TN, and FN represent the number of true positive, false positive, true negative, and false negative samples, respectively.

[0064] Step 5: Input the real-time collected elevator door opening and closing video signals and vibration signals into the trained spatiotemporal fusion fault detection model to detect abnormal states.

[0065] The present invention has the following key features:

[0066] 1. Fully explore multi-dimensional data features using spatiotemporal information fusion methods.

[0067] First, the KNN method is used to construct a graph structure from the time series signal. The time series segments extracted by the sliding window are used as graph nodes, and the Euclidean distance between nodes is used to define the edge weights. This transforms the original temporal relationship into a quantifiable spatial structure relationship, providing a foundation for GCN to extract spatial features.

[0068] Subsequently, CNN is used to extract instantaneous features such as abrupt signals during the opening and closing process, BiLSTM captures the long-term trend of speed changes, and multi-head attention mechanism is used to extract salient features of key time steps to enhance the model's ability to perceive key temporal patterns.

[0069] By employing graph convolution operations and combining them with a symmetrically normalized graph propagation matrix, the structural correlations between different time points are strengthened, further extracting global spatial features. Finally, the temporal and spatial features are fused, improving the accuracy of anomaly detection and avoiding information loss caused by single-dimensional features.

[0070] 2. Multi-source sensors compensate for the insufficient information from a single sensor.

[0071] Video signals can intuitively obtain the door's motion state and physical position, making them suitable for detecting structural anomalies such as doors not closing. Compared to vibration signals, their spatial structural characteristics are more obvious. Vibration signals are not affected by environmental interference such as light or obstruction, have strong stability, and obvious temporal characteristics, making them suitable for detecting trend anomalies such as slower door opening and closing speeds. The combination of the two overcomes the shortcomings of poor robustness of single vision methods and limited generalization ability of single sensor methods.

[0072] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for anomaly detection in an elevator door system based on multi-source spatiotemporal information fusion, characterized in that, include: Step 1: Collect video and vibration signals of the elevator doors opening and closing over a period of time. Randomly select 60% of the data to build a training set, 20% as a validation set, and 20% as a test set. Step 2: Preprocess the acquired signals; Step 3: Establish a spatiotemporal fusion fault detection model; Step 4: Train the spatiotemporal fusion fault detection model using the training set; Step 5: Input the real-time collected elevator door opening and closing video signals and vibration signals into the trained spatiotemporal fusion fault detection model to detect abnormal states.

2. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 1, characterized in that, Step 1 specifically involves: collecting vibration signals by installing an attitude measurement sensor in the gap between the landing door and the car door on one side of the car door, and deploying a camera in the corner of the car to collect video signals of the door opening and closing.

3. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 1, characterized in that, Step 2 specifically involves: Step 2.1: Perform stereo correction on the video signal: Extract the edge lines of the elevator door in the video, statistically analyze the distribution characteristics of straight lines in the horizontal and vertical directions, and determine the coordinates of the four vertices of the elevator door frame accordingly. Step 2.2: Calculate the perspective transformation matrix M and perform geometric correction on each frame of the video to eliminate distortion errors caused by shooting angle or tilt. The specific formula for calculating the perspective transformation matrix is ​​as follows: Among them, P src To extract the coordinates of the four vertices of the initial elevator door frame, P dst These are the corrected coordinates of the four ideal vertices in the target image; Step 2.3: Based on the geometrically corrected video, determine the position of the gate edge in each frame using template matching and inter-frame difference methods; Step 2.4: Calculate the time difference between two frames and the door opening / closing curve using the video capture frame rate, and then calculate the operating speed; Step 2.5: Use the KNN method to construct a graph structure, converting the time series signals of velocity and vibration into graph structure data suitable for graph convolutional network modeling.

4. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 3, characterized in that, Step 2.5 specifically involves: Step 2.5.1: Perform global standardization on the time series signal to eliminate numerical scale differences between different samples. The standardization formula is as follows: in, x is the standardized sampled value at time t. t Let μ be the sampled value at time t in the time series signal, and let μ and σ be the global mean and standard deviation of the time series signal at all time points, respectively. Step 2.5.2: Extract graph node features using the sliding window method: For time t, a sliding window is formed using itself and the subsequent w-1 time points as the feature of the t-th graph node in the graph structure, represented as: Among them, v t The features of the t-th graph node; Step 2.5.3: To establish structural connections between nodes, the KNN method based on Euclidean distance is used to construct an edge set. For each graph node i, the k nearest non-self nodes in the feature space of the graph node are selected to form a neighbor set N. k (i), which takes the following form: in, and The standardized node features; ||·||2 represents the Euclidean distance; Top-k represents selecting the k neighboring nodes with the smallest distance.

5. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 1, characterized in that, The spatiotemporal fusion fault detection model in step 3 includes: a temporal feature extraction module, a spatial feature extraction module, and a feature fusion module; The time series signals of velocity and vibration are input into the time feature extraction module to extract multi-source time features; the graph structure data converted from the time series signals of velocity and vibration are input into the spatial feature extraction module to obtain multi-source spatial features; the multi-source time features and multi-source spatial features are concatenated and spliced ​​through the feature fusion module to form spatiotemporal features; the constructed spatiotemporal feature set is input into the fully connected layer to build the mapping relationship between the fused features and the state mode and output the elevator door operation state classification result.

6. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 5, characterized in that, The temporal feature extraction module consists of CNN, BiLSTM, and multi-head attention pooling. The feature extraction process is as follows: By leveraging the sliding window characteristic of the local receptive field of convolutional neural networks, time-series signals of velocity and vibration are captured, enhancing the ability to extract local features by capturing abrupt changes, texture or instantaneous pattern changes within a short period of time. The convolutional time series features are input into a BiLSTM network composed of a bidirectional LSTM network to integrate contextual information and build long-term temporal dependencies. BiLSTM effectively fuses the information before and after the current moment by performing parallel computations of two LSTM units, forward and backward, and captures the global temporal structure of the signal more comprehensively. Its calculation process is formally represented as follows: Among them, X t y represents the input vector at time step t. t This represents the output vector at time step t. The hidden layer state at time step t during forward propagation. For the hidden state at time step t during backpropagation, w k Let f(·) be the weight, f(·) be the activation function, and g(·) be the output transformation. A multi-head attention pooling mechanism is introduced, which learns different attention weights and selects the time step information that is most discriminative for the classification task.

7. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 5, characterized in that, The spatial feature extraction module uses a GCN network, and the feature extraction process is as follows: First, linear mapping is used to increase the dimensionality of the original low-dimensional graph structure data, thereby improving information representation. Then, the upgraded features are input into the GCN network to construct the structural relationships between nodes, and the adjacency matrix of the graph is... Where A ij =1 indicates that node i is connected to node j. The graph propagation matrix is ​​constructed using the symmetric normalization method and is expressed as follows: Where i is the identity matrix. This is the corresponding degree matrix. The graph propagation matrix; The propagation and update process for the l-th layer graph convolution is expressed as follows: in, Represents the node features of the l-th layer. Let σ be the learnable parameter of this layer, and σ(·) be the nonlinear activation function.

8. The elevator door system anomaly detection method based on multi-source spatiotemporal information fusion according to claim 5, characterized in that, The structure of the feature fusion module is represented as follows: With fused =ReLU(W1[z time ;With graph ]+b1) Among them, z fused For the fused spatiotemporal feature set, z time For a multi-source temporal feature set, z graph For multi-source spatial feature set, W1 is the linear transformation weight of the fusion layer, and b1 is the bias of the fusion layer; in, The output represents the probability distribution for each state category, including four operating states: normal, slower door opening / closing speed, jamming fault, and inability to open / close the door normally; W2 is the classifier weight, and b2 is the classifier bias.

Images