A deep learning-based elevator abnormal behavior real-time detection method and system

Abstract

The application provides a kind of based on deep learning's elevator abnormal behavior real-time detection method and system, the method includes: obtaining the video stream data, sensor dataset and running state sequence of target elevator in first detection period;The video stream data and running state sequence are input to preset passenger behavior identification model, to enable the passenger behavior identification model to identify the passenger behavior type existing in the target elevator;The sensor dataset and the running state sequence are converted into multi-modal time series data and input to preset vibration prediction model, to enable the vibration prediction model to predict the vibration mode of the target elevator in second detection period;According to each passenger behavior type, vibration mode and preset rule base, determine the first detection result of the target elevator in first detection period, and control the target elevator in second detection period to execute corresponding first protection measures, improve the safety in the process of elevator operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and multi-modal data processing, and particularly relates to a deep learning-based elevator abnormal behavior real-time detection method and system. BACKGROUND

[0002] As a key vertical transportation tool of modern urban buildings, the safe operation of elevators is directly related to the safety of people's lives and property. Traditional elevator safety monitoring mainly relies on mechanical protection devices such as speed limiters and safety clamps, as well as electrical alarm systems based on fixed thresholds, which have significant limitations: these passive protection mechanisms only trigger after a fault occurs, and cannot achieve pre-emptive warning of risks.

[0003] With the development of Internet of Things technology, some intelligent elevators attempt to introduce a multi-sensor fusion solution, but still face core bottlenecks: on the one hand, traditional machine learning methods are difficult to effectively handle the collaborative modeling of video time series behavior features and vibration frequency domain signals, and the recognition accuracy of complex patterns such as "door opening triggering door machine current surge accompanied by abnormal vibration in a specific frequency band" is insufficient; on the other hand, real-time detection data and historical maintenance records are mutually isolated, and maintenance personnel cannot quickly associate the current anomaly with the fault case library of similar elevators, resulting in low efficiency of predictive maintenance. More seriously, intelligent analysis solutions relying on cloud computing are constrained by network delays, resulting in slow response of active protection measures, which cannot meet the millisecond-level safety response requirements in emergency situations.

[0004] The above-mentioned defects highlight the "perception fragmentation, analysis shallow, decision isolation" triple dilemma of existing technical systems, and urgently need technical breakthroughs in multi-modal data deep fusion, edge intelligence real-time computing, and historical knowledge collaborative reasoning to build a new paradigm for active elevator safety protection. SUMMARY

[0005] To solve the above technical problems, the present application provides a deep learning-based elevator abnormal behavior real-time detection method and system, which improves the safety during the operation of the elevator.

[0006] In a first aspect, the embodiments of the present application provide a deep learning-based elevator abnormal behavior real-time detection method, comprising:

[0007] obtaining video stream data, a sensor data set and an elevator operation instruction data set of a target elevator in a first detection period, the sensor data set including an acceleration data set and a door machine current signal data set;

[0008] According to the sequence of each instruction data in the elevator operation instruction data set, the instructions that the target elevator needs to execute in the first detection period are inferred, and then the operation state sequence of the target elevator is determined, the operation state sequence including the operation state of the target elevator at each time in the first detection period;

[0009] The video stream data and the operation state sequence are input into a preset passenger behavior identification model, so that the passenger behavior identification model identifies the passenger behavior type existing in the target elevator according to the operation state sequence and the video stream data;

[0010] The sensor data set and the operation state sequence are converted into multi-modal time series data and input into a preset vibration prediction model, so that the vibration prediction model predicts the vibration mode of the target elevator in the second detection period according to the multi-modal time series data;

[0011] According to each of the passenger behavior type, the vibration mode and a preset rule base, a first detection result of the target elevator in the first detection period is determined, and the target elevator is controlled to execute a corresponding first protection measure in the second detection period according to the first detection result;

[0012] The passenger behavior identification model is obtained based on a convolutional neural network model, and the vibration prediction model is obtained based on a time convolution network model.

[0013] The embodiment of the application provides a kind of based on deep learning's elevator abnormal behavior real-time detection method, by real-time acquisition video stream data, sensor data set and elevator operation instruction data set, and according to the output result of behavior recognition model and vibration prediction model of TCN constructed by CNN construction respectively Comprehensive judgment, realize the full-dimensional real-time monitoring of elevator abnormal behavior, break through the adaptability limitation of traditional single mode detection to complex scene, improve the safety in the process of elevator operation.Specifically, the embodiment first determines the operation state sequence of target elevator by elevator operation instruction data set and then determines the passenger behavior type existing in target elevator according to video stream data and operation state sequence, i.e.determines the external factor affecting the safe operation of elevator in current cycle;On the other hand, multi-modal time series data is constructed by combining sensor data set and operation state sequence, and the vibration mode of target elevator in the future is further predicted according to multi-modal time series data, i.e.determines whether there is abnormal state in the next cycle that affects the elevator itself.Finally, the current passenger behavior type and future vibration mode are combined, and the corresponding first detection result is generated by pre-set rule base, to realize the accurate prediction of elevator operation risk, determine the first protection measure to be executed in the second detection period in advance, overcome the problem of low recognition accuracy of elevator abnormal behavior and slow response of active protection measure in prior art, effectively improve the safety in the process of elevator operation.

[0014] Further, when the transmission of the video stream data is interrupted:

[0015] determine whether to execute the emergency protection measure according to the vibration mode of the target elevator in the second detection period;

[0016] If it is determined to execute the emergency protection measure, the target elevator is controlled to execute the emergency protection measure immediately, and the vibration mode and pre-set first text information are sent to the specified terminal device;

[0017] If it is determined not to execute the emergency protection measure, determine the first detection result of the target elevator in the first detection period according to the vibration mode and pre-set rule base, and control the target elevator to execute the corresponding first protection measure in the second detection period according to the first detection result;The time period information corresponding to the video stream data is sent to the specified terminal device with pre-set second text information.

[0018] The extreme situation that may occur in the elevator environment is further considered in the embodiments of the present application. Since the elevator is in a closed operating environment, the network quality is poor, and the video stream data has a high requirement on the transmission bandwidth, the video stream data transmission may be interrupted in the normal operation process of the elevator, and the video stream data transmission may also be interrupted when the elevator has an emergency failure. Therefore, when the video stream data transmission is interrupted, the embodiments of the present application first need to determine the interruption reason according to the vibration mode. If the transmission interruption is caused by the emergency failure of the elevator, the emergency protection measure is started, and the maintenance personnel is notified to perform maintenance. If the transmission interruption is caused by the network reason during the normal operation of the elevator, and considering that the subsequent network may be automatically recovered, the normal operation of the elevator is not disturbed, the real-time detection is continued according to the vibration mode, and the maintenance personnel is reminded that there is network fluctuation in the current period, so that the safety during the operation of the elevator and the use experience of the user are considered.

[0019] In a possible implementation manner, the passenger behavior recognition model identifies the passenger behavior type existing in the target elevator according to the operation state sequence and the video stream data, and the identification includes:

[0020] According to the duration of each operation state in the operation state sequence, the main operation state of the target elevator is determined.

[0021] The video stream data is respectively subjected to feature extraction through the preset 2D-CNN branch and 3D-CNN branch, and the corresponding spatial feature vector and space-time feature vector are obtained.

[0022] According to the main operation state, the weight distribution of the spatial feature vector and the space-time feature vector is determined.

[0023] According to the weight distribution of the spatial feature vector and the space-time feature vector, the spatial feature vector and the space-time feature vector are weighted and fused, and then input to the preset full connection layer, so that the full connection layer outputs the passenger behavior type existing in the target elevator.

[0024] The embodiment of the application provides a passenger behavior recognition method, which extracts spatial features and space-time features from video stream data through 2D-CNN branches and 3D-CNN, respectively, wherein, the spatial static features are focused on extracting, and the key frame visual information in the elevator scene is efficiently captured; the 3D-CNN is good at space-time dynamic feature modeling, and solves the recognition problem of continuous behavior evolution. By combining the spatial feature vector and the space-time feature vector for comprehensive recognition, the accuracy of passenger behavior recognition can be effectively improved. In addition, the embodiment of the application also provides an adaptive feature fusion mechanism based on the main running state of the elevator, which dynamically allocates the weights of the 2D-CNN spatial features and the 3D-CNN space-time features according to the recognized main running state, significantly improves the environmental adaptability of passenger behavior recognition, and further improves the accuracy of passenger behavior recognition.

[0025] Further, the feature extraction of the video stream data through the preset 2D-CNN branch obtains a corresponding spatial feature vector, comprising:

[0026] According to the main running state, a corresponding sampling frequency is determined;

[0027] According to the sampling frequency, a plurality of key video frames are collected from the video stream data;

[0028] Each of the key video frames is input into the 2D-CNN branch for image feature extraction, and each corresponding initial image feature vector is obtained;

[0029] Based on the time sequence, each of the initial image feature vectors is combined to obtain a spatial feature sequence containing a time dimension;

[0030] The spatial feature sequence is subjected to a global average pooling operation along the time dimension to obtain a corresponding spatial feature vector.

[0031] The embodiment of the application provides a spatial feature vector extraction method, which realizes the balance between calculation efficiency and feature integrity through an intelligent sampling mechanism driven by a main running state. Specifically, by collecting key video frames from video stream data, feature extraction is avoided for all video stream data, the advantages of a 2D-CNN model are fully utilized, and the operation efficiency of the model is improved. In the collection process of the key video frames, the sampling frequency is dynamically adjusted for different main running states. For example, in a stable running state, the video picture does not change greatly, and at this time, the sampling frequency can be reduced to save system computing resources. In the open door hovering state, passengers will have the behavior of entering and exiting the elevator, and the video picture changes greatly, and at this time, the sampling frequency can be increased to avoid missing key video frames and ensure the feature extraction effect of the model. Finally, the spatial features of the key frames extracted by the 2D-CNN are compressed through time sequence combination and global pooling, which not only retains the essential features of the behavior, but also eliminates redundant space-time information. The above series of designs enable the behavior recognition model to maintain high response speed and recognition accuracy in resource-constrained embedded devices, thereby improving the safety of the elevator running process.

[0032] Further, the feature extraction of the video stream data through the preset 3D-CNN branch includes:

[0033] extracting a plurality of video clips from the video stream data based on a preset length sliding window;

[0034] inputting each of the video clips into the 3D-CNN branch, so that the 3D-CNN branch simultaneously captures the features of the video clips in the spatial dimension and the time dimension, and obtains each corresponding initial video feature vector;

[0035] performing a global average pooling operation on each of the initial video feature vectors to obtain a corresponding space-time feature vector.

[0036] The embodiment of the application provides a space-time feature vector extraction method, each video clip is extracted through a preset sliding window, and a 3D-CNN is used for double-scale modeling of the spatial dimension and the time dimension of each video clip, so as to simultaneously analyze the spatial posture change and the time dynamic evolution in each video clip, overcome the inherent defects of traditional 2D-CNN for modeling continuous behavior, and effectively solve the time domain complexity of elevator scene behavior recognition. Especially for long-time abnormal behavior, the design extracts essential features across clips through global average pooling, so that the occurrence rate of "short-time action misjudgment as accidental behavior" is greatly reduced, the misjudgment rate of the model for dangerous behavior is effectively reduced, and the accuracy of the subsequent first detection result is ensured.

[0037] In a possible implementation, the converting the sensor data set and the operation state sequence into the multi-modal time series data comprises:

[0038] synchronizing the acceleration data set, the door machine current signal data set and the operation state sequence to a unified time sequence point through an interpolation or resampling operation, and performing data normalization to obtain corresponding acceleration time series data, door machine current signal time series data and operation state time series data;

[0039] splicing and fusing the acceleration time series data, the door machine current signal time series data and the operation state time series data along a feature dimension to obtain the multi-modal time series data.

[0040] In a possible implementation, the vibration prediction model predicts a vibration mode of the target elevator in a second detection period according to the multi-modal time series data, which comprises:

[0041] projecting features of the multi-modal time series data through a plurality of 1*1 convolution kernels respectively to obtain initial feature maps, a feature dimension of the initial feature maps being equal to a number of the convolution kernels;

[0042] performing a plurality of dilated convolution operations on the initial feature maps to obtain a plurality of corresponding dilated feature maps, wherein dilated rates in the dilated convolution operations are different from each other, and input data of a current dilated convolution operation is a dilated feature map of a previous dilated convolution operation;

[0043] performing feature pyramid aggregation according to the dilated feature maps to obtain an aggregated feature matrix;

[0044] determining a corresponding pooling strategy according to a change in the operation state in the multi-modal time series data;

[0045] performing a pooling operation on the aggregated feature matrix according to the pooling strategy to obtain a vibration feature vector;

[0046] inputting the vibration feature vector into a preset Softmax classification head, so that the Softmax classification head outputs the vibration mode of the target elevator in the second detection period.

[0047] The embodiment of the application provides a vibration mode prediction method, after a plurality of 1*1 convolution kernels are used for feature projection on multimodal time series data, continuous multiple inflation convolution operations are performed according to different inflation rates, and vibration modes of different frequencies are captured in layers, wherein low inflation rate focuses on high-frequency impact features, and high inflation rate captures low-frequency periodic features. Then, different aggregation matrix pooling strategies are adaptively adopted according to the running state changes of the target elevator, the sensitivity of the model to the vibration spectrum is further improved, and hierarchical extraction of frequency domain features is realized. The design successfully predicts early hidden faults that cannot be detected by the traditional threshold method, improves the timeliness and accuracy of elevator abnormal behavior detection, and further improves the safety during the running of the elevator.

[0048] In a possible implementation manner, the elevator abnormal behavior real-time detection method further includes inputting the elevator number of the target elevator, the passenger behavior type and the vibration mode to a preset knowledge graph when the target elevator executes corresponding first protection measures in a second detection period, so that the knowledge graph obtains a second detection result of the target elevator in the first detection period based on historical fault database reasoning;

[0049] comparing the first detection result with the second detection result, and determining a second protection measure of the target elevator in the second detection period according to a comparison result;

[0050] controlling the target elevator to execute the second protection measure in a remaining time period in the second detection period;

[0051] The knowledge graph is deployed in the cloud, and the passenger behavior recognition model and the vibration prediction model are deployed in the local control system of the target elevator.

[0052] The embodiment of the application further introduces a knowledge graph technology, combines local real-time detection with cloud-driven knowledge graph reasoning, constructs a double detection mechanism, and further improves the safety during the running of the elevator. Considering the hysteresis caused by knowledge graph reasoning and cloud data transmission, after obtaining the first detection result, the target elevator is controlled to execute the first protection measure, and then the second detection result is generated by accurate reasoning according to the historical database and the knowledge graph. Because the reasoning accuracy of the knowledge graph is higher than that of the local lightweight model, the first protection measure can be modified by comparing the first detection result with the second detection result, so that the target elevator executes more accurate second protection measures, and the safety during the running of the elevator is further improved.

[0053] Further, in a plurality of preset detection periods, if the consistency rate of each first detection result and the corresponding second detection result is lower than a preset threshold, the rule base is adjusted according to the reasoning process of each second detection result.

[0054] The embodiment of the application provides a rule base updating method, when the detection accuracy of a local model is low, valid information in a knowledge graph and a historical fault database is fully utilized, the rule base is adjusted according to a plurality of reasoning processes of the knowledge graph, intelligent continuous iteration of a detection system is realized, high accuracy of elevator abnormal behavior detection is ensured, and safety in an elevator operation process is improved.

[0055] In a second aspect, the embodiment of the application provides an elevator abnormal behavior real-time detection system based on deep learning, comprising an acquisition module, a running state determination module, a behavior recognition module, a vibration prediction module and an active protection module.

[0056] The acquisition module is configured to acquire video stream data, a sensor data set and an elevator operation instruction data set of a target elevator in a first detection period, wherein the sensor data set comprises an acceleration data set and a door machine current signal data set.

[0057] The running state determination module is configured to infer an instruction to be executed by the target elevator in the first detection period according to the sequence of each instruction data in the elevator operation instruction data set, and then determine a running state sequence of the target elevator, wherein the running state sequence comprises a running state of the target elevator at each time point in the first detection period.

[0058] The behavior recognition module is configured to input the video stream data and the running state sequence into a preset passenger behavior recognition model, so that the passenger behavior recognition model recognizes a passenger behavior type existing in the target elevator according to the running state sequence and the video stream data.

[0059] The vibration prediction module is configured to convert the sensor data set and the running state sequence into multi-modal time series data and input the multi-modal time series data into a preset vibration prediction model, so that the vibration prediction model predicts a vibration mode of the target elevator in a second detection period according to the multi-modal time series data.

[0060] The active protection module is configured to determine a first detection result of the target elevator in the first detection period according to each passenger behavior type, the vibration mode and a preset rule base, and control the target elevator to execute a corresponding first protection measure in the second detection period according to the first detection result.

[0061] The passenger behavior recognition model is constructed based on a convolutional neural network model, and the vibration prediction model is constructed based on a time convolution network model. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1A flowchart illustrating a real-time elevator abnormal behavior detection method based on deep learning, provided for an embodiment of this application;

[0063] Figure 2 A schematic diagram of the model architecture of the passenger behavior recognition model in a deep learning-based real-time elevator abnormal behavior detection method provided in this application embodiment;

[0064] Figure 3 A schematic diagram of the vibration prediction model in a deep learning-based real-time detection method for abnormal elevator behavior provided in this application embodiment;

[0065] Figure 4 A flowchart illustrating the feature pyramid aggregation process in a deep learning-based real-time elevator abnormal behavior detection method provided in this application embodiment;

[0066] Figure 5 This is a schematic diagram of the structure of a real-time elevator abnormal behavior detection system based on deep learning, provided in an embodiment of this application. Detailed Implementation

[0067] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0068] It should be noted that the step numbers in this document are only for the convenience of explaining the specific embodiments and are not intended to limit the order in which the steps are performed. In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0069] Example 1:

[0070] like Figure 1 As shown, Embodiment 1 provides a real-time elevator abnormal behavior detection method based on deep learning, including steps S1-S5:

[0071] Step S1: Obtain the video stream data, sensor dataset, and elevator operation command dataset of the target elevator during the first detection cycle. The sensor dataset includes an acceleration dataset and a door operator current signal dataset.

[0072] Step S2, according to the sequence of each instruction data in the elevator operation instruction data set, reasoning the instruction that the target elevator needs to execute in the first detection period, and further determining the operation state sequence of the target elevator, the operation state sequence including the operation state of the target elevator at each time in the first detection period;

[0073] Step S3, inputting the video stream data and the operation state sequence into a preset passenger behavior recognition model, so that the passenger behavior recognition model identifies the passenger behavior type existing in the target elevator according to the operation state sequence and the video stream data;

[0074] Step S4, converting the sensor data set and the operation state sequence into multi-modal time series data and inputting into a preset vibration prediction model, so that the vibration prediction model predicts the vibration mode of the target elevator in the second detection period according to the multi-modal time series data;

[0075] Step S5, according to each of the passenger behavior type, the vibration mode and a preset rule base, determining the first detection result of the target elevator in the first detection period, and according to the first detection result, controlling the target elevator to execute the corresponding first protection measure in the second detection period;

[0076] Among them, the passenger behavior recognition model is obtained based on a convolutional neural network model, and the vibration prediction model is obtained based on a time convolution network model.

[0077] The embodiment of the application provides a kind of based on deep learning's elevator abnormal behavior real-time detection method, by real-time acquisition video stream data, sensor data set and elevator operation instruction data set, and according to the output result of behavior recognition model and vibration prediction model of TCN constructed by CNN construction respectively Comprehensive judgment, realize the full-dimensional real-time monitoring of elevator abnormal behavior, break through the adaptability limitation of traditional single mode detection to complex scene, improve the safety in the process of elevator operation.Specifically, the embodiment first determines the running state sequence of target elevator by elevator operation instruction data set, then determines the passenger behavior type existing in target elevator according to video stream data and running state sequence, i.e.determines the external factor that affects the safe operation of elevator in current period;On the other hand, multi-modal time series data is constructed by combining sensor data set and running state sequence, and the vibration mode of target elevator in the future is further predicted according to multi-modal time series data, i.e.determines whether there is abnormal state in the next period that affects the elevator itself.Finally, the current passenger behavior type and future vibration mode are combined, and the corresponding first detection result is generated by pre-set rule base, to realize the accurate prediction of elevator operation risk, determine the first protection measure to be executed in the second detection period in advance, overcome the problem that the recognition accuracy of elevator abnormal behavior is low and the active protection measure is slow in prior art, effectively improve the safety in the process of elevator operation.

[0078] In a possible implementation manner, in step S1, the video stream data and sensor data set of the target elevator in the first detection period are acquired, including:

[0079] initial video stream data is acquired by the camera inside the car of the target elevator;

[0080] initial acceleration data set is acquired by the three-axis acceleration sensor of the target elevator;

[0081] initial door machine current signal data set is acquired by the current sensor of the target elevator;

[0082] initial elevator operation instruction data set is acquired from the elevator control system of the target elevator;

[0083] The initial video stream data, initial acceleration data set, initial door machine current signal data set and initial elevator operation instruction data set are time-synchronized based on pre-set time stamp, to generate video stream data, acceleration data set, door machine current signal data set and elevator operation instruction data set with time sequence label;

[0084] The acceleration data set and door machine current signal data set are combined to obtain the sensor data set.

[0085] The embodiment of the application provides a data acquisition method, initial video stream data, initial acceleration data set, initial door machine current signal data set and initial elevator operation instruction data set are acquired through a camera, an acceleration sensor, a current sensor and a control system respectively, and time synchronization is performed through a preset time stamp, so that the timing consistency between multi-source data is ensured, the core problem of heterogeneous data timing alignment in an elevator scene is effectively solved, accurate space-time correlation basis is provided for subsequent identification and prediction of two models, and the accuracy of elevator abnormal behavior detection is ensured.

[0086] In a preferred embodiment, the initial video stream data is acquired through a camera inside the target elevator car, the initial acceleration data set is collected through a three-axis acceleration sensor with a sampling frequency of 100Hz at the bottom of the car, the initial door machine current signal data set is collected through a current sensor with a sampling frequency of 1kHz of the door machine system, and the elevator operation instruction data set is read in real time from the CAN bus of the elevator control system, including floor instructions, door opening and closing instructions and the like. Then, the timing of the collected data is aligned, the NTP protocol is used to stamp a uniform time stamp on all data sources, the sampling points of the acceleration and current data are aligned to the time axis of the video stream through cubic spline interpolation, the acceleration data is scaled to [-1g, 1g], and the current data is scaled to [0, 5A]. Finally, the multi-source data set with timing label is output, including the video stream, the acceleration timing, the door machine current timing and the elevator state timing.

[0087] Further, when the transmission of the video stream data is interrupted:

[0088] According to the vibration mode of the target elevator in the second detection period, whether to execute an emergency protection measure is determined;

[0089] If it is determined to execute the emergency protection measure, the target elevator is controlled to immediately execute the emergency protection measure, and the vibration mode and a preset first text information are sent to a specified terminal device;

[0090] If it is determined not to execute the emergency protection measure, a first detection result of the target elevator in the first detection period is determined according to the vibration mode and a preset rule base, and a corresponding first protection measure is executed by the target elevator in the second detection period according to the first detection result; the time period information corresponding to the video stream data is sent to a specified terminal device together with a preset second text information.

[0091] The extreme situation that may occur in the elevator environment is further considered in the embodiments of the present application. Since the elevator is in a closed operating environment, the network quality is poor, and the video stream data has a high requirement on the transmission bandwidth, the video stream data transmission may be interrupted in the normal operation process of the elevator, and the video stream data transmission may also be interrupted when the elevator has an emergency failure. Therefore, when the video stream data transmission is interrupted, the embodiments of the present application first need to determine the interruption reason according to the vibration mode. If the transmission interruption is caused by the emergency failure of the elevator, an emergency protection measure is started, and the maintenance personnel are notified to repair. If the transmission interruption is caused by the network reason in the normal operation of the elevator, and considering that the subsequent network may be automatically recovered, the normal operation of the elevator is not disturbed, the real-time detection is continued according to the vibration mode, and the maintenance personnel are reminded that there is network fluctuation in the current period, so that the safety in the operation process of the elevator and the use experience of the user are considered.

[0092] In a preferred embodiment, in step S2, the corresponding instruction sequence is obtained by analyzing the elevator operation instruction data set, then the state change of the target elevator is inferred according to the instruction logic and time sequence relationship in the instruction sequence, and finally the operation state sequence of the target elevator is determined according to the duration of each state. For example, when the instruction sequence is [hover, open door, close door, and go up to 5th floor], the state change of the target elevator can be obtained as [standby, open door, close door, accelerate upward, uniform speed running, and decelerate stop], and finally the state sequence in the time dimension is obtained by combining the execution time of each instruction and the fixed time of some states (for example, the fixed time of accelerate upward is 1 second), for example, [t=0-2s: open door, t=2-3s: accelerate upward, t=3-8s: uniform speed running...].

[0093] In a possible implementation manner, in step S3, the passenger behavior recognition model recognizes the passenger behavior type existing in the target elevator according to the operation state sequence and the video stream data, including:

[0094] According to the duration of each operation state in the operation state sequence, the main operation state of the target elevator is determined;

[0095] The preset 2D-CNN branch and 3D-CNN branch are used to respectively extract features of the video stream data, to obtain corresponding spatial feature vectors and space-time feature vectors;

[0096] According to the main operation state, the weight distribution of the spatial feature vector and the space-time feature vector is determined;

[0097] According to the weight distribution of the spatial feature vector and the spatio-temporal feature vector, the spatial feature vector and the spatio-temporal feature vector are weighted and fused and then input to a preset full connection layer, so that the full connection layer outputs the passenger behavior type existing in the target elevator.

[0098] The embodiment of the application provides a passenger behavior recognition method, which extracts spatial features and spatio-temporal features from video stream data through 2D-CNN branches and 3D-CNN, respectively. The 2D-CNN branches focus on extracting spatial static features and efficiently capturing key frame visual information in an elevator scene. The 3D-CNN branches specialize in modeling spatio-temporal dynamic features and solving the recognition problem of continuous behavior evolution. By combining spatial feature vectors and spatio-temporal feature vectors for comprehensive recognition, the accuracy of passenger behavior recognition can be effectively improved. In addition, the embodiment of the application also provides an adaptive feature fusion mechanism based on the main running state of an elevator. The weights of 2D-CNN spatial features and 3D-CNN spatio-temporal features are dynamically allocated according to the recognized main running state, which significantly improves the environmental adaptability of passenger behavior recognition and further improves the accuracy of passenger behavior recognition.

[0099] In a preferred embodiment, the model architecture of the passenger behavior recognition model is as shown in Figure 2 The passenger behavior recognition model is based on a convolutional neural network model and includes 2D-CNN branches and 3D-CNN branches. The 2D-CNN branches can select EfficientNet-B3 as the backbone network, and the 3D-CNN branches can select Inflated 3D ConvNet as the backbone network. During training, an end-to-end training strategy is used, a training data set is constructed using artificially labeled historical video stream data and corresponding historical running state sequences in a time period, and the initial passenger behavior recognition model is supervised learning trained based on a preset loss function. During the training process, the neural network parameters are updated through back propagation to obtain the final passenger behavior recognition model.

[0100] Further, the feature extraction of the video stream data through the preset 2D-CNN branch to obtain the corresponding spatial feature vector includes:

[0101] According to the main running state, a corresponding sampling frequency is determined;

[0102] According to the sampling frequency, a plurality of key video frames are collected from the video stream data;

[0103] Each of the key video frames is input to the 2D-CNN branch for image feature extraction to obtain each corresponding initial image feature vector;

[0104] combining each of the initial image feature vectors in time sequence to obtain a spatial feature sequence containing a time dimension;

[0105] performing a global average pooling operation on the spatial feature sequence along the time dimension to obtain a corresponding spatial feature vector.

[0106] The embodiment of the application provides a spatial feature vector extraction method, which realizes the balance between calculation efficiency and feature integrity through an intelligent sampling mechanism driven by a main running state. Specifically, by collecting key video frames from video stream data, feature extraction is avoided for all video stream data, the advantages of a 2D-CNN model are fully utilized, and the operation efficiency of the model is improved. In the collection process of the key video frames, the sampling frequency is dynamically adjusted for different main running states. For example, in a stable running state, the video picture will not change greatly, and at this time, the sampling frequency can be reduced to save system computing resources; in the door opening and hovering state, passengers will have the behavior of entering and exiting the elevator, and the video picture changes greatly, and at this time, the sampling frequency can be increased to avoid missing key video frames and ensure the feature extraction effect of the model. Finally, the spatial features of the key frames extracted by the 2D-CNN are combined in time sequence and compressed by global pooling, which not only retains the essential features of the behavior, but also eliminates redundant space-time information. The above series of designs enable the behavior recognition model to maintain high response speed and recognition accuracy in resource-constrained embedded devices, thereby improving the safety of the elevator running process.

[0107] In a preferred embodiment, the specific process of feature extraction using a 2D-CNN branch is as follows:

[0108] 1. Preprocessing: Perform standard image preprocessing operations on each key frame, such as adjusting the size to the input size required by the model (e.g., 224x224 pixels), and performing pixel value normalization (e.g., scaling to [0, 1] or subtracting the mean and dividing by the standard deviation).

[0109] 2. 2D-CNN inference: Independently input each preprocessed key frame into a pre-trained 2D CNN model (such as ResNet, EfficientNet).

[0110] 3. Feature extraction: 2D-CNN is responsible for extracting rich spatial features (such as object shape, texture, position, appearance details, spatial layout) from a single static image. After each key frame passes through the 2D-CNN, an initial image feature vector (e.g., 2048 dimensions) is generated, representing the visual content of the frame.

[0111] 4、Feature Integration: Save the initial image feature vectors corresponding to all key frames in the current cycle, forming a spatial feature sequence. The dimension of the spatial feature sequence is: number of key frames x feature dimension.

[0112] 5、Spatial Feature Aggregation: Aggregate the spatial feature sequence obtained in step 4 (number of key frames x feature dimension), usually by performing global average pooling along the time dimension, and compress it into a single spatial feature vector.

[0113] Further, the feature extraction of the video stream data by the preset 3D-CNN branch to obtain the corresponding spatio-temporal feature vector, comprising:

[0114] extracting several video clips from the video stream data based on a preset length sliding window;

[0115] inputting each of the video clips into the 3D-CNN branch to enable the 3D-CNN branch to simultaneously capture the features of the video clips in the spatial dimension and the time dimension, and obtain each corresponding initial video feature vector;

[0116] performing global average pooling operation on each of the initial video feature vectors to obtain the corresponding spatio-temporal feature vector.

[0117] The embodiment of the application provides a spatio-temporal feature vector extraction method, which extracts each video clip through a preset sliding window, and uses 3D-CNN to perform double-scale modeling of the spatial dimension and the time dimension of each video clip, and synchronously analyzes the spatial posture change and the temporal dynamic evolution in each video clip, thereby overcoming the inherent defects of traditional 2D-CNN in modeling continuous behavior, and effectively solving the time domain complexity of elevator scene behavior recognition. Especially for long-time abnormal behavior, the design extracts the essential features across the clips through global average pooling, greatly reduces the occurrence rate of "short-time action misjudgment as accidental behavior", effectively reduces the misjudgment rate of the model for dangerous behavior, and ensures the accuracy of the subsequent first detection result.

[0118] In a preferred embodiment, the specific process of feature extraction using a 3D-CNN branch is as follows:

[0119] 1、Video clip construction: Select continuous short clips from the original video stream, usually referred to as "video cubes" or "clips". The specific way is to use a fixed length sliding window to move on the time axis, and then obtain multiple video cubes.

[0120] 2. Preprocessing: Preprocess each video cube: resize each frame to the model requirement (e.g. 224x224) and normalize the pixel values. Finally, form a 4D tensor (num_frames x height x width x channels).

[0121] 3. 3D-CNN inference: Input the preprocessed video cube into a pre-trained 3D-CNN model (e.g. I3D, SlowFast).

[0122] 4. Feature extraction: The core capability of 3D-CNN is to model the features of video data in both spatial dimension (within a single frame) and temporal dimension (across frames). It can capture the dynamic changes of actions, motion patterns, and temporal relationships across multiple frames. After passing through the 3D-CNN, each input video cube will generate an initial video feature vector.

[0123] 5. Segment feature aggregation: During the entire detection period, multiple overlapping or non-overlapping video cubes will be processed. To obtain a global spatio-temporal feature representing the entire detection period, an average pooling (Global Average Pooling) operation will be performed on the initial video feature vectors output by all video cubes, finally obtaining a single spatio-temporal feature vector.

[0124] After obtaining the spatial feature vector and spatio-temporal feature vector through 2D-CNN and 3D-CNN respectively, according to the previously determined main running state of the elevator, different fusion weights are assigned to the spatial feature vector and spatio-temporal feature vector. The following is an exemplary weight allocation method:

[0125] When the main running state of the elevator is door opening or door closing, a higher weight is given to the spatial feature vector. Because at this time, passengers are more likely to engage in dangerous behaviors such as blocking doors, grabbing doors, etc., and the recognition of these behaviors usually depends more on spatial information such as object position, posture, and area occlusion within a single frame or short time.

[0126] When the main running state of the elevator is stable operation / acceleration start / deceleration braking, a higher weight is given to the spatio-temporal feature vector. Because at this time, passengers are more likely to engage in dangerous behaviors such as jumping, fighting, etc., and the recognition of these behaviors usually depends more on the continuity and dynamic changes of actions.

[0127] When the main running state of the elevator is the stop state, a relatively balanced weight is used. Because behaviors such as staying, improper operation, etc. may depend on both spatial information and short-time actions.

[0128] The weighted spatial feature vector and the weighted spatio-temporal feature vector are added (Element-wise Addition) to obtain a fusion feature vector. This vector integrates static appearance information and dynamic motion information, and is guided by the current state of the elevator, making the model pay more attention to the features most relevant to the current scene.

[0129] Finally, the fusion feature vector is input into one or more fully connected layers (FC Layers). The role of the fully connected layer is to map the fusion feature to the probability space of the behavior class. Since multiple behaviors may occur simultaneously in the elevator, a Sigmoid activation function is usually used for multi-label classification. Each node of the output layer corresponds to a behavior type, such as "normal", "block the door", "jump", "crowded push", "violent shaking of luggage", "stuck", "improper operation", "violent fighting", etc. The model outputs the probability value of the existence of each behavior class in the current detection period.

[0130] In a possible implementation manner, in step S4, the converting the sensor data set and the running state sequence into multi-modal time series data comprises:

[0131] The acceleration data set, the door machine current signal data set, and the running state sequence are synchronized to a unified time series point through an interpolation or resampling operation, and data normalization is performed to obtain corresponding acceleration time series data, door machine current signal time series data, and running state time series data;

[0132] The acceleration time series data, the door machine current signal time series data, and the running state time series data are spliced and fused along the feature dimension to obtain the multi-modal time series data.

[0133] In a possible implementation manner, in step S4, the vibration prediction model predicts the vibration mode of the target elevator in the second detection period according to the multi-modal time series data, which comprises:

[0134] The multi-modal time series data is projected by a plurality of 1x1 convolution kernels to obtain an initial feature map, and a feature dimension of the initial feature map is equal to a number of the convolution kernels;

[0135] A plurality of dilated convolution operations are performed on the initial feature map to obtain a plurality of corresponding dilated feature maps, wherein a dilation rate in each dilated convolution operation is different, and input data of a current dilated convolution operation is a dilated feature map of a previous dilated convolution operation;

[0136] According to each dilated feature map, a feature pyramid aggregation is performed to obtain an aggregated feature matrix;

[0137] determine a corresponding pooling strategy according to a running state change in the multi-modal time series data;

[0138] perform a pooling operation on the aggregated feature matrix according to the pooling strategy to obtain a vibration feature vector;

[0139] input the vibration feature vector into a preset Softmax classification head, so that the Softmax classification head outputs a vibration mode of the target elevator in a second detection period.

[0140] In a preferred embodiment, the vibration prediction model is constructed based on a TCN architecture, and a specific model architecture is as shown in Figure 3 includes a convolution fusion module, a dilated convolution pyramid module, a feature pyramid aggregation module, a state-aware pooling module, and a prediction module. By constructing historical multi-modal time series data and using a staged training strategy to train an initial model, a final vibration prediction model is obtained. The staged training strategy includes a dilated convolution pyramid module training stage, a feature pyramid aggregation module training stage, a prediction module stage, and an end-to-end training stage. In the first three training stages, only the module parameters corresponding to the current stage are updated, and the parameters of other modules are frozen; in the end-to-end training stage, all parameters of the vibration prediction model are updated.

[0141] In a preferred embodiment, the specific prediction process of the vibration prediction model is as follows:

[0142] 1. Initial feature projection: using a 7x64 1x1 convolution kernel to automatically learn the nonlinear mapping relationship between the multi-modal time series data and the vibration, and extracting an initial feature map.

[0143] 2. Multi-level dilated convolution: using a preset four-layer dilated convolution pyramid structure to extract features for each possible event, and obtaining a plurality of corresponding dilated feature maps. The four-layer dilated convolution pyramid structure is as shown in Table 1:

[0144] Table 1 Dilated Convolution Pyramid Parameter Setting Table

[0145]

[0146] 3. Feature pyramid aggregation: performing feature pyramid aggregation on each of the dilated feature maps to obtain an aggregated feature matrix. As shown in Figure 4 the initial fusion feature map is obtained. Then, the initial fusion feature map and the dilated feature map corresponding to level 4 are spliced in the channel and 1x1 convolution to obtain the aggregated feature matrix.

[0147] 4. Determine the corresponding pooling strategy according to the main running state of the elevator, and obtain the vibration feature vector by performing a pooling operation on the aggregated feature matrix. When the main running state is the door opening or closing state, the pooling strategy is the last 1 / 3 period maximum pooling, aiming to focus on the impact at the end of the door action; when the main running state is the uniform speed running state, the pooling strategy is the global average pooling, aiming to capture the overall vibration energy distribution; when the main running state is the acceleration or deceleration state, the pooling strategy is the first 1 / 4 period maximum pooling, aiming to capture the maximum load at the moment of elevator start or deceleration.

[0148] 5. Input the vibration feature vector into a pre-set 4-class Softmax classification head to make the Softmax classification head output the corresponding vibration intensity level. Those skilled in the art can further subdivide the classification number of the Softmax classification head according to actual needs, or further adopt a multi-task prediction head, i.e., output the vibration peak time period and fault attribution analysis while predicting the vibration intensity, to realize more accurate vibration prediction.

[0149] In one possible implementation, the elevator abnormal behavior real-time detection method further includes inputting the elevator number of the target elevator, the passenger behavior type and the vibration mode into a pre-set knowledge graph when the target elevator executes the corresponding first protection measure in the second detection period, so that the knowledge graph obtains a second detection result of the target elevator in the first detection period based on a historical fault database reasoning;

[0150] comparing the first detection result with the second detection result, and determining a second protection measure of the target elevator in the second detection period according to the comparison result;

[0151] controlling the target elevator to execute the second protection measure in the remaining time period in the second detection period;

[0152] The knowledge graph is deployed in the cloud, and the passenger behavior recognition model and the vibration prediction model are deployed in the local control system of the target elevator.

[0153] The embodiment of the application further introduces knowledge graph technology, combines local real-time detection with cloud-driven knowledge graph reasoning, constructs a double detection mechanism, and further improves the safety during the operation of the elevator. Considering the hysteresis caused by knowledge graph reasoning and cloud data transmission, after obtaining the first detection result, the control target elevator first executes the first protection measure, and then generates the second detection result according to the historical database and the knowledge graph. Since the reasoning accuracy of the knowledge graph is higher than that of the local lightweight model, the first protection measure can be modified by comparing the first detection result and the second detection result, so that the target elevator executes more accurate second protection measures, further improving the safety during the operation of the elevator.

[0154] Further, within a predetermined number of detection cycles, if the consistency rate of each first detection result and the corresponding second detection result is lower than a predetermined threshold, the rule base is adjusted according to the reasoning process of each second detection result.

[0155] The embodiment of the application provides a rule base updating method. When the detection accuracy of the local model is low, the effective information in the knowledge graph and the historical fault database is fully utilized, and the rule base is adjusted according to the multiple reasoning processes of the knowledge graph, realizing intelligent continuous iteration of the detection system, ensuring high accuracy of elevator abnormal behavior detection, and improving safety during operation of the elevator.

[0156] In a preferred embodiment, in each detection cycle, the elevator number, behavior type and vibration mode are uploaded to the cloud knowledge graph, so that the cloud knowledge graph obtains the second detection result based on the historical fault database. Then the local result (first detection result) is compared with the cloud result (second detection result) to obtain the second protection measure. If the local result is consistent with the cloud result, the second protection measure is the same as the first protection measure; if the comparison result is inconsistent, the second protection measure is obtained from the knowledge graph according to the second detection result.

[0157] Embodiment two:

[0158] As shown in Figure 5 Embodiment two provides an elevator abnormal behavior real-time detection system based on deep learning, which includes an acquisition module 10, an operation state determination module 20, a behavior recognition module 30, a vibration prediction module 40 and an active protection module 50.

[0159] The acquisition module 10 is configured to acquire video stream data, a sensor data set and an elevator operation instruction data set of a target elevator in a first detection cycle, and the sensor data set includes an acceleration data set and a door machine current signal data set.

[0160] The operation state determination module 20 is configured to infer the instructions to be executed by the target elevator in the first detection period according to the sequence of the instruction data in the elevator operation instruction data set, and determine an operation state sequence of the target elevator, the operation state sequence including the operation state of the target elevator at each time point in the first detection period.

[0161] The behavior recognition module 30 is configured to input the video stream data and the operation state sequence into a preset passenger behavior recognition model, so that the passenger behavior recognition model recognizes the passenger behavior type existing in the target elevator according to the operation state sequence and the video stream data.

[0162] The vibration prediction module 40 is configured to convert the sensor data set and the operation state sequence into multi-modal time series data and input the multi-modal time series data into a preset vibration prediction model, so that the vibration prediction model predicts the vibration mode of the target elevator in the second detection period according to the multi-modal time series data.

[0163] The active protection module 50 is configured to determine a first detection result of the target elevator in the first detection period according to each of the passenger behavior type, the vibration mode and a preset rule base, and control the target elevator to execute a corresponding first protection measure in the second detection period according to the first detection result.

[0164] The passenger behavior recognition model is constructed based on a convolutional neural network model, and the vibration prediction model is constructed based on a time convolution network model.

[0165] In a possible implementation, when the transmission of the video stream data is interrupted:

[0166] According to the vibration mode of the target elevator in the second detection period, it is determined whether to execute an emergency protection measure;

[0167] If it is determined to execute the emergency protection measure, the target elevator is controlled to immediately execute the emergency protection measure, and the vibration mode and a preset first text information are sent to a designated terminal device;

[0168] If it is determined not to execute the emergency protection measure, a first detection result of the target elevator in the first detection period is determined according to the vibration mode and a preset rule base, and a corresponding first protection measure is executed in the second detection period according to the first detection result; the time period information corresponding to the video stream data and a preset second text information are sent to a designated terminal device.

[0169] In a possible implementation, the elevator abnormal behavior real-time detection system further includes a knowledge graph detection module, a comparison module and a second protection module.

[0170] The knowledge graph detection module is configured to input the elevator number of the target elevator, the passenger behavior type and the vibration mode to a preset knowledge graph when the target elevator executes the corresponding first protection measure in the second detection period, so that the knowledge graph obtains a second detection result of the target elevator in the first detection period based on a historical fault database reasoning;

[0171] The comparison module is configured to compare the first detection result with the second detection result, and determine a second protection measure of the target elevator in the second detection period according to a comparison result;

[0172] The second protection module is configured to control the target elevator to execute the second protection measure in a remaining time period in the second detection period;

[0173] The knowledge graph is deployed in the cloud, and the passenger behavior recognition model and the vibration prediction model are deployed in a local control system of the target elevator.

[0174] Further, in a plurality of preset detection periods, if a consistency rate of each first detection result and a corresponding second detection result is lower than a preset threshold, the rule base is adjusted according to a reasoning process of each second detection result.

[0175] The embodiment of the present application provides a real-time elevator abnormal behavior detection system based on deep learning. The system realizes full-dimensional real-time monitoring of elevator abnormal behavior by collecting video stream data, sensor data set and elevator operation instruction data set in real time, and comprehensively judging the output results of the behavior recognition model constructed by CNN and the vibration prediction model constructed by TCN, breaking through the adaptability limitation of traditional single-mode detection to complex scenes and improving the safety in the process of elevator operation. Specifically, the embodiment of the present application first determines the running state sequence of the target elevator by state reasoning based on the elevator operation instruction data set, and then determines the passenger behavior type existing in the target elevator according to the video stream data and the running state sequence, that is, determines the external factors affecting the safe operation of the elevator in the current period. On the other hand, the multi-modal time series data is constructed by combining the sensor data set and the running state sequence, and the vibration mode of the target elevator in the future is further predicted based on the multi-modal time series data, that is, it is determined whether there is an abnormal state of the elevator itself in the next period. Finally, the current passenger behavior type and the future vibration mode are combined to generate a corresponding first detection result through a preset rule base, so as to realize accurate prediction of the risk of elevator operation, determine the first protection measure to be executed in the second detection period in advance, overcome the problems of low recognition accuracy of elevator abnormal behavior and slow response of active protection measures in the prior art, and effectively improve the safety in the process of elevator operation.

[0176] The working principle and step flow of the embodiment can be but not limited to referring to the related description of embodiment one.

[0177] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only for specific embodiments of the present application and is not intended to limit the protection scope of the present application. It is particularly pointed out that any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A deep learning-based elevator abnormal behavior real-time detection method, characterized in that, The method comprises: obtaining video stream data, a sensor data set and an elevator operation instruction data set of a target elevator in a first detection period, the sensor data set comprising an acceleration data set and a door machine current signal data set; inference of instructions to be executed by the target elevator in the first detection period according to the order of each instruction data in the elevator operation instruction data set, and determination of a running state sequence of the target elevator, the running state sequence comprising the running state of the target elevator at each time in the first detection period; inputting the video stream data and the running state sequence into a preset passenger behavior recognition model to enable the passenger behavior recognition model to recognize the passenger behavior type existing in the target elevator according to the running state sequence and the video stream data; converting the sensor data set and the running state sequence into multi-modal time series data and inputting the multi-modal time series data into a preset vibration prediction model to enable the vibration prediction model to predict the vibration mode of the target elevator in a second detection period according to the multi-modal time series data; the conversion of the sensor data set and the running state sequence into multi-modal time series data comprises: synchronizing the acceleration data set, the door machine current signal data set and the running state sequence to a uniform time sequence point through interpolation or resampling operation, and performing data normalization to obtain corresponding acceleration time series data, door machine current signal time series data and running state time series data; and splicing and fusing the acceleration time series data, the door machine current signal time series data and the running state time series data along the feature dimension to obtain the multi-modal time series data; the vibration prediction model predicting the vibration mode of the target elevator in the second detection period according to the multi-modal time series data comprises: projecting the multi-modal time series data through a plurality of 1x1 convolution kernels to obtain an initial feature map, the feature dimension of the initial feature map being equal to the number of the convolution kernels; performing a plurality of dilated convolution operations on the initial feature map to obtain a plurality of corresponding dilated feature maps, wherein the dilation rates in each dilated convolution operation are different, and the input data of the current dilated convolution operation is the dilated feature map of the last dilated convolution operation; performing feature pyramid aggregation according to each dilated feature map to obtain an aggregated feature matrix; determining a corresponding pooling strategy according to the running state change in the multi-modal time series data; performing a pooling operation on the aggregated feature matrix according to the pooling strategy to obtain a vibration feature vector; and inputting the vibration feature vector into a preset Softmax classification head to enable the Softmax classification head to output the vibration mode of the target elevator in the second detection period; determining a first detection result of the target elevator in the first detection period according to each passenger behavior type, the vibration mode and a preset rule base, and controlling the target elevator to execute a corresponding first protection measure in the second detection period according to the first detection result; wherein the passenger behavior recognition model is constructed based on a convolutional neural network model, and the vibration prediction model is constructed based on a temporal convolution network model.

2. The deep learning-based elevator abnormal behavior real-time detection method of claim 1, wherein, When transmission of the video stream data is interrupted: According to the vibration mode of the target elevator in the second detection period, it is determined whether to execute an emergency protection measure; If it is determined to execute the emergency protection measure, the target elevator is controlled to immediately execute the emergency protection measure, and the vibration mode and preset first text information are sent to a designated terminal device; If it is determined not to execute the emergency protection measure, according to the vibration mode and a preset rule base, a first detection result of the target elevator in a first detection period is determined, and according to the first detection result, the target elevator is controlled to execute a corresponding first protection measure in a second detection period; The time period information corresponding to the video stream data and preset second text information are sent to a designated terminal device.

3. The real-time elevator abnormal behavior detection method based on deep learning according to claim 1, characterized in that, The passenger behavior recognition model identifies a passenger behavior type existing in the target elevator according to the running state sequence and the video stream data, including: According to the duration of each running state in the running state sequence, a main running state of the target elevator is determined; According to the main running state, weight distribution of the spatial feature vector and the spatio-temporal feature vector is determined; According to the weight distribution of the spatial feature vector and the spatio-temporal feature vector, the spatial feature vector and the spatio-temporal feature vector are weighted and fused and then input into a preset fully connected layer, so that the fully connected layer outputs the passenger behavior type existing in the target elevator. According to the main running state, a corresponding sampling frequency is determined; 4. The real-time elevator abnormal behavior detection method based on deep learning according to claim 3, characterized in that, According to the sampling frequency, a plurality of key video frames are collected from the video stream data; Each key video frame is input into the 2D-CNN branch for image feature extraction to obtain a corresponding initial image feature vector; Based on time sequence, the initial image feature vectors are combined to obtain a spatial feature sequence containing a time dimension; Along the time dimension, a global average pooling operation is performed on the spatial feature sequence to obtain a corresponding spatial feature vector. According to the main running state, a corresponding sampling frequency is determined; According to the sampling frequency, a plurality of key video frames are collected from the video stream data; 5. The real-time elevator abnormal behavior detection method based on deep learning according to claim 3, characterized in that, Each key video frame is input into the 2D-CNN branch for image feature extraction to obtain a corresponding initial image feature vector; Based on time sequence, the initial image feature vectors are combined to obtain a spatial feature sequence containing a time dimension; Along the time dimension, a global average pooling operation is performed on the spatial feature sequence to obtain a corresponding spatial feature vector. According to the main running state, a corresponding sampling frequency is determined; According to the sampling frequency, a plurality of key video frames are collected from the video stream data; Each key video frame is input into the 2D-CNN branch for image feature extraction to obtain a corresponding initial image feature vector; Based on time sequence, the initial image feature vectors are combined to obtain a spatial feature sequence containing a time dimension; Along the time dimension, a global average pooling operation is performed on the spatial feature sequence to obtain a corresponding spatial feature vector.

6. The deep learning-based elevator abnormal behavior real-time detection method according to any one of claims 1-5, characterized in that, The elevator abnormal behavior real-time detection method further comprises inputting the elevator number of the target elevator, the passenger behavior type and the vibration mode to a preset knowledge graph when the target elevator executes corresponding first protection measures in a second detection period, so that the knowledge graph obtains a second detection result of the target elevator in the first detection period based on a historical fault database reasoning; comparing the first detection result with the second detection result, and determining a second protection measure of the target elevator in the second detection period according to the comparison result; controlling the target elevator to execute the second protection measure in the remaining time period in the second detection period; wherein the knowledge graph is deployed in the cloud, and the passenger behavior recognition model and the vibration prediction model are deployed in the local control system of the target elevator.

7. The deep learning-based elevator abnormal behavior real-time detection method of claim 6, wherein, In a preset number of detection periods, if the consistency rate of each first detection result and the corresponding second detection result is lower than a preset threshold, the rule base is adjusted according to the reasoning process of each second detection result.

8. A deep learning-based elevator abnormal behavior real-time detection system, characterized by, It comprises an acquisition module, a running state determination module, a behavior recognition module, a vibration prediction module and an active protection module; wherein the acquisition module is used to acquire video stream data, sensor data set and elevator running instruction data set of a target elevator in a first detection period, and the sensor data set comprises acceleration data set and door machine current signal data set; The running state determination module is used to infer the instructions that need to be executed by the target elevator in the first detection period according to the sequence of each instruction data in the elevator running instruction data set, and then determine the running state sequence of the target elevator, which comprises the running state of the target elevator at each time in the first detection period; The behavior recognition module is used to input the video stream data and the running state sequence into a preset passenger behavior recognition model, so that the passenger behavior recognition model recognizes the passenger behavior type existing in the target elevator according to the running state sequence and the video stream data; The vibration prediction module is used to convert the sensor data set and the running state sequence into multi-modal time series data and input them into a preset vibration prediction model, so that the vibration prediction model predicts the vibration mode of the target elevator in a second detection period according to the multi-modal time series data; The conversion of the sensor data set and the running state sequence into multi-modal time series data comprises synchronizing the acceleration data set, the door machine current signal data set and the running state sequence to a unified time sequence point through interpolation or resampling operation, and performing data normalization to obtain corresponding acceleration time series data, door machine current signal time series data and running state time series data; the acceleration time series data, the door machine current signal time series data and the running state time series data are spliced and fused along the feature dimension to obtain the multi-modal time series data; The vibration prediction model predicts a vibration mode of the target elevator in a second detection period according to the multi-modal time series data, including: performing feature projection on the multi-modal time series data through a plurality of 1x1 convolution kernels respectively to obtain initial feature maps, a feature dimension of the initial feature maps being equal to a number of the convolution kernels; performing a plurality of dilation convolution operations on the initial feature maps to obtain a plurality of corresponding dilation feature maps, wherein dilation rates in the respective dilation convolution operations are different, and input data of a current dilation convolution operation is a dilation feature map of a previous dilation convolution operation; performing feature pyramid aggregation according to the respective dilation feature maps to obtain an aggregated feature matrix; determining a corresponding pooling strategy according to a change in a running state in the multi-modal time series data; performing a pooling operation on the aggregated feature matrix according to the pooling strategy to obtain a vibration feature vector; and inputting the vibration feature vector into a preset Softmax classification head to enable the Softmax classification head to output the vibration mode of the target elevator in the second detection period; The active protection module is configured to determine a first detection result of the target elevator in a first detection period according to the respective passenger behavior types, the vibration mode, and a preset rule base, and to control the target elevator to perform a corresponding first protection measure in a second detection period according to the first detection result. The passenger behavior recognition model is constructed based on a convolutional neural network model, and the vibration prediction model is constructed based on a temporal convolution network model.

Images