Multi-modal feature fusion elevator early warning method in big data environment

Abstract

The invention relates to the technical field of elevator early warning, and discloses a multi-modal feature fusion elevator early warning method in a big data environment, which comprises the following steps: step S1, collecting video feature data, voiceprint feature data and operation feature data in an elevator operation process; s2, video feature extraction: extracting a video feature vector of a key video frame from the collected video feature data by using a video feature extraction model; and S3, voiceprint feature extraction: extracting voiceprint feature vectors from the collected voiceprint feature data by using a Mel frequency cepstrum coefficient. According to the method, the video feature data, the voiceprint feature data and the operation feature data are fused, so that the problems of low single-mode data recognition rate and high false alarm rate of a traditional mode are solved, the accuracy, reliability and robustness of early warning are improved, accurate prediction of elevator operation risks can be realized, and the safety of elevator operation is improved. Therefore, the quality safety of the elevator is effectively improved through preventive maintenance measures.

Description

Technical Field

[0001] The present invention relates to the technical field of elevator early warning, and in particular to an elevator early warning method for multimodal feature fusion in a big data environment. Background Art

[0002] The safe and reliable operation of elevators is closely related to people's normal work and life.

[0003] At present, there are also attempts to adopt some technical means to supervise and manage, such as actively capturing the abnormal video features of elevator operation through a single-mode camera device, and actively warning of abnormal elevator status through a single-mode operation monitoring device. However, since the single-mode feature data is affected by factors such as the use environment, installation conditions, elevator structure and passenger interference, the recognition rate is low and the false alarm rate is high, making it difficult to replicate and promote. Therefore, an elevator warning method based on multi-modal feature fusion in a big data environment is proposed. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present invention provides an elevator early warning method based on multimodal feature fusion in a big data environment.

[0005] The present invention is implemented by the following technical solution: an elevator early warning method based on multimodal feature fusion in a big data environment, comprising the following steps:

[0006] Step S1: Collecting video feature data, voiceprint feature data and operation feature data during the operation of the elevator;

[0007] Step S2: video feature extraction, using a video feature extraction model to extract video feature vectors of key video frames from the collected video feature data;

[0008] Step S3: voiceprint feature extraction, using Mel frequency cepstrum coefficients to extract voiceprint feature vectors from the collected voiceprint feature data;

[0009] Step S4: extracting operation features, using word embedding technology to extract operation feature vectors from the collected operation feature data;

[0010] Step S5: Fusing the video feature vector, the voiceprint feature vector and the operation feature vector to form a comprehensive feature data set;

[0011] Step S6: constructing an elevator early warning model;

[0012] Step S7: Model training and optimization:

[0013] Classify the comprehensive feature data set into a training set and a test set, and use the training set and the test set to train and optimize the model;

[0014] Step S8: Use the trained model to predict and judge the collected characteristic data of elevator operation.

[0015] Through the above technical scheme, the role and effect corresponding to the above-mentioned rights and obligations are pointed out.

[0016] As a further improvement of the above scheme, the video feature data in step S1 includes the elevator environment and personnel behavior feature information; the voiceprint feature data mainly includes sound intensity and sound frequency information based on sound perception; the operation feature data mainly includes the operation floor, speed, direction, door opening and closing, and three-dimensional acceleration and vibration information monitored by the operation perception sensor.

[0017] As a further improvement of the above scheme, the video feature extraction model in step S2 extracts features from the video feature data to form key operating modes, abnormal behaviors or object information during the operation of the elevator, thereby capturing abnormal visual information during the operation of the elevator and forming a video feature vector.

[0018] As a further improvement of the above solution, the voiceprint feature vector in step S3 effectively represents the spectrum envelope of the voiceprint signal, thereby realizing the capture of abnormal sounds in the car and abnormal noise information in the hoistway when the elevator is running.

[0019] As a further improvement of the above solution, the operation data collected by the operation perception module in step v4 is converted into an operation feature vector using word embedding technology, thereby capturing fault information during elevator operation.

[0020] As a further improvement of the above solution, in step S5, the following sampling method is used to perform vector fusion:

[0021] V=λ1x1+λ2x2+λ3x3

[0022] Among them, v is the fused comprehensive feature dataset, λ1x1 is the video feature vector, λ2x2 is the voiceprint feature vector, and λ3x3 is the running feature vector.

[0023] As a further improvement of the above solution, the elevator early warning model in step S6 includes:

[0024] Input layer: used to receive comprehensive feature data set detection data;

[0025] Multiple groups of convolutional layers: perform convolution calculations on the detection data;

[0026] Flattening layer: flattens the feature data calculated by convolution to form one-dimensional data;

[0027] Multiple groups of fully connected layers: extract features from the flattened one-dimensional data;

[0028] Output layer: used to output the predicted value calculated by the model;

[0029] When the predicted value after the elevator early warning model predicts contains all the vectors of λ1x1, λ2x2, and λ3x3, an early warning is issued; when the predicted value after the elevator early warning model predicts contains any combination vector of λ1x1, λ2x2 or λ1x1, λ3x3 or λ2x2, λ3x3, an early warning is suspected; when the predicted value after the elevator early warning model predicts contains any vector of λ1x1 or λ2x2 or λ3x3, it cannot be judged as an early warning for the time being, and an early warning can only be issued after confirmation by other conditions, otherwise this early warning judgment is terminated.

[0030] As a further improvement of the above scheme, the method for dividing the training set and the test set in step S7 is as follows: the data set is randomly divided into a training set and a test set in a ratio of 8:2; the model training inputs the content of the training set into the model and uses the optimizer and activation function to perform batch training. After the training, the accuracy is evaluated in the test set and compared with the previous model accuracy. No training is performed until the accuracy remains basically unchanged.

[0031] As a further improvement of the above scheme, the optimizer adopts either adam or sgd; the activation function adopts either Relu or Liner; the number of training rounds is 120 to 150 rounds, and the batch size is 20 to 50 times; the accuracy evaluation is evaluated using RMSE and MAE formulas.

[0032] As a further improvement of the above solution, the video feature extraction model in step S2 includes:

[0033] Input layer: accepts video frames as input;

[0034] Feature extraction layer: perform deep feature extraction;

[0035] Feature fusion layer: fuses features at different levels to improve detection accuracy;

[0036] Feature detection layer: Outputs the bounding box, category, and confidence of the object.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The present invention solves the problems of low single-modal data recognition rate and high false alarm rate in traditional methods by integrating video feature data, voiceprint feature data and operation feature data, improves the accuracy, reliability and robustness of early warning, and can achieve accurate prediction of elevator operation risks, thereby effectively improving elevator quality and safety through preventive maintenance measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1A schematic diagram of the structure provided by the present invention;

[0040] Figure 2 A schematic diagram of the structure provided by the present invention. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0042] Embodiment 1:

[0043] Please combine Figure 1-Figure 2 , an elevator early warning method for multimodal feature fusion in a big data environment of this embodiment includes the following steps:

[0044] Step S1: Collecting video feature data, voiceprint feature data and operation feature data during the operation of the elevator;

[0045] The video feature data includes the elevator environment and personnel behavior feature information; the voiceprint feature data mainly includes the sound intensity and sound frequency information based on sound perception; the operation feature data mainly includes the operation floor, speed, direction, door opening and closing, and three-dimensional acceleration and vibration information monitored by the operation perception sensor;

[0046] Step S2: video feature extraction, using a video feature extraction model to extract video feature vectors of key video frames from the collected video feature data;

[0047] The video feature extraction model extracts features from the video feature data to form key operation modes, abnormal behaviors or object information during the operation of the elevator, thereby capturing abnormal visual information during the operation of the elevator and forming a video feature vector;

[0048] Step S3: voiceprint feature extraction, using Mel frequency cepstrum coefficients to extract voiceprint feature vectors from the collected voiceprint feature data;

[0049] The voiceprint feature vector effectively represents the spectrum envelope of the voiceprint signal, thereby capturing abnormal sounds in the elevator car and abnormal noise information in the shaft when the elevator is running;

[0050] Step S4: extracting operation features, using word embedding technology to extract operation feature vectors from the collected operation feature data;

[0051] The operation data collected by the operation perception module are converted into operation feature vectors using word embedding technology to capture the fault information of the elevator during operation.

[0052] Step S5: Fusing the video feature vector, the voiceprint feature vector and the operation feature vector to form a comprehensive feature data set;

[0053] Sampling is performed as follows for vector fusion:

[0054] V=λ1x1+λ2x2+λ3x3

[0055] Among them, v is the integrated feature dataset after fusion, λ1x1 is the video feature vector, λ2x2 is the voiceprint feature vector, and λ3x3 is the operation feature vector;

[0056] Step S6: constructing an elevator early warning model;

[0057] Step S7: Model training and optimization:

[0058] Classify the comprehensive feature data set into a training set and a test set, and use the training set and the test set to train and optimize the model;

[0059] Step S8: Use the trained model to predict and judge the collected characteristic data of elevator operation.

[0060] Embodiment 2:

[0061] The elevator early warning model includes:

[0062] Input layer: used to receive comprehensive feature data set detection data;

[0063] Multiple groups of convolutional layers: perform convolution calculations on the detection data;

[0064] Flattening layer: flattens the feature data calculated by convolution to form one-dimensional data;

[0065] Multiple groups of fully connected layers: extract features from the flattened one-dimensional data;

[0066] Output layer: used to output the predicted value calculated by the model;

[0067] When the predicted value after the elevator early warning model predicts contains all the vectors of λ1x1, λ2x2, and λ3x3, an early warning is issued; when the predicted value after the elevator early warning model predicts contains any combination vector of λ1x1, λ2x2 or λ1x1, λ3x3 or λ2x2, λ3x3, an early warning is suspected; when the predicted value after the elevator early warning model predicts contains any vector of λ1x1 or λ2x2 or λ3x3, it cannot be judged as an early warning for the time being, and an early warning can only be issued after confirmation by other conditions, otherwise this early warning judgment is terminated.

[0068] Embodiment 3:

[0069] The video feature extraction model includes:

[0070] Input layer: accepts video frames as input;

[0071] Feature extraction layer: Use CSPDarknet53 in Yolov4 as the backbone network for deep feature extraction;

[0072] Feature fusion layer: Through structures such as SPP (spatial pyramid pooling) and PAN (path aggregation network), features at different levels are fused to improve detection accuracy;

[0073] Feature detection layer: Outputs the bounding box, category, and confidence of the object.

[0074] Embodiment 4:

[0075] An elevator early warning method based on multimodal feature fusion in a big data environment comprises the following steps:

[0076] S11. Install a set of data acquisition terminals in the elevator car, and the data acquisition terminal is connected to a video acquisition module, a voiceprint acquisition module, and an operation perception module installed on the top of the car. The data acquisition terminal collects video data in the video acquisition module, voiceprint data in the voiceprint acquisition module, and operation data in the operation perception module in real time;

[0077] S12. Extract feature vectors of key video frames from the camera acquisition module through the video feature extraction model embedded in the data acquisition terminal. The video feature extraction model learns and identifies feature data in the video, including key operating modes, abnormal behaviors or objects, etc., so as to capture visual information related to elevator abnormalities, such as elevator shaking, door blocking, door picking and other dangerous information in the elevator;

[0078] S13, extracting voiceprint feature vectors from the voiceprint acquisition module through the MFCC Mel-frequency cepstrum coefficients embedded in the data acquisition terminal. The voiceprint feature vectors effectively represent the spectrum envelope of the voiceprint signal, thereby capturing information such as abnormal sounds in the elevator car and abnormal noise in the shaft when the elevator is running;

[0079] S14. Extract operation feature data (such as door opening and closing records, floor stop information, operation time, start, braking and vibration, etc.) from the operation perception module through the operation feature recognition algorithm embedded in the data acquisition terminal, and convert these structured data into feature vectors using word embedding technology, so as to capture abnormalities in the elevator operation process, such as abnormal vibration of the elevator, non-level stop, door opening and running, repeated door opening and closing, and long-term door opening, etc.

[0080] S15. Through the vector concatenation operation embedded in the data acquisition terminal, the feature vectors of the three modes are fused into a comprehensive feature data set. On the fused comprehensive feature data set, a neural network regression model is trained to identify abnormal warning features of the elevator. The neural network regression model learns the abnormal pattern of the elevator from the labeled data through supervised learning, and can accurately predict new data;

[0081] S16. When the data acquisition terminal detects abnormal operation of the elevator, the data acquisition terminal generates elevator warning information and notifies the elevator management and maintenance personnel immediately so that preventive measures can be taken in time to improve the safety factor of the elevator operation;

[0082] Taking the typical abnormal vibration stop fault in elevator operation as an example, the camera acquisition module based on the video feature extraction model extracts the feature vector λ1x1 of the car video shaking, the voiceprint acquisition module based on MFCC extracts the feature vector λ2x2 of the car abnormal noise, and the Word Embedding extracts the feature vector λ3x3 of the elevator abnormal vibration from the operation perception module. The data acquisition terminal fuses the feature vectors of these three modes into a comprehensive feature data set v=λ1x1+λ2x2+λ3x3 based on the vector splicing technology. If the feature data set contains all the vectors of λ1x1, λ2x2, and λ3x3, the data acquisition terminal will determine it as an abnormal vibration elevator stop fault and issue a warning; if the feature data set contains any combination of vectors of λ1x1, λ2x2 or λ1x1, λ3x3 or λ2x2, λ3x3, the data acquisition terminal will determine it as a suspected abnormal vibration elevator stop fault and issue a warning; if the feature data set contains any vector of λ1x1 or λ2x2 or λ3x3, the data acquisition terminal cannot determine it as an abnormal vibration elevator stop fault for the time being, and will issue a warning only after confirmation with the assistance of other conditions, otherwise this warning judgment will be terminated.

[0083] The present invention solves the problems of low single-modal data recognition rate and high false alarm rate in traditional methods by integrating video feature data, voiceprint feature data and operation feature data, improves the accuracy, reliability and robustness of early warning, and can achieve accurate prediction of elevator operation risks, thereby effectively improving elevator quality and safety through preventive maintenance measures.

[0084] The above-mentioned embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of protection of the present invention. Any non-substantial changes and substitutions made by technicians in this field on the basis of the present invention shall fall within the scope of protection required by the present invention.

Claims

1. An elevator early warning method based on multimodal feature fusion in a big data environment, characterized in that: The following steps are involved: Step S1: Collecting video feature data, voiceprint feature data and operation feature data during the operation of the elevator; Step S2: video feature extraction, using a video feature extraction model to extract video feature vectors of key video frames from the collected video feature data; Step S3: voiceprint feature extraction, using Mel frequency cepstrum coefficients to extract voiceprint feature vectors from the collected voiceprint feature data; Step S4: extracting operation features, using word embedding technology to extract operation feature vectors from the collected operation feature data; Step S5: Fusing the video feature vector, the voiceprint feature vector and the operation feature vector to form a comprehensive feature data set; Step S6: constructing an elevator early warning model; Step S7: Model training and optimization: Classify the comprehensive feature data set into a training set and a test set, and use the training set and the test set to train and optimize the model; Step S8: Use the trained model to predict and judge the collected characteristic data of elevator operation.

2. The elevator early warning method based on multimodal feature fusion in a big data environment as claimed in claim 1, characterized in that: The video feature data in step S1 includes the environment inside the elevator and the behavior feature information of the personnel; The voiceprint feature data mainly includes the sound intensity and sound frequency information based on sound perception; the operation feature data mainly includes the operation floor, speed, direction, door opening and closing, and three-dimensional acceleration and vibration information monitored by the operation perception sensor.

3. The elevator early warning method based on multimodal feature fusion in a big data environment as claimed in claim 1, characterized in that: In step S2, the video feature extraction model extracts features from the video feature data to form key operation modes, abnormal behaviors or object information during the operation of the elevator, thereby capturing abnormal visual information during the operation of the elevator and forming a video feature vector.

4. The elevator early warning method of multimodal feature fusion in a big data environment according to claim 1, characterized in that: The voiceprint feature vector in step S3 effectively represents the spectrum envelope of the voiceprint signal, thereby capturing abnormal sounds in the elevator car and abnormal noise information in the hoistway when the elevator is running.

5. The elevator early warning method based on multimodal feature fusion in a big data environment as claimed in claim 1, characterized in that: In the step S4, the operation data collected by the operation perception module are converted into operation feature vectors using word embedding technology, thereby capturing fault information during elevator operation.

6. The elevator early warning method of multimodal feature fusion in a big data environment according to claim 1, characterized in that: In step S5, the following sampling method is used to perform vector fusion: v=λ1x1+λ2x2+λ3x3 Among them, v is the fused comprehensive feature dataset, λ1x1 is the video feature vector, λ2x2 is the voiceprint feature vector, and λ3x3 is the running feature vector.

7. The elevator early warning method of multimodal feature fusion in a big data environment according to claim 1, characterized in that: The elevator early warning model in step S6 includes: Input layer: used to receive comprehensive feature data set detection data; Multiple groups of convolutional layers: perform convolution calculations on the detection data; Flattening layer: flattens the feature data calculated by convolution to form one-dimensional data; Multiple groups of fully connected layers: extract features from the flattened one-dimensional data; Output layer: used to output the predicted value calculated by the model; When the predicted value after the elevator early warning model predicts contains all the vectors of λ1x1, λ2x2, and λ3x3, an early warning is issued; when the predicted value after the elevator early warning model predicts contains any combination vector of λ1x1, λ2x2 or λ1x1, λ3x3 or λ2x2, λ3x3, an early warning is suspected; when the predicted value after the elevator early warning model predicts contains any vector of λ1x1 or λ2x2 or λ3x3, it cannot be judged as an early warning for the time being, and an early warning can only be issued after confirmation by other conditions, otherwise this early warning judgment is terminated.

8. The elevator early warning method of multimodal feature fusion in a big data environment as claimed in claim 1, characterized in that: The training set and test set division method in step S7: randomly divide the data set into a training set and a test set in a ratio of 8:2; Model training inputs the training set content into the model and uses the optimizer and activation function to perform batch training. After the training, the accuracy is evaluated in the test set and compared with the previous model accuracy. Training will not be continued until the accuracy remains basically unchanged.

9. The elevator early warning method of multimodal feature fusion in a big data environment according to claim 8, characterized in that: The optimizer adopts any one of adam or sgd; the activation function adopts any one of Relu or Liner; The number of training rounds is 120 to 150, and the batch size is 20 to 50 times. The accuracy evaluation is performed using the RMSE and MAE formulas.

10. The elevator early warning method of multimodal feature fusion in a big data environment according to claim 1, characterized in that: The video feature extraction model in step S2 includes: Input layer: accepts video frames as input; Feature extraction layer: perform deep feature extraction; Feature fusion layer: fuses features at different levels to improve detection accuracy; Feature detection layer: Outputs the bounding box, category, and confidence of the object.

Images