A method and system for detecting abnormal noise and alarm sound of elevator based on deep learning

Through deep learning methods, the elevator audio signals are directly collected and processed, and the classification detection of elevator noise, alarm sound and normal situation is realized. The problem of cumbersome detection and inability to detect elevator noise and alarm sound at the same time in the existing technology is solved, and the problem of cumbersome detection and inability to detect elevator noise and alarm sound at the same time is realized. Fast and accurate elevator noise and alarm sound detection reduces the dependence and cost of hardware equipment.

CN118658490BActive Publication Date: 2024-11-26ZHEJIANG NEW ZAILING TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411133058.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-26
Estimated Expiration
2044-08-19

AI Technical Summary

Technical Problem

The prior art has cumbersome algorithm processes in the detection of elevator noise and alarm sounds, and it is impossible to detect elevator noise and alarm sounds at the same time, and it relies on hardware equipment, which is expensive.

Method used

By adopting deep learning method, by directly collecting audio signals in elevator monitoring for preprocessing, the detection model of integrated feature extraction is input to realize the classification detection of elevator noise, alarm sound and normal situation, and determine whether to report according to the detection probability. Monitoring platform.

Benefits of technology

It realizes fast and accurate detection of elevator noise and alarm sound, reduces dependence on hardware equipment, saves costs, and is suitable for complex elevator scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118658490B_ABST
    Figure CN118658490B_ABST
Patent Text Reader

Abstract

The present invention relates to a method and system for detecting abnormal noise and alarm sound of elevator based on deep learning. The method is as follows: S1, collect acceleration data, judge whether the acceleration exceeds the set acceleration threshold to judge whether there is abnormal vibration in the current elevator; S2, collect monitoring video data of i seconds before and j seconds after the start time of the corresponding abnormal vibration, a total of i+j seconds; S3, obtain audio signal from monitoring video data; S4, perform data preprocessing on audio signal; S5, input preprocessed audio signal into detection model; detection model outputs detected audio type and corresponding detection probability; audio type includes: abnormal noise of elevator, alarm sound or normal; S6, respectively judge whether the detection probability of abnormal noise or alarm sound of elevator exceeds the set detection probability threshold, if it exceeds the detection probability threshold, then the audio type and corresponding detection probability and corresponding elevator number and detection time are reported to the monitoring platform in real time. The detection method is fast and accurate.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to elevator abnormal noise and alarm sound detection, and in particular to an elevator abnormal noise and alarm sound detection method and system based on deep learning. Background Art

[0002] As an indispensable vertical transportation tool in modern high-rise buildings, elevators provide great convenience for people's lives. However, the frequent use of elevators may cause various abnormal faults, leading to elevator accidents and threatening the lives of passengers. Among them, abnormal noises may be generated due to equipment problems, mechanical vibrations, door failures, etc., and sometimes the elevator alarm system may be triggered to generate alarm sounds. Abnormal noises or alarm sounds generated by elevators indicate that the elevator may have a fault and require maintenance and repair. In the absence of monitoring technology, it can only rely on manual regular inspections, which is extremely inefficient and difficult to detect and deal with problems in a timely manner. By automatically monitoring the abnormal noises and alarm sounds of elevators through monitoring technology, the elevator maintenance department can be informed in time to repair the elevator and eliminate safety hazards. However, the elevator operating environment is complex and there are many interfering sounds, such as mechanical noise, environmental noise, and electronic equipment noise. In addition, the types of abnormal noises and alarm sounds of elevators are diverse and may be similar to the normal operating sounds, making them difficult to distinguish.

[0003] Traditional monitoring technology requires additional hardware equipment to perform special detection. For example, patent CN208907833U discloses a method for detecting abnormal noises on elevator anti-ropes by installing a detection device containing a sound sensor. This method only detects local components, has limitations, and relies on hardware equipment, which is costly. At present, a more convenient monitoring technology can obtain audio signals from monitoring equipment and further extract audio features for detection. For example, patent CN112193959A discloses a method and system for detecting abnormal noises in elevators, which collects audio data monitored by elevators, extracts and selects audio features, and then identifies whether abnormal noises occur based on the detection model. Although this method reduces the requirements for hardware equipment, the fusion of multiple feature selection algorithms and multiple detection models results in a cumbersome algorithm process, and can only identify abnormal noises in elevators, and cannot be used to detect alarm sounds. Patent CN112960506A discloses an elevator alarm sound detection system based on audio features. After collecting elevator monitoring audio data, a first audio feature and a second audio feature are constructed. The first audio feature detection model uses an image classification model VGGNet, and the second audio feature detection model is a deep neural network model. This method combines two detection models and has a good recognition effect on alarm sounds, but it is still relatively cumbersome and requires two types of feature extraction and model fusion, and it is not suitable for monitoring abnormal noises in elevators. Summary of the invention

[0004] The method proposed in the present invention directly collects the audio signal in the elevator monitoring, performs simple preprocessing on the audio, and directly inputs it into the deep learning detection model integrated with feature extraction for detection. The model can realize the classification detection of abnormal noise, alarm sound and normal situation of the elevator, and further determine whether to report to the monitoring platform according to whether the detection probability exceeds the set detection threshold, thereby realizing the whole process. It solves the problem that the existing technology is cumbersome to detect abnormal noise and alarm sound of the elevator and cannot detect abnormal noise and alarm sound of the elevator at the same time.

[0005] The present invention provides the following technical solution: a method for detecting abnormal noise and alarm sound of an elevator based on deep learning, comprising the following steps:

[0006] S1. Collect acceleration data and determine whether the acceleration exceeds the set acceleration threshold to determine whether the current elevator has abnormal vibration;

[0007] S2, collecting monitoring video data from i seconds before and j seconds after the start time of the abnormal vibration, a total of i+j seconds;

[0008] S3, obtaining an audio signal from the surveillance video data;

[0009] S4, performing data preprocessing on the audio signal;

[0010] S5. Input the preprocessed audio signal into the detection model; the detection model outputs the detected audio type and the corresponding detection probability; the audio type includes: abnormal elevator sound, alarm sound or normal;

[0011] S6. Determine whether the detection probability of the elevator abnormal sound or alarm sound exceeds the set detection probability threshold. If it exceeds the detection probability threshold, report the audio type and the corresponding detection probability and the corresponding elevator number and detection time to the monitoring platform in real time.

[0012] Furthermore, the steps of data preprocessing are as follows:

[0013] Remove abnormal audio data, which refers to abnormal data with all zeros obtained due to audio extraction failure caused by problems with the acquisition device;

[0014] 1) Use the average mixing method to convert the two-channel audio into a mono audio; the average mixing method means taking the average of the left channel audio and the right channel audio at each sampling point as the converted mono audio;

[0015] 2) Determine whether the sampling frequency of the mono audio is the set sampling frequency fs; if not, resample to obtain an audio signal with a sampling frequency of fs; if so, do nothing;

[0016] 3) Use a sliding window to segment the audio signal. The window length is w, the step length is s, and the end of the audio signal that is less than the sliding window length is filled with 0, and finally a batch of sub-audio segments of equal length are obtained;

[0017] A single sub-audio segment refers to one of a batch of fixed-length audio sub-segments obtained after data preprocessing of an audio segment.

[0018] The data dimension after data preprocessing is (n,fs×w), where n is the number of sub-audio segments, and n is calculated according to the following formula: n=ceil((tw) / s)+1, t is the original audio duration, w is the window duration, s is the window step, and ceil() means taking the smallest integer greater than or equal to the number in the brackets.

[0019] Furthermore, the detection model is obtained by training an open-source audio classification model on a self-made elevator audio dataset;

[0020] The self-made elevator audio data set is derived from the annotated elevator abnormal sound, alarm sound and normal audio data in different elevator scenarios;

[0021] The model training refers to modifying the network structure and performing additional training based on the open source audio classification model using a self-made elevator audio data set, and the initial parameters of the model come from the open source audio classification model;

[0022] After model training, perform performance testing on the model. When the model test accuracy meets the actual usage requirements, it can be used for model deployment. Otherwise, the model structure and parameters need to be optimized and then retrained until it meets the actual usage requirements.

[0023] The actual usage requirement means that the detection accuracy of the model on the test set reaches more than 90%.

[0024] Furthermore, the detection model adopts a neural network model network, including a logarithmic Mel spectrum feature extraction module, 6 convolution modules and 1 fully connected layer.

[0025] Furthermore, the logarithmic Mel spectrum feature extraction module: applies the short-time Fourier transform to the time domain waveform of the audio signal to calculate the time-frequency spectrum; then passes through the Mel filter bank to obtain the logarithmic Mel spectrum feature; the window function of the short-time Fourier transform selects the Hamming window, the window length is n_windows sampling points, the window step is step, and the number of Mel filter banks is n_mels; the single sub-audio signal with an input shape of (fs×w,) passes through the logarithmic Mel spectrum feature extraction module and becomes (ceil(fs×w / step),n_mels,1), where ceil() represents the smallest integer greater than or equal to the number in the brackets;

[0026] The first convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n1 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (ceil(fs×w / step),n_mels,1) is transformed into a feature with dimension (floor(fs×w / (2×step)),n_mels / 2,n1). The floor(x) function represents rounding down, that is, the maximum integer not greater than x.

[0027] The second convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n2 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The feature of dimension (floor(fs×w / (2×step)),n_mels / 2,n1) is transformed into a feature of dimension (floor(fs×w / (4×step)),(n_mels) / 4,n2).

[0028] The third convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n3 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (floor(fs×w / (4×step)),n_mels / 4,n2) are transformed into features of dimension (floor(fs×w / (8×step)),n_mels / 8,n3).

[0029] The fourth convolutional module consists of 2 convolutional layers with a kernel size of 3×3 and n4 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (floor(fs×w / (8×step)),n_mels / 8,n3) is transformed into a feature with dimension (floor(fs×w / (16×step)),n_mels / 16,n4).

[0030] The fifth convolutional module consists of 2 convolutional layers with a kernel size of 3×3 and n5 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (floor(fs×w / (16×step)),n_mels / 16,n4) is transformed into the feature with dimension (floor(fs×w / (32×step)),n_mels / 32,n5).

[0031] The sixth convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n6 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then global pooling downsampling and dimensionality reduction are applied. The feature with dimension (floor(fs×w / (32×step)),n_mels / 32,n6) is converted into a 1D feature with dimension (n6,).

[0032] The fully connected layer has two sublayers. The first sublayer has n6 nodes, the activation function is ReLU, and the second sublayer has 3 nodes. The probability value of each category is obtained by mapping with the Softmax function. The 1-dimensional feature with dimension (n6,) passes through the fully connected layer to obtain a 3-dimensional vector output out=[p1,p2,p3] T , T represents transposition; p1, p2, and p3 correspond to the detection probabilities of abnormal noise, alarm sound, and normal sound of the elevator, respectively.

[0033] A deep learning-based elevator abnormal noise and alarm sound detection system, comprising:

[0034] A first acquisition module, used for acquiring acceleration data;

[0035] The first judgment module is used to judge whether the acceleration exceeds a set acceleration threshold to judge whether there is abnormal vibration in the current elevator;

[0036] The second acquisition module is used to collect monitoring video data from i seconds before and j seconds after the start time of the abnormal vibration, a total of i+j seconds;

[0037] An acquisition module, used for acquiring audio signals from surveillance video data;

[0038] A first processing module, used for performing data preprocessing on the audio signal;

[0039] The calculation module is used to input the preprocessed audio signal into the detection model, and the detection model outputs the detected audio type and the corresponding detection probability; the audio type includes: abnormal sound of elevator, alarm sound or normal sound;

[0040] The second judgment module is used to judge whether the detection probability of the elevator abnormal sound or alarm sound exceeds the set detection probability threshold. If it exceeds the detection probability threshold, the audio type and the corresponding detection probability and the corresponding elevator number and detection time are reported to the monitoring platform in real time.

[0041] An electronic device, comprising:

[0042] one or more processors;

[0043] A memory for storing one or more programs;

[0044] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods.

[0045] A computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the steps of any of the above methods.

[0046] The beneficial effects of the present invention are:

[0047] (1) By collecting acceleration data and monitoring videos, abnormal noises and alarm sounds of elevators can be judged without the need for additional hardware equipment, saving costs.

[0048] (2) The detection model adopts a deep learning method and is trained on a self-made elevator audio dataset based on an open source pre-trained audio classification model. It has strong generalization performance and robustness and is suitable for various complex elevator scenarios. It is applicable to elevator abnormal noises and alarm sounds. The detection method is fast and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is a flow chart of a method for detecting abnormal noise and alarm sound of an elevator based on deep learning of the present invention;

[0050] Figure 2 It is a module diagram of an elevator abnormal noise and alarm sound detection system based on deep learning of the present invention;

[0051] Among them, there are a first acquisition module 1, a first judgment module 2, a second acquisition module 3, an acquisition module 4, a first processing module 5, a calculation module 6, and a second judgment module 7. DETAILED DESCRIPTION

[0052] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation mode, structure, characteristics and effects of the present invention are described in detail below in combination with the accompanying drawings and preferred embodiments.

[0053] The method proposed in the present invention directly collects audio signals from elevator monitoring, performs simple preprocessing on the audio, and directly inputs it into a deep learning detection model that integrates feature extraction for detection. The model can realize the classification detection of abnormal noises, alarm sounds, and normal situations of elevators, and further determine whether to report to the monitoring platform based on whether the detection probability exceeds the set detection threshold, thereby realizing the entire process.

[0054] Example 1

[0055] A method for detecting abnormal noise and alarm sound of an elevator based on deep learning comprises the following steps:

[0056] S1. Collect acceleration data and determine whether the acceleration exceeds the set acceleration threshold to determine whether the current elevator has abnormal vibration;

[0057] Abnormal vibration is detected by judging whether the vertical acceleration value of the elevator exceeds a set threshold. Under normal conditions, the acceleration of the elevator will not exceed a certain value. If it exceeds a certain value, it can be considered that the elevator has abnormal vibration. In this embodiment, the acceleration threshold is set to 2000 cm / s 2 When the vertical acceleration value of the elevator is detected to be greater than 2000cm / s 2 , it is determined that the elevator has abnormal vibration at the current moment.

[0058] S2, collect monitoring video data for 30 seconds before and after the start time of the abnormal vibration, a total of 60 seconds;

[0059] The acquisition period of the collected monitoring video can be set under the premise of including the start time of the abnormal vibration of the elevator. In this embodiment, the monitoring video data of 30 seconds before and 30 seconds after the start time of the abnormal vibration of the elevator, a total of 60 seconds, is collected.

[0060] S3, obtaining an audio signal from the surveillance video data;

[0061] S4, performing data preprocessing on the audio signal;

[0062] The steps of data preprocessing are as follows:

[0063] Remove abnormal audio data, which refers to abnormal data with all zeros obtained due to audio extraction failure caused by problems with the acquisition device;

[0064] 1) Use the average mixing method to convert the two-channel audio into a mono audio; the average mixing method means taking the average of the left channel audio and the right channel audio at each sampling point as the converted mono audio;

[0065] 2) Determine whether the sampling frequency of the mono audio is the set sampling frequency fs; if not, resample to obtain an audio signal with a sampling frequency of fs; if so, do nothing;

[0066] The sampling frequency fs is 44100 Hz;

[0067] 3) Use a sliding window to segment the audio signal. The window length is w, the step length is s, and the end of the audio signal that is less than the sliding window length is filled with 0, and finally a batch of sub-audio segments of equal length are obtained;

[0068] The window duration is 10s and the step length is 10s;

[0069] The data dimension after data preprocessing is (n,fs×w), where n is the number of sub-audio segments, and n is calculated according to the following formula: n=ceil((tw) / s)+1, t is the original audio duration, i.e. 60s, w is the window duration, i.e. 10s, s is the window step, i.e. 10s, so n=6, ceil() means taking the smallest integer greater than or equal to the number in the brackets. The data shape after data preprocessing is (6,441000).

[0070] S5. Input the preprocessed audio signal into the detection model; the detection model outputs the detected audio type and the corresponding detection probability; the audio type includes: abnormal elevator sound, alarm sound or normal;

[0071] The detection model is obtained by training an open source audio classification model on a self-made elevator audio dataset;

[0072] The self-made elevator audio data set is derived from the annotated elevator abnormal sound, alarm sound and normal audio data in different elevator scenarios;

[0073] The model training is obtained by training the model on the basis of the open source audio classification model "KONG Q, CAO Y, IQBAL T, etc. PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition [M / OL].", using a self-made elevator audio data set to modify the network structure and perform additional training, and the initial parameters of the model come from the open source audio classification model;

[0074] After model training, perform performance testing on the model. When the model test accuracy meets the actual usage requirements, it can be used for model deployment. Otherwise, the model structure and parameters need to be optimized and then retrained until it meets the actual usage requirements.

[0075] The actual usage requirement means that the detection accuracy of the model on the test set reaches more than 90%.

[0076] The detection model adopts a neural network model network, including a logarithmic Mel spectrum feature extraction module, 6 convolution modules and 1 fully connected layer.

[0077] Logarithmic Mel spectrum feature extraction module: Short-time Fourier transform is applied to the time domain waveform of the audio signal to calculate the time-frequency spectrum; then the logarithmic Mel spectrum feature is obtained through the Mel filter bank; the window function of the short-time Fourier transform selects the Hamming window, the window length is 1024 sampling points, the window step is 320, and the number of Mel filter banks is 64; the single sub-audio signal with an input shape of (441000,) is changed to (ceil(441000 / 320),64,1) after passing through the logarithmic Mel spectrum feature extraction module, ceil() means taking the smallest integer greater than or equal to the number in the brackets, ceil(441000 / 320)=1379;

[0078] The first convolution module consists of 2 convolutional layers with a kernel size of 3×3 and 64 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The feature with dimension (1379, 64, 1) is transformed into a feature with dimension (689, 32, 64). The floor(x) function represents rounding down, that is, the maximum integer not greater than x.

[0079] The second convolution module consists of 2 convolutional layers with a kernel size of 3×3 and 128 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (689, 32, 64) are transformed into features of dimension (344, 16, 128).

[0080] The third convolution module consists of 2 convolutional layers with a kernel size of 3×3 and 256 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (344, 16, 128) are converted to features of dimension (172, 8, 256).

[0081] The fourth convolution module consists of 2 convolutional layers with a kernel size of 3×3 and 512 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (172, 8, 256) are transformed into features of dimension (86, 4, 512).

[0082] The fifth convolutional module consists of 2 convolutional layers with a kernel size of 3×3 and 1024 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (86, 4, 512) are transformed into features of dimension (43, 2, 1024).

[0083] The sixth convolution module consists of 2 convolutional layers with a convolution kernel size of 3×3 and 2048 convolution kernels. Batch normalization is used between each convolution layer, and ReLU is used as the activation function. Then global pooling downsampling and dimensionality reduction are applied. The features with dimensions (43, 2, 1024) are converted into 1D features with dimensions (2048,).

[0084] The fully connected layer has two sublayers. The first sublayer has 2048 nodes, and the activation function is ReLU. The second sublayer has 3 nodes. The probability value of each category is obtained by mapping with the Softmax function. The 1-dimensional feature with a dimension of (2048,) is passed through the fully connected layer to obtain a 3-dimensional vector output out=[0.93,0.02,0.05] T , T represents transposition; 0.93, 0.02, and 0.05 correspond to the detection probabilities of abnormal noise, alarm sound, and normal sound of the elevator, respectively.

[0085] S6. Determine whether the detection probability of the elevator abnormal sound or alarm sound exceeds the set detection probability threshold. If it exceeds the detection probability threshold, report the audio type and the corresponding detection probability and the corresponding elevator number and detection time to the monitoring platform in real time.

[0086] The detection probability threshold value can be set to different threshold values ​​for elevator abnormal noise and alarm sound according to the actual detection effect. In this embodiment, the detection probability threshold value of elevator abnormal noise is 0.85, and the detection probability threshold value of alarm sound is 0.9.

[0087] Therefore, in this embodiment, the elevator abnormal sound detection probability value of 0.93 exceeds the corresponding detection probability threshold of 0.85, so the abnormal sound type and the corresponding detection probability of 0.93 and the corresponding detection code and detection time are reported to the monitoring platform.

[0088] The corresponding elevator number refers to the unique identification number of each elevator, and the reported elevator can be uniquely determined based on the number.

[0089] Example 2

[0090] Figure 2 A system for detecting abnormal noise and warning sound of an elevator based on deep learning is shown according to an exemplary embodiment, comprising:

[0091] The first acquisition module 1 is used to obtain acceleration data;

[0092] The first judgment module 2 is used to judge whether the acceleration exceeds the set acceleration threshold to judge whether there is abnormal vibration in the current elevator; in this embodiment, the acceleration threshold is set to 2000cm / s 2 When the vertical acceleration value of the elevator is detected to be greater than 2000cm / s2 , it is determined that the elevator has abnormal vibration at the current moment.

[0093] The second acquisition module 3 is used to collect monitoring video data of 30 seconds before and after the start time of the corresponding abnormal vibration, a total of 60 seconds;

[0094] An acquisition module 4 is used to acquire an audio signal from the surveillance video data;

[0095] A first processing module 5, used for performing data preprocessing on the audio signal;

[0096] The calculation module 6 is used to input the preprocessed audio signal into the detection model, and the detection model outputs the detected audio type and the corresponding detection probability; the audio type includes: abnormal sound of the elevator, alarm sound or normal sound;

[0097] The second judgment module 7 is used to judge whether the detection probability of the elevator abnormal sound or alarm sound exceeds the set detection probability threshold. If it exceeds the detection probability threshold, the audio type and the corresponding detection probability and the corresponding elevator number and detection time are reported to the monitoring platform in real time.

[0098] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0099] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.

[0100] Example 3

[0101] Accordingly, the present application also provides an electronic device, including:

[0102] one or more processors;

[0103] A memory for storing one or more programs;

[0104] When the one or more programs are executed by the one or more processors, the one or more processors implement any of the above methods. In addition to the processor and the memory, any device with data processing capability in which the system is located in the embodiment may also include other hardware according to the actual function of the device with data processing capability, which will not be described in detail.

[0105] Accordingly, the present application also provides a computer-readable storage medium on which computer instructions are stored, and when the instructions are executed by a processor, the steps of any of the above methods are implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the above embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of a wind turbine, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.

[0106] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A method for detecting abnormal noise and alarm sound of elevator based on deep learning, It is characterized in that The steps include: S1. Collect acceleration data and determine whether the acceleration exceeds the set acceleration threshold to determine whether the current elevator has abnormal vibration; S2, collecting monitoring video data from i seconds before and j seconds after the start time of the abnormal vibration, a total of i+j seconds; S3, obtaining an audio signal from the surveillance video data; S4, performing data preprocessing on the audio signal; S5, inputting the preprocessed audio signal into the detection model; The detection model outputs the detected audio type and the corresponding detection probability; the audio types include: abnormal elevator sound, alarm sound or normal; S6. Determine whether the detection probability of the elevator abnormal sound or alarm sound exceeds the set detection probability threshold. If it exceeds the detection probability threshold, report the audio type and the corresponding detection probability and the corresponding elevator number and detection time to the monitoring platform in real time; The steps of data preprocessing are as follows: Remove abnormal audio data, which refers to abnormal data with all zeros obtained due to audio extraction failure caused by problems with the acquisition device; 1) Use the average mixing method to convert the two-channel audio into a mono audio; the average mixing method means taking the average of the left channel audio and the right channel audio at each sampling point as the converted mono audio; 2) Determine whether the sampling frequency of the mono audio is the set sampling frequency fs; if not, resample to obtain an audio signal with a sampling frequency of fs; if so, do nothing; 3) Use a sliding window to segment the audio signal. The window length is w, the step length is s, and the end of the audio signal that is less than the sliding window length is filled with 0, and finally a batch of sub-audio segments of equal length are obtained; The data dimension after data preprocessing is (n,fs×w), where n is the number of sub-audio segments, and n is calculated according to the following formula: n=ceil((tw) / s)+1, t is the original audio duration, w is the window duration, s is the window step, and ceil() means taking the smallest integer greater than or equal to the number in the brackets; The detection model is obtained by training an open source audio classification model on a self-made elevator audio dataset; The self-made elevator audio data set is derived from the annotated elevator abnormal sound, alarm sound and normal audio data in different elevator scenarios; The model training refers to modifying the network structure and performing additional training based on the open source audio classification model using a self-made elevator audio data set, and the initial parameters of the model come from the open source audio classification model; After model training, perform performance testing on the model. When the model test accuracy meets the actual usage requirements, it can be used for model deployment. Otherwise, the model structure and parameters need to be optimized and then retrained until it meets the actual usage requirements. The actual usage requirement means that the detection accuracy of the model on the test set reaches more than 90%; The detection model uses a neural network model network, including a logarithmic Mel spectrum feature extraction module, 6 convolution modules and 1 fully connected layer; Logarithmic Mel spectrum feature extraction module: Short-time Fourier transform is applied to the time domain waveform of the audio signal to calculate the time-frequency spectrum; then the logarithmic Mel spectrum feature is obtained through the Mel filter bank; the window function of the short-time Fourier transform selects the Hamming window, the window length is n_windows sampling points, the window step is step, and the number of Mel filter banks is n_mels; the single sub-audio signal with an input shape of (fs×w) is changed to (ceil(fs×w / step),n_mels,1) after passing through the logarithmic Mel spectrum feature extraction module, where ceil() means taking the smallest integer greater than or equal to the number in the brackets; The first convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n1 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (ceil(fs×w / step),n_mels,1) is transformed into a feature with dimension (floor(fs×w / (2×step)),n_mels / 2,n1). The floor(x) function represents rounding down, that is, the maximum integer not greater than x. The second convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n2 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (floor(fs×w / (2×step)),n_mels / 2,n1) are transformed into features of dimension (floor(fs×w / (4×step)),n_mels / 4,n2). The third convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n3 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, an average pooling layer of 2×2 is applied for downsampling. The features of dimension (floor(fs×w / (4×step)),n_mels / 4,n2) are transformed into features of dimension (floor(fs×w / (8×step)),n_mels / 8,n3). The fourth convolutional module consists of 2 convolutional layers with a kernel size of 3×3 and n4 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (floor(fs×w / (8×step)),n_mels / 8,n3) is transformed into a feature with dimension (floor(fs×w / (16×step)),n_mels / 16,n4). The fifth convolutional module consists of 2 convolutional layers with a kernel size of 3×3 and n5 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then, a 2×2 average pooling layer is applied for downsampling. The feature with dimension (floor(fs×w / (16×step)),n_mels / 16,n4) is transformed into the feature with dimension (floor(fs×w / (32×step)),n_mels / 32,n5). The sixth convolution module consists of 2 convolutional layers with a kernel size of 3×3 and n6 kernels. Batch normalization is used between each convolutional layer, and ReLU is used as the activation function. Then global pooling downsampling and dimensionality reduction are applied. The feature with dimension (floor(fs×w / (32×step)),n_mels / 32,n6) is converted into a 1D feature with dimension (n6). The fully connected layer has two sublayers. The first sublayer has n6 nodes, the activation function is ReLU, and the second sublayer has 3 nodes. The probability value of each category is obtained through Softmax function mapping. The 1-dimensional feature with dimension (n6) passes through the fully connected layer to obtain a 3-dimensional vector output out=[p1,p2,p3] T , T represents transposition; p1, p2, and p3 correspond to the detection probabilities of abnormal noise, alarm sound, and normal sound of the elevator, respectively.

2. An electronic device, It is characterized in that include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as claimed in claim 1.

3. A computer-readable storage medium having computer instructions stored thereon, It is characterized in that When the instruction is executed by a processor, the steps of the method as claimed in claim 1 are implemented.