Escalator abnormal sound detection method and system, medium and product

Abstract

The invention discloses an escalator abnormal sound detection method and system, a medium and a product, and relates to the field of data processing. The method comprises the following steps: acquiring a sound signal and a vibration signal in the running process of the escalator, and determining a reference running period of the escalator based on the vibration signal; segmenting the sound signal into a plurality of sound signal segments based on the reference operation period, and segmenting the vibration signal into a plurality of vibration signal segments; performing synchronous superposition processing on the sound signal segments and the vibration signal segments to obtain enhanced periodic sound signals and enhanced periodic vibration signals; performing time-frequency analysis on the enhanced periodic sound signal to obtain a periodic spectrogram feature, and calculating a frequency spectrum coherence degree between the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain a coherence degree feature; and multi-modal features are constructed, the multi-modal features are input into a preset abnormal sound detection model, and the abnormal sound type of the escalator is obtained. By implementing the technical scheme provided by the invention, the accuracy of escalator abnormal sound detection is improved.

Description

Technical Field

[0001] This application relates to the field of data processing, specifically to a method, system, medium, and product for detecting abnormal noises in escalators. Background Technology

[0002] As a crucial component of modern urban public transportation systems, the safety and reliability of escalators are paramount. To ensure stable operation and prevent potential malfunctions, real-time monitoring of their critical mechanical components is typically required. Collecting sound signals during escalator operation to diagnose abnormal noises is a common and effective monitoring method. However, escalators are usually installed in crowded and complex public places such as shopping malls and subway stations, inevitably resulting in a significant amount of background noise mixed in with the collected sound signals. This strong environmental noise often drowns out weak abnormal noise signals caused by early equipment failures, leading to an extremely low signal-to-noise ratio. This severely impacts the accuracy of subsequent abnormal noise identification, posing a significant challenge to early fault warning.

[0003] To address the aforementioned environmental noise interference, several signal denoising methods have been proposed in existing technologies. One approach involves preprocessing the acquired raw sound signal using wavelet thresholding. This method decomposes the signal into different scales and frequencies through wavelet transform, then sets an appropriate threshold, zeroing or reducing wavelet coefficients below this threshold that are considered noise. Finally, the signal is reconstructed through inverse wavelet transform to suppress noise and enhance the useful signal. The denoised sound signal exhibits a higher signal-to-noise ratio, which is beneficial for subsequent feature extraction and pattern recognition.

[0004] While the aforementioned noise reduction methods can filter out some background noise to a certain extent, they still have significant technical problems in practical applications. These methods primarily rely on the time-frequency characteristics of the sound signal itself for processing. When the characteristics of certain environmental noises are similar to those of abnormal equipment noises, the noise reduction algorithm struggles to distinguish them effectively, and may even mistakenly suppress useful abnormal noise components as noise. More importantly, a single sound signal cannot accurately determine the physical source of a sound event; that is, it cannot confirm whether the detected abnormal noise is indeed generated by the mechanical movement inside the escalator or is merely an external sound source unrelated to the escalator. This uncertainty stemming from single-modal information makes existing detection methods prone to false alarms, misjudging environmental interference as equipment malfunctions, thereby reducing the reliability and accuracy of fault diagnosis. Summary of the Invention

[0005] This application provides a method, system, medium, and product for detecting abnormal noises in escalators, which improves the accuracy of escalator noise detection.

[0006] A first aspect of this application provides a method for detecting abnormal noise in escalators. The method includes: acquiring sound signals and vibration signals during escalator operation, and determining a reference operating cycle of the escalator based on the vibration signals; dividing the sound signals into multiple sound signal segments and the vibration signals into multiple vibration signal segments based on the reference operating cycle; synchronously superimposing the sound signal segments and the vibration signal segments to obtain an enhanced periodic sound signal and an enhanced periodic vibration signal; performing time-frequency analysis on the enhanced periodic sound signal to obtain periodic spectrogram features, and calculating the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain coherence features; constructing multimodal features based on the periodic spectrogram features and the coherence features, and inputting the multimodal features into a preset abnormal noise detection model to obtain the type of abnormal noise from the escalator.

[0007] By employing the above technical solution, sound and vibration signals during escalator operation are acquired. Based on the vibration signals, the escalator's baseline operating cycle is determined, and the sound and vibration signals are segmented into multiple signal segments using this baseline operating cycle. Synchronous superposition of these signal segments enhances the periodic abnormal noise components and weakens non-periodic background noise interference, resulting in enhanced periodic sound and vibration signals. Furthermore, time-frequency analysis is performed on the enhanced periodic sound signal to obtain periodic spectrogram features, which reflect the energy distribution of the abnormal noise signal. The spectral coherence of the enhanced periodic sound and vibration signals is calculated, and the resulting coherence feature measures the correlation between these two signals; under abnormal conditions, the correlation will significantly change. Finally, the periodic spectrogram features and coherence features are used to construct a multimodal feature input to a pre-defined abnormal noise detection model. Through the discrimination rules learned during model training, the type of abnormal noise from the escalator can be accurately identified. In summary, this method fully utilizes the complementarity of sound and vibration signals, suppresses background noise, extracts features that reflect abnormal characteristics, and uses multimodal machine learning methods to achieve accurate detection of escalator noises.

[0008] Optionally, determining the reference operating cycle of the escalator based on the vibration signal specifically includes: performing low-pass filtering on the vibration signal to obtain a filtered vibration signal; performing autocorrelation analysis on the filtered vibration signal to obtain an autocorrelation coefficient sequence; determining the time interval corresponding to the first local maximum value of the autocorrelation coefficient sequence, and determining the time interval as the reference operating cycle of the escalator.

[0009] By employing the above technical solution, the vibration signal is first subjected to low-pass filtering to remove high-frequency noise interference, resulting in a stable low-frequency vibration component. This facilitates accurate estimation of the escalator's baseline operating cycle. Then, autocorrelation analysis is performed on the filtered vibration signal. The autocorrelation function measures the similarity of signals at different time delays. For periodic signals, the autocorrelation coefficient exhibits a local maximum at a time delay equal to the period length. By determining the time interval corresponding to the first local maximum of the autocorrelation coefficient sequence and using it as the escalator's baseline operating cycle, the operating cycle of different escalator devices can be adaptively estimated, demonstrating good robustness and applicability. This vibration signal-based cycle estimation method does not require escalator structural parameters or operating speed; it is entirely data-driven, simple, and reliable. It provides crucial prior information for subsequent signal segmentation and feature extraction, forming the foundation of the entire abnormal noise detection method.

[0010] Optionally, the step of performing time-frequency analysis on the enhanced periodic sound signal to obtain periodic spectrogram features specifically includes: dividing the enhanced periodic sound signal into multiple data frames along the time axis, wherein each data frame has a preset frame length and adjacent data frames have a preset overlap rate; performing window function processing on each data frame, and performing a fast Fourier transform on each data frame after window function processing to obtain the complex spectrum corresponding to the data frame; calculating the amplitude of the complex spectrum to obtain the amplitude spectrum, and stacking the amplitude spectra of all data frames along the time dimension to form a two-dimensional time-frequency matrix with time as one axis and frequency as the other axis; performing a logarithmic transformation on the amplitude in the two-dimensional time-frequency matrix and performing normalization processing to generate the periodic spectrogram features.

[0011] By employing the above technical solution, time-frequency analysis is performed on the enhanced periodic sound signal. First, the signal is divided into multiple data frames, each with an appropriate length and overlap rate to balance time and frequency resolution. Windowing is applied to each data frame to reduce spectral leakage and improve spectral resolution. Then, a Fast Fourier Transform is performed on the windowed data frames to obtain the complex spectrum, and the amplitude spectrum is calculated. The amplitude spectra of each frame are stacked along the time direction to form a two-dimensional time-frequency matrix. This matrix can characterize the evolution of the spectral structure over time and highlight the time-frequency energy distribution pattern of periodic abnormal sounds. Furthermore, taking the logarithm and normalizing the amplitude values ​​in the time-frequency matrix stretches the dynamic range of the amplitude, highlighting the low-to-medium energy abnormal sound components and compressing the high-energy background components. The resulting periodic spectrogram features facilitate subsequent pattern recognition.

[0012] Optionally, calculating the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain the coherence feature specifically includes: performing synchronous frame segmentation processing on the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain corresponding data frame pairs; performing a fast Fourier transform on each data frame pair to obtain a first frequency domain representation of the enhanced periodic sound signal and a second frequency domain representation of the enhanced periodic vibration signal; calculating the cross-power spectral density and the auto-power spectral density of each data frame pair based on the first frequency domain representation and the second frequency domain representation; calculating the spectral coherence value at different frequency points according to the cross-power spectral density and the auto-power spectral density, and combining the spectral coherence values ​​of all data frames along the time dimension to form a two-dimensional coherence matrix with time as one axis and frequency as the other axis, and using the two-dimensional coherence matrix as the coherence feature.

[0013] By employing the above technical solution, the coherence of enhanced periodic sound signals and enhanced periodic vibration signals in the time-frequency domain is calculated, yielding coherence characteristics reflecting their correlation. First, the two signals are synchronously framed to ensure time alignment. Then, a Fast Fourier Transform is performed on each data frame pair to obtain a frequency domain representation. Based on the frequency domain representation, the cross-power spectral density and auto-power spectral density are calculated, which respectively measure the cross-correlation and auto-correlation of the two signals. Further, based on the cross-power spectral density and auto-power spectral density, a coherence function at different frequency points can be obtained. The coherence value ranges from 0 to 1, with a larger value indicating a stronger correlation between the two signals. Combining the coherence values ​​of each data frame into a two-dimensional coherence matrix yields the coherence characteristics that vary with time and frequency. When an escalator makes abnormal noise, the abnormal excitation will produce a consistent response pattern in both the sound and vibration signals, significantly increasing their coherence in the abnormal frequency band. Therefore, the coherence characteristics are an important basis for identifying abnormal noise. In summary, spectral coherence analysis can extract anomalous information from the coupling perspective of acoustic and vibration signals. The features obtained are highly sensitive to abnormal noises and complement time-frequency analysis, together forming a multimodal characterization of abnormal noises.

[0014] Optionally, the step of constructing multimodal features based on the periodic spectrogram features and the coherence features specifically includes: calculating the spectral slope and kurtosis of the amplitude spectrum of the enhanced periodic sound signal to obtain spectral morphology features; extracting the modulation frequency bandwidth based on the coherence features to obtain modulation features used to distinguish between structured sound and airborne sound; combining the periodic spectrogram features, the coherence features, the spectral morphology features, and the modulation features as multiple feature channels along the feature dimension to form an input tensor, and using the input tensor as the multimodal features.

[0015] By employing the aforementioned technical solution, based on the periodic spectrogram features and coherence features, further spectral morphology features and modulation features are extracted to construct multimodal features. These features can characterize the inherent properties of abnormal noise signals from multiple perspectives, including energy, correlation, morphology, and modulation. Specifically, the spectral slope reflects the changing trend of the spectral envelope, and the spectral kurtosis reflects the flatness of the spectrum; both together determine the shape characteristics of the spectrum, corresponding to the texture of the abnormal noise. The modulation frequency bandwidth reflects the characteristics of modulation phenomena such as howling and whistling in abnormal noises, helping to distinguish between airborne sound and structured sound. Combining the periodic spectrogram, coherence, spectral morphology, and modulation features into an input tensor along the feature dimension yields a compact multi-channel time-frequency representation. This multimodal feature can capture the energy distribution of abnormal noise signals macroscopically, extract detailed features locally, uncover the coupling laws of acoustic vibration signals from a correlational perspective, and analyze the modulation phenomena of abnormal noise signals dynamically. Therefore, it has the advantages of large information content, strong discriminative power, and comprehensive description. Inputting it into an abnormal noise detection model can significantly improve detection and classification performance.

[0016] Optionally, the step of inputting the multimodal features into a preset abnormal noise detection model to obtain the abnormal noise type of the escalator specifically includes: taking the multimodal features as input and processing them through at least one convolutional layer and pooling layer of the preset abnormal noise detection model to extract local time-frequency features from the multimodal features and generate a high-dimensional feature map; inputting the high-dimensional feature map into the deep autoencoder of the preset abnormal noise detection model to perform nonlinear dimensionality reduction mapping to obtain multiple corresponding low-dimensional feature vectors; sorting the multiple low-dimensional feature vectors in chronological order to form a feature sequence, and inputting the feature sequence into the long short-term memory network of the preset abnormal noise detection model to obtain output features; inputting the output features into at least one fully connected layer and processing them using the Softmax function to output the probability distribution of the escalator on multiple preset abnormal noise categories, and determining the category corresponding to the highest probability as the abnormal noise type of the escalator.

[0017] By employing the above technical solution, a deep learning model is used to achieve discriminative feature learning and abnormal noise type identification of multimodal features, offering end-to-end advantages. Multimodal features are input into the abnormal noise detection model. First, local time-frequency features are extracted through convolutional and pooling layers. Convolutional operations adaptively learn multi-scale, multi-directional convolutional kernels sensitive to abnormal noise, while pooling operations aggregate local responses, enhancing the translation invariance of features. The generated high-dimensional feature map contains discriminative diagnostic information. Next, a deep autoencoder performs dimensionality reduction mapping on the high-dimensional feature map. Through nonlinear transformation and information compression, a low-dimensional feature vector is obtained, removing redundancy and noise while retaining key semantic information. To model the dynamic evolution of abnormal noise signals, the low-dimensional feature vector sequence is input into a Long Short-Term Memory (LSTM) network to learn temporal dependencies. Finally, the output features of the LSM network are processed by fully connected layers and a Softmax function to predict the posterior probability of the abnormal noise type. This deep abnormal noise detection model can automatically learn discriminative hierarchical features, mining deep diagnostic information from time-frequency data and overcoming the limitations of manual features. Meanwhile, the end-to-end training method enables the modules to work together to optimize and maximize the role of multimodal information, thereby significantly improving the performance of abnormal noise detection and recognition.

[0018] Optionally, the Long Short-Term Memory (LSTM) network includes a forward LSM network and a backward LSM network. The step of inputting the feature sequence into the LSM network of the preset abnormal noise detection model to obtain output features specifically includes: inputting the feature sequence into the forward LSM network and the backward LSM network; processing the feature sequence in forward chronological order through the forward LSM network to obtain a first output result; processing the feature sequence in reverse chronological order through the backward LSM network to obtain a second output result; concatenating the first output result and the second output result at each time step to form a bidirectional context representation, and using the bidirectional context representation as the output feature.

[0019] By adopting the above technical solution, a bidirectional mechanism is introduced into the Long Short-Term Memory (LSTM) network. The forward and backward recursive processing of the feature sequence through the forward and backward networks, respectively, fully explores the bidirectional long-short-term dependencies and obtains global contextual information. The forward network models sequentially from past to future, learning the impact of historical states on the present; the backward network models in reverse order from future to past, learning the impact of future states on the present. Concatenating the hidden states obtained from both at each time step yields a feature representation carrying bidirectional semantic information. This bidirectional encoding method enhances the model's ability to characterize the sequential correlation of abnormal noises. Simultaneously, the complementary modeling of forward and backward sequences allows for more efficient use of network information and richer feature expression, thereby further improving the abnormal noise recognition rate. In summary, the abnormal noise detection model integrating bidirectional LSM networks can learn hierarchical representations of temporal data from local to global perspectives, revealing the inherent temporal structure of abnormal noise signals. It exhibits stronger adaptability and generalization to complex and variable abnormal noise situations, providing a new approach for escalator status monitoring and fault early warning.

[0020] Secondly, embodiments of this application provide an escalator noise detection system, which includes one or more processors and a memory; the memory is coupled to the one or more processors and is used to store computer program code, which includes computer instructions, and the one or more processors call the computer instructions to cause the escalator noise detection system to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an escalator noise detection system, cause the escalator noise detection system to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an escalator noise detection system, cause the escalator noise detection system to perform the method described in the first aspect and any possible implementation thereof.

[0023] In summary, one or more technical solutions provided in this application have at least the following technical effects or advantages: This technical solution effectively solves the problem of distinguishing the source of escalator noise in complex background noise by combining sound and vibration signals and utilizing multimodal data fusion. Specifically, the reference operating cycle of the escalator is determined by the vibration signal. The sound and vibration signals are synchronously segmented and periodically superimposed for enhancement. Periodic spectrogram features are extracted, and the spectral coherence of the two signals is calculated, thereby strengthening the correlation between the noise signal and mechanical motion and reducing background noise interference. Based on the multimodal features of sound-vibration fusion, a noise detection model is further constructed to achieve accurate identification and classification of escalator noise. This method significantly improves the reliability, accuracy, and robustness of escalator noise detection in complex environments, effectively reduces the false alarm rate, and provides technical support for early warning and maintenance of equipment failures. Attached Figure Description

[0024] Figure 1 This is a flowchart illustrating a method for detecting abnormal noises from an escalator, as disclosed in an embodiment of this application. Figure 2 This is another flowchart illustrating a method for detecting abnormal noises from escalators disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an escalator noise detection system provided in an embodiment of this application.

[0025] Explanation of reference numerals in the attached drawings: 301, Central Processing Unit; 302, Read-Only Memory; 303, Random Access Memory; 304, Bus; 305, Input / Output Interface; 306, Input Section; 307, Output Section; 308, Storage Section; 309, Communication Section; 310, Driver; 311, Removable Media. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0027] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0028] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0029] This application provides a method for detecting abnormal noises in escalators, referring to... Figure 1 , Figure 1 This is a flowchart illustrating a method provided in an embodiment of this application. The method is applied to a system that can execute a detection procedure for escalator noise. The method includes steps S101 to S105, as follows: Step S101: Acquire sound and vibration signals during the operation of the escalator, and determine the reference operating cycle of the escalator based on the vibration signals.

[0030] In step S101, the sound signal refers to the data stream collected by the sound sensor, which reflects the airborne sound wave information during the operation of the escalator; the vibration signal refers to the data stream collected by the vibration sensor, which reflects the structural vibration information generated by the mechanical structure of the escalator during operation; the reference operating cycle is used to represent the time required for the escalator step chain to complete one complete cycle, and is the time reference for subsequent periodic signal analysis.

[0031] Specifically, the system first deploys high-sensitivity microphones and accelerometers at key locations on the escalator to continuously collect sound and vibration signals during normal operation. To determine the baseline operating cycle, the system processes the collected vibration signals. The system applies a low-pass filter to the vibration signals, removing high-frequency noise and interference components unrelated to the overall periodic motion of the escalator, while retaining the low-frequency principal components related to the cyclical motion of the step chain. Next, the system performs autocorrelation analysis on the filtered vibration signals, calculating the similarity between the signal and itself at different time delays, generating an autocorrelation coefficient sequence. This sequence exhibits local peaks at delay points corresponding to integer multiples of the signal period. The system automatically detects and locates the first local maximum value (excluding the zero-delay point), and the time interval corresponding to this maximum value is determined as the baseline operating cycle of the escalator.

[0032] In one possible implementation, the reference operating cycle of the escalator is determined based on the vibration signal, specifically including steps S1011-S1013, as follows: Step S1011: Perform low-pass filtering on the vibration signal to obtain the filtered vibration signal.

[0033] In step S1011, low-pass filtering refers to a signal processing method that allows components with frequencies below a set cutoff frequency to pass through while attenuating or blocking components with frequencies above the cutoff frequency. The filtered vibration signal refers to the original vibration signal after low-pass filtering, which mainly contains low-frequency components related to the overall periodic motion of the escalator.

[0034] Specifically, the system first performs low-pass filtering on the acquired raw vibration signal. The raw vibration signal typically contains a mixture of low-frequency main vibrations generated by the overall cyclic motion of the escalator chain, and high-frequency noise and random interference generated by roller rolling, motor operation, and passenger movement. To accurately extract the signal components representing the overall operating cycle of the escalator, the system needs to design a suitable cutoff frequency for the low-pass filter. The cutoff frequency is selected based on the fact that the operating cycle of an escalator is typically on the order of tens of seconds, corresponding to a very low frequency, such as below 0.1 Hz. Therefore, the system can set the cutoff frequency to around 1 Hz, which effectively filters out most of the high-frequency noise while preserving the core low-frequency vibration waveform related to the baseline operating cycle without loss, thus obtaining the filtered vibration signal.

[0035] Step S1012: Perform autocorrelation analysis on the filtered vibration signal to obtain the autocorrelation coefficient sequence.

[0036] In step S1012, autocorrelation analysis is a mathematical tool used to reveal the periodicity of a signal, which is achieved by calculating the similarity between the signal and itself at different time delays; the autocorrelation coefficient sequence is a sequence of correlation values ​​at different time delay points obtained after performing autocorrelation analysis on the filtered vibration signal.

[0037] Specifically, the system performs autocorrelation analysis on the filtered vibration signal obtained in step S1011. The system compares the filtered vibration signal with its copies at different time delays, calculating the correlation at each time delay. The calculation process iterates through a series of consecutive time delay values ​​starting from zero, with each delay value corresponding to an autocorrelation coefficient. All these coefficient values ​​are arranged in order of time delay, forming an autocorrelation coefficient sequence. This sequence is characterized by the fact that when the time delay is zero, the signal is identical to itself, exhibiting the strongest correlation, thus generating a global maximum. If the original signal is periodic, then when the time delay is an integer multiple of the signal period, the signal waveform will again be highly aligned with itself, resulting in a series of significant peaks in the autocorrelation coefficient sequence.

[0038] Step S1013: Determine the time interval corresponding to the first local maximum value of the autocorrelation coefficient sequence, and determine the time interval as the reference operating cycle of the escalator.

[0039] In step S1013, the first local maximum value refers to the first peak point that appears in the autocorrelation coefficient sequence in order of time delay, excluding the global maximum value where the time interval is zero; the time interval refers to the amount of time delay corresponding to a specific coefficient value in the autocorrelation coefficient sequence.

[0040] Specifically, the system analyzes the autocorrelation coefficient sequence generated in step S1012 to determine the baseline operating cycle. The system first locates the global maximum value at a time interval of zero, and then searches in the direction of increasing time from this starting point. The system detects and records the positions of all local maximum values. The system seeks the first local maximum value immediately following the global maximum value at zero. The time interval corresponding to this local maximum value accurately reflects the length of the principal periodic component in the vibration signal. The system extracts this time interval value and ultimately determines it as the baseline operating cycle of the escalator for subsequent signal segmentation processing.

[0041] To facilitate understanding, a specific example is given below to explain steps S1011-S1013: Suppose an escalator is in operation, and the raw vibration signal waveform acquired by the system is chaotic. The system first applies a digital low-pass filter with a cutoff frequency of 1 Hz to the signal. The processed signal waveform becomes smooth and exhibits obvious periodic fluctuations, which is the vibration generated by the escalator chain completing one cycle. Subsequently, the system performs autocorrelation analysis on this smoothed vibration signal, obtaining an autocorrelation coefficient curve. This curve reaches a peak of 1.0 at a time interval of 0 seconds, and then the value drops rapidly. As the time interval increases, the curve bulges again at 25.8 seconds, forming a significant local peak, after which the value begins to decrease again. The system program automatically ignores the peak at 0 seconds and identifies the peak at 25.8 seconds as the first local maximum. Based on this, the system determines the escalator's baseline operating cycle to be 25.8 seconds and records this value for use in subsequent steps.

[0042] Step S102: Based on the reference operating cycle, the sound signal is divided into multiple sound signal segments, and the vibration signal is divided into multiple vibration signal segments.

[0043] In step S102, a sound signal segment refers to a data segment of a single cycle length obtained by dividing a continuous sound signal according to a reference operating cycle length; a vibration signal segment refers to a data segment of a single cycle length obtained by dividing a continuous vibration signal according to a reference operating cycle length.

[0044] Specifically, the system segments the acquired long-term continuous sound and vibration signals based on the baseline operating period value determined in step S101. For example, if the determined baseline operating period is T seconds, the system will sequentially segment the continuous sound signal starting from time 0 into multiple sound signal segments of length T. The first segment is from 0 to T seconds, the second segment is from T to 2T seconds, and so on. Simultaneously, the system also segments the continuous vibration signal into multiple vibration signal segments of length T using the exact same starting time and segmentation method. Through this operation, the system ensures that each sound signal segment corresponds precisely to a vibration signal segment in time, laying the foundation for subsequent synchronization analysis.

[0045] Step S103: Perform synchronous superposition processing on the sound signal segment and the vibration signal segment respectively to obtain the enhanced periodic sound signal and the enhanced periodic vibration signal.

[0046] In step S103, synchronous superposition processing refers to the process of aligning multiple signal segments in time and adding them point by point to calculate the average. This process aims to enhance the periodic components in the signal and suppress random noise. Enhanced periodic sound signal refers to a sound signal with a higher signal-to-noise ratio and more significant periodic characteristics obtained after synchronous superposition processing of multiple sound signal segments. Enhanced periodic vibration signal refers to a vibration signal with a higher signal-to-noise ratio and more significant periodic characteristics obtained after synchronous superposition processing of multiple vibration signal segments.

[0047] Specifically, the system aligns all the sound signal segments obtained in step S102. Then, the system performs an arithmetic mean of the amplitudes of all sound signal segments at the same relative time point. This process iterates through every time point from 0 to T, ultimately generating a new sound signal of length T, i.e., an enhanced periodic sound signal. This processing method, also known as time-synchronized averaging, effectively amplifies abnormal noise signals appearing at fixed positions in each operating cycle, while significantly reducing random background noise unrelated to the escalator's operating cycle, such as passenger conversations and environmental announcements. The system uses the exact same synchronous superposition processing method to process all vibration signal segments, obtaining an enhanced periodic vibration signal, thereby highlighting the vibration characteristics associated with periodic faults.

[0048] Step S104: Perform time-frequency analysis on the enhanced periodic sound signal to obtain the periodic spectrogram features, and calculate the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain the coherence features.

[0049] In step S104, time-frequency analysis is a signal processing technique used to analyze how the frequency components of a signal change over time, generating a representation of the signal in both time and frequency dimensions. The periodic spectrogram feature refers to the two-dimensional feature generated by the time-frequency analysis method, which can display the frequency energy distribution of the enhanced periodic sound signal as a function of time in a single operating cycle in the form of an image. The spectral coherence is an index used to measure the degree of linear correlation between two signals at different frequency points. Its value is between 0 and 1, and the closer the value is to 1, the stronger the correlation at that frequency. The coherence feature is a two-dimensional feature that reflects the homology between sound and vibration, formed by combining the spectral coherence values ​​of the enhanced periodic sound signal and the enhanced periodic vibration signal at different frequency points.

[0050] Specifically, the system performs a Short-Time Fourier Transform (STFT) on the enhanced periodic sound signal for time-frequency analysis. The system divides the signal into multiple short-time data frames with a certain overlap rate. After applying a window function to each frame, a Fast Fourier Transform is performed to calculate the spectrum of each frame. Then, the system arranges the spectral amplitudes of all frames along the time dimension, forming a two-dimensional matrix with time as the horizontal axis, frequency as the vertical axis, and color or grayscale representing energy magnitude. After performing a logarithmic transformation and normalization on this matrix, the periodic spectrogram features are obtained. Simultaneously, the system synchronously frames the enhanced periodic sound signal and the enhanced periodic vibration signal, and calculates the spectral coherence of each pair of data frames. The calculation process involves the cross-power spectral density and the individual self-power spectral density of each data frame pair. The system calculates the coherence value at each frequency point according to the formula and combines the coherence values ​​of all data frames along the time dimension to form a two-dimensional coherence matrix, also with time and frequency as the axes. This matrix is ​​the coherence feature.

[0051] In one possible implementation, time-frequency analysis is performed on the enhanced periodic sound signal to obtain periodic spectrogram features, specifically including steps S1041-S1044, as follows: Step S1041: Divide the enhanced periodic sound signal into multiple data frames along the time axis, wherein each data frame has a preset frame length and adjacent data frames have a preset overlap rate.

[0052] In step S1041, a data frame refers to a short-segment signal segment into which the enhanced periodic sound signal is divided along the time axis; the preset frame length indicates the number of sampling points or the duration of each data frame; and the preset overlap rate indicates the proportion of overlapping parts between two adjacent data frames.

[0053] Specifically, the system first acquires the enhanced periodic audio signal. To perform short-time Fourier transform analysis, the system needs to discretize the continuous signal into a series of static segments. Based on a preset frame length, for example, 1024 sampling points, the system extracts the first data frame starting from the beginning of the signal. Subsequently, based on a preset overlap rate, for example, 50%, the system calculates the starting position of the next data frame. This starting position is equal to the starting position of the previous data frame plus the product of the frame length and the non-overlap rate, i.e., 1024 multiplied by (1-0.5) equals 512 sampling points. The system slides a window along the time axis in this manner, continuously extracting overlapping data frames until the entire enhanced periodic audio signal is covered, thus decomposing the complete signal into a set of multiple data frames.

[0054] Step S1042: Perform window function processing on each data frame, and perform Fast Fourier Transform on each data frame after window function processing to obtain the complex spectrum corresponding to the data frame.

[0055] In step S1042, window function processing refers to multiplying each data frame by a window function to smooth the signal amplitude at both ends of the data frame, thereby reducing spectral leakage caused by subsequent Fourier transform. Fast Fourier Transform is an efficient algorithm for calculating Discrete Fourier Transform, used to convert time-domain signals to the frequency domain. The complex spectrum represents the result obtained after the data frame undergoes Fast Fourier Transform, containing the amplitude and phase information of each frequency component.

[0056] Specifically, the system processes each data frame generated in step S1041 independently. The system applies a window function, such as a Hanning window, to each data frame. This processing is achieved by multiplying each sample point in the data frame by the value at the corresponding position in the window function, causing the signal amplitude at the ends of the data frame to gradually decay to zero, while the signal in the middle is less affected. After completing the window function processing, the system performs a Fast Fourier Transform on the windowed data frame. This transform maps the N-length data frame from the time domain to a frequency domain representation containing N complex values. This set of complex values ​​is the complex spectrum corresponding to the data frame, where each complex number represents the intensity and phase of a specific frequency component.

[0057] Step S1043: Calculate the amplitude of the complex spectrum to obtain the amplitude spectrum, and stack the amplitude spectra of all data frames along the time dimension to form a two-dimensional time-frequency matrix with time as one axis and frequency as the other axis.

[0058] In step S1043, amplitude refers to the magnitude or intensity of each frequency component in the complex spectrum, without phase information; amplitude spectrum refers to the spectral representation composed of the amplitudes of all frequency components; a two-dimensional time-frequency matrix is ​​a data structure in which one dimension represents time and the other dimension represents frequency, and the element values ​​in the matrix are the signal energy or amplitude at the corresponding time and frequency point.

[0059] Specifically, the system receives the complex spectrum generated for each data frame in step S1042. For each complex value in the complex spectrum, the system calculates its modulus, that is, calculates the square root of the sum of the squares of the real and imaginary parts of the complex number, thereby obtaining the amplitude of that frequency component. The set of amplitudes of all frequency components constitutes the amplitude spectrum of the data frame. The system repeats this process for all data frames, obtaining multiple amplitude spectra corresponding one-to-one with each data frame. Finally, the system arranges all amplitude spectra according to their temporal order in the original signal, stacking each amplitude spectrum as a column vector along the time dimension to form a two-dimensional time-frequency matrix. The row index of this matrix corresponds to the frequency, and the column index corresponds to the time frame.

[0060] Step S1044: Perform logarithmic transformation and normalization on the amplitude values ​​in the two-dimensional time-frequency matrix to generate periodic spectrogram features.

[0061] In step S1044, logarithmic transformation refers to applying a logarithmic function to calculate the amplitude in the two-dimensional time-frequency matrix to compress the numerical range and make it more consistent with the characteristics of human auditory perception; normalization refers to scaling the values ​​in the matrix proportionally to a fixed range, such as 0 to 1.

[0062] Specifically, the system performs subsequent processing on the two-dimensional time-frequency matrix generated in step S1043. First, the system performs a logarithmic transformation on each amplitude element in the matrix. Typically, a very small positive number is added to each amplitude before taking the logarithm to avoid errors caused by calculating values ​​of zero. This transformation converts linear-scale amplitudes to logarithmic scales, such as decibels, allowing both weak and strong components of the signal to be clearly visualized. Next, the system performs normalization on the entire matrix after the logarithmic transformation. The system finds the maximum and minimum values ​​in the matrix and then rescales all element values ​​to the range of 0 to 1 using a linear mapping formula. The resulting two-dimensional matrix after logarithmic transformation and normalization is the final generated periodic spectrogram feature.

[0063] To facilitate understanding, a specific example is given below to explain steps S1041-S1044: For example, the system acquires an enhanced periodic audio signal and divides it into multiple data frames, each 20 milliseconds long with a 50% overlap. For each data frame, the system first applies a Hanning window and then performs a Fast Fourier Transform to obtain the complex spectrum of that frame. The system then calculates the amplitude of each complex spectrum, resulting in a series of amplitude spectra. These amplitude spectra are then arranged chronologically to form a two-dimensional time-frequency matrix with time on the horizontal axis and frequency on the vertical axis. Next, the system performs a base-10 logarithmic transform on all amplitude values ​​in this matrix, converting linear amplitudes to decibels. Finally, the system performs min-max normalization on the entire logarithmic amplitude matrix, scaling all values ​​to between 0 and 1, thereby generating a standardized periodic spectrogram that visually reflects the time-frequency variation of the audio signal's energy.

[0064] In one possible implementation, the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal is calculated to obtain coherence characteristics, specifically including steps S1045-S1048, as follows: Step S1045: Perform synchronous framing processing on the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain the corresponding data frame pairs.

[0065] In step S1045, synchronous framing processing refers to performing framing operations on two or more signals at the same time point using the same frame length and overlap rate parameters to ensure that the generated data frames correspond precisely in time; a data frame pair represents a set of data consisting of enhanced periodic sound signal data frames and enhanced periodic vibration signal data frames captured within the same time window after synchronous framing processing.

[0066] Specifically, the system first acquires time-aligned enhanced periodic sound and vibration signals. Using the same framing parameters, such as a preset frame length of 1024 sampling points and a preset overlap rate of 50%, the system simultaneously divides the two signals. The system extracts the first data frame from the beginning of the two signals, forming the first data frame pair. Subsequently, the system calculates the beginning position of the next data frame based on the overlap rate and again extracts data frames from the corresponding positions of the two signals, forming the second data frame pair. This process is repeated until the end of the two signals, thereby generating a series of time-corresponding data frame pairs.

[0067] Step S1046: Perform a fast Fourier transform on each data frame pair to obtain a first frequency domain representation of the enhanced periodic sound signal and a second frequency domain representation of the enhanced periodic vibration signal.

[0068] In step S1046, the first frequency domain representation refers to the complex spectrum obtained after the enhanced periodic sound signal data frame undergoes a fast Fourier transform; the second frequency domain representation refers to the complex spectrum obtained after the enhanced periodic vibration signal data frame undergoes a fast Fourier transform.

[0069] Specifically, the system iterates through each data frame pair generated in step S1045. For the enhanced periodic sound signal data frame in the data frame pair, the system performs a Fast Fourier Transform (FFT) on it, converting the time-domain data frame into a frequency-domain complex sequence containing amplitude and phase information; this sequence is the first frequency-domain representation. Simultaneously, the system performs the same FFT operation on the enhanced periodic vibration signal data frame in the same data frame pair, also converting the vibration time-domain data frame into a frequency-domain complex sequence; this sequence is the second frequency-domain representation. This process prepares the necessary frequency-domain data for subsequent spectral density calculations.

[0070] Step S1047: Based on the first frequency domain representation and the second frequency domain representation, calculate the cross power spectral density and self power spectral density of each data frame pair.

[0071] In step S1047, the cross power spectral density is used to represent a measure of the correlation between two different signals in the frequency domain, reflecting the degree of correlation between the energy at a certain frequency in one signal and the energy at the same frequency in another signal; the self power spectral density is used to represent the distribution of the power of a single signal at different frequencies.

[0072] Specifically, for each data frame pair, the system performs calculations using the first and second frequency domain representations generated in step S1046. To calculate the auto-power spectral density of the enhanced periodic sound signal, the system multiplies each complex value in the first frequency domain representation by its own complex conjugate. Similarly, to calculate the auto-power spectral density of the enhanced periodic vibration signal, the system multiplies each complex value in the second frequency domain representation by its own complex conjugate. To calculate the cross-power spectral density between the two signals, the system multiplies each complex value in the first frequency domain representation by the complex conjugate of the corresponding frequency complex value in the second frequency domain representation. After the calculations are completed, each data frame pair will yield two auto-power spectral density results and one cross-power spectral density result.

[0073] Step S1048: Calculate the spectral coherence values ​​at different frequency points based on the cross power spectral density and the self power spectral density, and combine the spectral coherence values ​​of all data frames along the time dimension to form a two-dimensional coherence matrix with time as one axis and frequency as the other axis, and use the two-dimensional coherence matrix as the coherence feature.

[0074] In step S1048, the spectral coherence value represents the degree of linear correlation between two signals at a specific frequency. Its value is between 0 and 1, where 1 indicates complete linear correlation and 0 indicates no correlation. The two-dimensional coherence matrix is ​​a data structure in which one dimension represents time and the other dimension represents frequency. The element values ​​in the matrix are the spectral coherence values ​​at the corresponding time and frequency points.

[0075] Specifically, the system uses the spectral density result obtained in step S1047 to calculate the spectral coherence. For each data frame pair, at each discrete frequency point, the system calculates the square of the amplitude of the cross-power spectral density, and then divides this result by the product of the auto-power spectral density of the enhanced periodic sound signal and the auto-power spectral density of the enhanced periodic vibration signal at that frequency point. The result of this calculation is the spectral coherence value at that frequency point. The system repeats this calculation for all frequency points, thereby generating a complete spectral coherence value vector for the data frame pair. Finally, the system combines and arranges the spectral coherence value vectors of all data frame pairs in chronological order to form a two-dimensional coherence matrix with time as one axis and frequency as the other axis, and outputs this two-dimensional coherence matrix as the coherence feature.

[0076] To facilitate understanding, a specific example is given below to explain steps S1045-S1048: For example, the system receives a synchronized enhanced periodic audio signal and an enhanced periodic vibration signal. Using a frame length of 20 milliseconds and an overlap rate of 50%, the system synchronously frames the two signals, resulting in a series of data frame pairs. For one data frame pair, the system performs a Fast Fourier Transform on both the audio and vibration frames to obtain their respective complex spectra, i.e., the first frequency domain representation and the second frequency domain representation. Then, based on these two frequency domain representations, the system calculates the auto-power spectral density of the audio signal, the auto-power spectral density of the vibration signal, and the cross-power spectral density between them. Next, for each frequency component within that time frame, the system uses the coherence calculation formula, which divides the square of the cross-power spectral density amplitude by the product of the two auto-power spectral densities, resulting in a coherence value between 0 and 1. The coherence values ​​at all frequency points constitute a coherence vector. The system repeats this process for all data frame pairs and arranges the resulting coherence vectors along the time axis, ultimately forming a two-dimensional coherence matrix. This matrix visually demonstrates the strength of the linear correlation between sound and vibration signals at any time and frequency point; this matrix is ​​the coherence characteristic.

[0077] Step S105: Based on the periodic acoustic spectrum features and coherence features, construct multimodal features and input the multimodal features into the preset abnormal noise detection model to obtain the abnormal noise type of the escalator.

[0078] In step S105, multimodal features refer to a comprehensive feature representation formed by fusing data from different sources or types, which is used to provide richer information than a single data source; the preset abnormal noise detection model refers to a deep learning or machine learning algorithm model that has been pre-trained with a large amount of known escalator abnormal noise data and can analyze input features and automatically classify them; the abnormal noise type refers to a specific fault category predefined according to the escalator fault mechanism, such as step roller wear, drive chain slack, handrail belt friction, etc.

[0079] Specifically, the system first fuses the periodic spectrogram features and coherence features generated in step S104 to construct multimodal features. The system treats these two two-dimensional matrices as two independent channels, similar to the red, green, and blue channels of a color image, and stacks them along the feature dimension to form a three-dimensional tensor as input. Subsequently, the system inputs this multimodal feature into a preset abnormal noise detection model. This model has a complex internal structure and may include convolutional layers for extracting local time-frequency patterns, a deep autoencoder for data dimensionality reduction and abstract representation, and a long short-term memory network for capturing temporal dependencies. After layer-by-layer processing and information extraction within the model, the multimodal features finally pass through a fully connected layer with a Softmax activation function. This layer outputs a probability distribution vector, where each element corresponds to the probability that the escalator belongs to a certain preset abnormal noise type. The system selects the category with the highest probability value as the final detection result, thereby determining the abnormal noise type of the escalator.

[0080] In one possible implementation, multimodal features are constructed based on periodic spectrogram features and coherence features, specifically including steps S1051-S1053, as follows: Step S1051: Calculate the spectral slope and kurtosis of the amplitude spectrum of the enhanced periodic sound signal to obtain the spectral morphology features.

[0081] In step S1051, spectral slope is an index that describes the trend of spectral energy change with frequency, usually reflecting the attenuation rate of high-frequency components relative to low-frequency components; spectral kurtosis is a statistical measure of the sharpness or flatness of the spectral distribution shape, and a high kurtosis value indicates that the spectral energy is concentrated on a few frequency points; spectral morphology features are composed of spectral slope and spectral kurtosis, and are a set of features used to describe the overall shape of the spectrum.

[0082] Specifically, the system first performs the same synchronous framing processing on the enhanced periodic audio signal as in step S1045. For each audio data frame, the system performs a Fast Fourier Transform and calculates the amplitude of the transform result to obtain the amplitude spectrum of that data frame. Subsequently, the system calculates the spectral slope based on this amplitude spectrum. One implementation method is to take the logarithm of both the amplitude spectrum and the frequency axis, and then perform linear regression analysis on the transformed data; the slope of the resulting straight line is the spectral slope. Simultaneously, the system treats the amplitude spectrum as a probability distribution and calculates its fourth standard moment to obtain the spectral kurtosis value. The system repeats this process for all data frames, generating a spectral slope sequence and a spectral kurtosis sequence corresponding to each time frame. These two sequences together constitute the spectral morphology features.

[0083] Step S1052: Based on the coherence characteristics, extract the modulation frequency bandwidth to obtain the modulation characteristics used to distinguish between structured sound and airborne sound.

[0084] In step S1052, the modulation frequency bandwidth refers to the width of a continuous frequency range in which the spectral coherence value is consistently higher than a preset threshold in the coherence feature. This feature is used to quantify the frequency range in which sound and vibration signals are strongly correlated. The modulation feature is calculated based on the modulation frequency bandwidth and is a specific index used to distinguish sounds from different propagation paths. Structure sound usually exhibits high coherence over a wide frequency range, while air sound does not.

[0085] Specifically, the system uses the two-dimensional coherence matrix generated in the previous steps, i.e., the coherence feature. Each column or row of this matrix represents the spectral coherence value within a time frame. The system iterates through each time frame, analyzing the corresponding coherence value vector. The system sets a coherence threshold, for example, 0.8. For the coherence vector of the current time frame, the system searches for all frequency points from low to high frequencies that have coherence values ​​exceeding the threshold, and determines the start and end frequencies of the widest continuous bandwidth formed by these frequency points. The modulation frequency bandwidth is the difference between the end frequency and the start frequency. The system performs this calculation for all time frames, ultimately generating a time-varying modulation frequency bandwidth sequence, which is used as the modulation feature.

[0086] Step S1053: Combine the periodic spectrogram features, coherence features, spectral morphology features, and modulation features as multiple feature channels along the feature dimension to form an input tensor, and use the input tensor as a multimodal feature.

[0087] In step S1053, the feature channel refers to treating features from different sources or types as independent dimensions when constructing multidimensional input data, similar to the red, green, and blue channels in a color image; the feature dimension is the dimension used to stack different feature channels; the input tensor is a multidimensional array that integrates the data from all feature channels and serves as the input to the subsequent processing model.

[0088] Specifically, the system first prepares four sets of feature data: the first set is the periodic spectrogram feature, a two-dimensional time-frequency matrix; the second set is the coherence feature, i.e., the two-dimensional coherence matrix calculated in step S1048; the third set is the spectral morphology feature, including the spectral slope and spectral kurtosis time series calculated in step S1051; and the fourth set is the modulation feature, i.e., the modulation frequency bandwidth time series calculated in step S1052. To ensure that all features have the same dimension for combination, the system expands the three time series of spectral slope, spectral kurtosis, and modulation frequency bandwidth. For each time frame, the system copies the spectral slope value, spectral kurtosis value, and modulation frequency bandwidth value of that frame to all frequency units corresponding to that time frame, thereby expanding these three one-dimensional time series into three two-dimensional time-frequency matrices with the same size as the periodic spectrogram feature. Then, the system stacks the five two-dimensional matrices—the periodic spectrogram feature matrix, the coherence feature matrix, the expanded spectral slope matrix, the expanded spectral kurtosis matrix, and the expanded modulation feature matrix—along a new feature dimension. Finally, an input tensor with three dimensions—time, frequency, and feature channels—is generated, which is the multimodal feature.

[0089] To facilitate understanding, a specific example is given below to explain steps S1051-S1053: For example, after processing a signal, the system obtains a periodic spectrogram feature with 100 frames and 512 frequency points per frame. Simultaneously, the system also calculates a coherence feature matrix of the same size (100x512). Next, the system calculates the spectral slope and kurtosis for each frame of the audio signal, resulting in two one-dimensional arrays of length 100. The system then calculates the modulation frequency bandwidth based on the coherence vector of each frame, also resulting in a one-dimensional array of length 100. To integrate all the information, the system creates three new 100x512 matrices: the nth row of the first matrix is ​​filled with the spectral slope values ​​of the nth frame; the nth row of the second matrix is ​​filled with the spectral kurtosis values ​​of the nth frame; and the nth row of the third matrix is ​​filled with the modulation frequency bandwidth values ​​of the nth frame. Finally, the system stacks the original periodic spectrogram, the coherence matrix, and these three newly created matrices together like a deck of cards, forming a 100x512x5 three-dimensional tensor. This five-channel tensor contains multimodal features including sound spectrum, acoustic vibration correlation, spectral morphology, and modulation characteristics, which can be input into a deep learning model for subsequent analysis.

[0090] Please refer to Figure 2 In one possible implementation, the multimodal features are input into a preset abnormal noise detection model to obtain the abnormal noise type of the escalator, specifically including steps S201-S208, as follows: Step S201: Take the multimodal features as input and process them through at least one convolutional layer and pooling layer of the preset abnormal noise detection model to extract local time-frequency features from the multimodal features and generate a high-dimensional feature map.

[0091] In step S201, a convolutional layer is a neural network layer that uses a convolutional kernel to perform sliding computation on the input data, used to effectively extract spatial local patterns in images or time-frequency maps; a pooling layer usually follows the convolutional layer, reducing the data dimensionality by downsampling the feature map, while retaining the most salient features and enhancing the translation invariance of the features; local time-frequency features refer to specific patterns or textures existing in a small neighborhood in the time and frequency dimensions of the multimodal feature tensor; a high-dimensional feature map is intermediate data composed of multiple channels after convolution and pooling operations, which abstractly represents the original input features.

[0092] Specifically, the system uses the multimodal feature input tensor generated in the preceding steps as the input to the preset anomaly detection model. This tensor has three dimensions: time, frequency, and feature channels. The input tensor first passes through a first convolutional layer, which contains multiple three-dimensional convolutional kernels. Each kernel performs a sliding window convolution operation on the time and frequency dimensions of the input tensor, while integrating information from all feature channels to capture local time-frequency correlations in the multimodal features. The result of the convolution operation is then passed through a non-linear activation function, such as the ReLU function, to increase the model's non-linear expressive power and generate preliminary feature maps. Subsequently, these feature maps are fed into a pooling layer, such as a max pooling layer. This pooling layer slides a window with a specific stride on the time and frequency dimensions of the feature map, retaining only the maximum value within each window, thereby reducing the feature map resolution while preserving the most significant local time-frequency features. The system can repeatedly stack multiple convolutional and pooling layers, with each layer extracting more abstract and complex features from the high-dimensional feature map output by the previous layer. After processing by the last pooling layer, the final generated tensor is the high-dimensional feature map.

[0093] Step S202: Input the high-dimensional feature map into the deep autoencoder of the preset abnormal noise detection model, perform nonlinear dimensionality reduction mapping, and obtain the corresponding multiple low-dimensional feature vectors.

[0094] In step S202, a deep autoencoder is an unsupervised neural network consisting of an encoder and a decoder. Its encoder part can compress and map high-dimensional input data to a low-dimensional latent space. Nonlinear dimensionality reduction mapping refers to the process of transforming data from a high-dimensional space to a low-dimensional space through multi-layer neural networks and nonlinear activation functions. This process can learn the complex structure inherent in the data. The low-dimensional feature vector is a compact representation of the high-dimensional feature map after processing by the encoder part of the deep autoencoder.

[0095] Specifically, the system first performs dimensionality rearrangement on the high-dimensional feature map generated in step S201. The original high-dimensional feature map has dimensions of time, frequency, and number of channels. The system flattens the frequency and channel dimensions at each time step, forming a time series composed of multiple high-dimensional vectors, with the length of the series equal to the number of time steps. Then, the system inputs each high-dimensional vector in this series one by one into the encoder part of a pre-trained deep autoencoder in the preset anomaly detection model. This encoder consists of multiple fully connected layers, which, by progressively reducing the number of neurons and using nonlinear transformations, maps the input high-dimensional vectors to a latent space with significantly reduced dimensions. The output of the last hidden layer of the encoder is a low-dimensional feature vector corresponding to the current time step. The system repeats this process for the high-dimensional vectors of all time steps, ultimately obtaining multiple low-dimensional feature vectors with the same time length as the high-dimensional feature map.

[0096] Step S203: Sort multiple low-dimensional feature vectors in chronological order to form a feature sequence.

[0097] In step S203, the time order refers to the natural time flow order of the input signal from start to finish; the feature sequence is a one-dimensional sequence data structure composed of multiple low-dimensional feature vectors arranged according to their corresponding time order.

[0098] Specifically, the system collects all low-dimensional feature vectors generated for each time step in step S202. Since these vectors are generated step-by-step during the processing of the high-dimensional feature map, they naturally retain the time index corresponding to the original signal. Based on these time indices, the system arranges all low-dimensional feature vectors in an ordered manner, starting from the vector corresponding to the first time step and ending with the vector corresponding to the last time step. This forms a two-dimensional tensor, where one dimension is the time step and the other dimension is the dimension of the low-dimensional feature vectors; this tensor is the feature sequence.

[0099] Step S204: Input the feature sequence into the forward long short-term memory network and the backward long short-term memory network.

[0100] In step S204, the forward long short-term memory network is a recurrent neural network that can process and remember long-term dependencies in sequence information. It processes the input sequence in the forward order of time. The backward long short-term memory network has the same structure as the forward network, but it processes the input sequence in the reverse order of time.

[0101] Specifically, the system treats the feature sequence constructed in step S203 as a single data unit. This feature sequence is simultaneously fed into two parallel components of the preset abnormal noise detection model: a forward long short-term memory network and a backward long short-term memory network. This means that a copy of the same feature sequence is used as input to these two independent networks to extract contextual information from two different temporal directions.

[0102] Step S205: Process the feature sequence in ascending time order through a feedforward long short-term memory network to obtain the first output result.

[0103] In step S205, the first output result refers to the set of hidden state vectors generated by the feedforward long short-term memory network for each time step after processing the entire feature sequence.

[0104] Specifically, after receiving the feature sequence, the feedforward Long Short-Term Memory (FSS) network begins processing from the first feature vector of the sequence, corresponding to time step t=1. At each time step t, the network integrates the input feature vector of the current time step with the hidden state and cell state of the previous time step t-1, updating the cell state and calculating the hidden state output for the current time step t through its internal input gate, forget gate, and output gate. This process proceeds forward along the time axis until the last feature vector in the sequence has been processed. The hidden state vectors generated at all time steps are combined sequentially to form the first output result.

[0105] Step S206: Process the feature sequence in reverse chronological order using a backward long short-term memory network to obtain the second output result.

[0106] In step S206, the second output result refers to the set of hidden state vectors generated by the backward long short-term memory network for each time step after processing the entire feature sequence.

[0107] Specifically, after receiving the feature sequence, the backward Long Short-Term Memory (LSTM) network begins processing from the last feature vector in the sequence, corresponding to time step t=T. At each time step t, the network integrates the input feature vector of the current time step with the hidden state and cell state of the next time step t+1, updating the state and calculating the hidden state output for the current time step t through its internal gating mechanism. This process proceeds backward along the time axis until the first feature vector in the sequence has been processed. The hidden state vectors generated at all time steps are also arranged in the original chronological order, forming the second output result.

[0108] Step S207: Concatenate the first output result and the second output result at each time step to form a bidirectional context representation, and use the bidirectional context representation as the output feature.

[0109] In step S207, a time step refers to a single point in time in the feature sequence; concatenation is an operation that connects two or more vectors end to end to form a longer vector; bidirectional context representation is an enhanced feature representation that simultaneously contains past and future information at each time step; and the output feature is the sequence feature used for final classification after being processed by a bidirectional long short-term memory network.

[0110] Specifically, the system obtains the first output result generated in step S205 and the second output result generated in step S206. Both results are hidden state sequences of the same length as the original feature sequence. The system iterates through all time steps. For each time step t, the system extracts the forward hidden state vector corresponding to time step t from the first output result and the backward hidden state vector corresponding to time step t from the second output result. Then, the system concatenates these two vectors along the feature dimension to generate a new combined vector with a dimension equal to the sum of the dimensions of the two hidden state vectors. This process is performed once for all time steps, thus forming a new feature sequence in which each vector encodes the bidirectional temporal dependency information at that time point. This new sequence is the bidirectional context representation and is designated as the output feature.

[0111] Step S208: Input the output features into at least one fully connected layer and process them using the Softmax function to output the probability distribution of the escalator on multiple preset abnormal noise categories, and determine the category corresponding to the highest probability as the abnormal noise type of the escalator.

[0112] In step S208, the fully connected layer is a neural network layer in which every neuron in each layer is connected to all neurons in the previous layer, and is used to perform high-order combination and mapping of features; the Softmax function is a function that transforms a real number vector into a probability distribution, ensuring that each value of the output is between 0 and 1 and the sum is 1; the probability distribution represents the likelihood of the input signal belonging to each preset abnormal sound category; the abnormal sound type is the category with the highest probability that the model finally determines.

[0113] Specifically, the system extracts a representative vector for classification from the output feature sequence generated in step S207. One approach is to extract only the vector from the last time step of the sequence, as it already incorporates information from the entire sequence. This representative vector is then input into at least one fully connected layer. The first fully connected layer performs a linear transformation on the input vector and applies a non-linear activation function. If multiple fully connected layers exist, the output of the previous layer serves as the input for the next layer, and feature transformations are performed layer by layer. The last fully connected layer outputs a vector with the number of neurons equal to the total number of preset abnormal noise categories, and its output is a numerical vector, with each value corresponding to the original score of a category. This score vector is then input into a Softmax function. The Softmax function exponentializes each element in the vector and then divides each exponentialized value by the sum of all exponentialized values, thus converting the score vector into a probability distribution vector. Each element in this vector represents the probability that the input signal belongs to the corresponding abnormal noise category. Finally, the system finds the element with the largest value in this probability distribution vector and determines the preset abnormal noise category corresponding to this element as the final abnormal noise type of the escalator, then outputs it.

[0114] To facilitate understanding, a specific example is given below to explain steps S201-S208: Suppose that a segment of escalator operation signal, after preprocessing, forms a multimodal feature tensor containing 100 time steps, 512 frequency points, and 5 feature channels. This 100x512x5 tensor is first input into a network consisting of several convolutional and pooling layers. After feature extraction and dimensionality reduction, a high-dimensional feature map of 25x64x32 is output. Next, for each of the 25 time steps, the 64x32 feature map is flattened and mapped to a 128-dimensional low-dimensional feature vector by a deep autoencoder, thus forming a 25x128 feature sequence. This sequence is simultaneously fed into a forward LSTM and a backward LSTM. The forward LSTM processes from step 1 to step 25, and the backward LSTM processes from step 25 to step 1, each generating a 25x64 hidden state sequence. At each time step, the system concatenates the 64-dimensional forward state and the 64-dimensional backward state to form a 25x128 bidirectional context representation sequence. The system takes the 128-dimensional vector from the last time step of this sequence and inputs it into a fully connected layer. This fully connected layer outputs a 4-dimensional vector corresponding to four preset categories: normal, chain noise, step friction, and motor humming. This 4-dimensional vector, for example [0.5, 3.2, -1.1, 0.9], is then processed by the Softmax function to convert it into a probability distribution [0.09, 0.84, 0.01, 0.06]. Because the probability of the second element, 0.84, is the highest, the system ultimately determines the abnormality type of this signal segment as "chain noise".

[0115] The following describes an escalator noise detection system according to an embodiment of the present invention from a hardware processing perspective. Please refer to [link / reference]. Figure 3 This is a schematic diagram of the structure of an escalator noise detection system according to an embodiment of this application.

[0116] It should be noted that, Figure 3 The structure of the escalator noise detection system shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0117] like Figure 3 As shown, an escalator noise detection system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 302 or a program loaded from storage section 308 into random access memory (RAM) 303, such as performing the methods described in the above embodiments. The RAM 303 also stores various programs and data required for system operation. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0118] The following components are connected to I / O interface 305: input section 306 including audio input devices, push-button switches, etc.; output section 307 including a liquid crystal display (LCD) and audio output devices, indicator lights, etc.; storage section 308 including a hard disk, etc.; and communication section 309 including a network interface card such as a LAN (Local Area Network) card, modem, etc. Communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0119] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing computer programs for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311. When the computer program is executed by central processing unit (CPU) 301, it performs the various functions defined in the present invention.

[0120] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings.

[0122] Specifically, the escalator noise detection system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, it implements the escalator noise detection method provided in the above embodiment.

[0123] In another aspect, the present invention also provides a computer-readable storage medium, which may be included in the escalator noise detection system described in the above embodiments; or it may exist independently and not assembled into the escalator noise detection system. The storage medium carries one or more computer programs, which, when executed by a processor of the escalator noise detection system, cause the escalator noise detection system to implement the escalator noise detection method based on IoT data encryption transmission provided in the above embodiments.

Claims

1. A method for detecting abnormal noise in escalators, characterized in that, The method includes: Acquire sound and vibration signals during the operation of the escalator, and determine the reference operating cycle of the escalator based on the vibration signals; Based on the aforementioned baseline operating cycle, the sound signal is divided into multiple sound signal segments, and the vibration signal is divided into multiple vibration signal segments; The sound signal segment and the vibration signal segment are synchronously superimposed to obtain an enhanced periodic sound signal and an enhanced periodic vibration signal. Time-frequency analysis is performed on the enhanced periodic sound signal to obtain periodic spectrogram features, and the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal is calculated to obtain coherence features; Based on the periodic acoustic spectrum features and the coherence features, a multimodal feature is constructed, and the multimodal feature is input into a preset abnormal noise detection model to obtain the abnormal noise type of the escalator.

2. The method according to claim 1, characterized in that, Determining the reference operating cycle of the escalator based on the vibration signal specifically includes: The vibration signal is subjected to low-pass filtering to obtain the filtered vibration signal; Autocorrelation analysis was performed on the filtered vibration signal to obtain an autocorrelation coefficient sequence; The time interval corresponding to the first local maximum value of the autocorrelation coefficient sequence is determined, and the time interval is determined as the reference operating cycle of the escalator.

3. The method according to claim 1, characterized in that, The step of performing time-frequency analysis on the enhanced periodic sound signal to obtain periodic spectrogram features specifically includes: The enhanced periodic sound signal is divided into multiple data frames along the time axis, wherein each data frame has a preset frame length and adjacent data frames have a preset overlap rate. Each data frame is processed by a window function, and a fast Fourier transform is performed on each data frame after window function processing to obtain the complex spectrum corresponding to the data frame. The amplitude of the complex spectrum is calculated to obtain the amplitude spectrum, and the amplitude spectra of all data frames are stacked along the time dimension to form a two-dimensional time-frequency matrix with time as one axis and frequency as the other axis. The amplitude values ​​in the two-dimensional time-frequency matrix are logarithmically transformed and normalized to generate the periodic spectrogram features.

4. The method according to claim 1, characterized in that, The calculation of the spectral coherence between the enhanced periodic sound signal and the enhanced periodic vibration signal to obtain coherence characteristics specifically includes: The enhanced periodic sound signal and the enhanced periodic vibration signal are synchronously framed to obtain corresponding data frame pairs. Perform a Fast Fourier Transform on each data frame pair to obtain a first frequency domain representation of the enhanced periodic sound signal and a second frequency domain representation of the enhanced periodic vibration signal; Based on the first frequency domain representation and the second frequency domain representation, calculate the cross power spectral density and the self power spectral density of each data frame pair; Based on the cross-power spectral density and the self-power spectral density, the spectral coherence values ​​at different frequency points are calculated, and the spectral coherence values ​​of all data frames are combined along the time dimension to form a two-dimensional coherence matrix with time as one axis and frequency as the other axis. The two-dimensional coherence matrix is ​​used as the coherence feature.

5. The method according to claim 1, characterized in that, The construction of multimodal features based on the periodic spectrogram features and the coherence features specifically includes: The spectral slope and kurtosis of the amplitude spectrum of the enhanced periodic sound signal are calculated to obtain the spectral morphology features; Based on the coherence characteristics, the modulation frequency bandwidth is extracted to obtain modulation characteristics used to distinguish between structured sound and airborne sound. The periodic spectrogram features, coherence features, spectral morphology features, and modulation features are combined as multiple feature channels along the feature dimension to form an input tensor, and the input tensor is used as the multimodal features.

6. The method according to claim 1, characterized in that, The step of inputting the multimodal features into a preset abnormal noise detection model to obtain the abnormal noise type of the escalator specifically includes: The multimodal features are taken as input and processed through at least one convolutional layer and pooling layer of the preset abnormal noise detection model to extract local time-frequency features from the multimodal features and generate a high-dimensional feature map. The high-dimensional feature map is input into the deep autoencoder of the preset abnormal noise detection model to perform nonlinear dimensionality reduction mapping, thereby obtaining multiple corresponding low-dimensional feature vectors. The multiple low-dimensional feature vectors are sorted in chronological order to form a feature sequence, and the feature sequence is input into the long short-term memory network of the preset abnormal noise detection model to obtain the output features; The output features are input into at least one fully connected layer and processed using the Softmax function to output the probability distribution of the escalator in multiple preset abnormal noise categories, and the category corresponding to the highest probability is determined as the abnormal noise type of the escalator.

7. The method according to claim 6, characterized in that, The Long Short-Term Memory (LSTM) network includes a forward LSTM network and a backward LSTM network. The step of inputting the feature sequence into the LSTM network of the preset abnormal noise detection model to obtain output features specifically includes: The feature sequence is input into the forward long short-term memory network and the backward long short-term memory network; The feature sequence is processed in chronological order by the feedforward long short-term memory network to obtain the first output result; The feature sequence is processed in reverse chronological order by the backward long short-term memory network to obtain a second output result. The first output result and the second output result are concatenated at each time step to form a bidirectional context representation, and the bidirectional context representation is used as the output feature.

8. A detection system for abnormal noise in escalators, characterized in that, The escalator noise detection system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the escalator noise detection system to perform the method as described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the escalator noise detection system, the escalator noise detection system performs the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, When the computer program product is run on the escalator noise detection system, the escalator noise detection system performs the method as described in any one of claims 1-7.

Images