Escalator step missing detection method, system and device based on multi-feature fusion

Abstract

The application discloses an escalator step missing detection method, system and device based on multi-feature fusion, the method comprising: obtaining escalator step image data, extracting the step belt area ROI; constructing a one-dimensional stripe signal, using a short-time Fourier transform algorithm to perform local frequency domain analysis on the one-dimensional stripe signal, calculating the main cycle frequency, phase information and cycle energy distribution corresponding to each position, judging the step cycle structure anomaly, obtaining the cycle anomaly evidence of step missing; extracting the step edge information of the step belt area through an edge detection algorithm, judging the step geometric structure discontinuous feature, obtaining the geometric anomaly evidence of step missing; segmenting the cavity or dark area in the step belt area, obtaining the spatial distribution result of the cavity area as the cavity anomaly evidence of step missing; and fusing the results to generate an alarm state output. The application can realize real-time and high-reliability detection of escalator step missing under complex operating environment.

Description

Technical Field

[0001] This application relates to the field of escalator fault detection technology, and in particular to a method, system and equipment for detecting missing escalator steps based on multi-feature fusion. Background Technology

[0002] With the continuous acceleration of urbanization in my country, escalators have been widely installed in public places such as shopping malls, subway stations, airports, and transportation hubs to improve pedestrian traffic efficiency. Escalators, due to their advantages of continuous operation, high load-bearing capacity, and high efficiency, have become an indispensable transportation facility in densely populated areas. However, due to factors such as long-term high-frequency operation, complex operating environments, and limited maintenance cycles, key mechanical components of escalators are prone to wear, aging, and even loss during use. Missing steps, in particular, pose a significant safety hazard that can easily lead to serious personal injury.

[0003] When an escalator step is missing, the continuous structure of the escalator treads is disrupted. Users are highly susceptible to accidents such as misstepping, falling, or being trapped inside the escalator if they are not aware of the situation in time. This danger is significantly increased, especially in crowded areas, where upward and downward movements converge, or where visibility is limited. In recent years, safety accidents caused by missing escalator steps or damaged treads have occurred frequently, posing a serious threat to public safety. Therefore, real-time and accurate monitoring of the escalator step operation status and timely detection of anomalies such as missing steps are crucial for ensuring the safety of public places.

[0004] Currently, the main methods for detecting abnormalities in escalator steps include the following types:

[0005] Regular manual inspections: Maintenance personnel conduct manual inspections of the escalators according to a predetermined schedule, using visual inspection or simple tools to determine whether the steps are damaged or missing. However, this method relies on manual experience, has a long inspection cycle, and is difficult to detect sudden defects in a timely manner, resulting in a significant lag.

[0006] Mechanical or electrical sensing detection: Position sensors, limit switches, or operational status detection devices are installed inside the escalator to indirectly determine whether a fault exists by detecting abnormalities in the movement of the steps. However, this type of method is mostly for overall operational anomalies and is difficult to accurately identify local missing steps, and the installation and maintenance costs are relatively high.

[0007] Structural contact detection: This method senses the integrity of the steps through mechanical contacts or guide structures. However, it is susceptible to wear and environmental interference, has low reliability, and is not suitable for existing escalator systems that are already in operation.

[0008] Visual detection methods involve installing cameras in the escalator's operating area to identify the status of each step through image or video analysis. This method offers advantages such as being non-contact and providing a large amount of information, demonstrating significant development potential. However, current visual detection methods largely rely on simple features or general target recognition models, making them susceptible to factors such as changes in lighting, shadows, and human occlusion, leading to prominent false alarms and missed alarms.

[0009] Machine vision-based escalator step missing detection systems typically involve installing cameras around or above the escalator to collect and analyze image or video data during operation. Anomalies are determined by identifying the surface structure, edge features, or continuity of the steps. Image and video-based detection methods offer advantages such as high accuracy, no need to modify the escalator's mechanical structure, and minimal impact on normal passenger behavior. However, escalator steps have a strong periodic structure, and operation is accompanied by complex factors such as passenger obstruction, reflections, shadows, and changing viewing angles. Traditional machine vision algorithms still have shortcomings in real-time performance and robustness. Furthermore, the computational overhead of complex visual models limits the application of such methods in real-time safety monitoring scenarios.

[0010] Therefore, considering the characteristics of escalator steps during operation, such as strong structural periodicity, high safety risks after missing steps, and complex on-site environment, this study proposes a machine vision-based escalator step missing detection method with high reliability, low false alarm rate, and suitability for real-time operation. This method has significant research value and practical significance for improving the operational safety level of escalators in public places. Summary of the Invention

[0011] This application provides a method, system, and device for detecting missing escalator steps based on multi-feature fusion. The purpose is to address the problems of existing methods for detecting missing escalator steps, which mainly rely on manual inspection, mechanical or electrical sensor detection, and single visual target detection algorithms. These methods suffer from issues such as detection lag, insufficient real-time performance, susceptibility to occlusion and changes in lighting, and high false alarm rates. The application proposes an intelligent method for detecting missing escalator steps that can achieve real-time and highly reliable detection in complex operating environments.

[0012] On the one hand, this application provides a method for detecting missing escalator steps based on multi-feature fusion, including the following steps:

[0013] S1: Acquire escalator step image data, preprocess the image data, and extract the ROI (Region of Interest) of the step area in the image;

[0014] S2: Construct a one-dimensional stripe signal based on the ROI, and use the short-time Fourier transform algorithm to perform local frequency domain analysis on the one-dimensional stripe signal. Calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the ladder periodic structure, and obtain evidence of periodic anomalies with missing ladders.

[0015] S3: Extract the edge information of the tiered zone using an edge detection algorithm, fit the leading edge line of the tiered zone using a straight line transformation, determine the discontinuity features of the tiered geometric structure, and obtain geometric anomaly evidence of missing tiered zones.

[0016] S4: Segment the cavity or dark area in the step zone to obtain the spatial distribution of the cavity area as evidence of cavity anomaly in the missing step.

[0017] S5: Integrate and output evidence of periodic anomalies, geometric anomalies, and cavity anomalies with missing cascades.

[0018] Furthermore, in step S1, the preprocessing of the image data includes converting the image into a grayscale image and performing Gaussian filtering on the grayscale image for noise reduction.

[0019] Furthermore, in step S1, the step zone area is defined as including the step surface, the leading edge line, and the side skirt / yellow line area.

[0020] Furthermore, in step S2, a one-dimensional fringe signal is constructed through the following steps:

[0021] In the ROI, one or more fixed sampling bands are selected, and multi-row averaging is used to enhance noise reduction. m rows near the center of the stepped band region are selected, and a one-dimensional stripe signal is constructed by averaging the pixels of the selected m rows in the vertical direction.

[0022] Furthermore, in step S2, the step of determining the anomaly of the tiered periodic structure is as follows:

[0023] Three local anomalies are calculated based on the main period frequency, phase information, and period energy distribution:

[0024] The main frequency stability is abnormal, which is the first-order difference of the main frequency.

[0025] The phase continuity anomaly is caused by the phase increment deviating from the median.

[0026] The energy collapse anomaly is a normalized energy ratio;

[0027] Normalize the three terms to [0,1];

[0028] The anomaly intensity of the periodic phase coherence map (PCM) is calculated as a weighted sum of three terms;

[0029] The periodic anomaly fraction is obtained by summing the PCM anomaly intensities. The PCM anomaly intensity is interpolated back to the pixel x-coordinate to obtain the fracture zone mask. .

[0030] Furthermore, in step S3, the step of determining the discontinuity characteristics of the ladder geometry is as follows:

[0031] The line parameters obtained after the Hough line transform are:

[0032]

[0033] Project each line onto the ROI coordinate system and calculate its relationship with the ROI centerline y=y. c The intersection point x i , and press x i Sort to obtain the sequence:

[0034]

[0035] The spacing between adjacent leading edges is:

[0036]

[0037] Define the spacing anomaly ratio:

[0038]

[0039] Define geometric fault strength:

[0040]

[0041] in This is the upper limit of experience.

[0042] Calculate the geometric anomaly score:

[0043]

[0044] Map the locations corresponding to abnormal spacing to pixel fault band masks, that is, mark the interval [x(i), x(i+1)] as faults:

[0045]

[0046] This is the adjustment constant.

[0047] Furthermore, in step S4, the cavity mask is obtained by segmentation. Given area A and perimeter P, calculate the cavity compactness:

[0048]

[0049] Calculate the semantic anomaly score:

[0050]

[0051]

[0052] in , All of these are adjustable constants.

[0053] Furthermore, step S5 includes the following steps:

[0054] Extend the fracture zone mask and pixel tomographic zone mask obtained in steps S2 and S3 from 1D masks to 2D:

[0055]

[0056] Define the overlapping region of the three evidence spaces:

[0057]

[0058] Calculate the overlap area ratio:

[0059]

[0060] Calculate the spatial consistency gating coefficient:

[0061]

[0062] in For reference only;

[0063] Calculate the instantaneous fusion score:

[0064]

[0065] α, β, γ are weighting coefficients;

[0066] The final confidence level At is calculated using sequential cumulative calculation:

[0067]

[0068] Where λ is the attenuation coefficient;

[0069] Calculate and update alarm status y t :

[0070]

[0071] Θ on Θoff and Θoff are the trigger threshold and recovery threshold, respectively.

[0072] On another front, this application provides an escalator step missing detection system based on multi-feature fusion, comprising:

[0073] The information acquisition unit is used to acquire escalator step image data, preprocess the image data, and extract the ROI (Region of Interest) of the step area in the image.

[0074] The first information processing unit is used to construct a one-dimensional stripe signal based on the ROI, perform local frequency domain analysis on the one-dimensional stripe signal using the short-time Fourier transform algorithm, calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the ladder periodic structure, and obtain evidence of periodic anomalies with missing ladders.

[0075] The second information processing unit is used to extract the edge information of the ladder zone through the edge detection algorithm, fit the leading edge line of the ladder with a straight line transformation, judge the discontinuity features of the ladder geometric structure, and obtain geometric anomaly evidence of missing ladders.

[0076] The third information processing unit is used to segment the cavity or dark area in the step zone and obtain the spatial distribution result of the cavity area as evidence of cavity anomaly due to missing steps.

[0077] The fusion output unit is used to fuse the output results of the information processing unit and generate alarm status output.

[0078] In another aspect, this application provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and when the computer program is invoked and executed by the processor, it implements the steps in the method described above.

[0079] In summary, the beneficial effects of this application are:

[0080] (1) This invention introduces a periodic phase consistency analysis method based on short-time Fourier transform to quantitatively model the periodic structural characteristics of escalator steps during operation. It identifies step structural anomalies from multiple dimensions such as frequency stability, phase continuity and periodic energy change. Compared with existing technologies that rely solely on appearance features or target detection, this invention can directly reflect the true physical missing state of the steps.

[0081] (2) This invention combines Canny edge detection and Hough line transformation to construct a perspective geometric consistency constraint model. By analyzing the spacing continuity and structural integrity of the ladder leading edge sequence, the missing ladder is verified from the mechanical structure level, which can effectively avoid misjudgment caused by shadows, reflections and surface dirt.

[0082] (3) The present invention introduces a semantic segmentation algorithm based on YOLOv8-seg to perform pixel recognition on the tiered cavity region, and uses the semantic information as auxiliary evidence to perform spatial consistency verification with periodic anomalies and geometric anomalies, so that the semantic segmentation result does not trigger an alarm alone, thereby reducing the impact of false detection by a single visual model on the final judgment result.

[0083] (4) This invention uses a multi-feature spatial consistency constraint and a sequential cumulative decision mechanism to jointly constrain the detection results in the spatial and temporal dimensions, effectively suppressing false alarms caused by instantaneous occlusion, short-term noise and occasional interference. Without modifying the existing mechanical structure of the escalator, it can achieve real-time and stable detection of missing steps.

[0084] The technical solution proposed in this invention does not require modification of the existing mechanical structure of escalators. It can achieve real-time and stable detection of missing steps in complex operating environments, significantly improve detection accuracy and system robustness, effectively reduce safety risks during escalator operation, and enhance the intelligence and safety protection level of escalator operation. It has significant engineering application value and promotion significance. Attached Figure Description

[0085] Figure 1 This is a flowchart illustrating the escalator step missing detection method based on multi-feature fusion described in this application;

[0086] Figure 2 This is a schematic diagram of a periodic signal and an abnormal segment in one embodiment of this application;

[0087] Figure 3 This is a schematic diagram of edge / line extraction and geometric consistency in one embodiment of this application;

[0088] Figure 4 This is a schematic diagram of YOLOv8-seg identifying key regions in one embodiment of this application;

[0089] Figure 5 This is a schematic diagram of multi-feature fusion and spatial gating decision-making in one embodiment of this application;

[0090] Figure 6 This is a schematic diagram of time accumulation and lag threshold decision in one embodiment of this application;

[0091] Figure 7 This is a comparison chart of the detection accuracy and 95% confidence interval of different methods in one embodiment of this application;

[0092] Figure 8 This is a comparison chart of the Precision / Recall / F1 indices for different methods in one embodiment of this application. Detailed Implementation

[0093] The specific embodiments of this application are described in detail below with reference to the accompanying drawings.

[0094] A specific embodiment of this application provides a method for detecting missing escalator steps based on multi-feature fusion. The purpose is to address the problems of existing methods for detecting missing escalator steps, which mainly rely on manual inspection, mechanical or electrical sensor detection, and single visual target detection algorithms. These methods suffer from issues such as detection lag, insufficient real-time performance, susceptibility to occlusion and changes in lighting, and high false alarm rates. The proposed method is an intelligent method for detecting missing escalator steps that can achieve real-time and highly reliable detection in complex operating environments.

[0095] Existing machine vision-based detection methods typically treat missing steps as ordinary targets or abnormal areas, relying mainly on appearance features for identification. They fail to fully utilize the inherent physical characteristics of escalator steps during operation, such as the equidistant periodic texture structure, stable geometric arrangement, and consistent mechanical structure. This leads to the misjudgment of normal step gaps or shadowed areas as missing steps under conditions such as tread reflection, shadow interference, and personnel obstruction, making it difficult to meet the actual needs of public places for the safe operation of escalators.

[0096] The method described is an escalator step missing detection method based on the Periodic-Physics-Consistency Guard (PPC-Guard). This method focuses on the periodic texture structure of escalator steps as seen from a camera's perspective, combining periodic phase analysis algorithms, geometric structure consistency constraint algorithms, deep learning semantic segmentation algorithms, and multi-feature sequential cumulative decision algorithms. Through multi-source detection evidence fusion, it achieves stable and reliable determination of step missingness.

[0097] Specifically, this method first extracts the region of interest (ROI) of each step from the escalator's operating video and performs one-dimensional sampling of the step surface texture along the step's operating direction to form step stripe signals. Subsequently, the short-time Fourier transform (STFT) algorithm is used to perform local frequency domain analysis on the stripe signals, calculating the principal period frequency, phase information, and periodic energy distribution at each position. Based on the phase information, a phase-consistency map of the step structure is constructed. By detecting features such as phase continuity disruption, significant attenuation of periodic energy, and unstable principal frequency, anomalies in the step's periodic structure are determined, thereby obtaining evidence of periodic anomalies such as missing steps.

[0098] Secondly, this method introduces a ladder geometric consistency detection algorithm based on perspective geometric constraints. The Canny edge detection algorithm is used to extract ladder edge information, and the Hough line transform is applied to fit the ladder leading edge line. Within the perspective-corrected geometric framework, the parallelism, adjacent spacing consistency, and sequence continuity of the ladder leading edge line are analyzed. When a ladder is missing, obvious discontinuities or abrupt changes in spacing will appear in the ladder leading edge line sequence, thus obtaining geometrical evidence of ladder missingness through geometric structural discontinuities.

[0099] Meanwhile, to semantically confirm the cavity region formed by missing steps, this method employs a real-time semantic segmentation algorithm based on YOLOv8 to segment the surface region of the steps, the side structure region of the steps, and the cavity or dark region. The semantic segmentation algorithm only outputs the spatial distribution result of the cavity region. This result is not directly used to trigger alarms, but rather serves as one of the semantic pieces of evidence for determining missing steps.

[0100] Building upon this foundation, this method further proposes a sequential cumulative decision algorithm based on multi-feature weighted fusion. Anomaly scores are calculated for periodic structural anomaly evidence, geometric structural anomaly evidence, and semantic segmentation cavity evidence, and then fused using a linear weighting method. Subsequently, an exponentially decaying time-series cumulative method is used to update the fused score. When the cumulative score continuously exceeds a preset threshold, it is determined that the escalator has a missing step, and an alarm is output; when the cumulative score falls below a recovery threshold, the alarm is deactivated, thereby effectively suppressing false alarms caused by momentary occlusion, short-term noise, or environmental disturbances.

[0101] The specific steps of this method are as follows:

[0102] S1: Acquire escalator step image data. This embodiment uses escalator monitoring video data as an example for explanation. The input escalator monitoring video stream is I. t ∈R H×W×3 Frame rate F. I t Let R represent the video frame at time t, where R is the set of real numbers, H is the image height (number of pixel rows), W is the image width (number of pixel columns), and 3 is the number of color channels.

[0103] The image data is preprocessed, including converting the image to grayscale and applying Gaussian filtering to reduce noise in the grayscale image.

[0104] In this detection task, the color frame images first need to be processed to ensure data consistency and validity. To do this, the color information of the image needs to be converted to grayscale for subsequent processing and analysis. Secondly, to remove noise from the image, Gaussian filtering is performed. Typically, a standard kernel size and standard deviation are chosen to smooth the image and reduce the impact of noise.

[0105] Convert color frames to grayscale:

[0106]

[0107] G(x,y): Represents the grayscale value of a pixel (x,y) in a grayscale image. R(x,y): Represents the red channel value of the (x,y)th pixel in a color image. t (x,y): Represents the green channel value of the (x,y)th pixel in the color image. B(x,y): Represents the blue channel value of the (x,y)th pixel in the color image.

[0108] Apply Gaussian filtering to the grayscale image for noise reduction (kernel size 5×5, standard deviation σ=1.0):

[0109]

[0110] Next, the Region of Interest (ROI) within the step zone is extracted from the image. By defining a fixed ROI, important regions of interest can be extracted from each frame of the image. The ROI is defined as including the step surface, leading edge line, and side skirts / yellow lines, ensuring that the extracted data is relevant to the analysis target.

[0111] In this embodiment, a fixed ROI is used: after installation, a rectangular region Ω⊂[1,W]×[1,H] is manually selected once, and then directly cropped in each subsequent frame.

[0112]

[0113] The ROI covers the "stepped surface + leading edge area + side skirt / yellow line area". Output: ROI grayscale image R t ∈R h ×w .

[0114] S2: Construct a one-dimensional stripe signal based on the ROI, and use the short-time Fourier transform algorithm to perform local frequency domain analysis on the one-dimensional stripe signal. Calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the stepped periodic structure, and obtain evidence of periodic anomalies with missing steps.

[0115] In the ROI, one or more fixed sampling bands are selected, and multi-row averaging is used to enhance noise reduction. m rows near the center of the stepped band region are selected, and a one-dimensional stripe signal is constructed by averaging the pixels in the selected m rows in the vertical direction.

[0116]

[0117] Where y0 is the starting row coordinate of the sampling band.

[0118] To reduce brightness drift, perform one-dimensional detrending:

[0119]

[0120] Where L is the moving average half-window.

[0121] right Perform an STFT on the spatial axis. Let the window length N and step size Hs be fixed, and the window function be Hann:

[0122]

[0123] The Hann window is a window function, where w(n) is the weight value of the window function, n represents the nth sampling point in the window, and N represents the window length.

[0124] The center position of the kth window is x k =kH s STFT definition:

[0125]

[0126] Dominant frequency, phase, and energy extraction; amplitude spectrum and energy spectrum:

[0127]

[0128] In the pre-set frequency band [ω min ,ω max [Search for the main frequency within]

[0129]

[0130] Main frequency amplitude, main frequency energy, main frequency phase:

[0131]

[0132] Phase unfolding:

[0133]

[0134] Three local anomalies are calculated based on the main period frequency, phase information, and period energy distribution:

[0135] The main frequency stability is abnormal, which is due to the first-order difference of the main frequency:

[0136]

[0137] Phase continuity anomaly, characterized by a phase increment deviating from the median:

[0138]

[0139]

[0140] Energy collapse anomaly, normalized energy ratio:

[0141]

[0142] Represents the energy anomaly ratio of the k-th window. Represents: the spectral energy at time t and position x. The median energy of all windows in the current frame is given by ε, which is a constant to prevent the denominator from being 0.

[0143] When a step is missing, a "deep, dark cavity" is created, which is usually... .

[0144] Normalize the three terms to [0,1] (fixed min-max is determined by calibration data):

[0145]

[0146]

[0147]

[0148] in This is the normal energy ratio reference value.

[0149] The PCM anomaly intensity is calculated as a weighted sum of three terms:

[0150]

[0151] Where α p +β p +γ p =1, and its ratio can be flexibly adjusted according to the actual situation.

[0152] The periodic anomaly fraction is obtained by summing the PCM anomaly intensities. :

[0153]

[0154] Interpolate the PCM anomaly intensity back to the pixel x-coordinate to obtain the fracture zone mask. :

[0155]

[0156] Where τ p It is a fixed threshold.

[0157] S3: Extract the edge information of the tiered zone using an edge detection algorithm, fit the leading edge line of the tiered zone using a linear transformation, determine the discontinuity features of the tiered geometric structure, and obtain geometric anomaly evidence of missing tiers.

[0158] First, perform Canny edge detection on the ROI:

[0159] gradient:

[0160]

[0161] Used to calculate the rate of change of an image in the x and y directions.

[0162] R t : Input image, ∂ x R t The change of the image in the x-direction, ∂ y R t The change of the image in the y-direction, ∇R t : Gradient vector.

[0163] Amplitude:

[0164]

[0165] Used to calculate the magnitude of the gradient, ||R| t | indicates the edge intensity of the pixel, varying in the x-direction and y-direction.

[0166] Double threshold (fixed) T low ,T high :

[0167]

[0168] This formula applies to image R. t Use the Canny edge detection algorithm. t This represents the edge map obtained after Canny edge detection of the t-th frame image.

[0169] Then, Hough line detection is performed on the edge map E. t The parameters of the line obtained after performing the Hough transform are:

[0170]

[0171] The peak set L={(ρ) is obtained by accumulating in the parameter space. i ,θ i )}. Filter the direction of the "step leading edge": Based on the installation calibration, obtain the center angle θ0 of the leading edge, and retain only:

[0172]

[0173] Project each line onto the ROI coordinate system and calculate its relationship with the ROI centerline y=y. c The intersection point x i , and press x i Sort to obtain the sequence:

[0174]

[0175] Geometric spacing consistency and missing faults, the spacing between adjacent leading edges is:

[0176]

[0177] Under normal circumstances d i Approximately constant after perspective stabilization. Define the spacing anomaly ratio:

[0178]

[0179] r d (i) represents the distance anomaly ratio at the i-th location, d i represents the actual distance to the i-th point, median(d) represents the median of all distance values, and ε is a constant to prevent division by zero.

[0180] When a line is missing, it appears as "a line that should appear is missing", causing a certain d i Significantly increased. Define geometric fault strength:

[0181]

[0182] in This is the upper limit of experience.

[0183] Calculate the geometric anomaly score:

[0184]

[0185] Map the locations corresponding to abnormal spacing to pixel fault band masks, that is, mark the interval [x(i), x(i+1)] as faults:

[0186]

[0187] This is the adjustment constant.

[0188] S4: Segment the cavity or dark area in the step zone to obtain the spatial distribution of the cavity area as evidence of cavity anomaly in the missing step.

[0189] Input the ROI color cropping image (same as the RGB of the Ω cropping) into YOLOv8-seg and segment it into three fixed classes:

[0190] 1.C1: Stepped surface;

[0191] 2.C2: Side structure / yellow line;

[0192] 3.C3: Cavity / Dark Area.

[0193] Cavity mask obtained by segmentation Given area A and perimeter P, calculate the cavity compactness:

[0194]

[0195] Cavities are typically more "blocky" with a larger κ; shadow stripes have a smaller κ.

[0196] Calculate the semantic anomaly score:

[0197]

[0198]

[0199] in , All of these are adjustable constants.

[0200] S5: The output results of merging steps S2-S4 are used to generate alarm status output.

[0201] Spatial consistency extends the fracture zone mask and pixel tomographic zone mask obtained in steps S2 and S3 from 1D masks to 2D:

[0202]

[0203] Define the overlapping region of the three evidence spaces:

[0204]

[0205] Calculate the overlap area ratio:

[0206]

[0207] Calculate the spatial consistency gating coefficient:

[0208]

[0209] in For reference only;

[0210] Calculate the instantaneous fusion score:

[0211]

[0212] α, β, γ are weighting coefficients;

[0213] The final confidence level At is calculated using sequential cumulative calculation:

[0214]

[0215] Where λ is the attenuation coefficient;

[0216] Calculate and update alarm status y t :

[0217]

[0218] Θ on Θoff and Θoff are the trigger threshold and recovery threshold, respectively.

[0219] In this embodiment, the attenuation coefficient is fixed at λ=0.85, and A0 is initialized to 0. Hysteresis threshold: trigger threshold Θ on =0.65, recovery threshold Θoff=0.45.

[0220] In summary, the steps of this method can be summarized as follows:

[0221] Get video frame I t Cut the ROI to get R t And grayscale filtering.

[0222] Construction stripe signal Perform STFT to obtain ω(k), ϕ~(k), E(k), and generate PCM. t , , .

[0223] For R t The edge map is obtained by performing Canny calculation, the leading edge sequence is obtained by performing Hough calculation, and the gap anomaly is calculated. t ,generate , .

[0224] Input the ROI RGB image into YOLOv8-seg to obtain the cavity mask. And calculate .

[0225] Computational space overlap Combined with the gated η(t), we obtain S(t).

[0226] Sequential cumulative calculation At Output alarm y according to the hysteresis threshold t .

[0227] In this embodiment, the above method is verified based on a typical escalator scenario. The implementation process of the method can be found in the appendix. Figure 1 In this embodiment, images of actual escalators in operation are collected to construct an escalator missing step detection dataset, and the proposed method is experimentally verified based on this dataset. The dataset consists of 4000 images of actual escalator operation, covering complex conditions such as different lighting conditions, shooting angles, escalator materials, step texture differences, and passenger occlusion, demonstrating strong engineering representativeness. During the experiment, the dataset is divided into a training set, a validation set, and a test set in a 6:1:1 ratio, with 3000 images in the training set, 500 in the validation set, and 500 in the test set. In this experiment, the semantic segmentation module is trained using the YOLOv8-seg model, with 20 training iterations, an initial learning rate of 0.001, and the Adam optimizer with parameters β1=0.9 and β2=0.999. During the inference phase, the segmentation confidence threshold of the YOLOv8-seg model is set to 0.5 to output the segmentation results of the stepped surface, side structure, and cavity region. It should be noted that the missing step detection method described in this embodiment is not an end-to-end training structure. Instead, the periodic analysis module, geometric consistency detection module, and semantic segmentation module each output abnormal evidence, which is then uniformly judged in the multi-evidence fusion and time-accumulation decision module.

[0228] As attached Figure 1 As shown, the method described in this embodiment adopts a vertical process structure, specifically including: first, acquiring images of escalator operation and preprocessing the acquired images, including region of interest extraction, perspective correction, and noise suppression; then, performing periodic analysis on the step texture or brightness change sequence, using short-time Fourier transform to extract the energy and phase features of the step periodic structure to determine the continuity of the step structure; simultaneously, performing edge detection and straight line extraction on the leading edge line of the step, using the Canny operator and Hough line transform, and using the parallelism and spacing consistency of the leading edge line for geometric consistency constraint judgment; further, using a semantic segmentation model to perform pixel-level segmentation of the step surface, side edges, and potential missing regions to obtain spatial mask evidence; then, aligning the periodic analysis results, geometric consistency detection results, and semantic segmentation results spatially, and performing multi-feature fusion through a spatial gating mechanism; based on this, performing time accumulation processing on the fusion results, using exponential decay and hysteresis threshold strategies to suppress instantaneous noise interference, and finally outputting the missing step detection result.

[0229] As attached Figure 2As shown, the periodic signal of the escalator steps changing over time and its corresponding short-time Fourier transform spectrum are presented. When the escalator steps are continuous, their spectral energy distribution and phase remain stable. However, when missing steps occur, the stability of the spectral energy distribution and phase changes significantly, thus providing an effective periodic criterion for missing step detection. (See attached image) Figure 3 The diagram illustrates the extraction and geometric consistency detection process of the leading edge line of the ladder step. By constraining the parallelism and interval continuity of the leading edge line, structural faults caused by missing steps can be effectively identified. (See attached diagram) Figure 4 The diagram illustrates the segmentation results of a semantic segmentation model on key structural areas of an escalator, including the step surface, side edges, and missing or hollow areas. The segmentation mask serves as supplementary evidence in subsequent judgments, rather than as a single triggering condition, thereby reducing the risk of misjudgment by a single model. (See attached diagram.) Figure 5-6 As shown, the multi-feature fusion and time-cumulative decision-making process is illustrated. Periodic evidence, geometric consistency evidence, and semantic segmentation evidence are spatially aligned to form a fusion score, and a stable alarm result is output through a time-cumulative and hysteresis threshold mechanism. (Attached) Figure 5-6 The illustration further shows the relationship between the fusion score, cumulative value, and alarm status over time, demonstrating that the mechanism can effectively suppress false triggering caused by transient noise.

[0230] In terms of experimental evaluation, this embodiment selects accuracy, precision, recall, and F1 score as evaluation indicators to compare and analyze different detection strategies. The experimental results are attached. Figure 7-8 As shown. (From the appendix) Figure 7-8 As can be seen, using only periodic analysis, geometric consistency detection, or semantic segmentation, the accuracy is approximately 90%–92%, and the F1 score is approximately 65%–69%, respectively. After adopting a multi-feature fusion strategy, the detection accuracy is improved to approximately 93%, the recall rate reaches approximately 80%, and the F1 score is improved to approximately 72%. Furthermore, by introducing a time-cumulative decision-making mechanism, the fusion scheme effectively reduces the false positive rate while maintaining a high recall rate (approximately 83%). Its overall accuracy remains stable in the 93%–95% range, and the F1 score is improved to approximately 74%, indicating that this method has superior detection stability and overall performance under complex field conditions.

[0231] Based on the above experimental results, the advancement of this invention can be summarized as follows:

[0232] (1) The core technology of this invention focuses on the periodic structural characteristics of escalator steps and constructs an engineering-applicable method for detecting missing steps. Escalator steps have characteristics such as strong structural repeatability, fixed perspective relationships, and stable running direction. At the same time, there are complex interferences such as shadows, reflections, stains, obstructions, and wear on site. This invention does not simply adopt general target detection or image anomaly detection, but rather proposes a detection technology system based on the structural laws of escalator steps, with periodic consistency, geometric consistency, and semantic evidence collaborative verification as the core, thereby ensuring that the detection results can reflect the true physical missing state.

[0233] (2) This invention introduces a periodic phase consistency analysis mechanism based on short-time Fourier transform (STFT) to quantitatively model the periodic signal presented during the operation of the ladder, and to identify structural anomalies from dimensions such as frequency stability, phase continuity and periodic energy distribution. This technology enables the system to directly characterize the periodic structural pattern of the ladder, avoids misjudgment caused by relying solely on appearance changes, and significantly improves the sensitivity and reliability to "structural defects".

[0234] (3) This invention constructs a perspective geometric consistency constraint model based on Canny edge detection and Hough line transformation. By analyzing the parallelism, spacing continuity and fault characteristics of the ladder leading edge sequence, it realizes physical constraint verification of missing anomalies from the mechanical structure level. The key technology of this part can effectively suppress false alarms of grayscale anomalies caused by shadows, reflections, local dirt, etc., and improve the detection from "appearance recognition" to "structural consistency verification".

[0235] (4) This invention introduces a pixel-level semantic segmentation model based on YOLOv8-seg to finely segment the escalator surface region, side structure region, and cavity region. A spatial consistency gating mechanism is used to verify the overlap between semantic evidence and periodic anomalies and geometric anomalies. Simultaneously, an exponentially decaying sequential cumulative decision and a hysteresis threshold strategy are combined to time-stabilize the detection results. This technology significantly reduces the impact of single-model false detections on the final alarm, suppresses false alarms caused by short-term occlusion and transient noise, and achieves real-time, stable detection without modifying the escalator's mechanical structure.

[0236] Another specific embodiment of this application provides an escalator step missing detection system based on multi-feature fusion, including:

[0237] The information acquisition unit is used to acquire escalator step image data, preprocess the image data, and extract the ROI (Region of Interest) of the step area in the image.

[0238] The first information processing unit is used to construct a one-dimensional stripe signal based on the ROI, perform local frequency domain analysis on the one-dimensional stripe signal using the short-time Fourier transform algorithm, calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the ladder periodic structure, and obtain evidence of periodic anomalies with missing ladders.

[0239] The second information processing unit is used to extract the edge information of the ladder zone through the edge detection algorithm, fit the leading edge line of the ladder with a straight line transformation, judge the discontinuity features of the ladder geometric structure, and obtain geometric anomaly evidence of missing ladders.

[0240] The third information processing unit is used to segment the cavity or dark area in the step zone and obtain the spatial distribution result of the cavity area as evidence of cavity anomaly due to missing steps.

[0241] The fusion output unit is used to fuse the output results of the information processing unit and generate alarm status output.

[0242] This system uses the aforementioned multi-feature fusion-based escalator step missing detection method to accurately detect escalator steps.

[0243] Another specific embodiment of this application provides an electronic device, including a processor and a memory. The memory stores a computer program, which, when executed by the processor, implements the steps in the method described above. This electronic device uses the aforementioned multi-feature fusion-based escalator step missing detection method to accurately detect missing escalator steps.

[0244] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the inventive concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for detecting missing escalator steps based on multi-feature fusion, characterized in that, Includes the following steps: S1: Acquire escalator step image data, preprocess the image data, and extract the ROI (Region of Interest) of the step area in the image; S2: Construct a one-dimensional stripe signal based on the ROI, and use the short-time Fourier transform algorithm to perform local frequency domain analysis on the one-dimensional stripe signal. Calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the ladder periodic structure, and obtain evidence of periodic anomalies with missing ladders. The steps for determining anomalies in tiered periodic structures are as follows: Three local anomalies are calculated based on the main period frequency, phase information, and period energy distribution: The main frequency stability is abnormal, which is the first-order difference of the main frequency. The phase continuity anomaly is caused by the phase increment deviating from the median. The energy collapse anomaly is a normalized energy ratio; Normalize the three terms to [0,1]; The anomaly intensity of the periodic phase coherence map (PCM) is calculated as a weighted sum of three terms; The periodic anomaly fraction is obtained by summing the PCM anomaly intensities. The PCM anomaly intensity is interpolated back to the pixel x-coordinate to obtain the fracture zone mask. ; S3: Extract the edge information of the tiered zone using an edge detection algorithm, fit the leading edge line of the tiered zone using a straight line transformation, determine the discontinuity features of the tiered geometric structure, and obtain geometric anomaly evidence of missing tiered zones. S4: Segment the cavity or dark area in the step zone to obtain the spatial distribution of the cavity area as evidence of cavity anomaly in the missing step. S5: Integrate and output evidence of periodic anomalies, geometric anomalies, and cavity anomalies with missing cascades.

2. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, In step S1, the preprocessing of the image data includes converting the image into a grayscale image and performing Gaussian filtering on the grayscale image for noise reduction.

3. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, In step S1, the step zone area is defined as including the step surface, the leading edge line, and the side skirt / yellow line area.

4. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, In step S2, a one-dimensional fringe signal is constructed through the following steps: In the ROI, one or more fixed sampling bands are selected, and multi-row averaging is used to enhance noise reduction. m rows near the center of the stepped band region are selected, and a one-dimensional stripe signal is constructed by averaging the pixels of the selected m rows in the vertical direction.

5. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, In step S3, the steps for determining the discontinuity characteristics of the ladder-like geometric structure are as follows: The line parameters obtained after the Hough line transform are: Project each line onto the ROI coordinate system and calculate its relationship with the ROI centerline y=y. c The intersection point x i , and press x i Sort to obtain the sequence: The spacing between adjacent leading edges is: Define the spacing anomaly ratio: Define geometric fault strength: in This is the upper limit of experience. Calculate the geometric anomaly score: Map the locations corresponding to abnormal spacing to pixel fault band masks, that is, mark the interval [x(i), x(i+1)] as faults: This is the adjustment constant.

6. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, In step S4, the cavity mask is obtained by segmentation. Given area A and perimeter P, calculate the cavity compactness: Calculate the semantic anomaly score: in , All of these are adjustable constants.

7. The escalator step missing detection method based on multi-feature fusion according to claim 1, characterized in that, Step S5 includes the following steps: Extend the fracture zone mask and pixel tomographic zone mask obtained in steps S2 and S3 from 1D masks to 2D: Define the overlapping region of the three evidence spaces: Calculate the overlap area ratio: Calculate the spatial consistency gating coefficient: in For reference only; Calculate the instantaneous fusion score: α, β, γ are weighting coefficients; The final confidence level At is calculated using sequential cumulative calculation: Where λ is the attenuation coefficient; Calculate and update alarm status y t : Θ on Θoff and Θoff are the trigger threshold and recovery threshold, respectively.

8. A multi-feature fusion-based escalator step missing detection system, characterized in that, For implementing the method as described in any one of claims 1-7, comprising: The information acquisition unit is used to acquire escalator step image data, preprocess the image data, and extract the ROI (Region of Interest) of the step area in the image. The first information processing unit is used to construct a one-dimensional stripe signal based on the ROI, perform local frequency domain analysis on the one-dimensional stripe signal using the short-time Fourier transform algorithm, calculate the main period frequency, phase information and period energy distribution corresponding to each position, determine the abnormality of the ladder periodic structure, and obtain evidence of periodic anomalies with missing ladders. The second information processing unit is used to extract the edge information of the ladder zone through the edge detection algorithm, fit the leading edge line of the ladder with a straight line transformation, judge the discontinuity features of the ladder geometric structure, and obtain geometric anomaly evidence of missing ladders. The third information processing unit is used to segment the cavity or dark area in the step zone and obtain the spatial distribution result of the cavity area as evidence of cavity anomaly due to missing steps. The fusion output unit is used to fuse the output results of the information processing unit and generate alarm status output.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, implements the steps of the method as described in any one of claims 1-7.

Images