A method and terminal device for escalator stop recognition based on multimodal displacement detection

Abstract

This invention discloses an escalator stop detection method and terminal device based on multimodal displacement detection, belonging to the field of escalator stop detection technology. It can solve the problems of high false alarms and false negatives in escalator stop detection under scenarios such as crowd obstruction, complex lighting, and unclear step texture. The method includes: S1, acquiring a sequence of step images and a sequence of handrail images from the real-time video stream of the escalator; S2, calculating the displacement between every two adjacent frames of step images in the step image sequence, obtaining multiple sub-pixel displacements of the steps and the phase correlation peak of the current frame step, and determining the step confidence level; S3, calculating the displacement vector of each effective corner point in the handrail image sequence between the current frame handrail image and the previous frame handrail image, and determining the handrail confidence level; S4, determining a comprehensive confidence level based on the step confidence level and the handrail confidence level, and generating an escalator stop alarm signal based on the comprehensive confidence level. This invention is used for escalator stop detection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent escalator stop recognition technology, and discloses an escalator stop recognition method and terminal equipment based on multimodal displacement detection. Background Technology

[0002] Existing vision-based escalator stagnation detection systems face significant challenges in complex real-world scenarios, with severely limited detection accuracy and robustness, resulting in both high false alarm and high false negative rates. Specific problems manifest in the following aspects: (1) The problem of crowd obstruction leads to the loss of target features. During peak hours, dense crowds completely obscure the escalator steps. The key moving carriers (steps) and their texture details disappear from the monitoring screen, and the system cannot extract effective motion features. This renders methods that rely on step textures or edges for optical flow calculation and background modeling completely ineffective, easily leading to false negatives due to "not being able to see the target," making it impossible to determine whether the escalator is truly stagnant. (2) Complex lighting changes undermine the image consistency assumption. On-site lighting conditions are complex and variable, such as shadow changes caused by sunlight movement, lobby light switching, and glass curtain wall reflections. These sudden changes in lighting can cause drastic changes in pixel values ​​over a large area, which are highly similar to pixel changes caused by object movement. This directly undermines the core assumption of "constant grayscale / brightness" in optical flow methods, causing the system to incorrectly identify changes in light and shadow as abnormal movement or stagnation, resulting in a large number of false alarms. (3) Indistinct escalator texture leads to inaccurate motion estimation. Many escalator steps are solid-colored or severely worn, lacking distinct and stable texture features. Traditional vision methods (such as optical flow and feature point tracking) rely on sufficient gradient information in the image to establish constraint equations. In areas lacking texture, the system cannot calculate reliable motion vectors, resulting in highly noisy and inconsistent motion estimations. This makes it difficult to make robust distinctions between "slow movement" and "complete stagnation," while also increasing the risk of false alarms and false negatives.

[0003] Most existing methods rely on detecting "meaningful pixel-level changes" from consecutive frames of images. Under the aforementioned multi-interference conditions, "passenger occlusion" leads to a loss of useful information, "illumination interference" introduces a large number of spurious change signals, and "texture loss" weakens the system's ability to perceive real motion. The combined effect of these three factors makes the detection paradigm that simply analyzes pixel changes extremely fragile, and the system performance drops sharply in complex scenes. Summary of the Invention

[0004] The purpose of this invention is to provide an escalator stop detection method and terminal device based on multimodal displacement detection, so as to solve the problem that existing escalator stop detection is prone to high false alarms and false negatives in scenarios such as many people blocking the view, complex lighting, and unclear step texture.

[0005] The first aspect of this invention provides a method for identifying escalator stoppages based on multimodal displacement detection, the method comprising:

[0006] S1. Obtain the real-time video stream of the escalator, and extract the step area and handrail area from the continuous multiple frames of the real-time video stream respectively to obtain the step image sequence and handrail image sequence.

[0007] S2. Calculate the displacement between each two adjacent frames of the step image sequence to obtain multiple sub-pixel displacements of the steps and the phase correlation peak of the current frame of the steps, and determine the step confidence based on the multiple sub-pixel displacements of the steps and the phase correlation peak of the current frame of the steps.

[0008] S3. Extract all valid corner points of the handrail image sequence, calculate the displacement vector of each valid corner point between the current frame handrail image and the previous frame handrail image, and determine the handrail confidence level based on the number of valid corner points and the displacement vector.

[0009] S4. Determine the overall confidence level based on the confidence level of the steps and the confidence level of the handrail, and generate an escalator stop alarm signal based on the overall confidence level.

[0010] Optionally, prior to S4, the method further includes:

[0011] Narrowband vertical texture regions on the side of the steps are extracted from multiple consecutive frames of the real-time video stream to obtain a narrowband image sequence;

[0012] Calculate the displacement between each two adjacent narrowband images in the narrowband image sequence to obtain multiple narrowband subpixel displacements, and determine the vertical texture confidence based on the multiple narrowband subpixel displacements and the texture sharpness of the current frame narrowband image;

[0013] Accordingly, in step S4, determining the overall confidence level based on the confidence level of the step and the confidence level of the handrail is specifically as follows: determining the overall confidence level based on the confidence level of the step, the confidence level of the handrail, and the confidence level of the vertical texture.

[0014] Optionally, the calculation of the displacement between every two adjacent frame images of the steps in S2, resulting in multiple sub-pixel displacements of the steps, specifically includes:

[0015] Calculate the cross power spectrum between every two adjacent frames of step images in the step image sequence, perform an inverse Fourier transform on the cross power spectrum to obtain the step phase correlation peak, and use the integer pixel displacement corresponding to the peak position of the step phase correlation peak as the initial sub-pixel displacement between the corresponding two adjacent frames of step images.

[0016] The displacement correction amount is determined based on the peak neighborhood of the phase correlation peak of the step, and the initial subpixel displacement is corrected using the displacement correction amount to obtain the subpixel displacement of the step between two adjacent frames of step images.

[0017] Optionally, the displacement correction is determined based on the peak neighborhood of the step phase correlation peak, specifically including:

[0018] The peak neighborhood of the step phase correlation peak is fitted with a quadratic surface to obtain the fitting equation;

[0019] Solve for the extreme points of the fitted equation and use these extreme points as displacement corrections.

[0020] Optionally, the current frame step phase correlation peak is the step phase correlation peak between the current frame step image and the previous frame step image;

[0021] The step confidence level determination in S2 based on the multiple step sub-pixel displacements and the current frame step phase correlation peak specifically includes:

[0022] The standard deviation of the sub-pixel displacements of the steps is calculated to obtain the stability of the step displacements;

[0023] Calculate the peak intensity of the current frame step phase correlation peak and the mean value of the step displacement stability, and use the mean value as the step confidence level.

[0024] Optionally, calculating the displacement vector of each valid corner point between the current frame handrail image and the previous frame handrail image in step S3 specifically includes:

[0025] An optical flow constraint equation is constructed for the surrounding neighboring pixels of each effective corner point in the current frame handrail image, and the optical flow constraint equation is solved using the least squares method to obtain the displacement vector of the corresponding effective corner point.

[0026] Optionally, determining the handrail confidence level based on the number of effective corner points and the displacement vector in step S3 specifically includes:

[0027] The displacement consistency of the current frame is determined based on the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image.

[0028] Calculate the mean of the consistency between the number of effective corner points and the current frame displacement, and use the mean as the confidence level of the handrail.

[0029] Optionally, the vertical texture confidence is determined based on multiple narrowband subpixel displacements and the texture sharpness of the current frame's narrowband image, specifically including:

[0030] The standard deviation of multiple narrowband subpixel displacements is calculated to obtain the narrowband displacement stability;

[0031] Calculate the mean of the texture sharpness and the narrowband displacement stability of the current frame narrowband image, and use the mean as the vertical texture confidence.

[0032] Optionally, the step S4 of generating an escalator stop alarm signal based on the comprehensive confidence level specifically includes:

[0033] When the overall confidence level is greater than or equal to the first threshold, an escalator stop alarm signal is generated;

[0034] When the overall confidence level is greater than or equal to the third threshold and less than the first threshold, wait for a preset time and then re-execute S1~S4 to obtain the reviewed overall confidence level. When the reviewed overall confidence level is greater than or equal to the second threshold, generate an escalator stop alarm signal.

[0035] Wherein, the second threshold is less than the first threshold and greater than the third threshold.

[0036] Optionally, the displacement consistency of the current frame is determined based on the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image, specifically including:

[0037] Calculate the mean and standard deviation of the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image, and use the ratio of the standard deviation to the mean as the displacement consistency of the current frame.

[0038] Optionally, a comprehensive confidence level is determined based on the confidence levels of the steps, the handrail, and the vertical texture, specifically including:

[0039] The confidence scores of the steps, the handrails, and the vertical textures are weighted and fused to obtain a comprehensive confidence score.

[0040] A second aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the escalator stop recognition method based on multimodal displacement detection as described above.

[0041] The escalator stop recognition method based on multimodal displacement detection of the present invention has the following advantages compared with the prior art:

[0042] This invention innovatively constructs a multimodal fusion detection system consisting of "periodic texture phase detection of escalators + sparse optical flow tracking of handrails + vertical structural displacement compensation." Through cross-validation and dynamic fusion of three complementary detection methods, it effectively overcomes the limitations of single detection methods in complex scenarios. First, step edge phase tracking technology is used to perform targeted detection on the escalator step area. By extracting Regions of Interest (ROIs) and applying frequency-domain bandpass filtering to periodic textured areas such as step gaps, sub-pixel-level displacement estimation is achieved using phase correlation to accurately capture the movement state of the steps. Secondly, for the handrail's movement characteristics, corner points within the handrail's ROI are extracted, and the LK optical flow algorithm is used to track corner point displacements to determine the handrail's movement state, forming cross-validation. Subsequently, for cases where step textures are severely obscured, vertical structures (threads / comb teeth / mold release seams) on the side or middle of the steps are selected as detection targets, and phase correlation detection within narrow-band ROIs is used for supplementary validation. Finally, through a multi-dimensional confidence-weighted fusion mechanism combined with temporal stability analysis, dynamic feedback and precise correction are applied to the three detection results to achieve the final determination and graded early warning of escalator stagnation. Attached Figure Description

[0043] Figure 1 The flowchart illustrates the escalator stop recognition method based on multimodal displacement detection provided in this embodiment of the invention. Detailed Implementation

[0044] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0045] This invention provides a method for identifying escalator stoppages based on multimodal displacement detection, such as... Figure 1 As shown, the method includes:

[0046] S1. Obtain the real-time video stream of the escalator, and extract the step area and handrail area from the continuous multiple frames of the real-time video stream to obtain the corresponding step image sequence and handrail image sequence.

[0047] For periodic textures on the surface of escalator steps (such as gaps between steps), a high-definition camera fixed to the side of the escalator acquires real-time video streams. After preprocessing the video frames, the camera focuses on the step area for precise displacement detection, solving the problem of insufficient detection accuracy at the single pixel level.

[0048] (1) Extraction and preprocessing of the step area.

[0049] An adaptive threshold segmentation algorithm was used to extract the Region of Interest (ROI) of the escalator steps, eliminating irrelevant background interference such as the side panels and the ground. Contrast enhancement and Gaussian noise suppression were applied to the extracted step images to enhance the periodic texture features of the step gaps and improve the accuracy of subsequent frequency domain analysis.

[0050] (2) Frequency domain bandpass filter.

[0051] Since the periodic texture of the step gaps exhibits a specific frequency distribution in the frequency domain, and environmental interference (such as sudden changes in lighting and small debris) is mostly low-frequency or high-frequency noise, frequency domain bandpass filtering technology is introduced to separate effective texture features from interference signals.

[0052] Specifically: Let the preprocessed step image be... Its two-dimensional Fourier transform is The bandpass filter function is The filtered frequency domain image is The spatial domain image after inverse Fourier transform is ,but:

[0053] ;

[0054] ;

[0055] ;

[0056] in, This represents the two-dimensional Fourier transform operation. This represents the two-dimensional inverse Fourier transform operation.

[0057] Bandpass filter function Defined as:

[0058] ;

[0059] in, This is the low-frequency cutoff frequency. This is the high-frequency cutoff frequency, adaptively adjusted based on the periodic characteristics of the step texture (typically within a certain range). , , (Nyquist frequency of the image).

[0060] S2. Calculate the displacement between each two adjacent step images in the step image sequence to obtain multiple step sub-pixel displacements and the current frame step phase correlation peak, and determine the step confidence based on the multiple step sub-pixel displacements and the current frame step phase correlation peak.

[0061] The above calculation of the displacement between each two adjacent frames of the step image sequence yields multiple sub-pixel displacements of the steps, specifically including:

[0062] (1) Calculate the cross power spectrum between each two adjacent step images in the step image sequence, perform inverse Fourier transform on the cross power spectrum to obtain the step phase correlation peak, and take the integer pixel displacement corresponding to the peak position of the step phase correlation peak as the initial sub-pixel displacement between the corresponding two adjacent step images.

[0063] (2) Determine the displacement correction amount based on the peak neighborhood of the step phase correlation peak, and use the displacement correction amount to correct the initial subpixel displacement to obtain the step subpixel displacement between two adjacent frame step images.

[0064] Specifically, determining the displacement correction based on the peak neighborhood of the step phase correlation peak involves: first, fitting the peak neighborhood of the step phase correlation peak with a quadratic surface to obtain the fitting equation; then, solving the extreme points of the fitting equation and using the extreme points as the displacement correction.

[0065] For the filtered step image, the phase correlation method is used to calculate the displacement between two consecutive frames, achieving sub-pixel level motion estimation. The core is to accurately capture minute displacements by utilizing the linear characteristics of phase information.

[0066] Specifically: Let the processed step images of two consecutive frames be... (No. (frame) and (No. (frame), its cross-power spectrum Defined as:

[0067] ;

[0068] in, and They are respectively and Fourier transform, for .

[0069] Performing an inverse Fourier transform on the cross-power spectrum yields the step phase correlation peak. :

[0070] ;

[0071] The peak position of the step phase correlation peak corresponds to an integer pixel displacement. Let this be the initial subpixel displacement. To achieve subpixel-level accuracy, a quadratic surface is fitted to the 3×3 neighborhood around the peak. The displacement correction is obtained by solving the extreme points of this fitted equation. Final step subpixel displacement for:

[0072] ;

[0073] .

[0074] The current frame step phase correlation peak is the step phase correlation peak between the current frame step image and the previous frame step image; the above determination of step confidence based on multiple step sub-pixel displacements and the current frame step phase correlation peak specifically includes:

[0075] (1) Calculate the standard deviation of multiple step subpixel displacements to obtain the step displacement stability;

[0076] (2) Calculate the peak intensity of the current frame step phase correlation peak and the mean value of the step displacement stability, and use the mean value as the step confidence.

[0077] The step confidence level C1 is calculated based on the peak intensity of the step phase correlation peak in the current frame and the step displacement stability, and its value ranges from [0,1].

[0078] Furthermore, in practical applications, when the average vertical displacement of multiple sub-pixel displacements of steps calculated from consecutive multi-frame step images (or 4 sub-pixel displacements if there are 5 frames of step images) is less than a preset threshold, it is determined that the step is suspected of being stagnant.

[0079] S3. Extract all valid corner points from the handrail image sequence, calculate the displacement vector of each valid corner point between the current frame handrail image and the previous frame handrail image, and determine the handrail confidence level based on the number of valid corner points and the displacement vector.

[0080] The above calculation of the displacement vector of each valid corner point between the current frame of the handrail image and the previous frame of the handrail image specifically includes:

[0081] For each effective corner point in the current frame of the handrail image, construct the optical flow constraint equation around the neighboring pixels, and solve the optical flow constraint equation using the least squares method to obtain the displacement vector of the corresponding effective corner point.

[0082] The above determination of the handrail confidence level based on the number of effective corner points and displacement vectors specifically includes:

[0083] (1) Determine the displacement consistency of the current frame based on the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image;

[0084] Specifically, this includes: calculating the mean and standard deviation of the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image, and using the ratio of the standard deviation to the mean as the displacement consistency of the current frame.

[0085] (2) Calculate the mean of the number of effective corner points and the consistency of the current frame displacement, and use the mean as the confidence level of the handrail.

[0086] In this embodiment of the invention, for scenarios where the step texture may be severely obscured, the motion state of the handrail is introduced as a cross-validation criterion. The Handrail Texture Tracking technology is used to track the movement of the corner points (i.e. texture points) on the surface of the handrail, avoiding false alarms caused by relying solely on step detection.

[0087] A method combining color space segmentation and edge detection is used to extract the Region of Interest (ROI) of the armrest strap (black leather or rubber strap) area from video frames, eliminating interference from armrest supports, human figures, etc. The Shi-Tomasi corner detection algorithm is then used to extract corner points (i.e., texture points) in the armrest strap area, requiring corner response values... Furthermore, the spacing between adjacent corner points is ≥10 pixels to ensure the uniqueness and stability of the corner points.

[0088] The Lucas-Kanade (LK) optical flow algorithm is used to perform cross-frame tracking of the extracted corner points to obtain the effective corner points of the handrail image sequence; based on the assumption of image grayscale invariance, an optical flow constraint equation is constructed to solve for the displacement vector of the effective corner points. ( (Index for corner points).

[0089] Specifically: The grayscale of LK optical flow remains unchanged as assumed:

[0090] ;

[0091] Performing a Taylor expansion on the left side and neglecting higher-order terms, we obtain the optical flow constraint equation:

[0092] ;

[0093] in , For the first The partial derivative of the handrail image in the spatial domain describes the first frame. Frame of handrail images in At the location, along axis, The grayscale change rate in the axial space direction reflects the grayscale distribution gradient within a single frame; For the first Partial derivatives of the frame handrail image in the time domain.

[0094] For each effective corner point, establish optical flow constraint equations for pixels within a 3×3 neighborhood, and solve for the displacement vector using the least squares method. :

[0095] .

[0096] The handrail confidence score C2 is calculated based on the number of successful tracking corner points and the consistency of displacement in the current frame, and its value ranges from [0,1].

[0097] Furthermore, in practical applications, to avoid misjudgment caused by individual abnormal corner points (such as occlusion or noise interference), the median of the horizontal and vertical displacement vectors of all successfully tracked valid corner points in each frame of the handrail image is taken after sorting them separately to obtain the representative displacement of the corresponding frame of the handrail image.

[0098] The number of successfully tracked valid corner points generally needs to be greater than or equal to [the number of successful corner points]. Otherwise, the detection of the handrail image frame is deemed invalid; when the average vertical displacement of the representative displacement of multiple consecutive handrail images is less than a preset threshold, the handrail is suspected of being stationary.

[0099] S4. Determine the overall confidence level based on the confidence levels of the steps and the handrail, and generate an escalator stop alarm signal based on the overall confidence level.

[0100] Furthermore, prior to S4, the method further includes:

[0101] Narrowband vertical texture regions on the side of the steps are extracted from consecutive frames of images from a real-time video stream to obtain a narrowband image sequence.

[0102] Calculate the displacement between each two adjacent narrowband images in the narrowband image sequence to obtain multiple narrowband subpixel displacements, and determine the vertical texture confidence based on the multiple narrowband subpixel displacements and the texture sharpness of the current frame narrowband image;

[0103] Accordingly, in S4, the overall confidence level is determined based on the confidence levels of the steps and the handrail, specifically: the overall confidence level is determined based on the confidence levels of the steps, the handrail, and the vertical texture.

[0104] The above determination of vertical texture confidence based on multiple narrowband subpixel displacements and the texture sharpness of the current frame's narrowband image specifically includes:

[0105] (1) Calculate the standard deviation of multiple narrowband subpixel displacements to obtain the narrowband displacement stability;

[0106] (2) Calculate the mean of texture sharpness and narrowband displacement stability of the current frame narrowband image, and use the mean as the vertical texture confidence.

[0107] As a supplement to the first two detection methods, this method is specifically designed to detect the vertical structures (threads / comb teeth / mold release seams) on the side or middle of the steps in extreme scenarios where the step texture is completely obscured and the handrail detection fails. It utilizes the strong vertical periodicity of these structures to achieve displacement detection.

[0108] By performing morphological operations and edge detection, vertical texture regions on the sides of steps are identified. Narrow vertical texture regions (vertical strips) with a width of 8-12 pixels can be selected to ensure that the narrow vertical texture regions contain only vertical texture features and eliminate horizontal interference. The selection of narrow vertical texture regions follows the principle of "vertical alignment and continuous distribution," and at least two narrow vertical texture regions are retained for each step to ensure detection redundancy.

[0109] Apply the phase correlation method (same as S2) directly to each narrowband image to calculate the narrowband subpixel displacement between two consecutive narrowband images. Because the vertical stripes in narrowband images have strong vertical periodicity, their phase correlation peak signal intensity is high, resulting in strong anti-interference ability.

[0110] The vertical texture confidence C3 is calculated based on the texture sharpness and narrow-band displacement stability of the narrow-band image, and its value ranges from [0,1].

[0111] Furthermore, in practical applications, when the average vertical displacement of the narrowband subpixel displacement of multiple consecutive narrowband images is less than a preset threshold, it is determined that the vertical structure is suspected of stagnation.

[0112] The above-mentioned comprehensive confidence level is determined based on the confidence levels of the steps, handrails, and vertical texture, specifically including:

[0113] The confidence scores of the steps, handrails, and vertical textures are weighted and fused to obtain the overall confidence score.

[0114] The overall confidence level C is calculated using a weighted fusion formula:

[0115] C = W1×C1 + W2×C2 + W3×C3;

[0116] Among them, W1, W2 and W3 are dynamic weights (satisfying W1+W2+W3=1), which are adaptively adjusted according to the scene. Specifically, W3 is increased to 0.4 in scenes with many people and occlusion, W1 is increased to 0.5 in scenes with complex lighting, and W1=0.4, W2=0.4 and W3=0.2 in normal scenes.

[0117] The above-mentioned generation of escalator stop alarm signals based on comprehensive confidence level specifically includes:

[0118] When the overall confidence level is greater than or equal to the first threshold, an escalator stop alarm signal is generated.

[0119] When the overall confidence level is greater than or equal to the third threshold and less than the first threshold, wait for a preset time and then re-execute S1~S4 to obtain the reviewed overall confidence level. When the reviewed overall confidence level is greater than or equal to the second threshold, generate an escalator stop alarm signal.

[0120] The second threshold is less than the first threshold and greater than the third threshold.

[0121] In practical applications, the first threshold can be set to 85%; the second threshold can be set to 75%; and the third threshold can be set to 70%.

[0122] High confidence level (C≥85%): If the escalator is determined to be "stopped", a tiered alarm will be triggered immediately (audio-visual alarm + push notification from the backend system).

[0123] Medium confidence level (70% ≤ C < ): Extract images of multiple consecutive frames (e.g., 8 frames) from the future, re-execute the three detection methods, and perform a second verification by combining temporal stability analysis (e.g., displacement change trend). If the confidence level is still ≥75% after verification, trigger an alarm.

[0124] Low confidence level (C < If the sample is determined to be "no stagnation (interference scene)", the sample (including scene parameters and detection results) will be automatically stored in the interference sample library for subsequent fine-tuning of model parameters (such as bandpass filter frequency range and corner detection threshold) to form a closed-loop optimization.

[0125] This invention discloses an intelligent escalator stagnation detection method based on multimodal displacement detection fusion, aiming to solve the problems of high false alarms and false negatives in escalator stagnation detection under scenarios such as heavy crowds, complex lighting, and unclear step textures. The method constructs a three-stage closed-loop technology system: "step phase tracking detection + handrail optical flow detection + local vertical texture supplementary detection," achieving reliable identification of escalator stagnation through "sub-pixel-level precise positioning + multi-dimensional cross-validation + dynamic fusion decision-making." First, stepping edge phase tracking technology is used to perform frequency domain analysis and sub-pixel displacement estimation on the periodic texture of the step surface; second, sparse optical flow technology is used to track the movement state of the handrail texture points; subsequently, local vertical texture technology is used to detect the vertical structural displacement of the steps as supplementary verification; finally, a multi-dimensional confidence weighted fusion mechanism is used to output the final judgment result, significantly improving the robustness and accuracy of escalator stagnation detection in complex scenarios.

[0126] A second aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the escalator stop recognition method based on multimodal displacement detection as described above.

[0127] The above descriptions are merely a few embodiments of the present invention and are not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any modifications or alterations made by those skilled in the art without departing from the scope of the technical solution of the present invention using the disclosed technical content are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A method for identifying escalator stoppages based on multimodal displacement detection, characterized in that, The method includes: S1. Obtain the real-time video stream of the escalator, and extract the step area and handrail area from the continuous multiple frames of the real-time video stream respectively to obtain the step image sequence and handrail image sequence. S2. Calculate the displacement between each two adjacent frames of the step image sequence to obtain multiple sub-pixel displacements of the steps and the phase correlation peak of the current frame of the steps, and determine the step confidence based on the multiple sub-pixel displacements of the steps and the phase correlation peak of the current frame of the steps. S3. Extract all valid corner points of the handrail image sequence, calculate the displacement vector of each valid corner point between the current frame handrail image and the previous frame handrail image, and determine the handrail confidence level based on the number of valid corner points and the displacement vector. S4. Determine the overall confidence level based on the confidence level of the steps and the confidence level of the handrail, and generate an escalator stop alarm signal based on the overall confidence level.

2. The method according to claim 1, characterized in that, Prior to S4, the method further includes: Narrowband vertical texture regions on the side of the steps are extracted from multiple consecutive frames of the real-time video stream to obtain a narrowband image sequence; Calculate the displacement between each two adjacent narrowband images in the narrowband image sequence to obtain multiple narrowband subpixel displacements, and determine the vertical texture confidence based on the multiple narrowband subpixel displacements and the texture sharpness of the current frame narrowband image; Accordingly, in step S4, determining the overall confidence level based on the confidence level of the step and the confidence level of the handrail is specifically as follows: determining the overall confidence level based on the confidence level of the step, the confidence level of the handrail, and the confidence level of the vertical texture.

3. The method according to claim 1, characterized in that, S2 calculates the displacement between every two adjacent frame images of the step image sequence to obtain multiple sub-pixel displacements of the steps, specifically including: Calculate the cross power spectrum between every two adjacent frames of step images in the step image sequence, perform an inverse Fourier transform on the cross power spectrum to obtain the step phase correlation peak, and use the integer pixel displacement corresponding to the peak position of the step phase correlation peak as the initial sub-pixel displacement between the corresponding two adjacent frames of step images. The displacement correction amount is determined based on the peak neighborhood of the phase correlation peak of the step, and the initial subpixel displacement is corrected using the displacement correction amount to obtain the subpixel displacement of the step between two adjacent frames of step images.

4. The method according to claim 3, characterized in that, The displacement correction is determined based on the peak neighborhood of the step phase correlation peak, specifically including: The peak neighborhood of the step phase correlation peak is fitted with a quadratic surface to obtain the fitting equation; Solve for the extreme points of the fitted equation and use these extreme points as displacement corrections.

5. The method according to claim 3, characterized in that, The current frame step phase correlation peak is the step phase correlation peak between the current frame step image and the previous frame step image; The step confidence level determination in S2 based on the multiple step sub-pixel displacements and the current frame step phase correlation peak specifically includes: The standard deviation of the sub-pixel displacements of the steps is calculated to obtain the stability of the step displacements; Calculate the peak intensity of the current frame step phase correlation peak and the mean value of the step displacement stability, and use the mean value as the step confidence level.

6. The method according to claim 1, characterized in that, The calculation of the displacement vector of each effective corner point between the current frame handrail image and the previous frame handrail image in S3 specifically includes: An optical flow constraint equation is constructed for the surrounding neighboring pixels of each effective corner point in the current frame handrail image, and the optical flow constraint equation is solved using the least squares method to obtain the displacement vector of the corresponding effective corner point.

7. The method according to claim 1, characterized in that, The step S3, determining the handrail confidence level based on the number of effective corner points and the displacement vector, specifically includes: The displacement consistency of the current frame is determined based on the displacement vectors of all valid corner points between the current frame handrail image and the previous frame handrail image. Calculate the mean of the consistency between the number of effective corner points and the current frame displacement, and use the mean as the confidence level of the handrail.

8. The method according to claim 2, characterized in that, The vertical texture confidence is determined based on multiple narrowband subpixel displacements and the texture sharpness of the current frame's narrowband image, specifically including: The standard deviation of multiple narrowband subpixel displacements is calculated to obtain the narrowband displacement stability; Calculate the mean of the texture sharpness and the narrowband displacement stability of the current frame narrowband image, and use the mean as the vertical texture confidence.

9. The method according to claim 2, characterized in that, The step S4, generating an escalator stop alarm signal based on the comprehensive confidence level, specifically includes: When the overall confidence level is greater than or equal to the first threshold, an escalator stop alarm signal is generated; When the overall confidence level is greater than or equal to the third threshold and less than the first threshold, wait for a preset time and then re-execute S1~S4 to obtain the reviewed overall confidence level. When the reviewed overall confidence level is greater than or equal to the second threshold, generate an escalator stop alarm signal. Wherein, the second threshold is less than the first threshold and greater than the third threshold.

10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the escalator stop recognition method based on multimodal displacement detection as described in any one of claims 1 to 9.

Images