Escalator Abnormality Detection Method

Abstract

The present invention provides a method for detecting escalator anomalies, the method comprising: obtaining at least two target images including a region of interest, the region of interest including an entrance region or an exit region in the escalator; respectively performing feature extraction on the depth images corresponding to each frame of the target images to obtain a depth feature sequence; determining the height feature of the target steps in each frame of the target images to obtain a step height feature sequence, the target step being the first step in the region of interest with a step height greater than zero; and performing anomaly detection on the escalator based on the depth feature sequence and the step height feature sequence. The present invention can improve the accuracy of escalator anomaly detection.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method for detecting escalator anomalies. Background Art

[0002] In public places, the use of escalators has become increasingly common. However, sometimes anomalies such as sudden stops and reverse rotations of escalators may occur due to human or machine failures, etc., which can easily cause serious consequences.

[0003] In the prior art, when detecting whether an escalator has an anomaly, it is necessary to collect a video stream containing the escalator, and determine the motion vector and the escalator area of the escalator through the video stream, so as to estimate the real-time running direction of the escalator, realize the state detection of the escalator, and thus judge whether the escalator has an anomaly.

[0004] However, in the above method, the calculation of the motion vector is usually realized by calculating the optical flow formed by sampling points on different frames, and the optical flow method is easily affected by light and inaccurate. Therefore, the accuracy of escalator anomaly detection is relatively low. Summary of the Invention

[0005] The present invention provides a method for detecting escalator anomalies, which is used to solve the defect of relatively low accuracy in escalator anomaly detection in the prior art, and realizes the improvement of the accuracy of escalator anomaly detection.

[0006] The present invention provides a method for detecting escalator anomalies, including:

[0007] Obtain at least two target images including the region of interest, where the region of interest includes the entrance area or the exit area of the escalator;

[0008] Extract features from the depth-of-field images corresponding to each frame of the target images respectively to obtain a depth-of-field feature sequence;

[0009] Determine the height feature of the target steps in each frame of the target images to obtain a step height feature sequence, where the target step is the first step with a step height greater than zero in the region of interest;

[0010] Based on the depth-of-field feature sequence and the step height feature sequence, perform anomaly detection on the escalator.

[0011] According to the method for detecting escalator anomalies provided by the present invention, the step of extracting features from the depth-of-field images corresponding to each frame of the target images respectively to obtain a depth-of-field feature sequence includes:

[0012] For each depth-of-field image, determine the gray value of each pixel point in the depth-of-field image;

[0013] Determine at least one grayscale value feature of the depth-of-field image based on the grayscale values of all pixel points in the depth-of-field image;

[0014] Perform normalization processing on each of the grayscale value features to obtain each normalized grayscale value feature;

[0015] Determine the depth-of-field feature sequence based on the normalized grayscale value features corresponding to each of the depth-of-field images.

[0016] According to an escalator anomaly detection method provided by the present invention, the at least one grayscale value feature includes a maximum grayscale value feature, a minimum grayscale value feature, an average grayscale value feature, and a median grayscale value feature;

[0017] The determining the depth-of-field feature sequence based on the normalized grayscale value features corresponding to each of the depth-of-field images includes:

[0018] Determine the depth-of-field feature value of the depth-of-field image based on the normalized maximum grayscale value feature, the normalized minimum grayscale value feature, the normalized average grayscale value feature, and the normalized median grayscale value feature corresponding to the depth-of-field image;

[0019] Combine the depth-of-field feature values of each of the depth-of-field images to obtain the depth-of-field feature sequence.

[0020] According to an escalator anomaly detection method provided by the present invention, the determining the depth-of-field feature value of the depth-of-field image based on the normalized maximum grayscale value feature, the normalized minimum grayscale value feature, the normalized average grayscale value feature, and the normalized median grayscale value feature corresponding to the depth-of-field image includes:

[0021] Determine the depth-of-field feature value of the depth-of-field image based on the following formula (1):

[0022] (1)

[0023] where, represents the depth-of-field feature value, DA represents the normalized average grayscale value feature, DMid represents the normalized median grayscale value feature, Dmax represents the normalized maximum grayscale value feature, Dmin represents the normalized minimum grayscale value feature, represents the correction coefficient, 。

[0024] According to an escalator anomaly detection method provided by the present invention, the determining the height feature of the target step in each frame of the target image to obtain the step height feature sequence includes:

[0025] For each frame of the target image, determine a first line segment and a second line segment at the height plane edge of the target step in the target image, wherein the extension directions of the first line segment and the second line segment are both perpendicular to the escalator running direction;

[0026] Normalize the distance between the first line segment and the second line segment to obtain a normalized distance;

[0027] Determine the normalized distance as the height feature of the target step in the target image;

[0028] Combine the height features of the target steps in each of the target images to obtain the step height feature sequence.

[0029] According to an escalator anomaly detection method provided by the present invention, the escalator anomaly detection based on the depth of field feature sequence and the step height feature sequence includes:

[0030] Input the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier to obtain the current escalator state output by the time feature sequence classifier;

[0031] Based on the current escalator state and a preset state, perform anomaly detection on the escalator, where the preset state is the state expected for the escalator to be in.

[0032] According to an escalator anomaly detection method provided by the present invention, the escalator anomaly detection based on the current escalator state and a preset state includes:

[0033] When the current escalator state is the running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determine that the escalator is abnormal;

[0034] When the current escalator state is the stop state and the preset state indicates that the escalator is in the running state, determine that the escalator is abnormal;

[0035] When the current escalator state is the running state and the running acceleration is different from the acceleration during normal operation of the escalator corresponding to the preset state, determine that the escalator is abnormal.

[0036] According to an escalator anomaly detection method provided by the present invention, the obtaining of at least two frames of target images including the region of interest includes:

[0037] Obtain at least two initial images including the escalator;

[0038] For each frame of the initial image, based on the position of the skirt board of the escalator in the initial image, determine the region of interest in the initial image;

[0039] Extract the region of interest in the initial image to obtain a target image corresponding to the initial image.

[0040] According to an escalator anomaly detection method provided by the present invention, determining the region of interest in the initial image based on the position of the skirt board of the escalator in the initial image includes:

[0041] Determine a first step position at the bottom end position of the skirt board in the length direction and a second step position at the top end position in the initial image;

[0042] Determine the region between the first step position and the second step position in the initial image and including the entire step width as the region of interest.

[0043] According to an escalator anomaly detection method provided by the present invention, the method further includes:

[0044] Input the target image into a step detection model to obtain the target steps output by the step detection model; the step detection model is trained by using a plurality of image samples including escalator samples and labels for characterizing step samples in each of the image samples for an initial step detection model.

[0045] The present invention also provides an escalator anomaly detection device, including:

[0046] An acquisition module for acquiring at least two target images including a region of interest, where the region of interest includes an entrance region or an exit region in the escalator;

[0047] An extraction module for respectively performing feature extraction on the depth-of-field images corresponding to each frame of the target images to obtain a depth-of-field feature sequence;

[0048] A determination module for determining the height feature of the target steps in each frame of the target images to obtain a step height feature sequence, where the target steps are the first steps with a step height greater than zero in the region of interest;

[0049] A detection module for performing anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence.

[0050] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the escalator anomaly detection method as described in any one of the above is implemented.

[0051] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the abnormal detection method for escalators as described in any one of the above is implemented.

[0052] The present invention also provides a computer program product, including a computer program, and when the computer program is executed by a processor, the abnormal detection method for escalators as described in any one of the above is implemented.

[0053] In the abnormal detection method for escalators provided by the embodiments of the present invention, by obtaining at least two target images including a region of interest, where the region of interest includes an entrance region or an exit region in the escalator, feature extraction is respectively performed on the depth images corresponding to each frame of the target images to obtain a depth feature sequence, and the height feature of the target steps in each frame of the target images is determined to obtain a step height feature sequence. The target step is the first step with a height greater than zero in the region of interest. Based on the depth feature sequence and the step height feature sequence, abnormal detection of the escalator is performed. Since the running state and running direction of the escalator can be determined based on the depth image corresponding to the target image and the height change of the target steps, abnormal detection of the escalator can be performed, avoiding the phenomenon in the prior art that the running vector of the escalator needs to be determined by the optical flow method, and improving the accuracy of abnormal state detection of the escalator. Description of the Drawings

[0054] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0055] Figure 1 It is a schematic diagram of the operation of the escalator at the t-th moment provided by the embodiments of the present invention.

[0056] Figure 2 It is a schematic diagram of the operation of the escalator at the (t + 1)-th moment provided by the embodiments of the present invention.

[0057] Figure 3 It is a schematic flowchart of the abnormal detection method for escalators provided by the embodiments of the present invention.

[0058] Figure 4 It is the target image corresponding to the t-th moment provided by the embodiments of the present invention.

[0059] Figure 5 It is the target image corresponding to the (t + 1)-th moment provided by the embodiments of the present invention.

[0060] Figure 6 It is the depth image corresponding to the target image at the t-th moment provided by the embodiments of the present invention.

[0061] Figure 7 It is the depth-of-field image corresponding to the target image at the (t + 1)-th moment provided by the embodiment of the present invention.

[0062] Figure 8 It is the schematic diagram of step detection provided by the embodiment of the present invention.

[0063] Figure 9 It is the schematic diagram of the change of the depth-of-field feature sequence when the escalator is running upward provided by the embodiment of the present invention.

[0064] Figure 10 It is the schematic diagram of the change of the depth-of-field feature sequence when the escalator is running downward provided by the embodiment of the present invention.

[0065] Figure 11 It is the schematic diagram of the change of the depth-of-field feature sequence when the escalator stops provided by the embodiment of the present invention.

[0066] Figure 12 It is the schematic diagram of the change of the depth-of-field feature sequence when the escalator has mechanical abnormalities provided by the embodiment of the present invention.

[0067] Figure 13 It is the schematic diagram of the change of the step height feature sequence when the escalator is running upward provided by the embodiment of the present invention.

[0068] Figure 14 It is the schematic diagram of the change of the step height feature sequence when the escalator is running downward provided by the embodiment of the present invention.

[0069] Figure 15 It is the schematic diagram of the change of the step height feature sequence when the escalator stops provided by the embodiment of the present invention.

[0070] Figure 16 It is the schematic diagram of the change of the step height feature sequence when the escalator has mechanical abnormalities provided by the embodiment of the present invention.

[0071] Figure 17 It is the schematic diagram of the escalator abnormal state detection provided by the embodiment of the present invention.

[0072] Figure 18 It is the schematic diagram of the escalator abnormal detection provided by the embodiment of the present invention.

[0073] Figure 19 It is the structural schematic diagram of the escalator abnormal detection device provided by the embodiment of the present invention.

[0074] Figure 20 It is the physical structure schematic diagram of an electronic device provided by the embodiment of the present invention. Specific embodiments

[0075] To make the objectives, technical solutions and advantages of the present invention more clear, the following will, in conjunction with the accompanying drawings in the present invention, clearly and completely describe the technical solutions in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the scope of protection of the present invention.

[0076] When an escalator experiences anomalies such as sudden stops or reverse rotations, serious consequences will occur. If the anomalies of the escalator can be automatically detected and reported to the monitoring center, passengers can be protected from harm. Therefore, how to accurately detect the anomalies of the escalator has become an important part of escalator status monitoring. Currently, by collecting the video stream containing the escalator and using the optical flow method to determine the motion vector and the escalator area based on this video stream, the real-time running direction of the escalator is estimated, and the status of the escalator is detected by judging the estimated running direction of the escalator, so as to determine whether the escalator has anomalies. Since the optical flow method is easily affected by light and inaccurate, the accuracy of escalator anomaly detection is relatively low.

[0077] In view of the above problems, an embodiment of the present invention provides an escalator anomaly detection method. Among them, the escalator includes multiple steps, and each step is composed of horizontal and vertical plates. Passengers step on the horizontal plates, and the height of the vertical plates at the bottom of the escalator changes periodically during the operation of the escalator. Figure 1 It is a schematic diagram of the operation of the escalator at the t-th moment provided by an embodiment of the present invention. Figure 2 It is a schematic diagram of the operation of the escalator at the (t + 1)-th moment provided by an embodiment of the present invention. As Figure 1 and Figure 2 shown, when the escalator is going up, the steps at the bottom will gradually appear from the flat surface, and the height will gradually increase from low to high. When the escalator is going down, the height of the steps at the bottom will gradually decrease from high to low and finally become a flat surface. When the escalator is stationary, the height of the steps does not change. If the escalator experiences mechanical shaking anomalies, such as sudden up and down vibrations, the height of the steps will oscillate. At the same time, during the operation of the escalator, the depth-of-field image of the steps will also change periodically. For example, when the escalator is going up, the bottom steps are in the process from near to far, so the overall depth of field of the steps will change from bright to dark. Similarly, when the escalator is going down, the bottom steps are in the process from far to near, so the overall depth of field of the steps will change from dark to bright. When the escalator is stationary, the depth-of-field image of the steps does not change significantly. If the escalator experiences mechanical shaking anomalies, such as sudden up and down vibrations, the overall depth of field of the steps will oscillate.

[0078] Based on the above principle, in the escalator anomaly detection method provided by an embodiment of the present invention, by using the variation law of the height of a target step on the lower side of the escalator during the operation of the escalator and combining it with the variation law of the depth of field caused by the height change of the target step, the current operating state and the running direction of the escalator are determined. Then, the operating state and the running direction are compared with the preset operating state and the preset running direction to determine whether the escalator is abnormal. In the above manner, based only on the depth-of-field image corresponding to the target image and the height feature of the target step, the operating state and the running direction of the escalator can be determined, so as to detect anomalies in the escalator, avoiding the phenomenon in the prior art that the running vector of the escalator needs to be determined by the optical flow method and improving the accuracy of escalator anomaly state detection.

[0079] The following Figures 3 to 18 describes the escalator anomaly detection method provided by an embodiment of the present invention. The embodiment of the present invention is applicable to detecting the anomaly state of an escalator in any scenario where an escalator is installed. Among them, the escalator anomalies in the embodiment of the present invention include escalator reverse rotation, escalator emergency stop, or mechanical shaking of the escalator. The execution subject of this method can be an electronic device such as an intelligent camera, an escalator control device, a computer, a server, a server cluster, or a specially designed escalator anomaly detection device, or it can be an escalator anomaly detection device set in the electronic device, and the escalator anomaly detection device can be implemented by software, hardware, or a combination of both.

[0080] Figure 3 is a schematic flowchart of the escalator anomaly detection method provided by an embodiment of the present invention. As Figure 3 shown, the method includes:

[0081] Step 301: Obtain at least two target images including the region of interest, where the region of interest includes the entrance region or the exit region in the escalator.

[0082] In this step, at least one initial image including the escalator region can be collected by an image acquisition device. For each frame of the initial image, a target image including the region of interest can be intercepted from the initial image. It should be understood that since only the step heights at the entrance and exit of the escalator usually change when the escalator is in operation, while the step heights in the middle part of the escalator remain unchanged, in order to reduce the computational complexity of image processing, only the target images including the region of interest need to be detected.

[0083] It should be noted that when the escalator is in the upward state, the area at the bottom of the escalator can be understood as the entrance area of the escalator, and the area at the top of the escalator can be understood as the exit area of the escalator. When the escalator is in the downward state, the area at the bottom of the escalator can be understood as the exit area of the escalator, and the area at the top of the escalator can be understood as the entrance area of the escalator. In the embodiments of the present invention, it is assumed that the image acquisition device can collect the area below the escalator, and the region of interest includes the area below the escalator or the area at the bottom of the escalator for description.

[0084] Among them, at least two consecutive target images need to ensure that the height of at least one step changes from 0 to the maximum height value.

[0085] Step 302: Perform feature extraction on the depth-of-field images corresponding to each frame of target images to obtain a depth-of-field feature sequence.

[0086] In this step, for each frame of target image, a depth-of-field estimation algorithm can be used to process the target image to obtain the depth-of-field image corresponding to the target image, where the depth-of-field image is a grayscale image. Exemplarily, the depth-of-field estimation algorithm can adopt Depth Anything, which is a monocular depth estimation algorithm that can combine labeled and unlabeled images for depth-of-field estimation. First, use the labeled set to learn the teacher model T, and then use the teacher model T to assign pseudo-depth labels to the unlabeled images. Finally, train the student model S on the combination of the labeled set and the pseudo-labeled set. To make the depth estimation model have rich semantic priors, an auxiliary constraint can be enforced between the online student model and the frozen encoder to maintain semantic capabilities.

[0087] After determining the depth-of-field image corresponding to each frame of target image, feature extraction will be performed on each frame of depth-of-field image to obtain the depth-of-field feature sequence of the steps in the region of interest. Among them, the depth-of-field feature sequence includes depth-of-field feature values determined based on each frame of depth-of-field image, and this depth-of-field feature sequence can characterize the height change of the same step in at least two consecutive frames of target images.

[0088] Step 303: Determine the height feature of the target step in each frame of target image to obtain a step height feature sequence, where the target step is the first step with a height greater than zero in the region of interest.

[0089] In this step, a step detection algorithm can be used to detect the single-step area in each frame of target image, and determine the target step from all the detected single-step areas, where the target step is the first step with a height greater than zero in the region of interest. For example, when the escalator is in the upward running state, the region of interest can be the area below the escalator, and the target step is the first step with a height greater than zero below the escalator.

[0090] Assume that when the escalator is in operation, the height of the target step changes within a preset time duration. Therefore, based on the height feature of the target step in each frame of the target image, a step height feature sequence can be determined. Among them, this step height feature sequence can be used to characterize the height change of the target step in at least two consecutive frames of target images.

[0091] Step 304: Based on the depth-of-field feature sequence and the step height feature sequence, perform abnormal detection on the escalator.

[0092] In this step, after determining the depth-of-field feature sequence and the step height feature sequence, by analyzing the depth-of-field feature sequence, the depth-of-field change of the step can be determined, and by analyzing the step height feature sequence, the height change of the target step can be determined. Compare the obtained depth-of-field change and height change with the depth-of-field change of the step and the height change of the target step during normal operation of the escalator, so as to determine whether the escalator has an abnormality, for example, whether it has reversed, stopped suddenly, or mechanically shaken, etc.

[0093] The escalator abnormal detection method provided by the embodiments of the present invention obtains at least two frames of target images including the region of interest, where the region of interest includes the entrance region or the exit region of the escalator, performs feature extraction on the depth-of-field images corresponding to each frame of the target images to obtain a depth-of-field feature sequence, determines the height feature of the target step in each frame of the target image to obtain a step height feature sequence, the target step is the first step with a height greater than zero in the region of interest, and based on the depth-of-field feature sequence and the step height feature sequence, perform abnormal detection on the escalator. Since the operating state and operating direction of the escalator can be determined based on the depth-of-field image corresponding to the target image and the height change of the target step, the escalator can be abnormally detected, avoiding the phenomenon of determining the operating vector of the escalator by the optical flow method in the prior art, and improving the accuracy of detecting the abnormal state of the escalator.

[0094] Exemplarily, on the basis of the above embodiments, when obtaining at least two frames of target images including the region of interest, at least two frames of initial images including the escalator can be obtained. For each frame of the initial image, based on the position of the skirt board of the escalator in the initial image, the region of interest in the initial image is determined, and the region of interest in the initial image is extracted to obtain the target image corresponding to the initial image.

[0095] Specifically, as Figure 1 and Figure 2As shown, multiple initial images including the escalator can be obtained by an image acquisition device. The initial images can include the entire escalator or most areas of the escalator. In practical applications, whether the escalator is going up or down, usually only the height of the steps in the entrance area or the exit area of the escalator changes, while the height of the steps in other areas remains unchanged. In the escalator anomaly detection solution provided in the embodiments of the present invention, it is based on the height change of the steps. Therefore, considering the long escalator area, in order to save the computational amount during subsequent image processing, image extraction can be performed through a region of interest (ROI) sub-image extraction module, and an image of the region of interest containing the image feature range to be utilized is intercepted from the initial image, that is, an image containing the entrance or exit area of the escalator. Specifically, when performing image interception, the position of the skirt board of the escalator can be used as a reference marker to determine the region of interest in each frame of the initial image, and the determined region of interest is extracted from the initial image to obtain the target image corresponding to each frame of the initial image.

[0096] In the above manner, since the target image containing the region of interest is intercepted from each frame of the initial image, and the region of interest contained in the target image is only a partial area of the escalator, the subsequent computational amount can be greatly reduced, and the efficiency of image quality detection is improved.

[0097] Exemplarily, when determining the region of interest in the initial image based on the position of the skirt board of the escalator in the initial image, the first step position at the bottom end position of the skirt board in the length direction and the second step position at the top end position in the initial image can be determined, and the area between the first step position and the second step position in the initial image and including the entire step width is determined as the region of interest.

[0098] Specifically, Figure 4 is the target image corresponding to the t-th moment provided by the embodiments of the present invention, Figure 5 is the target image corresponding to the (t + 1)-th moment provided by the embodiments of the present invention. As Figure 4 and Figure 5 shown, taking the escalator going up and the region of interest being the area at the bottom entrance of the escalator as an example, after performing image recognition on each frame of the initial image to determine the position of the skirt board in the initial image, the region of interest will be determined based on the position of the skirt board, and the target image will be intercepted. Among them, the step area range between the first step position at the bottom end position of the skirt board in the length direction and the second step position at the top end position of the skirt board needs to be covered in the height direction of the target image, and the step width of each step is covered in the width direction of the target image.

[0099] In this embodiment, taking the position of the apron panel as a reference, the region of interest can be determined quickly and accurately. Moreover, the region of interest determined in the above manner includes the range of image features that can be utilized in subsequent escalator anomaly detection, making the region of interest more precise.

[0100] Exemplarily, based on the above embodiment, when performing feature extraction on the depth images corresponding to each frame of the target image to obtain a depth feature sequence, the following method can be adopted:

[0101] For each depth image, determine the gray value of each pixel point in the depth image. Based on the gray values of all pixel points in the depth image, determine at least one gray value feature of the depth image. Perform normalization processing on each gray value feature to obtain each normalized gray value feature. Based on the normalized gray value features corresponding to each depth image, determine the depth feature sequence.

[0102] Specifically, Figure 6 is the depth image corresponding to the target image at the t-th moment provided by the embodiment of the present invention, Figure 7 is the depth image corresponding to the target image at the (t + 1)-th moment provided by the embodiment of the present invention. As Figure 6 and Figure 7 shown, the depth image is a grayscale image. For each depth image, the gray value of each pixel point in the depth image represents the distance between the actual object represented by the pixel point and the image acquisition device. The larger the gray value, the farther the actual object represented by the pixel point is from the image acquisition device. On the contrary, the smaller the gray value, the closer the actual object represented by the pixel point is to the image acquisition device. Additionally, continuing to refer to Figure 6 and Figure 7 shown, within the same step area range, the depth in the depth image at the (t + 1)-th moment is significantly greater than the depth in the depth image at the t-th moment, indicating that this part of the step area is moving away from the image acquisition device, that is, it can be determined that the escalator is in the upward state.

[0103] Based on the above principle, the depth features of the depth image can be determined through the gray values of the pixel points in the depth image, and thus the current state of the escalator can be judged based on the depth features. Specifically, for a single-frame depth image, the gray value features of the depth image can be determined based on the gray values of all pixel points in the depth image, and normalization processing is performed on the gray value features. Exemplarily, normalization can be performed using the maximum pixel gray value in the value range of the depth image. For example, divide the gray value feature by the maximum pixel gray value to obtain each normalized gray value feature. Further, the normalized gray value features corresponding to each depth image can be combined to form a depth feature sequence. It should be noted that at least two consecutive depth images need to ensure that the height of at least one step changes from 0 to the maximum height value.

[0104] In this embodiment, at least one gray value feature of the depth-of-field image is determined through the gray values of each pixel point in each depth-of-field image, and each gray value feature is normalized, so as to determine a depth-of-field feature sequence based on the normalized gray value features. Since the normalization process can normalize the data to the same interval, the comparability of the data is improved.

[0105] Exemplarily, the above at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature, and a median gray value feature. When determining the depth-of-field feature sequence based on the normalized gray value features corresponding to each depth-of-field image, the depth-of-field feature value of the depth-of-field image can be determined based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth-of-field image, and the depth-of-field feature values of each depth-of-field image are combined to obtain a depth-of-field feature sequence.

[0106] Specifically, for a single-frame depth-of-field image, the maximum gray value feature, the minimum gray value feature, the average gray value feature, and the median gray value feature can be determined based on the gray values of all pixel points in the depth-of-field image, and the maximum gray value feature, the minimum gray value feature, the average gray value feature, and the median gray value feature are respectively divided by the maximum pixel gray value in the value range of the depth-of-field image to obtain the normalized maximum gray value feature DMax, the normalized minimum gray value feature DMin, the normalized average gray value feature DA, and the normalized median gray value feature Dmid.

[0107] Exemplarily, the depth-of-field feature value of the depth-of-field image can be determined based on the following formula (1):

[0108] (1)

[0109] where represents the depth-of-field feature value, represents a correction coefficient, usually taking a value between 0.01 and 0.1, .

[0110] The depth-of-field feature value of the depth-of-field image can be calculated quickly and accurately through the above formula (1).

[0111] After determining the depth-of-field feature values of each frame of depth-of-field image in the above manner, the depth-of-field feature values of all depth-of-field images are combined according to the acquisition time sequence of the depth-of-field images to obtain a depth-of-field feature sequence.

[0112] In this embodiment, since the depth feature value can be calculated jointly based on four features: the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth image, the error problem caused by calculating the depth feature value using a single gray value feature is avoided, and the calculation accuracy of the depth feature value is improved.

[0113] Furthermore, on the basis of the above embodiments, before determining the height feature of the target step in the target image, it is also necessary to first determine the target step from the target image. Exemplarily, the target image can be input into the step detection model to obtain the target step output by the step detection model; the step detection model is obtained by training the initial step detection model based on multiple image samples including escalator samples and the labels in each image sample for characterizing the step samples in the escalator samples.

[0114] Specifically, in this embodiment, step detection uses an object detection algorithm based on a convolutional neural network. Among them, by pre-collecting images of multiple escalator samples to obtain multiple image samples including escalator samples and annotating the labels for characterizing the step samples in the escalator samples in each image sample, and inputting each image sample into the initial step detection model, the predicted steps output by the initial step detection model can be obtained. After determining the loss information based on the predicted steps and the above labels, the model parameters of the initial step detection model can be adjusted according to the loss information. By continuously repeating the above process until the number of repetitions reaches the preset number or the obtained model converges, the finally obtained model is determined as the step detection model. Among them, the backbone network responsible for feature extraction in the step detection model uses an improved Cross Stage Partial Network (CSPNet), which can optimize gradient propagation; the Neck network integrates the Path Aggregation Network layer (PAN), which realizes the effective fusion of multi-scale features. When training the step detection model and performing step detection based on the step detection model, a single best prediction result can be generated for each step, and the Non-Maximum Suppression (NMS) module is removed.

[0115] Figure 8 The step detection schematic diagram provided by the embodiment of the present invention is as Figure 8 shown. By inputting the target image into the step detection model, each step in the target image can be obtained, as Figure 8As shown by the yellow box in []. After selecting each step, the first step with a step height greater than zero is determined from all the steps as the target step. This target step is the longitudinal surface area of the first step at the bottom of the escalator with a step height, which can also be understood as the first step detected from bottom to top in the escalator area, or the first step at the bottom.

[0116] In this embodiment, the target image is input into the step detection model to obtain each step in the target image. This method can improve the speed and accuracy of step determination.

[0117] Exemplarily, on the basis of the above embodiments, when determining the height feature of the target step in each frame of the target image to obtain the step height feature sequence, it can be to determine, for each frame of the target image, the first line segment and the second line segment at the height plane edge of the target step in the target image. The extension directions of the first line segment and the second line segment are both perpendicular to the escalator running direction. After normalizing the distance between the first line segment and the second line segment, the normalized distance is determined as the height feature of the target step in the target image, and the height features of the target step in each target image are combined to obtain the step height feature sequence.

[0118] Specifically, for each frame of the target image, after determining the target step in the target image, it is necessary to use a line detection algorithm within the target step. For example, the Hough transform algorithm can be used to detect the first line segment and the second line segment at the height plane edge of the target step. As Figure 8 The red line segments shown in [] are the first line segment and the second line segment. The extension directions of the first line segment and the second line segment are both perpendicular to the escalator running direction.

[0119] Furthermore, the distance between the first line segment and the second line segment in the vertical plane can be determined to obtain the step height of the target step. The distance is normalized. For example, the maximum height when the step normally rises is fixed, and the distance can be divided by the maximum height of the known step to obtain the normalized distance, forming the height feature of the target step in a single frame of the target image.

[0120] After obtaining the height feature of the target step in each frame of the target image, all the height features are combined according to the acquisition time of the target image, and the step height feature sequence can be obtained. It should be understood that this step height feature sequence can be used to characterize the height change of the target step in multiple frames of the target image.

[0121] In this embodiment, by determining the first line segment and the second line segment of the target step in the target image and determining the height of the target step based on the first line segment and the second line segment, the determined height of the target step is relatively accurate. In addition, by combining the height features of the target steps in all target images and performing escalator anomaly detection based on the escalator height feature sequence obtained by the combination, which can characterize the change of the height of the target step, the accuracy of escalator anomaly detection can be improved.

[0122] In a possible implementation, when performing anomaly detection on the escalator based on the depth of field feature sequence and the escalator height feature sequence, the current operating state of the escalator can be determined according to the change rules of the depth of field feature values in the depth of field feature sequence and the escalator height in the escalator height feature sequence, so as to compare the current operating state with the preset state to determine whether the escalator has an anomaly.

[0123] Specifically, Figure 9 is a schematic diagram of the change of the depth of field feature sequence when the escalator is running upward according to the embodiment of the present invention. As Figure 9 shown, when the escalator is running upward, each depth of field feature value in the depth of field feature sequence gradually decreases. Figure 10 is a schematic diagram of the change of the depth of field feature sequence when the escalator is running downward according to the embodiment of the present invention. As Figure 10 shown, when the escalator is running downward, each depth of field feature value in the depth of field feature sequence gradually increases. Figure 11 is a schematic diagram of the change of the depth of field feature sequence when the escalator stops according to the embodiment of the present invention. As Figure 11 shown, when the escalator stops, each depth of field feature value in the depth of field feature sequence remains unchanged. Figure 12 is a schematic diagram of the change of the depth of field feature sequence when the escalator has a mechanical anomaly according to the embodiment of the present invention. As Figure 12 shown, when the escalator has a mechanical anomaly, each depth of field feature value in the depth of field feature sequence oscillates.

[0124] Figure 13 is a schematic diagram of the change of the escalator height feature sequence when the escalator is running upward according to the embodiment of the present invention. As Figure 13 shown, when the escalator is running upward, the escalator height in the escalator height feature sequence gradually increases. Figure 14 is a schematic diagram of the change of the escalator height feature sequence when the escalator is running downward according to the embodiment of the present invention. As Figure 14 shown, when the escalator is running downward, the escalator height in the escalator height feature sequence gradually decreases. Figure 15 is a schematic diagram of the change of the escalator height feature sequence when the escalator stops according to the embodiment of the present invention. As Figure 15 shown, when the escalator stops, the escalator height in the escalator height feature sequence remains unchanged. Figure 16Schematic diagram of the change of the step height feature sequence when a mechanical abnormality occurs in the escalator provided by the embodiment of the present invention, as Figure 16 shown, when a mechanical abnormality occurs in the escalator, the escalator heights in the step height feature sequence will oscillate.

[0125] In another possible implementation, the depth of field feature sequence and the step height feature sequence can also be input into the time feature sequence classifier to obtain the current escalator state output by the time feature sequence classifier, and based on the current escalator state and the preset state, the escalator is subjected to abnormality detection, and the preset state is the state in which the escalator is expected to be.

[0126] Specifically, inputting the depth of field feature sequence and the step height feature sequence into the time feature sequence classifier for classification can obtain the current escalator state, such as the escalator going up, the escalator going down, or the escalator stopping. Among them, the time feature sequence classifier can be, for example, a Long Short-Term Memory (LSTM) classification model. This LSTM classification model contains 100 hidden layer units, the output layer is a fully connected layer of 3 classes, and finally there is a softmax layer and a classification layer, and the loss function uses categorical cross-entropy.

[0127] After determining the current escalator state, the current escalator state can be compared with the preset state corresponding to the escalator. If they are inconsistent, it means that the escalator has an abnormality. Among them, the preset state of the escalator is the state in which the escalator is expected to be, that is, the state in which the escalator is when there is no abnormality.

[0128] In this embodiment, the depth of field feature sequence and the step height feature sequence can be input into the time feature sequence classifier, and the current escalator state analyzed by the time feature sequence classifier can be determined in combination with the depth of field feature and height feature of the escalator in the past period of time, making the result of the current escalator state more accurate.

[0129] Exemplarily, when performing abnormality detection on the escalator based on the current escalator state and the preset state, the following situations will be included:

[0130] When the current escalator state is the running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, it is determined that the escalator is abnormal. When the current escalator state is the stop state and the preset state indicates that the escalator is in the running state, it is determined that the escalator is abnormal. When the current escalator state is the running state and the running acceleration is different from the normal running acceleration of the escalator corresponding to the preset state, it is determined that the escalator is abnormal.

[0131] Specifically, Figure 17 Schematic diagram of the escalator abnormality state detection provided by the embodiment of the present invention, as Figure 17As shown, the depth-of-field feature sequence and the step height feature sequence are input into the LSTM classification model for classification, and the current escalator state can be obtained, such as the escalator going up, the escalator going down, or the escalator stopped. If the current escalator state is the running state, compare the running direction with the set running direction of the escalator corresponding to the preset state. If they are inconsistent, it means the escalator has reversed, and the escalator can be determined to be abnormal. For example, if the current escalator state is the escalator going up and the set running direction of the escalator is going down, it means the escalator has reversed.

[0132] If the current escalator state is the stopped state, while the preset state indicates that the escalator is currently set to the running state, it means the escalator has an abnormal stop currently, and the escalator can be determined to be abnormal.

[0133] In addition, it should be understood that when the escalator is running normally, it generally moves at a constant speed, and the corresponding acceleration should be 0. When the escalator has abnormal jitter or mechanical abnormality, an upward or downward acceleration will be generated, and the acceleration will continue to increase or increase and decrease suddenly, and it is also possible that the direction of the acceleration changes in the reverse direction. Therefore, if the current escalator state is the running state and the running acceleration of the current escalator is different from the acceleration of the escalator corresponding to the preset state during normal movement, it means the escalator has abnormal jitter or mechanical abnormality, and the escalator can be determined to be abnormal. Among them, the acceleration of the escalator corresponding to the preset state during normal movement is 0. The difference between the running acceleration of the current escalator and the acceleration of the escalator corresponding to the preset state during normal movement can include the difference in the magnitude of the acceleration and / or the difference in the direction.

[0134] In this embodiment, it is only necessary to compare the current escalator state with the preset state. When the current escalator state is inconsistent with the preset state, it can be determined that the escalator is abnormal, making the escalator abnormality detection method simpler and faster.

[0135] Next, in combination with the solutions in the foregoing embodiments, the overall solution for escalator abnormality detection provided by the embodiments of the present invention will be introduced.

[0136] Figure 18 It is a schematic diagram of escalator abnormality detection provided by the embodiments of the present invention. As Figure 18 shown, after obtaining the initial image sequence including the escalator, for at least two initial images in the initial image sequence, the region of interest is extracted according to the set ROI region to obtain the target image. Based on the obtained at least two target images, the depth-of-field images corresponding to each frame of the target image can be generated, and the depth-of-field features in each frame of the depth-of-field image are extracted. After normalizing the depth-of-field features in each frame of the depth-of-field image, the depth-of-field feature values are determined based on the normalized depth-of-field features, and the depth-of-field feature values in multiple frames of the depth-of-field image are combined in the acquisition time order of the depth-of-field images to obtain the depth-of-field feature sequence corresponding to multiple frames of the depth-of-field image.

[0137] In addition, after performing step detection on each frame of the target image, a target step can be determined from the detected steps, straight line detection can be performed on the target step, the step height of the target step can be determined based on the difference between two straight lines detected within the target step, and the normalized step height can be determined as the height feature of the target step. The height features in multiple frames of target images are combined in the order of the acquisition time of the target images to obtain a step height feature sequence corresponding to the multiple frames of target images.

[0138] Further, by inputting the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier, the current escalator state output by the time feature sequence classifier can be obtained, and the current escalator state can be compared with a preset state in which the escalator is expected to be located, so as to determine whether the escalator is abnormal, such as whether it undergoes reverse rotation, abnormal stop, or jitter.

[0139] The escalator abnormality detection method provided by the embodiment of the present invention obtains at least two frames of target images including an area of interest, where the area of interest includes an entrance area or an exit area in the escalator, respectively extracts features from the depth of field images corresponding to each frame of the target images to obtain a depth of field feature sequence, determines the height features of the target steps in each frame of the target images to obtain a step height feature sequence, the target step is the first step with a height greater than zero in the area of interest, and performs abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence. Since the operating state and operating direction of the escalator can be determined based on the depth of field image corresponding to the target image and the height change of the target step, the escalator can be detected for abnormalities, avoiding the phenomenon of determining the operating vector of the escalator through the optical flow method in the prior art, and improving the accuracy of detecting the abnormal state of the escalator.

[0140] Next, the escalator abnormality detection device provided by the embodiment of the present invention will be described. The escalator abnormality detection device described below can be correspondingly referred to the escalator abnormality detection method described above.

[0141] Figure 19 For the structural schematic diagram of the escalator abnormality detection device provided by the embodiment of the present invention, refer to Figure 19 As shown, the escalator abnormality detection device 1900 includes:

[0142] An acquisition module 11, configured to acquire at least two frames of target images including an area of interest, where the area of interest includes an entrance area or an exit area in the escalator;

[0143] An extraction module 12, configured to respectively extract features from the depth of field images corresponding to each frame of the target images to obtain a depth of field feature sequence;

[0144] A determination module 13, configured to determine the height features of the target steps in each frame of the target image, so as to obtain a step height feature sequence, where the target step is the first step in the region of interest whose step height is greater than zero;

[0145] A detection module 14, configured to perform anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence.

[0146] In an exemplary embodiment, the extraction module 12 is specifically configured to:

[0147] For each of the depth-of-field images, determine the gray value of each pixel point in the depth-of-field image;

[0148] Based on the gray values of all pixel points in the depth-of-field image, determine at least one gray value feature of the depth-of-field image;

[0149] Perform normalization processing on each of the gray value features to obtain each normalized gray value feature;

[0150] Based on the normalized gray value features corresponding to each of the depth-of-field images, determine the depth-of-field feature sequence.

[0151] In an exemplary embodiment, the at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature, and a median gray value feature;

[0152] The extraction module 12 is specifically configured to:

[0153] Based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth-of-field image, determine the depth-of-field feature value of the depth-of-field image;

[0154] Combine the depth-of-field feature values of each of the depth-of-field images to obtain the depth-of-field feature sequence.

[0155] In an exemplary embodiment, the extraction module 12 is specifically configured to:

[0156] Determine the depth-of-field feature value of the depth-of-field image based on the following formula (1):

[0157] (1)

[0158] Where represents the depth-of-field feature value, DA represents the normalized average gray value feature, DMid represents the normalized median gray value feature, Dmax represents the normalized maximum gray value feature, Dmin represents the normalized minimum gray value feature, represents the correction coefficient, 。

[0159] In an exemplary embodiment, the determining module 13 is specifically configured to:

[0160] For each frame of the target image, determine a first line segment and a second line segment at the height plane edge of the target step in the target image, and the extending directions of the first line segment and the second line segment are both perpendicular to the escalator running direction;

[0161] Normalize the distance between the first line segment and the second line segment to obtain a normalized distance;

[0162] Determine the normalized distance as the height feature of the target step in the target image;

[0163] Combine the height features of the target steps in each of the target images to obtain the step height feature sequence.

[0164] In an exemplary embodiment, the detecting module 14 is specifically configured to:

[0165] Input the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier to obtain the current escalator state output by the time feature sequence classifier;

[0166] Based on the current escalator state and a preset state, perform anomaly detection on the escalator, where the preset state is the state expected for the escalator to be in.

[0167] In an exemplary embodiment, the detecting module 14 is specifically configured to:

[0168] When the current escalator state is the running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determine that the escalator is abnormal;

[0169] When the current escalator state is the stopped state and the preset state indicates that the escalator is in the running state, determine that the escalator is abnormal;

[0170] When the current escalator state is the running state and the running acceleration is different from the acceleration during normal running of the escalator corresponding to the preset state, determine that the escalator is abnormal.

[0171] In an exemplary embodiment, the obtaining module 11 is specifically configured to:

[0172] Obtain at least two initial images including the escalator;

[0173] For each frame of the initial image, based on the position of the skirt board of the escalator in the initial image, determine the region of interest in the initial image;

[0174] Extract the region of interest in the initial image to obtain the target image corresponding to the initial image.

[0175] In an exemplary embodiment, the determining module 13 is specifically configured to:

[0176] Determine a first step position at the bottom end position of the apron plate in the length direction and a second step position at the top end position in the initial image;

[0177] Determine the region between the first step position and the second step position in the initial image and including the entire step width as the region of interest.

[0178] In an exemplary embodiment, the apparatus further includes: an input module, where:

[0179] The input module is configured to input the target image into a step detection model to obtain the target step output by the step detection model; the step detection model is obtained by training an initial step detection model based on a plurality of image samples including escalator samples and labels for characterizing step samples in each of the image samples.

[0180] The apparatus in this embodiment can be used to execute the method in any one of the embodiments on the side of the escalator anomaly detection method. The specific implementation process and technical effects are similar to those in the embodiments on the side of the escalator anomaly detection method. For details, reference can be made to the detailed introduction in the embodiments on the side of the escalator anomaly detection method, which will not be elaborated here.

[0181] Figure 20 As shown in the schematic physical structure diagram of an electronic device provided by an embodiment of the present invention, Figure 20 The electronic device may include: a processor 2010, a communication interface 2020, a memory 2030, and a communication bus 2040. Among them, the processor 2010, the communication interface 2020, and the memory 2030 communicate with each other through the communication bus 2040. The processor 2010 can call the logical instructions in the memory 2030 to execute the escalator anomaly detection method, which includes: obtaining at least two frames of target images including regions of interest, where the regions of interest include the entrance region or the exit region in the escalator; respectively performing feature extraction on the depth-of-field images corresponding to each frame of the target images to obtain a depth-of-field feature sequence; determining the height features of the target steps in each frame of the target images to obtain a step height feature sequence, where the target step is the first step with a step height greater than zero in the region of interest; and performing anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence.

[0182] In addition, when the logical instructions in the above-mentioned memory 2030 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The foregoing storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0183] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the escalator anomaly detection method provided by the above-mentioned various methods. The method includes: obtaining at least two target images including a region of interest, where the region of interest includes an entrance region or an exit region in the escalator; respectively performing feature extraction on the depth-of-field images corresponding to each frame of the target images to obtain a depth-of-field feature sequence; determining the height feature of the target step in each frame of the target images to obtain a step height feature sequence, where the target step is the first step with a step height greater than zero in the region of interest; and performing anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence.

[0184] On yet another hand, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the escalator anomaly detection method provided by the above-mentioned various methods. The method includes: obtaining at least two target images including a region of interest, where the region of interest includes an entrance region or an exit region in the escalator; respectively performing feature extraction on the depth-of-field images corresponding to each frame of the target images to obtain a depth-of-field feature sequence; determining the height feature of the target step in each frame of the target images to obtain a step height feature sequence, where the target step is the first step with a step height greater than zero in the region of interest; and performing anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence.

[0185] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0186] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0187] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An escalator abnormality detection method, characterized in that, Including: Obtain at least two target images including the region of interest, where the region of interest includes the entrance region or the exit region in the escalator; Extract features from the depth images corresponding to each frame of the target images respectively to obtain a depth feature sequence; Determine the height features of the target steps in each frame of the target images to obtain a step height feature sequence, where the target step is the first step with a step height greater than zero in the region of interest; Based on the depth feature sequence and the step height feature sequence, perform anomaly detection on the escalator; The extracting features from the depth images corresponding to each frame of the target images respectively to obtain a depth feature sequence includes: For each of the depth images, determine the gray value of each pixel point in the depth image; Based on the gray values of all pixel points in the depth image, determine at least one gray value feature of the depth image; Perform normalization processing on each of the gray value features to obtain each normalized gray value feature; Based on the normalized gray value features corresponding to each of the depth images, determine the depth feature sequence; The at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature, and a median gray value feature; The based on the normalized gray value features corresponding to each of the depth images, determine the depth feature sequence includes: Based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth image, determine the depth feature value of the depth image; Combine the depth feature values of each of the depth images to obtain the depth feature sequence.

2. The escalator anomaly detection method according to claim 1, wherein, The based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth image, determine the depth feature value of the depth image includes: Determine the depth feature value of the depth image based on the following formula (1): Among them, Depth_Feature represents the depth-of-field feature value, DA represents the average gray value feature after normalization, DMid represents the median gray value feature after normalization, Dmax represents the maximum gray value feature after normalization, Dmin represents the minimum gray value feature after normalization, and β represents the correction coefficient.

3. The escalator abnormal detection method according to claim 1, wherein The determining the height features of the target steps in each frame of the target images to obtain a step height feature sequence includes: For each frame of the target images, determine a first line segment and a second line segment at the edge of the height plane of the target step in the target image, where the extending directions of the first line segment and the second line segment are both perpendicular to the running direction of the escalator; Perform normalization processing on the distance between the first line segment and the second line segment to obtain a normalized distance; Determine the normalized distance as the height feature of the target step in the target image; Combine the height features of the target steps in each of the target images to obtain the step height feature sequence.

4. The escalator abnormal detection method according to claim 1, wherein The based on the depth feature sequence and the step height feature sequence, perform anomaly detection on the escalator includes: Input the depth feature sequence and the step height feature sequence into a time feature sequence classifier to obtain the current escalator state output by the time feature sequence classifier; Based on the current escalator state and the preset state, perform anomaly detection on the escalator, where the preset state is the state expected for the escalator to be in.

5. The escalator abnormal detection method according to claim 4, characterized in that, The performing anomaly detection on the escalator based on the current escalator state and the preset state includes: When the current escalator state is the running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determine that the escalator is abnormal; When the current escalator state is the stopped state and the preset state indicates that the escalator is in the running state, determine that the escalator is abnormal; When the current escalator state is the running state and the running acceleration is different from the acceleration during normal running of the escalator corresponding to the preset state, determine that the escalator is abnormal.

6. The escalator abnormality detection method according to any one of claims 1-5, characterized in that, The obtaining at least two target images including the region of interest includes: Obtain at least two initial images including the escalator; For each of the initial images, based on the position of the skirt board of the escalator in the initial image, determine the region of interest in the initial image; Extract the region of interest in the initial image to obtain the target image corresponding to the initial image.

7. The escalator abnormality detection method according to claim 6, characterized in that, The determining the region of interest in the initial image based on the position of the skirt board of the escalator in the initial image includes: Determine the first step position at the bottom position in the length direction of the skirt board and the second step position at the top position in the initial image; Determine the region between the first step position and the second step position in the initial image and including the entire step width as the region of interest.

8. The escalator abnormal detection method according to any one of claims 1-5, characterized in that, The method further includes: Input the target image into the step detection model to obtain the target steps output by the step detection model; the step detection model is obtained by training an initial step detection model based on multiple image samples including escalator samples and labels for characterizing the step samples in each of the image samples.

Images