Escalator anomaly detection method

Abstract

The invention provides an escalator anomaly detection method which comprises the steps that at least two frames of target images including areas of interest are obtained, and the areas of interest include an entrance area or an exit area in an escalator; performing feature extraction on the depth-of-field image corresponding to each frame of the target image to obtain a depth-of-field feature sequence; determining height features of a target step in each frame of the target image to obtain a step height feature sequence, the target step being a first step of which the step height is greater than zero in the region of interest; and performing anomaly detection on the escalator based on the depth-of-field feature sequence and the step height feature sequence. The escalator abnormity detection accuracy can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an escalator abnormality detection method. Background Art

[0002] In public places, the use of escalators has become more and more common. However, sometimes the escalators may stop suddenly or reverse due to human or machine failure, which may easily lead to serious consequences.

[0003] In the prior art, when detecting whether an escalator has an abnormality, it is necessary to collect a video stream containing the escalator, and determine the motion vector and escalator area of ​​the escalator through the video stream, so as to estimate the real-time running direction of the escalator, realize the state detection of the escalator, and thus determine whether the escalator has an abnormality.

[0004] However, in the above method, the calculation of motion vectors is usually achieved by calculating the optical flow formed by sampling points on different frames, and the optical flow method is easily affected by light and is inaccurate, thus resulting in low accuracy of escalator abnormality detection. Summary of the invention

[0005] The present invention provides an escalator abnormality detection method, which is used to solve the defect of low accuracy of escalator abnormality detection in the prior art and improve the accuracy of escalator abnormality detection.

[0006] The present invention provides an escalator abnormality detection method, comprising: Acquire at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; Extracting features of the depth of field images corresponding to the target image in each frame to obtain a depth of field feature sequence; Determine the height feature of the target step in the target image of each frame to obtain a step height feature sequence, wherein the target step is the first step in the region of interest whose step height is greater than zero; Based on the depth of field feature sequence and the step height feature sequence, abnormality detection is performed on the escalator.

[0007] According to an escalator abnormality detection method provided by the present invention, the feature extraction of the depth of field image corresponding to the target image of each frame is performed to obtain a depth of field feature sequence, including: For each of the depth of field images, determining the grayscale value of each pixel in the depth of field image; Determining at least one grayscale value feature of the depth of field image based on the grayscale values ​​of all pixels in the depth of field image; Normalizing each of the grayscale value features to obtain each normalized grayscale value feature; The depth of field feature sequence is determined based on the normalized grayscale value features corresponding to each of the depth of field images.

[0008] According to an escalator abnormality detection method provided by the present invention, the at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature and a median gray value feature; The determining the depth of field feature sequence based on the normalized grayscale value features corresponding to each of the depth of field images comprises: Determining a depth feature value of the depth image based on a normalized maximum gray value feature, a normalized minimum gray value feature, a normalized average gray value feature, and a normalized median gray value feature corresponding to the depth image; The depth-of-field feature values ​​of the depth-of-field images are combined to obtain the depth-of-field feature sequence.

[0009] According to an escalator abnormality detection method provided by the present invention, the depth feature value of the depth image is determined based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature and the normalized median gray value feature corresponding to the depth image, including: The depth characteristic value of the depth image is determined based on the following formula (1): (1)

[0010] in, represents the depth of field feature value, DA represents the normalized average grayscale value feature, DMid represents the normalized median grayscale value feature, Dmax represents the normalized maximum grayscale value feature, and Dmin represents the normalized minimum grayscale value feature. represents the correction factor, .

[0011] According to an escalator abnormality detection method provided by the present invention, the step of determining the height feature of the target step in each frame of the target image to obtain the step height feature sequence includes: For each frame of the target image, determine a first line segment and a second line segment at an edge of a height plane of the target step in the target image, wherein an extension direction of the first line segment and an extension direction of the second line segment are both perpendicular to a running direction of the escalator; Normalizing the distance between the first line segment and the second line segment to obtain a normalized distance; Determine the normalized distance as a height feature of the target step in the target image; The height features of the target steps in each of the target images are combined to obtain the step height feature sequence.

[0012] According to an escalator abnormality detection method provided by the present invention, the escalator is subjected to abnormality detection based on the depth of field feature sequence and the step height feature sequence, comprising: Inputting the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier to obtain a current escalator state output by the time feature sequence classifier; Based on the current escalator state and a preset state, an abnormality detection is performed on the escalator, wherein the preset state is a state in which the escalator is expected to be.

[0013] According to an escalator abnormality detection method provided by the present invention, the escalator is subjected to abnormality detection based on the current escalator state and the preset state, comprising: When the current escalator state is a running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determining that the escalator is abnormal; When the current escalator state is a stopped state and the preset state indicates that the escalator is in a running state, determining that the escalator is abnormal; When the current escalator state is a running state and the running acceleration is different from the acceleration of the escalator during normal operation corresponding to the preset state, it is determined that the escalator is abnormal.

[0014] According to an escalator abnormality detection method provided by the present invention, the step of acquiring at least two frames of target images including a region of interest comprises: Acquire at least two frames of initial images including the escalator; For each frame of the initial image, based on the position of the apron plate of the escalator in the initial image, determining the region of interest in the initial image; The region of interest in the initial image is extracted to obtain a target image corresponding to the initial image.

[0015] According to an escalator abnormality detection method provided by the present invention, the determining the region of interest in the initial image based on the position of the apron plate of the escalator in the initial image includes: Determine a first step position at a bottom end position and a second step position at a top end position of the apron plate in the length direction in the initial image; An area between the first step position and the second step position in the initial image and including the entire step width is determined as the region of interest.

[0016] According to a method for detecting abnormality of an escalator provided by the present invention, the method further comprises: The target image is input into a step detection model to obtain the target step output by the step detection model; the step detection model is obtained by training an initial step detection model based on multiple image samples including escalator samples and labels in each of the image samples for characterizing step samples in the escalator samples.

[0017] The present invention also provides an escalator abnormality detection device, comprising: An acquisition module, used for acquiring at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; An extraction module is used to extract features from the depth of field images corresponding to the target image in each frame to obtain a depth of field feature sequence; A determination module, used for determining the height feature of the target step in the target image of each frame to obtain a step height feature sequence, wherein the target step is the first step in the region of interest whose step height is greater than zero; A detection module is used to perform abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence.

[0018] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the escalator abnormality detection method described above is implemented.

[0019] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the escalator abnormality detection method as described in any one of the above is implemented.

[0020] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the escalator abnormality detection method as described above is implemented.

[0021] The escalator abnormality detection method provided by the embodiment of the present invention obtains at least two frames of target images including an area of ​​interest, and the area of ​​interest includes an entrance area or an exit area in the escalator, and extracts features from the depth of field images corresponding to each frame of the target image to obtain a depth of field feature sequence, and determines the height features of the target step in each frame of the target image to obtain a step height feature sequence, wherein the target step is the first step in the area of ​​interest whose height is greater than zero, and performs abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence. Since the operating state and operating direction of the escalator can be determined based on the depth of field image corresponding to the target image and the height change of the target step, the escalator can be detected abnormally, thereby avoiding the phenomenon in the prior art that the operating vector of the escalator needs to be determined by the optical flow method, and the accuracy of the abnormal state detection of the escalator is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 A schematic diagram of the operation of an escalator at time t provided by an embodiment of the present invention.

[0024] Figure 2 A schematic diagram of the operation of an escalator at time t+1 provided by an embodiment of the present invention.

[0025] Figure 3 A schematic flow chart of an escalator abnormality detection method provided in an embodiment of the present invention.

[0026] Figure 4 The target image corresponding to the tth moment provided by the embodiment of the present invention.

[0027] Figure 5 The target image corresponding to the time t+1 provided by the embodiment of the present invention.

[0028] Figure 6 The depth of field image corresponding to the target image at the tth moment provided by the embodiment of the present invention.

[0029] Figure 7 The depth of field image corresponding to the target image at the time t+1 provided by the embodiment of the present invention.

[0030] Figure 8 A schematic diagram of step detection provided by an embodiment of the present invention.

[0031] Fig. 9 A schematic diagram of the change of the depth of field feature sequence when the escalator goes up provided by an embodiment of the present invention.

[0032] Fig.10 A schematic diagram of the change of the depth of field feature sequence when an escalator goes down provided by an embodiment of the present invention.

[0033] Fig.11 A schematic diagram of changes in a depth of field feature sequence when an escalator stops provided by an embodiment of the present invention.

[0034] Fig.12 A schematic diagram of changes in a depth of field feature sequence when a mechanical abnormality occurs in an escalator provided by an embodiment of the present invention.

[0035] Fig.13A schematic diagram of the change of the step height characteristic sequence when the escalator is ascending provided by an embodiment of the present invention.

[0036] Fig.14 A schematic diagram of the change of the step height characteristic sequence when the escalator is descending provided by an embodiment of the present invention.

[0037] Fig.15 A schematic diagram of the change of the step height characteristic sequence when the escalator stops provided in an embodiment of the present invention.

[0038] Fig.16 A schematic diagram of the change of the step height characteristic sequence when a mechanical abnormality occurs in an escalator provided by an embodiment of the present invention.

[0039] Fig.17 A schematic diagram of escalator abnormal state detection provided by an embodiment of the present invention.

[0040] Fig.18 A schematic diagram of escalator abnormality detection provided by an embodiment of the present invention.

[0041] Fig.19 A schematic diagram of the structure of an escalator abnormality detection device provided in an embodiment of the present invention.

[0042] Fig. 20 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] When an escalator stops suddenly or reverses, it will cause serious consequences. If the escalator's abnormality can be automatically detected and reported to the monitoring center, passengers can be prevented from being hurt. Therefore, how to accurately detect escalator abnormalities has become an important part of escalator status monitoring. At present, by collecting a video stream containing an escalator and using the optical flow method to determine the escalator's motion vector and escalator area based on the video stream, the real-time running direction of the escalator is estimated. The state detection of the escalator is achieved by judging the estimated running direction of the escalator, thereby judging whether the escalator is abnormal. Since the optical flow method is easily affected by light and inaccurate, it will cause the accuracy of escalator abnormality detection to be low.

[0045] In response to the above problems, an embodiment of the present invention provides an escalator abnormality detection method, wherein the escalator includes multiple steps, each step is composed of horizontal and vertical plates, passengers step on the horizontal plates, and the height of the vertical plates at the bottom of the escalator will change periodically during the operation of the escalator. Figure 1 The schematic diagram of the operation of the escalator at the tth moment provided by the embodiment of the present invention, Figure 2 The operation schematic diagram of the escalator at the t+1th time provided by the embodiment of the present invention is as follows: Figure 1 and Figure 2 As shown in the figure, when the escalator is going up, the steps at the bottom will gradually appear from a plane, and the height will gradually increase from low to high. When the escalator is going down, the height of the steps at the bottom will gradually decrease from high to low, and finally become a plane. When the escalator is stationary, the step height will not change. If the escalator has abnormal mechanical shaking, such as rapid up and down vibration, the step height will oscillate. At the same time, during the operation of the escalator, the depth of field image of the steps will also change periodically. For example, when the escalator is going up, the bottom steps are from near to far, so the overall depth of field of the steps will change from bright to dark. Similarly, when the escalator is going down, the bottom steps are from far to near, so the overall depth of field of the steps will change from dark to bright. When the escalator is stationary, the depth of field image of the steps does not change significantly. If the escalator has abnormal mechanical shaking, such as rapid up and down vibration, the overall depth of field of the steps will oscillate.

[0046] Based on the above principle, in the escalator abnormality detection method provided in the embodiment of the present invention, the change law of the height of the target step on the lower side of the escalator when the escalator is running is used, combined with the change law of the depth of field caused by the height change of the target step, to determine the current running state and running direction of the escalator, so as to compare the running state and running direction with the preset running state and preset running direction to determine whether the escalator is abnormal. In the above method, the running state and running direction of the escalator can be determined only based on the depth of field image corresponding to the target image and the height characteristics of the target step, so as to detect the abnormality of the escalator, avoiding the phenomenon in the prior art that the running vector of the escalator needs to be determined by the optical flow method, and improving the accuracy of the abnormal state detection of the escalator.

[0047] Combine the following Figures 3 to 18 The escalator abnormality detection method provided by the embodiment of the present invention is described. The embodiment of the present invention can be applicable to the detection of abnormal conditions of escalators in any scenario where an escalator is installed, wherein the escalator abnormalities in the embodiment of the present invention include escalator reversal, escalator emergency stop or escalator mechanical shaking. The executor of the method can be an electronic device such as an intelligent camera, an escalator control device, a computer, a server, a server cluster or a specially designed escalator abnormality detection device, or an escalator abnormality detection device provided in the electronic device, and the escalator abnormality detection device can be implemented by software, hardware or a combination of both.

[0048] Figure 3 A schematic diagram of a flow chart of an escalator abnormality detection method provided by an embodiment of the present invention, such as Figure 3 As shown, the method includes: Step 301: Acquire at least two frames of target images including an area of ​​interest, where the area of ​​interest includes an entrance area or an exit area in an escalator.

[0049] In this step, at least one frame of initial image including the escalator area can be acquired by an image acquisition device, and for each frame of initial image, a target image including the region of interest can be intercepted from the initial image. It should be understood that, when the escalator is in operation, usually only the step heights at the entrance and exit of the escalator change, while the step heights in the middle of the escalator remain unchanged. Therefore, in order to reduce the amount of image processing calculations, only the target image including the region of interest can be detected.

[0050] It should be noted that, if the escalator is in an upward state, the area at the bottom of the escalator can be understood as the entrance area of ​​the escalator, and the area at the top of the escalator can be understood as the exit area of ​​the escalator, and when the escalator is in a downward state, the area at the bottom of the escalator can be understood as the exit area of ​​the escalator, and the area at the top of the escalator can be understood as the entrance area of ​​the escalator. In the embodiments of the present invention, the image acquisition device can capture the lower side area of ​​the escalator as an example, and the area of ​​interest includes the lower side of the escalator or the area at the bottom of the escalator for explanation.

[0051] Among them, at least two frames of continuous target images need to ensure that the height of at least one step changes from 0 to a maximum height value.

[0052] Step 302: extracting features from the depth of field images corresponding to the target images of each frame to obtain a depth of field feature sequence.

[0053] In this step, for each frame of the target image, the target image can be processed using a depth estimation algorithm to obtain a depth image corresponding to the target image, wherein the depth image is a grayscale image. Exemplarily, the depth estimation algorithm can use Depth Anything, which is a monocular depth estimation algorithm that can combine labeled and unlabeled images for depth estimation. First, the teacher model T is learned using the labeled set, and then the teacher model T is used to assign pseudo-depth labels to the unlabeled images. Finally, the student model S is trained on the combination of the labeled set and the pseudo-labeled set. In order to make the depth estimation model have rich semantic priors, auxiliary constraints can be enforced between the online student model and the frozen encoder to maintain semantic capabilities.

[0054] After determining the depth image corresponding to each frame of the target image, feature extraction is performed on each frame of the depth image to obtain a depth feature sequence of the steps in the region of interest. The depth feature sequence includes a depth feature value determined based on each frame of the depth image, and the depth feature sequence can characterize the height change of the same step in at least two consecutive frames of the target image.

[0055] Step 303: Determine the height features of the target steps in each frame of the target image to obtain a step height feature sequence, where the target step is the first step in the region of interest whose height is greater than zero.

[0056] In this step, a step detection algorithm can be used to detect a single step region in each frame of the target image, and a target step can be determined from all the detected single step regions, wherein the target step is the first step in the region of interest whose height is greater than zero. For example, when the escalator is in an upward operation state, the region of interest can be the lower side of the escalator, and the target step is the first step on the lower side of the escalator whose height is greater than zero.

[0057] Assuming that when the escalator is in operation, the height of the target step will change within a preset time period, therefore, a step height feature sequence can be determined based on the height features of the target step in each frame of the target image. The step height feature sequence can be used to characterize the height change of the target step in at least two consecutive frames of the target image.

[0058] Step 304: Based on the depth of field feature sequence and the step height feature sequence, perform abnormality detection on the escalator.

[0059] In this step, after determining the depth of field characteristic sequence and the step height characteristic sequence, the depth of field change of the step can be determined by analyzing the depth of field characteristic sequence, and the height change of the target step can be determined by analyzing the step height characteristic sequence. The obtained depth of field change and height change are compared with the depth of field change of the step during normal operation of the escalator and the height change of the target step during normal operation, so as to determine whether there is any abnormality in the escalator, for example, whether reversal, emergency stop or mechanical shaking occurs.

[0060] The escalator abnormality detection method provided by the embodiment of the present invention obtains at least two frames of target images including an area of ​​interest, and the area of ​​interest includes an entrance area or an exit area in the escalator, and extracts features from the depth of field images corresponding to each frame of the target image to obtain a depth of field feature sequence, and determines the height features of the target step in each frame of the target image to obtain a step height feature sequence, wherein the target step is the first step in the area of ​​interest whose height is greater than zero, and performs abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence. Since the operating state and operating direction of the escalator can be determined based on the depth of field image corresponding to the target image and the height change of the target step, the escalator can be detected abnormally, thereby avoiding the phenomenon in the prior art that the operating vector of the escalator needs to be determined by the optical flow method, and the accuracy of the abnormal state detection of the escalator is improved.

[0061] Exemplarily, based on the above embodiment, when acquiring at least two frames of target images including a region of interest, at least two frames of initial images including an escalator can be acquired, and for each frame of the initial image, the region of interest in the initial image can be determined based on the position of the apron plate of the escalator in the initial image, and the region of interest in the initial image can be extracted to obtain a target image corresponding to the initial image.

[0062] Specifically, Figure 1 and Figure 2 As shown, multiple frames of initial images including the escalator can be obtained by an image acquisition device, and the initial image can include the entire escalator or most of the escalator. In practical applications, whether the escalator is going up or down, usually only the height of the steps in the entrance area or exit area of ​​the escalator changes, and the height of the steps in other areas remains unchanged. In the escalator abnormality detection scheme provided in the embodiment of the present invention, it is based on the height change of the steps. Therefore, considering that the escalator area is long, in order to save the amount of calculation in subsequent image processing, the image can be extracted by the region of interest (ROI) sub-image extraction module, and the image of the region of interest containing the image feature range used is intercepted from the initial image, that is, the image containing the entrance or exit area of ​​the escalator. Specifically, when performing image interception, the position of the apron plate of the escalator can be used as a reference marker to determine the region of interest in each frame of the initial image, and the determined region of interest is extracted from the initial image to obtain the target image corresponding to each frame of the initial image.

[0063] In the above method, since the target image containing the region of interest is captured from each frame of the initial image, the region of interest contained in the target image is only a partial area of ​​the escalator. Therefore, the subsequent calculation amount can be greatly reduced and the efficiency of image quality detection is improved.

[0064] Exemplarily, when determining the area of ​​interest in the initial image based on the position of the apron plate of the escalator in the initial image, the first step position at the bottom position and the second step position at the top position of the apron plate in the length direction in the initial image can be determined, and the area between the first step position and the second step position in the initial image and including the entire step width can be determined as the area of ​​interest.

[0065] Specifically, Figure 4 The target image corresponding to the t-th moment provided by the embodiment of the present invention, Figure 5 The target image corresponding to the t+1th moment provided by the embodiment of the present invention is as follows: Figure 4 and Figure 5 As shown, taking the escalator going up and the region of interest being the region at the bottom entrance of the escalator as an example, after performing image recognition on each frame of the initial image and determining the position of the apron board in the initial image, the region of interest is determined based on the position of the apron board, and the target image is captured. Among them, the height direction of the target image needs to cover the step area range between the first step position at the bottom position of the apron board in the length direction and the second step position at the top position of the apron board, and the width direction of the target image covers the step width of each step.

[0066] In this embodiment, the position of the apron plate is used as a reference to quickly and accurately determine the region of interest, and the region of interest determined in the above manner includes the image feature range that can be used in subsequent escalator abnormality detection, making the region of interest more accurate.

[0067] Exemplarily, based on the above embodiment, when extracting features from the depth images corresponding to the target images of each frame to obtain a depth feature sequence, the following method can be used: For each depth of field image, the grayscale value of each pixel in the depth of field image is determined, based on the grayscale values ​​of all pixels in the depth of field image, at least one grayscale value feature of the depth of field image is determined, each grayscale value feature is normalized to obtain each normalized grayscale value feature, and based on the normalized grayscale value features corresponding to each depth of field image, a depth of field feature sequence is determined.

[0068] Specifically, Figure 6 The depth of field image corresponding to the target image at time t provided by the embodiment of the present invention, Figure 7 The depth of field image corresponding to the target image at time t+1 provided by the embodiment of the present invention. Figure 6 and Figure 7As shown, the depth of field image is a grayscale image. For each depth of field image, the grayscale value of each pixel in the depth of field image represents the distance between the actual object represented by the pixel and the image acquisition device. The larger the grayscale value, the farther the actual object represented by the pixel is from the image acquisition device. Conversely, the smaller the grayscale value, the closer the actual object represented by the pixel is to the image acquisition device. In addition, continue to refer to Figure 6 and Figure 7 As shown, within the same step area, the depth of field in the depth of field image at time t+1 is significantly greater than the depth of field image at time t, indicating that this part of the step area is far away from the image acquisition device, which means that the escalator is in an upward state.

[0069] Based on the above principle, the depth of field feature of the depth of field image can be determined by the grayscale value of each pixel in the depth of field image, so that the current state of the escalator can be judged based on the depth of field feature. Specifically, for a single-frame depth of field image, the grayscale value feature of the depth of field image can be determined based on the grayscale values ​​of all pixels in the depth of field image, and the grayscale value feature can be normalized. For example, the maximum grayscale value of the pixel in the depth of field image value range can be used for normalization, such as dividing the grayscale value feature by the maximum grayscale value of the pixel to obtain each normalized grayscale value feature. Furthermore, the normalized grayscale value features corresponding to each depth of field image can be combined to form a depth of field feature sequence. It should be noted that at least two consecutive frames of depth of field images need to ensure that the height of at least one step changes from 0 to the maximum height value.

[0070] In this embodiment, at least one grayscale value feature of the depth of field image is determined by the grayscale value of each pixel in each depth of field image, and each grayscale value feature is normalized, so as to determine the depth of field feature sequence based on the normalized grayscale value features. Since the normalization process can normalize the data to the same interval, the comparability of the data is improved.

[0071] Exemplarily, the at least one grayscale value feature mentioned above includes a maximum grayscale value feature, a minimum grayscale value feature, an average grayscale value feature and a median grayscale value feature. When determining the depth feature sequence based on the normalized grayscale value features corresponding to each depth of field image, the depth feature value of the depth image can be determined based on the normalized maximum grayscale value feature, the normalized minimum grayscale value feature, the normalized average grayscale value feature and the normalized median grayscale value feature corresponding to the depth image, and the depth feature values ​​of each depth of field image are combined to obtain the depth feature sequence.

[0072] Specifically, for a single-frame depth of field image, the maximum grayscale value feature, the minimum grayscale value feature, the average grayscale value feature and the median grayscale value feature can be determined based on the grayscale values ​​of all pixels in the depth of field image, and the maximum grayscale value feature, the minimum grayscale value feature, the average grayscale value feature and the median grayscale value feature are respectively divided by the maximum grayscale value of the pixel in the depth of field image value range to obtain the normalized maximum grayscale value feature DMax, the normalized minimum grayscale value feature DMin, the normalized average grayscale value feature DA and the normalized median grayscale value feature Dmid.

[0073] For example, the depth feature value of the depth image can be determined based on the following formula (1): (1)

[0074] in, represents the depth of field eigenvalue, It represents the correction factor, usually taking a value between 0.01 and 0.1. .

[0075] The depth of field characteristic value of the depth of field image can be calculated quickly and accurately by using the above formula (1).

[0076] After the depth feature value of each frame of the depth image is determined in the above manner, the depth feature values ​​of all the depth images are combined according to the acquisition time sequence of the depth images to obtain a depth feature sequence.

[0077] In this embodiment, since the depth of field feature value can be calculated based on the four features corresponding to the depth of field image, namely the normalized maximum grayscale value feature, the normalized minimum grayscale value feature, the normalized average grayscale value feature and the normalized median grayscale value feature, the error problem caused by using a single grayscale value feature to calculate the depth of field feature value is avoided, and the calculation accuracy of the depth of field feature value is improved.

[0078] Further, based on the above embodiments, before determining the height features of the target steps in the target image, it is also necessary to first determine the target steps from the target image. Exemplarily, the target image can be input into a step detection model to obtain the target steps output by the step detection model; the step detection model is obtained by training an initial step detection model based on multiple image samples including escalator samples and labels of step samples in each image sample used to characterize the escalator samples.

[0079] Specifically, in this embodiment, the step detection adopts a target detection algorithm based on a convolutional neural network. Among them, by pre-collecting images of multiple escalator samples, multiple image samples including escalator samples are obtained, and the labels of the step samples representing the escalator samples are marked in each image sample. Each image sample is input into the initial step detection model, and the predicted step output by the initial step detection model can be obtained. After the loss information is determined based on the predicted step and the above-mentioned label, the model parameters of the initial step detection model can be adjusted according to the loss information. By continuously repeating the above process until the number of repetitions reaches a preset number or the obtained model converges, the final model is determined as the step detection model. Among them, the backbone network responsible for extracting features in the step detection model adopts an improved cross-stage partial network (CSPNet), which can optimize gradient propagation; the neck network (Neck) integrates a path aggregation network layer (Path Aggregation Network, PAN), which realizes the effective fusion of multi-scale features. When the step detection model is trained and step detection is performed based on the step detection model, a single best prediction result can be generated for each step, and a non-maximum suppression module (NMS) is removed.

[0080] Figure 8 A schematic diagram of step detection provided by an embodiment of the present invention, such as Figure 8 As shown, by inputting the target image into the step detection model, each step in the target image can be obtained, such as Figure 8 As shown in the yellow box in the figure. After selecting each step, the first step with a step height greater than zero is determined from all the steps as the target step. The target step is the longitudinal surface area of ​​the first step at the bottom of the escalator with a step height. It can also be understood as the first step detected from bottom to top in the escalator area, or the first step at the bottom.

[0081] In this embodiment, the target image is input into the step detection model to obtain each step in the target image. This method can improve the speed and accuracy of step determination.

[0082] Exemplarily, on the basis of the above-mentioned embodiments, when determining the height features of the target step in each frame of the target image to obtain a step height feature sequence, the first line segment and the second line segment at the edge of the height plane of the target step in the target image can be determined for each frame of the target image, and the extension direction of the first line segment and the extension direction of the second line segment are both perpendicular to the running direction of the escalator. The distance between the first line segment and the second line segment is normalized to obtain the normalized distance, and then the normalized distance is determined as the height feature of the target step in the target image, and the height features of the target step in each target image are combined to obtain a step height feature sequence.

[0083] Specifically, for each frame of the target image, after determining the target step in the target image, it is necessary to use a straight line detection algorithm within the target step, such as the Hough transform algorithm, to detect the first line segment and the second line segment at the edge of the height plane of the target step. Figure 8 The red line segments shown in are the first line segment and the second line segment. The extending directions of the first line segment and the second line segment are both perpendicular to the running direction of the escalator.

[0084] Furthermore, the distance between the first line segment and the second line segment in the vertical plane can be determined to obtain the step height of the target step. The distance is normalized. For example, if the maximum height of the step is fixed when it is normally raised, the distance can be divided by the maximum height of the known step to obtain the normalized distance, forming the height feature of the target step in the single-frame target image.

[0085] After obtaining the height features of the target steps in each frame of the target image, all the height features are combined according to the acquisition time of the target image to obtain the step height feature sequence. It should be understood that the step height feature sequence can be used to characterize the height change of the target steps in multiple frames of the target image.

[0086] In this embodiment, by determining the first line segment and the second line segment of the target step in the target image, and determining the height of the target step based on the first line segment and the second line segment, the height of the target step determined is relatively accurate. In addition, by combining the height features of the target steps in all target images, escalator anomaly detection is performed based on the step height feature sequence obtained by the combination that can characterize the change in the height of the target step, the accuracy of escalator anomaly detection can be improved.

[0087] In one possible implementation, when performing abnormality detection on an escalator based on a depth of field feature sequence and a step height feature sequence, the current operating state of the escalator can be determined according to the variation pattern of the depth of field feature value in the depth of field feature sequence and the step height in the step height feature sequence, thereby comparing the current operating state with the preset state to determine whether an abnormality has occurred in the escalator.

[0088] Specifically, Fig. 9 A schematic diagram of the change of the depth of field feature sequence when the escalator goes up provided by the embodiment of the present invention, such as Fig. 9 As shown, when the escalator goes up, the depth of field feature values ​​in the depth of field feature sequence will gradually decrease. Fig.10 A schematic diagram of the change of the depth of field feature sequence when the escalator goes down provided by the embodiment of the present invention, as shown in FIG. Fig.10 As shown, when the escalator goes down, the depth of field feature values ​​in the depth of field feature sequence will gradually increase. Fig.11A schematic diagram of a change in a depth of field feature sequence when an escalator stops provided by an embodiment of the present invention, as shown in FIG. Fig.11 As shown, when the escalator stops, each depth of field feature value in the depth of field feature sequence remains unchanged. Fig.12 A schematic diagram of the change of the depth of field feature sequence when a mechanical abnormality occurs in an escalator provided by an embodiment of the present invention, such as Fig.12 As shown, when a mechanical abnormality occurs in the escalator, the depth of field feature values ​​in the depth of field feature sequence will oscillate.

[0089] Fig.13 A schematic diagram of the change of the step height characteristic sequence when the escalator goes up provided by the embodiment of the present invention, such as Fig.13 As shown, when the escalator goes up, the escalator height in the step height characteristic sequence will gradually increase. Fig.14 A schematic diagram of the change of the step height characteristic sequence when the escalator is descending provided by an embodiment of the present invention, as shown in FIG. Fig.14 As shown, when the escalator goes down, the escalator height in the step height characteristic sequence will gradually decrease. Fig.15 A schematic diagram of a change in a characteristic sequence of step heights when an escalator stops provided by an embodiment of the present invention is shown in FIG. Fig.15 As shown, when the escalator stops, the escalator height in the step height characteristic sequence remains unchanged. Fig.16 A schematic diagram of the change of the step height characteristic sequence when a mechanical abnormality occurs in the escalator provided by the embodiment of the present invention, such as Fig.16 As shown, when a mechanical abnormality occurs in the escalator, the height of each escalator in the step height characteristic sequence will oscillate.

[0090] In another possible implementation, the depth of field feature sequence and the step height feature sequence can also be input into a time feature sequence classifier to obtain the current escalator state output by the time feature sequence classifier, and based on the current escalator state and a preset state, the escalator is detected for abnormalities, and the preset state is the state in which the escalator is expected to be.

[0091] Specifically, the depth of field feature sequence and the step height feature sequence are input into the time feature sequence classifier for classification, and the current escalator state, such as escalator up, escalator down, or escalator stopped, can be obtained. The time feature sequence classifier can be, for example, a long short-term memory network (LSTM) classification model, which contains 100 hidden units, the output layer is a fully connected layer of 3 categories, and finally a softmax layer and a classification layer, and the loss function uses classification cross entropy.

[0092] After determining the current escalator state, the current escalator state can be compared with the preset state corresponding to the escalator. If they are inconsistent, it means that the escalator has an abnormality. The preset state of the escalator is the state in which the escalator is expected to be, that is, the state in which the escalator is when no abnormality occurs.

[0093] In this embodiment, the depth of field feature sequence and the step height feature sequence can be input into the time feature sequence classifier. The current escalator state analyzed by the time feature sequence classifier can be determined in combination with the depth of field features and height features of the escalator in the past period of time, so that the result of the current escalator state is more accurate.

[0094] For example, when an escalator is detected to be abnormal based on the current escalator state and the preset state, the following situations are included: When the current escalator state is in the running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, the escalator is determined to be abnormal. When the current escalator state is in the stopped state and the preset state indicates that the escalator is in the running state, the escalator is determined to be abnormal. When the current escalator state is in the running state and the running acceleration is different from the acceleration of the escalator in the normal running state corresponding to the preset state, the escalator is determined to be abnormal.

[0095] Specifically, Fig.17 A schematic diagram of an escalator abnormal state detection provided by an embodiment of the present invention, such as Fig.17 As shown, the depth of field feature sequence and the step height feature sequence are input into the LSTM classification model for classification, and the current escalator state can be obtained, such as escalator up, escalator down, or escalator stopped. If the current escalator state is running, the running direction is compared with the running direction set for the escalator corresponding to the preset state. If they are inconsistent, it means that the escalator has reversed, and the escalator can be determined to be abnormal. For example, if the current escalator state is escalator up, and the running direction set for the escalator is down, it means that the escalator is reversed.

[0096] If the current escalator state is a stopped state, and the preset state indicates that the escalator is currently set to a running state, it means that the escalator has currently stopped abnormally, and it can be determined that the escalator is abnormal.

[0097] In addition, it should be understood that when the escalator is operating normally, it generally moves at a uniform speed, and the corresponding acceleration should be 0. When the escalator has abnormal shaking or mechanical abnormalities, it will generate upward or downward acceleration, and the acceleration will continue to increase or increase and decrease, and the acceleration direction may also change in the opposite direction. Therefore, if the current escalator state is in the running state, and the running acceleration is different from the acceleration of the escalator in normal movement corresponding to the preset state, it means that the escalator has abnormal shaking or mechanical abnormalities, and the escalator can be determined to be abnormal. Among them, the acceleration of the escalator in normal movement corresponding to the preset state is 0, and the running acceleration of the current escalator is different from the acceleration of the escalator in normal movement corresponding to the preset state, which may include different accelerations, and / or different directions.

[0098] In this embodiment, it is only necessary to compare the current escalator state with the preset state. When the current escalator state is inconsistent with the preset state, it can be determined that an abnormality has occurred in the escalator, making the escalator abnormality detection method simpler and faster.

[0099] Next, in combination with the solutions in the aforementioned embodiments, the overall solution for escalator abnormality detection provided by the embodiments of the present invention is introduced.

[0100] Fig.18 A schematic diagram of escalator abnormality detection provided by an embodiment of the present invention, such as Fig.18 As shown, after acquiring an initial image sequence including an escalator, a region of interest is extracted from at least two frames of the initial image in the initial image sequence according to a set ROI region to obtain a target image. Based on the obtained at least two frames of the target image, a depth of field image corresponding to each frame of the target image can be generated, and the depth of field features in each frame of the depth of field image can be extracted. After the depth of field features in each frame of the depth of field image are normalized, a depth of field feature value is determined based on the normalized depth of field features, and the depth of field feature values ​​in multiple frames of the depth of field image are combined according to the acquisition time sequence of the depth of field images to obtain a depth of field feature sequence corresponding to the multiple frames of the depth of field image.

[0101] In addition, after performing step detection on each frame of target image, the target step can be determined from the detected steps, and straight line detection can be performed on the target step. The step height of the target step can be determined based on the difference between two straight lines detected in the target step, and the normalized step height can be determined as the height feature of the target step. The height features in multiple frames of target images are combined according to the acquisition time sequence of the target images to obtain the step height feature sequence corresponding to the multiple frames of target images.

[0102] Furthermore, by inputting the depth of field feature sequence and the step height feature sequence into the time feature sequence classifier, the current escalator state output by the time feature sequence classifier can be obtained, and the current escalator state is compared with the preset state of the expected escalator to determine whether there is any abnormality in the escalator, such as whether it reverses, stops abnormally, or shakes.

[0103] The escalator abnormality detection method provided by the embodiment of the present invention obtains at least two frames of target images including an area of ​​interest, and the area of ​​interest includes an entrance area or an exit area in the escalator, and extracts features from the depth of field images corresponding to each frame of the target image to obtain a depth of field feature sequence, and determines the height features of the target step in each frame of the target image to obtain a step height feature sequence, wherein the target step is the first step in the area of ​​interest whose height is greater than zero, and performs abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence. Since the operating state and operating direction of the escalator can be determined based on the depth of field image corresponding to the target image and the height change of the target step, the escalator can be detected abnormally, thereby avoiding the phenomenon in the prior art that the operating vector of the escalator needs to be determined by the optical flow method, and the accuracy of the abnormal state detection of the escalator is improved.

[0104] The escalator abnormality detection device provided by an embodiment of the present invention is described below. The escalator abnormality detection device described below and the escalator abnormality detection method described above can be referred to each other.

[0105] Fig.19 The schematic diagram of the structure of the escalator abnormality detection device provided by the embodiment of the present invention is shown in FIG. Fig.19 As shown, the escalator abnormality detection device 1900 includes: An acquisition module 11 is used to acquire at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; An extraction module 12 is used to extract features of the depth of field images corresponding to the target image in each frame to obtain a depth of field feature sequence; A determination module 13 is used to determine the height feature of the target step in each frame of the target image to obtain a step height feature sequence, wherein the target step is the first step in the region of interest whose step height is greater than zero; The detection module 14 is used to perform abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence.

[0106] In an exemplary embodiment, the extraction module 12 is specifically configured to: For each of the depth of field images, determining the grayscale value of each pixel in the depth of field image; Determining at least one grayscale value feature of the depth of field image based on the grayscale values ​​of all pixels in the depth of field image; Normalizing each of the grayscale value features to obtain each normalized grayscale value feature; The depth of field feature sequence is determined based on the normalized grayscale value features corresponding to each of the depth of field images.

[0107] In an example embodiment, the at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature, and a median gray value feature; The extraction module 12 is specifically used for: Determining a depth feature value of the depth image based on a normalized maximum gray value feature, a normalized minimum gray value feature, a normalized average gray value feature, and a normalized median gray value feature corresponding to the depth image; The depth-of-field feature values ​​of the depth-of-field images are combined to obtain the depth-of-field feature sequence.

[0108] In an exemplary embodiment, the extraction module 12 is specifically configured to: The depth characteristic value of the depth image is determined based on the following formula (1): (1)

[0109] in, represents the depth of field feature value, DA represents the normalized average grayscale value feature, DMid represents the normalized median grayscale value feature, Dmax represents the normalized maximum grayscale value feature, and Dmin represents the normalized minimum grayscale value feature. represents the correction factor, .

[0110] In an exemplary embodiment, the determination module 13 is specifically configured to: For each frame of the target image, determine a first line segment and a second line segment at an edge of a height plane of the target step in the target image, wherein an extension direction of the first line segment and an extension direction of the second line segment are both perpendicular to a running direction of the escalator; Normalizing the distance between the first line segment and the second line segment to obtain a normalized distance; Determine the normalized distance as a height feature of the target step in the target image; The height features of the target steps in each of the target images are combined to obtain the step height feature sequence.

[0111] In an exemplary embodiment, the detection module 14 is specifically configured to: Inputting the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier to obtain a current escalator state output by the time feature sequence classifier; Based on the current escalator state and a preset state, an abnormality detection is performed on the escalator, wherein the preset state is a state in which the escalator is expected to be.

[0112] In an exemplary embodiment, the detection module 14 is specifically configured to: When the current escalator state is a running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determining that the escalator is abnormal; When the current escalator state is a stopped state and the preset state indicates that the escalator is in a running state, determining that the escalator is abnormal; When the current escalator state is a running state and the running acceleration is different from the acceleration of the escalator during normal operation corresponding to the preset state, it is determined that the escalator is abnormal.

[0113] In an exemplary embodiment, the acquisition module 11 is specifically configured to: Acquire at least two frames of initial images including the escalator; For each frame of the initial image, based on the position of the apron plate of the escalator in the initial image, determining the region of interest in the initial image; The region of interest in the initial image is extracted to obtain a target image corresponding to the initial image.

[0114] In an exemplary embodiment, the determination module 13 is specifically configured to: Determine a first step position at a bottom end position and a second step position at a top end position of the apron plate in the length direction in the initial image; An area between the first step position and the second step position in the initial image and including the entire step width is determined as the region of interest.

[0115] In an exemplary embodiment, the apparatus further comprises: an input module, wherein: An input module is used to input the target image into a step detection model to obtain the target step output by the step detection model; the step detection model is obtained by training an initial step detection model based on multiple image samples including escalator samples and labels in each of the image samples for characterizing step samples in the escalator samples.

[0116] The device of this embodiment can be used to execute the method of any one of the embodiments in the escalator abnormality detection method side embodiments. Its specific implementation process and technical effects are similar to those in the escalator abnormality detection method side embodiments. For details, please refer to the detailed introduction in the escalator abnormality detection method side embodiments, which will not be repeated here.

[0117] Fig. 20 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Fig. 20 As shown, the electronic device may include: a processor 2010, a communication interface 2020, a memory 2030 and a communication bus 2040, wherein the processor 2010, the communication interface 2020 and the memory 2030 communicate with each other through the communication bus 2040. The processor 2010 may call the logic instructions in the memory 2030 to execute the escalator abnormality detection method, which includes: acquiring at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; performing feature extraction on the depth of field images corresponding to each frame of the target image to obtain a depth of field feature sequence; determining the height feature of the target step in each frame of the target image to obtain a step height feature sequence, wherein the target step is the first step in the area of ​​interest whose step height is greater than zero; and performing abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence.

[0118] In addition, the logic instructions in the above-mentioned memory 2030 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the escalator abnormality detection method provided by the above-mentioned methods, which includes: acquiring at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; performing feature extraction on the depth of field images corresponding to the target images of each frame to obtain a depth of field feature sequence; determining the height features of the target step in the target images of each frame to obtain a step height feature sequence, wherein the target step is the first step in the area of ​​interest where the step height is greater than zero; and performing abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence.

[0120] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the escalator abnormality detection method provided by the above-mentioned methods, the method comprising: acquiring at least two frames of target images including an area of ​​interest, the area of ​​interest including an entrance area or an exit area in the escalator; performing feature extraction on depth of field images corresponding to the target images of each frame to obtain a depth of field feature sequence; determining the height features of the target step in the target images of each frame to obtain a step height feature sequence, the target step being the first step in the area of ​​interest whose step height is greater than zero; and performing abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting an escalator abnormality, characterized in that: include: Acquire at least two frames of target images including an area of ​​interest, wherein the area of ​​interest includes an entrance area or an exit area in the escalator; Extracting features of the depth of field images corresponding to the target image in each frame to obtain a depth of field feature sequence; Determine the height feature of the target step in the target image of each frame to obtain a step height feature sequence, wherein the target step is the first step in the region of interest whose step height is greater than zero; Based on the depth of field feature sequence and the step height feature sequence, abnormality detection is performed on the escalator.

2. The escalator abnormality detection method according to claim 1, characterized in that: The extracting features of the depth of field images corresponding to the target images of each frame to obtain a depth of field feature sequence comprises: For each of the depth of field images, determining the grayscale value of each pixel in the depth of field image; Determining at least one grayscale value feature of the depth of field image based on the grayscale values ​​of all pixels in the depth of field image; Normalizing each of the grayscale value features to obtain each normalized grayscale value feature; The depth of field feature sequence is determined based on the normalized grayscale value features corresponding to each of the depth of field images.

3. The escalator abnormality detection method according to claim 2, characterized in that: The at least one gray value feature includes a maximum gray value feature, a minimum gray value feature, an average gray value feature and a median gray value feature; The determining the depth of field feature sequence based on the normalized grayscale value features corresponding to each of the depth of field images comprises: Determining a depth feature value of the depth image based on a normalized maximum gray value feature, a normalized minimum gray value feature, a normalized average gray value feature, and a normalized median gray value feature corresponding to the depth image; The depth-of-field feature values ​​of the depth-of-field images are combined to obtain the depth-of-field feature sequence.

4. The escalator abnormality detection method according to claim 3, characterized in that: Determining the depth feature value of the depth image based on the normalized maximum gray value feature, the normalized minimum gray value feature, the normalized average gray value feature, and the normalized median gray value feature corresponding to the depth image includes: The depth characteristic value of the depth image is determined based on the following formula (1): （1） in, represents the depth of field feature value, DA represents the normalized average grayscale value feature, DMid represents the normalized median grayscale value feature, Dmax represents the normalized maximum grayscale value feature, and Dmin represents the normalized minimum grayscale value feature. represents the correction factor, .

5. The escalator abnormality detection method according to claim 1, characterized in that: The step of determining the height features of the target steps in the target image of each frame to obtain a step height feature sequence includes: For each frame of the target image, determine a first line segment and a second line segment at an edge of a height plane of the target step in the target image, wherein an extension direction of the first line segment and an extension direction of the second line segment are both perpendicular to a running direction of the escalator; Normalizing the distance between the first line segment and the second line segment to obtain a normalized distance; Determine the normalized distance as a height feature of the target step in the target image; The height features of the target steps in each of the target images are combined to obtain the step height feature sequence.

6. The escalator abnormality detection method according to claim 1, characterized in that: The performing abnormality detection on the escalator based on the depth of field feature sequence and the step height feature sequence comprises: Inputting the depth of field feature sequence and the step height feature sequence into a time feature sequence classifier to obtain a current escalator state output by the time feature sequence classifier; Based on the current escalator state and a preset state, an abnormality detection is performed on the escalator, wherein the preset state is a state in which the escalator is expected to be.

7. The escalator abnormality detection method according to claim 6, characterized in that: The performing abnormality detection on the escalator based on the current escalator state and the preset state includes: When the current escalator state is a running state and the running direction is opposite to the normal running direction of the escalator corresponding to the preset state, determining that the escalator is abnormal; When the current escalator state is a stopped state and the preset state indicates that the escalator is in a running state, determining that the escalator is abnormal; When the current escalator state is a running state and the running acceleration is different from the acceleration of the escalator during normal operation corresponding to the preset state, it is determined that the escalator is abnormal.

8. The escalator abnormality detection method according to any one of claims 1 to 7, characterized in that: The acquiring of at least two frames of target images including the region of interest comprises: Acquire at least two frames of initial images including the escalator; For each frame of the initial image, based on the position of the apron plate of the escalator in the initial image, determining the region of interest in the initial image; The region of interest in the initial image is extracted to obtain a target image corresponding to the initial image.

9. The escalator abnormality detection method according to claim 8, characterized in that: The determining the region of interest in the initial image based on the position of the apron plate of the escalator in the initial image comprises: Determine a first step position at a bottom end position and a second step position at a top end position of the apron plate in the length direction in the initial image; An area between the first step position and the second step position in the initial image and including the entire step width is determined as the region of interest.

10. The escalator abnormality detection method according to any one of claims 1 to 7, characterized in that: The method further comprises: The target image is input into a step detection model to obtain the target step output by the step detection model; the step detection model is obtained by training an initial step detection model based on multiple image samples including escalator samples and labels in each of the image samples for characterizing step samples in the escalator samples.

Images