Elevator door state detection method and device and medium
Through the combination of edge detection algorithm, image morphology and Hough transformation technology, the interference problem in elevator door state detection is solved, the detection accuracy and reliability are improved, and it is suitable for a variety of elevator brands and scenarios.
Patent Information
- Application Number
- CN202411007208.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-11-12
AI Technical Summary
There are a variety of interference information in the elevator door state detection, including external scene differences, lighting changes, light reflections and special textures, which lead to misidentification of the detection algorithm and reduced accuracy, affecting the detection accuracy and reliability.
The edge detection algorithm is used to combine image morphology and Hough transformation technology to obtain the opening and closing status of the elevator door through filtering processing and interference information exclusion strategies, and improve the algorithm's anti-interference ability.
Effectively reduce irrelevant edge interference, improve the accuracy and reliability of elevator door status detection, and achieve good detection results. It is suitable for different elevator brands and usage scenarios.
Smart Images

Figure CN118942009A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of elevator door state detection, and in particular to an elevator door state detection method, device and medium. Background Art
[0002] Elevator brands vary, especially in different elevator usage scenarios. The video images of the elevator doors are inevitably interfered by various factors. In the process of image processing of elevator doors, in order to achieve accurate elevator door status recognition, it is necessary to focus on the interference information of different types of elevators in different application scenarios. Through the analysis of multiple elevator monitoring videos, the following interference information in the elevator door status detection process is summarized:
[0003] 1. Changeable external scenes of elevators. Due to different installation scenarios of elevator doors, the external scenes of elevator doors are different. The door gap is the foreground information to be extracted, and the background information after the car door is opened is different. For example, after the elevator door is opened, the anti-theft door opposite is another user's. The edge features of the external door are easily confused, causing interference to the detection of the elevator door. The existence of objects with obvious straight line features such as walls in the external scene will also cause the algorithm to misidentify to a large extent, and make wrong conclusions about the elevator door status.
[0004] 2. Changes in lighting when the door is opened and closed. There are differences in the light inside and outside the elevator door, and the light intensity is inconsistent. When the elevator door is opened, the light changes suddenly. Some elevators are limited by the performance of monitoring equipment, and the imaging effect is poor, which will cause the picture to lose details and present an overexposed video effect, affecting the algorithm detection effect.
[0005] 3. Light reflection from the elevator door. The elevator door is usually made of thin steel plates or stainless steel. After polishing and subsequent processing and decoration, the steel material usually has a metallic luster and can reflect the elevator light. The reflected light will form a texture on the elevator door, which interferes with the detection of the elevator door.
[0006] 4. Special textures on elevator doors. In addition to being made of steel, some elevator doors will be decorated later to meet the design requirements of the elevator. This objectively causes these elevators to have special textures, which reduces the accuracy of the algorithm and increases misrecognition.
[0007] How to solve the interference problem in the above-mentioned door status detection process and improve the accuracy and reliability of the door status detection algorithm is crucial for the elevator door status monitoring system. Summary of the invention
[0008] The present invention provides an elevator door state detection method, device and medium, aiming to solve the above-mentioned problem.
[0009] An embodiment of the present invention provides an elevator door state detection method, comprising:
[0010] S1. Obtain elevator surveillance video, and obtain the area to be detected after image processing of the surveillance video;
[0011] S2, using edge detection algorithm to obtain edge features in the area to be detected;
[0012] S3, using an edge image processing method based on image morphology to filter the interference information of edge features to obtain straight line segment edge information;
[0013] S4, using a straight line detection algorithm based on Hough transform to obtain the door edge position information for the straight line segment edge information;
[0014] S5. Adopt an interference information elimination strategy for the door edge position to eliminate interference information and obtain the elevator door opening and closing status.
[0015] An embodiment of the present invention provides an electronic device, including:
[0016] processor; and,
[0017] A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the elevator door state detection method as described above.
[0018] An embodiment of the present invention provides a storage medium for storing computer executable instructions, wherein the computer executable instructions implement the steps of the elevator door state detection method as described above when executed.
[0019] In order to reduce the interference of irrelevant edges, the embodiment of the present invention adopts the opening operation in image morphology, and realizes the filtering operation of edge image through custom structure elements; the key information of straight line edge, the length, inclination, and relative position of the straight line are obtained in turn by using the straight line detection algorithm based on Hough transform; through the analysis of the key information of the straight line edge, the strategy of eliminating the interference edge information is proposed, and the anti-interference ability of the algorithm is improved; finally, the opening and closing state of the elevator door is obtained by fusing multiple frames of information, and a good detection effect is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0021] Figure 1 Flow chart of a method for detecting an elevator door state according to an embodiment of the present invention;
[0022] Figure 2 The original image of the elevator door area according to the embodiment of the present invention;
[0023] Figure 3 Schematic diagram of edge detection using the Prewitt operator according to an embodiment of the present invention;
[0024] Figure 4 Schematic diagram of edge detection using a Sobel operator according to an embodiment of the present invention;
[0025] Figure 5 Schematic diagram of Scharr operator edge detection according to an embodiment of the present invention;
[0026] Figure 6 A schematic diagram of edge detection using a Laplacian operator according to an embodiment of the present invention;
[0027] Figure 7 Schematic diagram of structural elements selected for embodiments of the present invention;
[0028] Figure 8 A filtering schematic diagram of the opening and closing operation of an embodiment of the present invention;
[0029] Fig. 9 is a schematic diagram of morphological filtering according to an embodiment of the present invention;
[0030] Fig.10 This is a diagram showing the detection effects of Hough space and Hough line in an embodiment of the present invention;
[0031] Fig.11 A detection result diagram of the optical flow in a video using the Horn-Schunck optical flow algorithm according to an embodiment of the present invention;
[0032] Fig.12 This is a detection result diagram of the optical flow in the video using the Farneback optical flow algorithm according to an embodiment of the present invention;
[0033] Fig.13 It is a diagram of the algorithm implementation process of the pyramid LK optical flow method according to an embodiment of the present invention;
[0034] Fig.14 This is a diagram showing the experimental results of the elevator door status detection algorithm No. 1-3 according to an embodiment of the present invention;
[0035] Fig.15 This is a diagram showing the experimental results of the No. 4-6 elevator door state detection algorithm according to an embodiment of the present invention;
[0036] Fig.16 This is a diagram of the experimental results of the elevator door status detection algorithm No. 7-9 according to an embodiment of the present invention. DETAILED DESCRIPTION
[0037] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.
[0038] Method Embodiment
[0039] According to an embodiment of the present invention, a method for detecting the state of an elevator door is provided. Figure 1 is a flow chart of the elevator door state detection method according to an embodiment of the present invention. Figure 1 As shown, the elevator door state detection method of the embodiment of the present invention specifically includes:
[0040] S1. Obtain elevator surveillance video, and obtain the area to be detected after image processing of the surveillance video;
[0041] Due to the shooting angle of the elevator surveillance video, the cabin scene recorded by the surveillance video will be tilted as a whole. In addition, since the edge detection algorithm will be applied in the door edge detection process, some edge detection operators will have vertical convolution templates. In order to achieve better detection results, the elevator door area needs to be rotated. When only the elevator door status is detected, the aforementioned image ROI can be used to select only the image of the appropriate area as the area to be detected for edge detection, thereby achieving good detection results and speeding up the calculation speed. The elevator surveillance video is a color image and needs to be grayed to facilitate the subsequent application of the edge detection algorithm.
[0042] Image filtering is also very important. In the elevator scene, in the embodiment of the present invention, Gaussian filtering with a 3×3 template is used.
[0043] S2, using edge detection algorithm to obtain edge features in the area to be detected;
[0044] Through in-depth research on elevator videos, we have sorted out typical elevator doors. The original image of a typical elevator door area is as follows: Figure 2 As shown in the figure, there are various interference information, including interference from the environment outside the door, changes in illumination during the door opening and closing process, reflection problems of floor doors, and different texture problems on the floor doors. By conducting edge detection tests on typical elevators in the figure and analyzing the edge detection effects, we can select edge detection operators suitable for elevator scenarios.
[0045] Four different edge detection operators are used in the embodiment of the present invention to verify their effectiveness, as follows:
[0046] 1. Prewitt operator
[0047] The Prewitt operator uses 3×3 templates in the x-axis and y-axis directions to calculate edge features in the horizontal and vertical directions respectively. This operator template has a certain anti-interference performance and can remove some false edges. In the Prewitt operator, the convolution templates of the x-axis and y-axis are:
[0048]
[0049] Among them, in the horizontal and vertical directions, the expressions are:
[0050] G x (x,y)=f(x+1,y-1)+f(x+1,y)+f(x+1,y+1)-f(x-1,y-1)-f(x-1,y)-f(x-1,y+1) Formula 3;
[0051] G y (x,y)=f(x-1,y+1)+f(x,y+1)+f(x+1,y+1)-f(x-1,y-1)-f(x,y-1)-f(x+1,y-1) Formula 4;
[0052] The approximate partial derivative is obtained by using Formula 3 and Formula 4. The gradient amplitude can be obtained by combining the calculation result with the gradient amplitude of the target pixel point, and compared with the set threshold T to obtain the target edge. Figure 3 FIG. 1 is a schematic diagram showing the effect of the Prewitt operator on door edge feature detection according to an embodiment of the present invention.
[0053] In the right Figure 2 After the above image preprocessing algorithm is used in the elevator door area shown, the edge detection is performed using the Prewitt operator on the preprocessed image. The detection effect of the Prewitt operator is shown in the figure below. Figure 3 As shown in the figure. It can be seen that for elevators 1, 2, 7, and 9, the edge features caused by patterns or lighting on the elevator doors are also detected by the Prewit operator; for elevators 1, 4, 5, and 6, there are people or other objects outside the elevator after the elevator door opens, and these background features are also captured by the operator. Overall, the Prewitt operator has a strong edge feature detection performance. However, for the elevator door state detection algorithm, a good door edge detection algorithm should minimize the detection of irrelevant edges while retaining clear door edges, so as to reduce the subsequent filtering of the image.
[0054] 2. Sobel operator
[0055] The Sobel operator combines the Gaussian smoothing idea with the differential derivation method. By performing weighted average calculation on the pixel values in the neighborhood of the desired pixel, it has good anti-interference performance and can reduce the impact of noise. The closer the neighborhood pixel is to the center pixel, the greater the correlation weight, and the greater the effect of the convolution template on the desired pixel. For example, if a 3×3 template is used, with the target pixel as the center pixel, its gradient on the x-axis and y-axis is:
[0056] G x (x,y)=f(x-1,y+1)+2f(x,y+1)+f(x+1,y+1)-f(x-1,y-1)-2f(x,y-1)-f(x+1,y-1);
[0057] G y (x,y)=f(x+1,y-1)+2f(x+1,y)+f(x+1,y+1)-f(x-1,y-1)-2f(x-1,y)-f(x-1,y+1);
[0058] For G x (x, y) and G y (x, y), the convolution templates taken are:
[0059]
[0060] The Sobel operator is calculated by H x With H y After convolution calculation is performed on the target pixel, the sum is calculated, and the gradient amplitude is compared with the preset threshold. Only when it is greater than the threshold, the calculation result is replaced with the original center pixel value, otherwise the center point is assigned to zero. Similar to the above, the Sobel operator is used to locate the edge of the elevator door on the preprocessed image. The detection effect is shown in the figure below. Figure 4 As shown. The Sobel operator has two types of templates, horizontal and vertical. The corresponding horizontal and vertical edge features can be calculated through the two direction templates. In the elevator door edge feature detection scenario, choosing to use the vertical edge can greatly reduce the influence of the background edge in the horizontal direction. It can be seen that compared with the Prewitt operator, in the areas of elevator doors 1, 2, 3, 4, 7, 8, and 9, the irrelevant features on the doors are significantly reduced; in elevators 5 and 6, although the outline of the person is still detected, compared with the Prewitt operator, there is less adhesion between the door edges and between the door edges and the background edges, and the door edges are better positioned. At the same time, there is incomplete detection of the right door edge of elevators 2 and 4.
[0061] 3. Scharr operator
[0062] The Scharr operator is similar to the Sobel operator, both are first-order differential operators, have good operation speed and are sensitive to changes in grayscale values. The Scharr operator uses a method of amplifying weight coefficients to enhance pixel differences and improve the detection effect accordingly. The templates in the horizontal and vertical directions are as follows:
[0063]
[0064] The Scharr operator is used to detect the preprocessed image. The detection effect is as follows Figure 5 As shown in the figure. In the elevator scenario, the overall detection effect of the Scharr operator is similar to that of the Prewitt operator. It has better edge detection capability, but is too sensitive to non-door edge features, resulting in too many noise edges, which require further processing by subsequent algorithms. Excessive noise will reduce the accuracy of edge detection.
[0065] 4. Laplacian operator
[0066] The Laplacian operator is a second-order differential operator. It uses the property that the second-order derivative forms a zero point at the edge point for detection. The basic idea is to compare the central pixel and the surrounding pixels. If the central pixel has a larger gray value, the gray value of the central pixel is increased, otherwise it is decreased. It does not need to calculate in different directions, but performs convolution operation on the target image through the Laplacian kernel.
[0067] The one-dimensional first-order difference and second-order difference formula are:
[0068]
[0069] The second-order difference formula of f(x, y) in the x-axis direction and the y-axis direction is:
[0070]
[0071] This leads to the Laplacian transformation formula of the function:
[0072]
[0073] The template of the Laplacian operator is:
[0074]
[0075] The two templates in the above formula can be converted to each other by multiplying them by -1. The template is isotropic in different directions.
[0076] After processing the original door area map, the Laplacian operator is used for subsequent edge detection. The door edge feature detection effect is as follows: Figure 6As shown in the figure. The Laplacian operator is different from the first-order differential operator mentioned above and is a second-order differential operator. When the detection object is the elevator door, the overall detection effect is not very ideal. The door edge information is intermittent or even incomplete. Moreover, some elevator doors, such as elevators 1, 2, 4, and 9, have too many background edges. The Laplacian operator is not ideal for door edge extraction and retains more noise edges.
[0077] Selection of edge detection operator By comparing the detection effects of four edge detection operators on elevators 1 to 9, the Scharr operator and the Prewitt operator are more sensitive to noise, and are more sensitive to interference information such as patterns and lights on the door; they are more sensitive to the edge features of objects outside the door. The Laplacian operator extracts door edge features discontinuously and incompletely. In terms of comprehensive detection effects, the embodiment of the present invention preferably uses the Sobel operator as the edge detection operator. The Sobel operator is less sensitive to irrelevant noise and can better detect the edge information of the elevator door. However, the following problems still exist when only using the Sobel operator for detection: 1) Although the Sobel operator has less noise, it can still detect noise, and further denoising processing is required; 2) There is incomplete detection of door edge features on the door edge of elevators 2 and 4, which requires further processing to ensure the reliability of door status detection.
[0078] S3, using an edge image processing method based on image morphology to filter the interference information of edge features to obtain straight line segment edge information;
[0079] The embodiment of the present invention further filters the interference information of edge features through an edge image processing method based on image morphology. Image morphology is one of the most important image processing technologies. It can extract image components based on defined structural elements. Its most basic operations are expansion and corrosion. The two are used in combination to form opening and closing operations, which play an important role in image filtering, image enhancement, etc. The definition of the structural element is the key to the operation effect.
[0080] In the morphological processing method, the structural element plays an important role. In the basic steps of the morphological operation, there is a key operation of moving the structural element in the original image, and performing the prescribed operation to correct the image accordingly. For the elevator edge image, since the elevator door area is rotated after the image is rotated, the elevator door edge can be basically perpendicular to the image. After being processed by the Sobel operator, the position of the edge information in the door edge image is also basically vertical. It is observed that the target door gap edge is a vertical straight line, and the target edge straight line is generally longer, longer than most noise edges. The schematic diagram of the structural element selected in the embodiment of the present invention is shown in FIG. Figure 7As shown in the figure, based on the characteristics of the elevator edge image, the selected structural element can maximize the retention of the target door edge while removing noise to the greatest extent. The structural element is a rectangle with a pixel width. The selection of the parameter n and the detailed value depend on the edge characteristics of different elevators. The maximum value of n cannot exceed the height of the original image, and the empirical minimum value is generally greater than half of the height of the original image.
[0081] Image morphology opening and closing operations are commonly used filtering methods. Both methods can remove clutter and make the target edge features clear. By defining the structure element, the image processed by the Sobel operator is filtered by opening and closing, where n is uniformly set to 50. The effect is as follows Figure 8 shown.
[0082] According to the detection effect of the opening and closing operations, the closing operation first dilates and then erodes, which makes it easier to retain the interference edges in the edge image and highlights the short edge information; the opening operation retains less interference information, and the operation sequence of first eroding and then dilating ensures the removal effect of smaller edges. The embodiment of the present invention preferably uses the opening operation with better effect as the filtering algorithm.
[0083] After the elevator door area is processed by the edge operator, there is still noise information in the image, including edge information of other objects outside the door, edge information of people when passengers enter and exit the door, edge information of the texture on the elevator door, etc. How to filter irrelevant interference information while retaining useful door edge information to the greatest extent is the key prerequisite for accurate door status recognition.
[0084] By selecting the opening operation in image morphology, the edge image detected by the Sobel operator is filtered. The structural elements are all rectangular elements with a width of one pixel and a height n selected according to different elevators. The selection range of n is [50,300].
[0085] The filtering effect is as follows Fig. 9 As shown in the figure, it can be seen that through the opening operation filtering, the contrast between the image edge lines and the image background is enhanced, the edge information is clear, the door edge is basically included in the image, and irrelevant and subtle edges are removed. Since the elevator door is composed of the floor door and the car door, it is inevitable that multiple edge information will appear in the door area. Which edge is selected as the door edge will also affect the recognition of the door state.
[0086] S4, using a straight line detection algorithm based on Hough transform to obtain the door edge position information for the straight line segment edge information;
[0087] After the door edge image is processed by the image morphological opening operation, the image only contains straight line segments, which include the edge of the landing door and the edge of the car door. In order to determine the state of the elevator door, it is necessary to locate the position of the door edge, so it is necessary to detect the position of the straight line. The embodiment of the present invention obtains the coordinates of the straight line through Hough transform, that is, the door edge position information.
[0088] Among the line detection methods, Hough transform is the most commonly used. The basic principle of Hough transform is as follows: suppose there is a straight line in the binary image, and find the coordinate position of the straight line. The general expression of the straight line equation is generally:
[0089] Where k and b are the slope and intercept of the straight line in the binary image.
[0090] Suppose there is a coordinate point (x0, y0), the straight line through this point is constrained by y0=kx0+b, and it is still a straight line in the kb (slope-intercept) coordinate system. A point in the xy plane can be mapped to a straight line in the kb plane. If there are m points on a straight line in the xy plane, then there are m straight lines on the kb plane, and these m straight lines will intersect at one point, such as Fig.10 shown.
[0091] In the figure, there are three points in the xy coordinate system on the same straight line. Let's assume the coordinates of the three points are A(x1,y1), B(x2,y2), and C(x3,y3). The straight line can be set to y=k1x+b1. By mapping these three points, it is not difficult to obtain three intersecting straight lines on the slope-intercept plane: l1, l2, and l3. The expression is as follows:
[0092] l1:b=y1-kx1
[0093] l2:b=y2-kx2
[0094] l3: b = y3 - kx3;
[0095] The intersection point is P and the coordinates are (k1, b1). If all the pixels in the edge detection image are mapped to the parameter space, the parameters of the target line can be obtained according to the peak points in the space. Since k cannot be 0, the ρ-θ coordinate system is generally selected in the application, and the parameter equation becomes:
[0096] ρ=xcos(θ)+ysin(θ);
[0097] According to the parameter equation of Hough transform, we first discretize and define a two-dimensional parameter space A with an initial value of 0. Then we calculate ρ based on each pixel point M(x, y) in the image space and each discrete value of θ, and round it to the discrete point of the nearest neighbor ρ. For each ρ and θ, we vote in the parameter space accumulation matrix A, that is, A(ρ, θ) = A(ρ, θ) + 1. Find the local maximum value A in A. max (ρ,θ). If the value of max is greater than the threshold, the image is considered to contain a straight line, and the image can be max (ρ,θ) determines the position of the line.
[0098] The elevator edge image contains the door edge straight line. The position of the door edge straight line determines the position of the elevator door. The coordinate position of the elevator door edge in the image is obtained by linear Hough transform. By performing linear detection on continuous video sequences, the movement trajectory of the elevator door can be obtained, and then the opening and closing states of the elevator door can be described. Taking elevator No. 3 as an example, the effects of Hough space and Hough line detection are shown in the figure below. Fig.10 shown.
[0099] Apply the linear Hough transform to the ladder No. 3 to obtain the corresponding Hough space. Each pixel corresponds to a curve in the coordinate system. The curves intersect at multiple points, and the coordinates of each point are the parameters of a line. Hough line detection can obtain the endpoint coordinates of multiple lines, and the endpoints are the crosses in the figure. According to the endpoint coordinates, the following information can be obtained:
[0100] 1) Number of straight lines.
[0101] 2) Line position: Based on the x-coordinate of each line in the image coordinate system. Since the edge of the elevator door is perpendicular to the entire image, the x-coordinate represents the relative position of the door in the image. Multiple frames of images can determine the direction of movement of the elevator door and thus the open and closed state of the elevator door.
[0102] 3) Straight line length: The length of the straight line can be determined based on the y-value of the endpoint coordinates of each straight line.
[0103] 4) Inclination angle of a straight line: It is not difficult to calculate the slope of a straight line based on the coordinates of its endpoints, and then the inclination angle.
[0104] After Hough line detection, more edge information will be detected. Only by further removing the interfering edge information can the door edge position be accurately located. Since the Sobel operator performs edge detection, it will not only detect the door edge information, but also irrelevant edge information. There may still be interfering information after morphological filtering. At present, there are still the following problems:
[0105] 1) After the elevator door opens, the scene outside the door is an uncontrollable factor. It is impossible to exclude vertical edge information such as the edge of the door, which greatly interferes with the recognition of the edge information of the door gap.
[0106] 2) There are two layers of doors inside and outside the elevator door, namely the car door and the floor door. When performing edge detection, due to the shooting angle problem, the edges of both doors will be retained in some elevators, which will interfere with the detection of the internal car door edge.
[0107] 3) During the detection process of the Sobel operator, there is a problem that the edge of one side of some elevators is incomplete.
[0108] Define the pixel coordinates of any point in the door gap detection area as (x, y), the coordinates of the upper left corner of the image as (0, 0), and the coordinates of the lower right corner are composed of the width w and height h of the image, that is, (w, h). Define Z as a detected straight line edge. Define the function S(z) to represent the angle between the detected straight line and the vertical direction of the image, in degrees. Define the function L(z) to represent the length of the detected straight line. In order to better locate the door gap, the embodiment of the present invention adopts the method of filtering interference information disclosed in S5:
[0109] S5. Adopt an interference information elimination strategy for the door edge position to eliminate interference information and obtain the elevator door opening and closing status.
[0110] The strategies for eliminating interference information include:
[0111] 1) Preliminary screening of straight line edges.
[0112] Eliminate edge information that is not a vertical line. The door edges should all be vertical lines, and filter the door edge lines that are close to vertical. Let the threshold angle be θ, and the qualified lines should satisfy the condition S(z)≤θ;
[0113] Secondly, exclude vertical lines that are not long enough. The length of the door edge is fixed in the elevator monitoring area and should be close to the height of the image. Considering that the edge may be incompletely extracted, it is necessary to set a reasonable length threshold l. The threshold value range is The straight line satisfies L(z)≤l.
[0114] 2) To address issues such as interference between the outer edge of the door and the edge of the floor door and the edge of the car door.
[0115] During the door opening process, the car door edge is usually on the far left and far right, while the floor door edge and the uncontrollable interference edge that appears during the door opening process are all in the middle area. Therefore, only the vertical lines on both sides are extracted, and the vertical lines in the middle are filtered out. For this purpose, the set M = {z1, ..., z1} is defined, where is the set of lines that satisfy S(z) ≤ θ and L(z) ≤ l. X1 is the minimum value in the set M, and X2 is the maximum value in the set M, representing the leftmost and rightmost door edge positions of the elevator, respectively.
[0116]
[0117] 3) To address the problem of incomplete edges on one side of some elevators.
[0118] Since the elevator door is symmetrical, in actual detection, as long as one of the left or right elevator door can be clearly detected, the problem of occasional incomplete edges at the edge of the elevator door can be solved by using the symmetrical characteristics of the elevator door.
[0119] 4) Comprehensive judgment of multi-frame fusion to eliminate single-frame anomalies
[0120] Since the opening and closing of the elevator door is a dynamic process, the operation status of the elevator door is comprehensively determined by analyzing multiple consecutive frames. Whether the elevator door is opened or closed, it conforms to the movement law and movement trend, and it is unlikely to have a sudden jump in position. According to the above-mentioned elevator door gap positioning algorithm, the elevator door gap position of each frame is determined. The position information and time information of multiple frames are continuously recorded, and abnormal frames can be excluded through continuous changes in position. Finally, the opening and closing state of the elevator door is determined based on the normal continuous frames, and the movement speed of the elevator door can be calculated in combination with the time information.
[0121] The embodiment of the present invention further performs door vibration detection by optical flow method, which specifically includes:
[0122] In two consecutive frames, the position of the same object in the two images may be different due to the movement of objects in the scene. This position difference can be expressed by calculating the change in the gray value of the pixel. By analyzing the change in the gray value of the pixel between adjacent frames, the movement trajectory of the pixel can be inferred, which is optical flow. Its basic idea is based on the following two assumptions:
[0123] (1) Constant brightness: In adjacent image frames, the difference in grayscale values of pixels on the same object surface in the two images is approximately equal.
[0124] (2) Time persistence (small movement): The small movement of the camera on the image changes slowly over time, so the pixel position will not change dramatically with time. This is to solve the partial derivative of the pixel position and obtain the corresponding grayscale value.
[0125] Based on these two assumptions, we get the constraint equation for each frame of the video:
[0126] I(x,y,t)=I(x+δx,y+δy,t+δt);
[0127] Where is the brightness of the image at position at time t.
[0128] Using Taylor's formula to expand the above function at (x, y, t) we can get:
[0129]
[0130] The higher-order remainder of the Taylor formula is approximately 0.
[0131] The constraint equations of the simultaneous graph and the expansion equations at (x, y, t) give:
[0132]
[0133] Equivalent to
[0134]
[0135] in and are the derivatives of the corresponding pixel points along the x and y directions, respectively. The velocity components along the x and y directions are recorded as u and v, respectively. The above formula can be simplified as:
[0136] I x u+I y v+I t =0;
[0137] Further simplified to matrix form:
[0138]
[0139] This is the basic constraint equation of optical flow. It is the gradient matrix of a two-dimensional image. The change in the gray value of the image at the same position at two consecutive times t and t+Δt is I t , can be expressed as the product of the gray gradient at that position and the speed relative to the observer. In other words, the change in the image gray value I t , is caused by the movement of the object at that position relative to the observer. The following is a specific analysis of the three optical flow algorithms:
[0140] 1. Horn-Schunck optical flow algorithm
[0141] The Horn-Schunck optical flow algorithm is a classic dense optical flow algorithm used to calculate the motion information of each pixel in an image sequence. Its basic assumption is that the change in the grayscale value of the image pixel mentioned above is caused by the movement of the object. The HS algorithm assumes that the motion between adjacent pixels in two adjacent frames is similar, and believes that the optical flow field should be as smooth as possible. Therefore, it defines the optical flow field as a vector field, in which each pixel corresponds to a vector, indicating the amount of motion of the pixel in the x and y directions. Then, this vector field is solved by minimizing the cost function to obtain the motion vector of each pixel.
[0142] The HS algorithm originally calculated the optical flow field by minimizing the cost function. Its cost function includes the difference between the temporal and spatial derivatives of the pixel gray value, and also considers the constraint of optical flow smoothness to reduce the noise sensitivity of the algorithm.
[0143] The smoothed error function is:
[0144]
[0145] E c (u,v)=∫∫[I x u+I y v+I t ] 2 dxdy;
[0146] In summary, the constraints are:
[0147]
[0148] E s (u, v) = ∫ ∫ F (x, y, u, u x ,u y ,v,v x , v y )dxdy;
[0149]
[0150] The above formula can be simplified into a differential equation:
[0151]
[0152] is the Laplace operator, discretizing the above equation:
[0153]
[0154] Use the Euler-Lagrange equation to minimize the above cost function and solve the equation. The result of the equation system is:
[0155]
[0156] Use the Euler-Lagrange equation to minimize this cost function and solve for the value of v. This process can be implemented using the Gauss-Seidel iteration method. The specific steps are as follows:
[0157] For each pixel, the gradient value of the cost function is calculated according to the value of the current optical flow vector, and the value of v is updated, that is:
[0158]
[0159] Among them, I x ,I y is the partial derivative of the image in the x and y directions.
[0160] Repeat the above process until the value of v converges. Generally speaking, convergence is determined by setting a tolerance value. When the change in v is less than the tolerance value, the algorithm is considered to have converged.
[0161] ∑ x ∑ v E S (i, j) < ò;
[0162] Where ∈ is the maximum tolerance value;
[0163] Take Elevator 5 as an example. Fig.11 This is the result of using the Horn-Schunck optical flow algorithm to detect the optical flow in the video. Fig.11 (a), (b) and (c) are the optical flow points detected. Fig.11 In (a), there are optical flow points detected at the edge of the elevator door gap, and Fig.11 (b) and Fig.11 The edge of the door gap in (c) is not detected, which may be related to its fast movement speed, and does not meet the "small movement" constraint of the optical flow method. Therefore, the detection effect of this algorithm is not ideal.
[0164] 2. Farneback optical flow algorithm
[0165] The Farneback algorithm is an optical flow calculation algorithm based on Gaussian pyramid decomposition. Its main idea is to assume that the change in the grayscale value of pixels on two adjacent frames between time t and t+1 is caused by their movement on the image plane. Specifically, it obtains images at different scales by performing Gaussian pyramid decomposition on the input image. At each scale, it uses the cross-correlation function to calculate the optical flow between the two images.
[0166] The optical flow equation assumes that the grayscale value of the same spatial point changes at two adjacent time points, and this change can be explained by the movement of the point on the image plane. The quadratic polynomial used by the algorithm to construct and model the image can effectively capture the motion information of the pixels in the image. The expression is as follows:
[0167] I1(x)=X T A1X+b1 T X+c1;
[0168] is the coordinate point, c1 is a constant, and the global displacement is added to the previous frame to build the model of the next image frame:
[0169] I2(X)=I1(Xd)=(Xd) T A1(Xd)+b1 T (Xd)+c1;
[0170] The above formula is expanded into:
[0171]
[0172] Arranged:
[0173] d is the horizontal and vertical offset of each pixel in the image.
[0174] Let's take Elevator 5 as an example. Fig.12 This is a detection result diagram of the optical flow in the video using the Farneback optical flow algorithm. It can be seen that no optical flow points are detected at the edge of the elevator door gap, but optical flow points are detected in the upper left corner of the image, which does not achieve the expected result of the present invention.
[0175] 3. Lucas-Kanade optical flow algorithm
[0176] The Lucas-Kanade (LK) optical flow algorithm is a method for calculating dense optical flow, proposed by Bruce D. Lucas and Takeo Kanade in 1981. The algorithm was originally designed to calculate dense optical flow, but was soon applied to sparse optical flow scenes. The algorithm uses a small window to extract local information, which allows it to track in sparse scenes. However, the disadvantage of the LK algorithm is that if the movement of the pixel exceeds the range of the window, the algorithm will not be able to find it. To solve this problem, the "pyramid-LK algorithm" was developed, which starts tracking from the lowest details of the image pyramid and gradually tracks to finer details. This method of tracking the image pyramid allows the algorithm to track large movements while still being able to extract information from the local window.
[0177] Based on the first two assumptions, the LK algorithm proposes a third assumption - spatial consistency assumption, which states that the positions of adjacent pixels in adjacent frames at two time points are the same. The significance of this assumption is that in order to calculate the speed in the x and y directions, multiple equations need to be established to solve. Through the spatial consistency assumption, n equations can be established using n pixels in the neighborhood.
[0178] Using the third assumption, the least squares method can be used to minimize the equation, and the above equation can be expressed in matrix form:
[0179]
[0180] Let AV = -b, and use the least squares method to get
[0181] A T AV=A T (-b);
[0182] Then the final optical flow (velocity vector) solved is:
[0183] V=(A T A) -1 A T (-b);
[0184] The matrix form is as follows:
[0185]
[0186] Finally, u and v are the components of the optical flow in the x and y directions corresponding to the LK algorithm.
[0187] The embodiment of the present invention is the door vibration of the elevator during the door opening and closing process. The constraint condition of the LK algorithm, i.e., the small speed, is not easily satisfied. When the motion displacement of the object between two frames is large, the algorithm will have a large error. In view of this situation, the original image can be decomposed layer by layer by using the image pyramid layering method to reduce the image size. Fig.13 This is the algorithm implementation process diagram of the pyramid LK optical flow method. First, find the optical flow vector d0 at a low scale, and then enter the next layer. After going down, first magnify it twice, and the "small movement" is satisfied at the blue position. Continue to calculate the optical flow, get the optical flow vector d1, and then throw (2d0+d1) to the next layer to continue moving forward, and so on.
[0188] Taking Elevator 5 as an example, the optical flow detection results calculated using the Lucas-Kanade algorithm are as follows: Fig.13 shown. Fig.13 (a), (b) and (c) are the optical flow points detected. Fig.13(d) is the motion trajectory of the optical flow point in a complete door opening process. According to the detection results, it can be seen that the Lucas-Kanade algorithm does not lose the detected optical flow points, and the detection effect is good.
[0189] By comparing the detection effects of the above three optical flow algorithms, the embodiment of the present invention preferably detects the vibration of the elevator during the door opening and closing process based on the pyramid layered Lucas-Kanade algorithm. By setting the appropriate number of pyramid layers, selecting the appropriate optical flow point quality level, fixing the detection area of the optical flow point at the edge of the elevator door gap, drawing the motion trajectory of the optical flow point, and obtaining the ordinate of the optical flow point in each frame image to analyze the vibration during the door opening and closing process.
[0190] Door system failures are generally related to mechanical system failures, the most common of which are as follows:
[0191] Fault 1: The door opening and closing speed slows down. This is usually caused by the door opening and closing machine belt slipping.
[0192] Fault 2: The elevator suddenly stops during the process of opening and closing the door. The door knife hits the door wheel, causing the lock arm to disengage, the door lock switch to disconnect, and the door lock relay to lose power.
[0193] Fault 3: The elevator door cannot be opened and closed normally. There are debris in the elevator door slide, the slider is deformed or falls off, the door hanging wheel is severely worn, etc., which will cause the elevator door to be unable to open and close normally.
[0194] For the above-mentioned fault 3, the embodiment of the invention obtains the video signal of the real elevator door fault, but for the remaining two faults, due to the limitation of conditions, this paper simulates the possible manifestations of various faults through artificial means. In the process of collecting video signals, a slight resistance is applied to the elevator door to simulate the increase of friction to create an abnormal video signal of the door opening and closing speed slowing down, and a person enters and exits during the door closing process, and presses the door closing button during the door opening process to create an abnormal video signal of the elevator suddenly pausing during the door opening and closing process (compared with the normal signal, this fault is manifested in that the elevator door is in a stationary state after the pause, instead of switching to the door state).
[0195] The embodiment of the invention experiments the door state detection algorithm through the door area images of elevators 1 to 9, where elevators 1 to 6 are video images of normal operation of the elevator, and elevators 7 to 9 represent the video images of the above-mentioned faults 1, 2, and 3, respectively. The average single-frame processing time of the video is 30ms. The experimental results of the door state detection algorithm are as follows: Fig.14 , Fig.15 and Fig.16As shown. By applying the interference information exclusion strategy to the detected edge information, the edge recognition error caused by the abnormal single frame detection can be greatly reduced. As can be seen from the table, the overall detection effect of the door state detection algorithm is good. When there are interference factors such as changes in the environment outside the car, changes in the lighting of the door opening and closing, light reflection of the elevator door caused by the lighting, and people outside the door, the algorithm has a good overall effect on the detection of the door state. The detection effect is also good for elevators with textures on the door, and it has anti-interference ability for the elevator door texture. The door state detection algorithm has fast calculation and processing speed and good real-time performance. At the same time, on the basis of accurately identifying the elevator door opening and closing state and the relative speed of the door opening and closing, it also realizes the detection of the vibration of the elevator door opening and closing. The vibration of the elevator door opening and closing can reflect many key information and is an important indicator for subsequent door state information processing and fault diagnosis.
[0196] By adopting the embodiments of the present invention, the following beneficial effects are achieved:
[0197] In order to reduce the interference of irrelevant edges, the embodiment of the present invention adopts the opening operation in image morphology, and realizes the filtering operation of edge image through custom structure elements; the key information of straight line edge, the length, inclination, and relative position of the straight line are obtained in turn by using the straight line detection algorithm based on Hough transform; through the analysis of the key information of the straight line edge, the strategy of eliminating the interference edge information is proposed, and the anti-interference ability of the algorithm is improved; finally, the opening and closing state of the elevator door is obtained by fusing multiple frames of information, and a good detection effect is achieved.
[0198] Device Example 1
[0199] According to an embodiment of the present invention, an electronic device is provided, including:
[0200] processor; and,
[0201] A memory is arranged to store computer executable instructions, which when executed cause the processor to perform the steps of the above method embodiments.
[0202] Device Example 2
[0203] According to an embodiment of the present invention, a storage medium is provided for storing computer executable instructions, wherein the computer executable instructions implement the steps of the above method embodiment when executed.
[0204] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for detecting the state of an elevator door, characterized in that: include: S1. Obtain elevator surveillance video, and obtain the area to be detected after performing image processing on the surveillance video; S2, using an edge detection algorithm to obtain edge features of the area to be detected; S3, using an edge image processing method based on image morphology to filter the interference information of the edge feature to obtain straight line segment edge information; S4, using a straight line detection algorithm based on Hough transform to obtain door edge position information for the straight line segment edge information; S5. Using an interference information elimination strategy for the door edge position to eliminate interference information and obtain the elevator door opening and closing state.
2. The method according to claim 1, characterized in that The S1 specifically includes: After horizontal correction is performed on each frame of the elevator video image in the monitoring video, an image ROI is used to select a region of interest, the region of interest is grayed to obtain a grayscale image, and the grayscale image is Gaussian filtered to obtain the region to be detected.
3. The method according to claim 1, characterized in that The S2 specifically includes: An edge detection operator that is insensitive to irrelevant noise is selected to extract the edge features of the elevator door.
4. The method according to claim 1, characterized in that: The S3 specifically includes: selecting a structural element based on the characteristics of the edge feature, performing an opening operation of first corroding and then dilating the edge feature according to the selected structural element, and then performing image binarization enhancement to obtain the edge information of the straight line segment.
5. The method according to claim 1, characterized in that The S5 specifically includes: Angle filtering is performed, specifically including: selecting door edge position information that meets a preset threshold angle as first door edge position information; Performing length filtering specifically includes: selecting first door edge position information that meets a preset length as second door edge position information; Performing filtering on both sides, specifically including: extracting the vertical second door edge position information on both sides as the third door edge position information, and filtering out the vertical straight line in the middle; Perform multi-frame fusion judgment, specifically including: executing steps S1-S5 for each frame of the elevator monitoring video, obtaining continuous third door edge position information to obtain the door opening and closing state of the elevator door.
6. The method according to claim 5, characterized in that The movement speed of the elevator door is obtained according to the third door edge position information combined with the time information of the elevator monitoring video.
7. The method according to claim 1, characterized in that The elevator door state detection method further includes: using an optical flow algorithm to implement elevator fault detection and diagnosis.
8. The method according to claim 6, characterized in that The use of optical flow algorithm to implement elevator fault detection and diagnosis specifically includes: The pyramid-based Lucas-Kanade algorithm is selected to detect the vibration of the elevator during the door opening and closing process. According to the preset number of pyramid layers and the quality level of optical flow points, the detection area of the optical flow points is fixed at the edge of the elevator door gap, so as to draw the movement trajectory of the optical flow points, and obtain the vertical coordinates of the optical flow points in each frame image to analyze the vibration during the door opening and closing process.
9. An electronic device, comprising: processor; as well as, A memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the elevator door state detection method according to any one of claims 1 to 8.
10. A storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed, implement the steps of the elevator door state detection method according to any one of claims 1 to 8.
