Elevator failure early warning method and device

Abstract

The application provides an elevator fault early warning method and device, and belongs to the technical field of fault early warning. The method comprises the following steps: photographing the elevator door and the passenger by using a camera to obtain image data; detecting the image data to obtain the edge of the elevator door, determining the movement speed of the elevator door at 2N position points in K return processes before the current time according to the edge, and using the 2N*K movement speeds to form a speed vector V; detecting the image data to identify the abnormal behavior of the passenger, determining the abnormal behavior vector A in a T time period before the current time according to the abnormal behavior of the passenger; fusing the abnormal behavior vector A and the speed vector V to obtain a fusion vector P; inputting the fusion vector P into a pre-trained fault detection model to output the fault category in a T1 time period after the current time; and sending the fault category to the elevator maintenance personnel. The application can early warn the elevator fault and ensure the personal safety of the passenger.

Description

Technical Field

[0001] This invention relates to the field of fault early warning technology, and in particular to an elevator fault early warning method and device. Background Technology

[0002] With the acceleration of urbanization, there are more and more mid- to high-rise buildings. In order to facilitate people's travel, the use of elevators has become widespread. At the same time, their operational safety is becoming increasingly important. The safe and reliable operation of elevators is directly related to the personal and life safety of passengers. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an elevator malfunction early warning method and device, which can provide early warning of elevator malfunctions and ensure the personal safety of passengers.

[0004] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0005] On the one hand, an elevator malfunction early warning method is provided, including:

[0006] The camera captures images of the elevator doors and passengers inside, obtaining image data.

[0007] The image data is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points in the K round trips before the current time is determined. The 2N*K movement speeds are used to form a velocity vector V, where K and N are positive integers.

[0008] The image data is detected to identify abnormal behavior of the passenger, and the abnormal behavior vector A is determined based on the passenger's abnormal behavior during the time period T before the current moment.

[0009] The abnormal behavior vector A is fused with the velocity vector V to obtain the fused vector P;

[0010] Input the fusion vector P into the pre-trained fault detection model and output the types of faults in the time period T1 after the current time.

[0011] Send the fault type to the elevator maintenance personnel.

[0012] In an optional embodiment of the present invention, the step of detecting the image data to obtain the edge of the elevator door includes:

[0013] The depth features of the image data are obtained to obtain the segmented region of the elevator door area on the image.

[0014] Connected component edge extraction is performed on the segmented region to obtain an edge coordinate linked list of the elevator door in the image. The edge coordinate linked list includes the coordinates of multiple points on the edge of the elevator door in the image.

[0015] In an optional embodiment of the present invention, determining the movement speed of the elevator door includes:

[0016] A reference point is determined on the edge of the elevator door, and the movement speed of the reference point is determined based on the position of the reference point in multiple consecutive frames of images.

[0017] The speed of the elevator door is determined based on the speed of movement of the reference point.

[0018] In an optional embodiment of the present invention, the step of detecting the image data and identifying abnormal behavior of the passenger includes:

[0019] The image data is analyzed to identify the passenger's joint points;

[0020] Input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model, and output the abnormal behavior of the rider.

[0021] In an optional embodiment of the present invention, the step of determining the abnormal behavior vector A within the time period T2 prior to the current moment based on the passenger's abnormal behavior includes:

[0022] Establish an initial vector of abnormal behaviors of length M, where M is the number of types of abnormal behaviors, and each element of the initial vector of abnormal behaviors represents an abnormal behavior;

[0023] Count the number of each type of abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each type of abnormal behavior of the passenger to obtain the abnormal behavior vector A.

[0024] In an optional embodiment of the present invention, the step of fusing the abnormal behavior vector A with the velocity vector V to obtain the fused vector P includes any one of the following:

[0025] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0026] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0027] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0028] In an optional embodiment of the present invention, before the step of inputting the fusion vector P into the pre-trained fault detection model, the method further includes a step of training the fault detection model, the step of training the fault detection model including:

[0029] Establish an initial model for fault detection;

[0030] The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds.

[0031] Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers.

[0032] The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1;

[0033] The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

[0034] This invention also provides an elevator fault early warning device, comprising:

[0035] The camera module is used to capture images of the elevator doors and passengers inside the elevator using a camera to obtain image data.

[0036] The first detection module is used to detect the image data, obtain the edge of the elevator door, determine the movement speed of the elevator door at 2N position points in the K round trips before the current time based on the edge, and use the 2N*K movement speeds to form a velocity vector V, where K and N are positive integers;

[0037] The second detection module is used to detect the image data, identify the abnormal behavior of the passenger, and determine the abnormal behavior vector A in the time period T before the current moment based on the abnormal behavior of the passenger.

[0038] The fusion module is used to fuse the abnormal behavior vector A with the velocity vector V to obtain a fused vector P;

[0039] The prediction module is used to input the fusion vector P into the pre-trained fault detection model and output the fault types within the time period T1 after the current time.

[0040] The fault management module is used to send the fault types to elevator maintenance personnel.

[0041] In an optional embodiment of the present invention, the first detection module includes:

[0042] The segmentation region acquisition unit is used to acquire the depth features of the image data to obtain the segmentation region of the elevator door area on the image.

[0043] The segmentation region processing unit is used to extract the connected component edges of the segmented region to obtain a linked list of edge coordinates of the elevator door in the image. The linked list of edge coordinates includes the coordinates of multiple points on the edge of the elevator door in the image.

[0044] In an optional embodiment of the present invention, the first detection module includes:

[0045] The first calculation unit is used to determine a reference point on the edge of the elevator door and to determine the motion speed of the reference point based on the position of the reference point in multiple consecutive frames of images.

[0046] The second calculation unit is used to determine the movement speed of the elevator door based on the movement speed of the reference point.

[0047] In an optional embodiment of the present invention, the second detection module includes:

[0048] The first processing unit is used to detect the image data and identify the joint points of the passenger;

[0049] The second processing unit is used to input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model and output the abnormal behavior of the passenger.

[0050] In an optional embodiment of the present invention, the second detection module includes:

[0051] A unit is established to establish an initial vector of abnormal behavior of length M, where M is the number of types of abnormal behavior, and each element of the initial vector of abnormal behavior represents an abnormal behavior.

[0052] The update unit is used to count the number of each abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passenger to obtain the abnormal behavior vector A.

[0053] In an optional embodiment of the present invention, the fusion module is specifically used to perform any of the following:

[0054] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0055] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0056] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0057] In an optional embodiment of the present invention, a training module is further included, used to train the fault detection model, wherein the training module is specifically used for:

[0058] Establish an initial model for fault detection;

[0059] The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds.

[0060] Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers.

[0061] The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1;

[0062] The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

[0063] This invention also provides an elevator fault early warning device, comprising:

[0064] processor; and

[0065] A memory, in which computer program instructions are stored,

[0066] When the computer program instructions are executed by the processor, the processor performs the steps in the elevator malfunction early warning method described above.

[0067] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps in the elevator fault early warning method described above.

[0068] The embodiments of the present invention have the following beneficial effects:

[0069] In the above scheme, a fault detection model is pre-trained, and the types of elevator faults in the future are predicted using the pre-trained fault detection model. The types of faults are then sent to elevator maintenance personnel so that they can take relevant measures to deal with them. This embodiment can provide early warning of elevator faults to ensure the personal safety of passengers. Attached Figure Description

[0070] Figure 1 This is a flowchart illustrating the elevator fault early warning method according to an embodiment of the present invention;

[0071] Figure 2 This is a system block diagram illustrating the application of this invention in an embodiment.

[0072] Figure 3 and Figure 4 This is a schematic diagram of data transmission according to an embodiment of the present invention;

[0073] Figure 5 This is a schematic diagram illustrating the process of training a fault detection model according to an embodiment of the present invention;

[0074] Figure 6 This is a structural block diagram of the elevator fault early warning device according to an embodiment of the present invention;

[0075] Figure 7 This is a structural block diagram of an elevator fault early warning device according to an embodiment of the present invention. Detailed Implementation

[0076] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0077] This invention provides an elevator malfunction early warning method and device, which can provide early warning of elevator malfunctions and ensure the personal safety of passengers.

[0078] Example 1

[0079] This invention provides an elevator fault early warning method, such as... Figure 1 As shown, this embodiment includes:

[0080] Step 101: Use a camera to photograph the elevator door and the passengers inside the elevator to obtain image data;

[0081] like Figure 2The diagram shown illustrates the system block diagram used in this embodiment. The technical solution of this embodiment can provide fault warnings for multiple elevators. A camera is installed inside each elevator car. The camera can capture images of the elevator doors and passengers inside, obtaining video images (i.e., image data), and then transmit the collected image data to the data management module. Additionally, each elevator is equipped with an elevator central control system, which can collect elevator fault information and transmit it to the data management module.

[0082] The camera can capture black and white images or color images.

[0083] Step 102: Detect the image data to obtain the edge of the elevator door, and determine the movement speed of the elevator door at 2N position points during the K round trips before the current time based on the edge. Use the 2N*K movement speeds to form a velocity vector V, where K and N are positive integers.

[0084] Specifically, deep learning algorithms can be used, but are not limited to, for elevator door detection. For example, Mask R-CNN can be used. The input is image data acquired by a camera. The depth features of the image data are obtained, leading to segmentation markers for the elevator door region on the image. These markers are then used to determine the segmented region. Connected component edge extraction is performed on the segmented region to obtain a linked list of edge coordinates for the elevator door in the image. This linked list includes the coordinates of multiple points on the edge of the elevator door in the image. For instance, a coordinate system can be established on the plane of the image, with one point as the origin. This allows obtaining the coordinates of k points on the edge of the elevator door in the image: (x1, y1), (x2, y2), (x3, y3), ..., (xk, yk), where k is a positive integer. The linked list can record the coordinates of all points on the edge of the elevator door, or it can record the coordinates of only a subset of these points.

[0085] Generally, elevators have two doors. If the x-axis of the coordinate system is horizontal, the left or right elevator door can be distinguished based on the x-coordinates of points in the edge coordinate chain. For example, multiple points on the edge of the elevator door can be divided into two groups based on the magnitude of their x-coordinates: the group with smaller x-coordinates is on the edge of the left elevator door, and the group with larger x-coordinates is on the edge of the right elevator door.

[0086] A reference point is determined on the edge of the elevator door, and the movement speed of the reference point is determined based on the position of the reference point in multiple consecutive frames of images; the movement speed of the elevator door is determined based on the movement speed of the reference point.

[0087] For example, the extreme points on the edges of the two elevator doors can be used as reference points, that is, the points with the maximum and minimum x-coordinates on the edge of the elevator. The movement speed of the reference point can be determined based on the positional changes of the reference point in multiple consecutive frames of images. The movement speed Vi of the reference point is the pixel distance moved / time. For example, if the pixel distance moved by the reference point in two adjacent frames is 5 pixels and the interval between two adjacent frames is 1 / 60 second, then the movement speed of the reference point is 300 pixels / second.

[0088] Measuring the elevator door's speed solely based on the speed of a reference point might not be accurate enough. When the elevator door is fully closed, we can obtain the door's width in pixels, L. For example, on the same y-axis, L = the maximum x-coordinate of the elevator door's edge point xmax – the minimum x-coordinate xmin. Using L and the speed of the reference point, we can calculate the elevator door's speed as Vr = Vi / L.

[0089] When the elevator doors are fully open, the position of the inner edge of the elevator doors is recorded as 0; when the elevator doors are fully closed, the position of the inner edge of the elevator doors is recorded as 1. N position points are sampled within the range [0,1], and the velocity of the elevator doors when they reach each position point is recorded. In this way, each opening and closing of the elevator doors, that is, one round trip of the elevator doors, can record a total of 2N velocities, forming a velocity vector of length 2N, with the following data format: Vr1, Vr2, ..., VrN, ..., Vr2N.

[0090] Starting from the current moment, trace back K door opening and closing events, and use 2N*K motion velocities to form a velocity vector V:

[0091] Vr11, Vr12, …, Vr1N, …, Vr12N, Vr21, Vr22, …, Vr22N, …, VrK1, …, VrK2N.

[0092] Where K and N are positive integers, they can be set as needed.

[0093] Step 103: Detect the image data, identify the abnormal behavior of the passenger, and determine the abnormal behavior vector A within the time period T before the current moment based on the abnormal behavior of the passenger.

[0094] Specifically, the image data can be detected to identify the passenger's key points; the coordinates of the key points in multiple consecutive frames of images can be input into the behavior recognition model to output the passenger's abnormal behavior.

[0095] Pedestrian detection algorithms can be used to obtain bounding boxes of pedestrians in images. Commonly used algorithms include the YOLO series and SSD. After obtaining the bounding boxes, sequential image analysis can be performed on each pedestrian to identify abnormal actions or behaviors, such as kicking elevator doors, reaching out to block elevator doors, or fighting near elevator doors. Specifically, the OpenPose algorithm can be used to obtain information about the joints of the human body, including elbows, wrists, and knees. Then, using a behavior recognition model, the coordinate vectors of each joint in multiple consecutive frames of images are used as time series input to obtain the action and behavior recognition results. The behavior recognition model can use the LSTM algorithm.

[0096] Establish an initial vector of abnormal behavior of length M, where M is the number of types of abnormal behavior. Each element of the initial vector represents a type of abnormal behavior. Count the number of each type of abnormal behavior of passengers in the time period T2 before the current time. Update the value of the corresponding element in the initial vector of abnormal behavior based on the number of each type of abnormal behavior of passengers to obtain the abnormal behavior vector A. Assuming there are M types of actions and behaviors, if a certain type of action or behavior occurs once in the time period T2 before the current time (e.g., one day, set as needed), the value of the corresponding element is incremented by 1, resulting in the abnormal behavior vector A: A1, A2, ..., AM, where the value of each element is the number of times the corresponding abnormal behavior occurs.

[0097] Step 104: Fuse the abnormal behavior vector A with the velocity vector V to obtain the fused vector P;

[0098] The abnormal behavior vector A and the velocity vector V can be fused using any of the following methods:

[0099] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0100] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0101] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0102] In a specific example, the fusion vector P can be: Vr11, Vr12, ..., Vr1N, ..., Vr12N, Vr21, Vr22, ..., Vr22N, ..., VrK1, ..., VrK2N, A1, A2, ..., AM.

[0103] Step 105: Input the fusion vector P into the pre-trained fault detection model and output the types of faults in the time period T1 after the current time.

[0104] Fault detection models can utilize traditional machine learning methods such as SVM, decision trees, and random forests. Deep networks can also be used. Taking a multi-layer deep network as an example, the input to the fault detection model is a vector P, and the output is whether a fault E will occur in the future after a certain period. The value of E ranges from 0 to Q, where 0 represents no fault, and Q represents the type of fault, including but not limited to inability to open, inability to close, intermediate jamming, and insufficient opening / closing speed.

[0105] Using a general deep learning training method, the input vector P is the network input, and the fault type variable E is the output, and training is performed iteratively.

[0106] Step 106: Send the fault type to the elevator maintenance personnel.

[0107] like Figure 3 The diagram shown is a data transmission schematic of this embodiment. Figure 3 As shown, the camera captures multiple frames of images to form a video stream. By detecting elevator doors in a single frame, the elevator door edges can be obtained. By analyzing the elevator door's speed, the elevator door speed vector can be obtained. By detecting pedestrians in a single frame, pedestrian detection boxes can be obtained. By recognizing abnormal actions and behaviors, action / behavior vectors, i.e., abnormal behavior vectors, can be obtained. The abnormal behavior vectors are fused with the speed vectors and input into the fault detection model to perform fault prediction. The prediction results are then reported to elevator maintenance personnel.

[0108] In this embodiment, a fault detection model is pre-trained and used to predict the types of elevator faults in the future. The types of faults are then sent to elevator maintenance personnel so that they can take relevant measures to deal with them. This embodiment can provide early warning of elevator faults to ensure the personal safety of passengers.

[0109] In this embodiment, a fault detection model needs to be pre-trained, such as... Figure 5 As shown, the steps for training the fault detection model include:

[0110] Step 201: Establish an initial fault detection model. The initial fault detection model can use traditional machine learning methods, such as SVM, decision tree, random forest, etc., or it can use deep networks.

[0111] Step 202: Detect the historical image data captured by the camera to obtain the edge of the elevator door. Based on the edge, determine the movement speed of the elevator door at 2N locations during K round trips in the historical time period. Use the 2N*K movement speeds to form a historical speed vector V1.

[0112] To detect elevator doors, historical image data from a previous period (e.g., within two weeks) can be obtained. Specifically, deep learning algorithms can be used, but are not limited to, to detect elevator doors. For example, Mask R-CNN can be used to detect elevator doors. The input is image data acquired by a camera. The depth features of the image data can be obtained to obtain segmentation markers for the elevator door area on the image. The segmentation region is determined based on the segmentation markers. Connected component edge extraction is performed on the segmented region to obtain a linked list of edge coordinates of the elevator door in the image. The linked list of edge coordinates includes the coordinates of multiple points on the edge of the elevator door in the image. For example, a coordinate system can be established on the plane where the image is located, with one point as the origin. The coordinates of k points on the edge of the elevator door in the image can be obtained (x1, y1), (x2, y2), (x3, y3), ..., (xk, yk), where k is a positive integer. The linked list of edge coordinates can record the coordinates of all points on the edge of the elevator door, or it can record the coordinates of some of the points.

[0113] Generally, elevators have two doors. If the x-axis of the coordinate system is horizontal, the left or right elevator door can be distinguished based on the x-coordinates of points in the edge coordinate chain. For example, multiple points on the edge of the elevator door can be divided into two groups based on the magnitude of their x-coordinates: the group with smaller x-coordinates is on the edge of the left elevator door, and the group with larger x-coordinates is on the edge of the right elevator door.

[0114] A reference point is determined on the edge of the elevator door, and the movement speed of the reference point is determined based on the position of the reference point in multiple consecutive frames of images; the movement speed of the elevator door is determined based on the movement speed of the reference point.

[0115] For example, the extreme points on the edges of the two elevator doors can be used as reference points, that is, the points with the maximum and minimum x-coordinates on the edge of the elevator. The movement speed of the reference point can be determined based on the positional changes of the reference point in multiple consecutive frames of images. The movement speed Vi of the reference point is the pixel distance moved / time. For example, if the pixel distance moved by the reference point in two adjacent frames is 5 pixels and the interval between two adjacent frames is 1 / 60 second, then the movement speed of the reference point is 300 pixels / second.

[0116] Measuring the elevator door's speed solely based on the speed of a reference point might not be accurate enough. When the elevator door is fully closed, we can obtain the door's width in pixels, L. For example, on the same y-axis, L = the maximum x-coordinate of the elevator door's edge point xmax – the minimum x-coordinate xmin. Using L and the speed of the reference point, we can calculate the elevator door's speed as Vr = Vi / L.

[0117] When the elevator doors are fully open, the position of the inner edge of the elevator doors is recorded as 0; when the elevator doors are fully closed, the position of the inner edge of the elevator doors is recorded as 1. N position points are sampled within the range [0,1], and the velocity of the elevator doors when they reach each position point is recorded. In this way, each opening and closing of the elevator doors, that is, one round trip of the elevator doors, can record a total of 2N velocities, forming a velocity vector of length 2N, with the following data format: Vr11, Vr2, ..., Vr1N, ..., Vr12N.

[0118] Count K door opening and closing events within a historical time period, and use 2N*K movement velocities to form a historical velocity vector V1:

[0119] Vr111, Vr112, ..., Vr11N, ..., Vr112N, Vr121, Vr122, ..., Vr122N, ..., Vr1K1, ..., Vr1K2N.

[0120] Step 203: Detect the historical image data captured by the camera, identify the abnormal behavior of the passenger, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of the passenger.

[0121] Specifically, the image data can be detected to identify the passenger's key points; the coordinates of the key points in multiple consecutive frames of images can be input into the behavior recognition model to output the passenger's abnormal behavior.

[0122] Pedestrian detection algorithms can be used to obtain bounding boxes of pedestrians in images. Commonly used algorithms include the YOLO series and SSD. After obtaining the bounding boxes, sequential image analysis can be performed on each pedestrian to identify abnormal actions or behaviors, such as kicking elevator doors, reaching out to block elevator doors, or fighting near elevator doors. Specifically, the OpenPose algorithm can be used to obtain information about the joints of the human body, including elbows, wrists, and knees. Then, using a behavior recognition model, the coordinate vectors of each joint in multiple consecutive frames of images are used as time series input to obtain the action and behavior recognition results. The behavior recognition model can use the LSTM algorithm.

[0123] An initial vector of abnormal behavior of length M is established, where M is the number of types of abnormal behavior. Each element of the initial vector represents a type of abnormal behavior. The number of each type of abnormal behavior by passengers is counted within a historical time period. The value of the corresponding element in the initial vector is updated based on the number of each type of abnormal behavior, resulting in an abnormal behavior vector A. Assuming there are M types of actions and behaviors, if a certain type of action or behavior occurs once within a historical time period, the value of the corresponding element is incremented by 1, resulting in a historical abnormal behavior vector A1: A11, A12, ..., A1M, where the value of each element represents the number of times the corresponding abnormal behavior occurred.

[0124] Step 204: Fuse the historical abnormal behavior vector A1 with the historical velocity vector V1 to obtain the historical fusion vector P1;

[0125] The historical abnormal behavior vector A1 and the historical velocity vector V1 can be fused using any of the following methods:

[0126] The historical abnormal behavior vector A1 and the historical velocity vector V1 are directly combined to obtain the historical fusion vector P1;

[0127] After reducing the dimension of the historical abnormal behavior vector A1, it is combined with the historical velocity vector V1 to obtain the historical fusion vector P1.

[0128] After reducing the dimensionality of the historical velocity vector V1, it is combined with the historical abnormal behavior vector A1 to obtain the historical fusion vector P1.

[0129] Step 205: Obtain the historical fault types of the elevator within the historical time period, and use the historical fault types and the historical fusion vector P1 to train the initial fault detection model to obtain the fault detection model.

[0130] Specifically, vector P can be used as the input to the initial fault detection model, and historical fault types E can be used as the output of the initial fault detection model. The initial fault detection model can be trained iteratively, for example, by stopping training after reaching a preset number of iterations.

[0131] like Figure 4 The diagram shown illustrates the data transfer process during the training of the fault detection model in this embodiment. Figure 4 As shown, the camera captures multiple frames of images, forming a video stream. Elevator door edges are obtained by detecting elevator doors in a single frame, and the elevator door velocity vector is obtained by analyzing the elevator door's velocity. Pedestrian detection bounding boxes are obtained by detecting pedestrians in a single frame, and action / behavior vectors, i.e., abnormal behavior vectors, are obtained by recognizing abnormal actions and behaviors. The abnormal behavior vectors are fused with the velocity vectors to obtain a fused vector. Elevator door fault records are acquired and classified. The classified fault types and abnormal behavior vectors are used to train the initial fault detection model, resulting in a fault detection model.

[0132] Example 2

[0133] This invention also provides an elevator fault early warning device, such as... Figure 6 As shown, the device includes:

[0134] The shooting module 31 is used to capture images of the elevator door and passengers inside the elevator using a camera to obtain image data.

[0135] The first detection module 32 is used to detect the image data, obtain the edge of the elevator door, determine the movement speed of the elevator door at 2N position points in the K round trips before the current time based on the edge, and use the 2N*K movement speeds to form a velocity vector V, where K and N are positive integers;

[0136] The second detection module 33 is used to detect the image data, identify the abnormal behavior of the passenger, and determine the abnormal behavior vector A in the time period T before the current moment based on the abnormal behavior of the passenger.

[0137] The fusion module 34 is used to fuse the abnormal behavior vector A with the velocity vector V to obtain a fusion vector P;

[0138] Prediction module 35 is used to input the fusion vector P into the pre-trained fault detection model and output the fault types within the time period T1 after the current time.

[0139] The fault management module 36 is used to send the fault types to elevator maintenance personnel.

[0140] In an optional embodiment of the present invention, the first detection module 32 includes:

[0141] The segmentation region acquisition unit is used to acquire the depth features of the image data to obtain the segmentation region of the elevator door area on the image.

[0142] The segmentation region processing unit is used to extract the connected component edges of the segmented region to obtain a linked list of edge coordinates of the elevator door in the image. The linked list of edge coordinates includes the coordinates of multiple points on the edge of the elevator door in the image.

[0143] In an optional embodiment of the present invention, the first detection module 32 includes:

[0144] The first calculation unit is used to determine a reference point on the edge of the elevator door and to determine the motion speed of the reference point based on the position of the reference point in multiple consecutive frames of images.

[0145] The second calculation unit is used to determine the movement speed of the elevator door based on the movement speed of the reference point.

[0146] In an optional embodiment of the present invention, the second detection module 33 includes:

[0147] The first processing unit is used to detect the image data and identify the joints of the passenger;

[0148] The second processing unit is used to input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model and output the abnormal behavior of the passenger.

[0149] In an optional embodiment of the present invention, the second detection module 33 includes:

[0150] A unit is established to establish an initial vector of abnormal behavior of length M, where M is the number of types of abnormal behavior, and each element of the initial vector of abnormal behavior represents an abnormal behavior.

[0151] The update unit is used to count the number of each abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passenger to obtain the abnormal behavior vector A.

[0152] In an optional embodiment of the present invention, the fusion module 34 is specifically configured to perform any of the following:

[0153] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0154] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0155] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0156] In an optional embodiment of the present invention, a training module is further included, used to train the fault detection model, wherein the training module is specifically used for:

[0157] Establish an initial model for fault detection;

[0158] The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds.

[0159] Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers.

[0160] The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1;

[0161] The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

[0162] Example 3

[0163] This invention also provides an elevator fault early warning device 50, such as... Figure 7 As shown, it includes:

[0164] Processor 52; and

[0165] Memory 54, in which computer program instructions are stored.

[0166] When the computer program instructions are executed by the processor, the processor 52 performs the following steps:

[0167] The camera captures images of the elevator doors and passengers inside, obtaining image data.

[0168] The image data is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points in the K round trips before the current time is determined. The 2N*K movement speeds are used to form a velocity vector V, where K and N are positive integers.

[0169] The image data is detected to identify abnormal behavior of the passenger, and the abnormal behavior vector A is determined based on the passenger's abnormal behavior during the time period T before the current moment.

[0170] The abnormal behavior vector A is fused with the velocity vector V to obtain the fused vector P;

[0171] Input the fusion vector P into the pre-trained fault detection model and output the types of faults in the time period T1 after the current time.

[0172] Send the fault type to the elevator maintenance personnel.

[0173] Furthermore, such as Figure 7 As shown, the elevator fault early warning device 50 also includes a network interface 51, an input device 53, a hard disk 55, and a display device 56.

[0174] The various interfaces and devices described above can be interconnected via a bus architecture. The bus architecture can include any number of interconnecting buses and bridges. Specifically, various circuits representing one or more central processing units (CPUs), represented by processor 52, and one or more memories, represented by memory 54, are connected together. The bus architecture can also connect various other circuits such as peripheral devices, voltage regulators, and power management circuits. It is understood that the bus architecture is used to implement communication between these components. In addition to the data bus, the bus architecture also includes a power bus, a control bus, and a status signal bus, all of which are well known in the art and will not be described in detail herein.

[0175] The network interface 51 can be connected to a network (such as the Internet, a local area network, etc.), obtain relevant data from the network, and save it to the hard disk 55.

[0176] The input device 53 can receive various commands input by the operator and send them to the processor 52 for execution. The input device 53 may include a keyboard or a clicking device (e.g., a mouse, trackball, touchpad, or touchscreen).

[0177] The display device 56 can display the results obtained by the processor 52 executing instructions.

[0178] The memory 54 is used to store programs and data necessary for the operation of the operating system, as well as intermediate results and other data during the calculation process of the processor 52.

[0179] It is understood that the memory 54 in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. The memory 54 of the apparatus and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0180] In some implementations, memory 54 stores elements such as executable modules or data structures, or subsets thereof, or extended sets thereof: operating system 541 and application programs 542.

[0181] The operating system 541 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 542 includes various applications, such as a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 542.

[0182] When the processor 52 calls and executes the application program and data stored in the memory 54, it specifically performs the following steps: It uses a camera to capture images of the elevator door and the passengers inside, obtaining image data; it detects the image data to obtain the edge of the elevator door, and determines the speed of the elevator door at 2N locations during K round trips before the current moment based on the edge; it uses 2N*K speeds to form a speed vector V, where K and N are positive integers; it detects the image data to identify abnormal behavior of the passengers, and determines the abnormal behavior vector A within a time period T before the current moment based on the abnormal behavior; it fuses the abnormal behavior vector A with the speed vector V to obtain a fused vector P; it inputs the fused vector P into a pre-trained fault detection model, outputting the fault types within a time period T1 after the current moment; and it sends the fault types to elevator maintenance personnel.

[0183] The methods disclosed in the above embodiments of the present invention can be applied to processor 52, or implemented by processor 52. Processor 52 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 52 or by instructions in software form. The processor 52 may be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), off-the-shelf programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, and can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of the hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 54, and processor 52 reads the information in memory 54 and completes the steps of the above method in combination with its hardware.

[0184] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0185] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0186] Further, the processor 52 is specifically used to acquire the depth features of the image data to obtain the segmented region of the elevator door area in the image; to perform connected component edge extraction on the segmented region to obtain the edge coordinate linked list of the elevator door in the image, the edge coordinate linked list including the coordinates of multiple points on the edge of the elevator door in the image.

[0187] Furthermore, the processor 52 is specifically configured to determine a reference point on the edge of the elevator door, determine the movement speed of the reference point based on the position of the reference point in multiple consecutive frames of images, and determine the movement speed of the elevator door based on the movement speed of the reference point.

[0188] Furthermore, the processor 52 is specifically used to detect the image data, identify the key points of the passenger; input the coordinates of the key points in multiple consecutive frames of images into the behavior recognition model, and output the abnormal behavior of the passenger.

[0189] Further, the processor 52 is specifically used to establish an abnormal behavior initial vector of length M, where M is the number of abnormal behavior types, and each element of the abnormal behavior initial vector represents an abnormal behavior; count the number of each abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each abnormal behavior of the passenger to obtain the abnormal behavior vector A.

[0190] Furthermore, processor 52 is specifically configured to perform any of the following:

[0191] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0192] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0193] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0194] Furthermore, the processor 52 is also used to train the fault detection model, including:

[0195] Establish an initial model for fault detection;

[0196] The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds.

[0197] Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers.

[0198] The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1;

[0199] The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

[0200] Example 4

[0201] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the following steps:

[0202] The camera captures images of the elevator doors and passengers inside, obtaining image data.

[0203] The image data is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points in the K round trips before the current time is determined. The 2N*K movement speeds are used to form a velocity vector V, where K and N are positive integers.

[0204] The image data is detected to identify abnormal behavior of the passenger, and the abnormal behavior vector A is determined based on the passenger's abnormal behavior during the time period T before the current moment.

[0205] The abnormal behavior vector A is fused with the velocity vector V to obtain the fused vector P;

[0206] Input the fusion vector P into the pre-trained fault detection model and output the types of faults in the time period T1 after the current time.

[0207] Send the fault type to the elevator maintenance personnel.

[0208] Furthermore, when the computer program is run by the processor, the processor also performs the following steps: acquiring the depth features of the image data to obtain the segmented region of the elevator door area on the image;

[0209] Connected component edge extraction is performed on the segmented region to obtain an edge coordinate linked list of the elevator door in the image. The edge coordinate linked list includes the coordinates of multiple points on the edge of the elevator door in the image.

[0210] Furthermore, when the computer program is run by the processor, it causes the processor to perform the following steps:

[0211] A reference point is determined on the edge of the elevator door, and the movement speed of the reference point is determined based on the position of the reference point in multiple consecutive frames of images.

[0212] The speed of the elevator door is determined based on the speed of movement of the reference point.

[0213] Furthermore, when the computer program is run by the processor, it causes the processor to perform the following steps:

[0214] The image data is analyzed to identify the passenger's joint points;

[0215] Input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model, and output the abnormal behavior of the rider.

[0216] Furthermore, when the computer program is run by the processor, it causes the processor to perform the following steps:

[0217] Establish an initial vector of abnormal behaviors of length M, where M is the number of types of abnormal behaviors, and each element of the initial vector of abnormal behaviors represents an abnormal behavior;

[0218] Count the number of each type of abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each type of abnormal behavior of the passenger to obtain the abnormal behavior vector A.

[0219] Furthermore, when the computer program is run by the processor, it also causes the processor to perform any of the following steps:

[0220] The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P;

[0221] The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P;

[0222] The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

[0223] Furthermore, when the computer program is run by the processor, it causes the processor to perform the following steps:

[0224] Establish an initial model for fault detection;

[0225] The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds.

[0226] Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers.

[0227] The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1;

[0228] The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

[0229] The above are preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the principles described in the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for early warning of elevator malfunctions, characterized in that, include: The camera captures images of the elevator doors and passengers inside, obtaining image data. The image data is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points in the K round trips before the current time is determined. The 2N*K movement speeds are used to form a velocity vector V, where K and N are positive integers. The image data is detected to identify abnormal behavior of the passenger, and the abnormal behavior vector A is determined based on the passenger's abnormal behavior during the time period T before the current moment. The abnormal behavior vector A is fused with the velocity vector V to obtain the fused vector P; Input the fusion vector P into the pre-trained fault detection model, and output the types of faults in the time period T1 after the current time. Send the fault type to the elevator maintenance personnel.

2. The elevator fault early warning method according to claim 1, characterized in that, The steps for detecting the edges of the elevator doors from the image data include: The depth features of the image data are obtained to obtain the segmented region of the elevator door area on the image. Connected component edge extraction is performed on the segmented region to obtain an edge coordinate linked list of the elevator door in the image. The edge coordinate linked list includes the coordinates of multiple points on the edge of the elevator door in the image.

3. The elevator fault early warning method according to claim 2, characterized in that, Determining the speed of the elevator doors includes: A reference point is determined on the edge of the elevator door, and the movement speed of the reference point is determined based on the position of the reference point in multiple consecutive frames of images. The speed of the elevator door is determined based on the speed of movement of the reference point.

4. The elevator fault early warning method according to claim 1, characterized in that, The steps for detecting and identifying abnormal behavior of passengers in the image data include: The image data is analyzed to identify the passenger's joint points; Input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model, and output the abnormal behavior of the rider.

5. The elevator fault early warning method according to claim 1, characterized in that, The steps to determine the abnormal behavior vector A for the time period T2 prior to the current moment based on the abnormal behavior of the riders include: Establish an initial vector of abnormal behaviors of length M, where M is the number of types of abnormal behaviors, and each element of the initial vector of abnormal behaviors represents an abnormal behavior; Count the number of each type of abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each type of abnormal behavior of the passenger to obtain the abnormal behavior vector A.

6. The elevator fault early warning method according to claim 1, characterized in that, The step of fusing the abnormal behavior vector A with the velocity vector V to obtain the fused vector P includes any one of the following: The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P; The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P; The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

7. The elevator fault early warning method according to any one of claims 1-6, characterized in that, Before inputting the fusion vector P into the pre-trained fault detection model, the method further includes a step of training the fault detection model, which includes: Establish an initial model for fault detection; The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds. Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers. The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1; The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

8. An elevator malfunction early warning device, characterized in that, include: The camera module is used to capture images of the elevator doors and passengers inside the elevator using a camera to obtain image data. The first detection module is used to detect the image data, obtain the edge of the elevator door, determine the movement speed of the elevator door at 2N position points in the K round trips before the current time based on the edge, and use the 2N*K movement speeds to form a velocity vector V, where K and N are positive integers; The second detection module is used to detect the image data, identify the abnormal behavior of the passenger, and determine the abnormal behavior vector A within the time period T before the current moment based on the abnormal behavior of the passenger. The fusion module is used to fuse the abnormal behavior vector A with the velocity vector V to obtain a fused vector P; The prediction module is used to input the fusion vector P into the pre-trained fault detection model and output the fault types within the time period T1 after the current time. The fault management module is used to send the fault types to elevator maintenance personnel.

9. The elevator fault early warning device according to claim 8, characterized in that, The first detection module includes: The segmentation region acquisition unit is used to acquire the depth features of the image data to obtain the segmentation region of the elevator door area on the image. The segmentation region processing unit is used to extract the connected component edges of the segmented region to obtain a linked list of edge coordinates of the elevator door in the image. The linked list of edge coordinates includes the coordinates of multiple points on the edge of the elevator door in the image.

10. The elevator fault early warning device according to claim 9, characterized in that, The first detection module includes: The first calculation unit is used to determine a reference point on the edge of the elevator door and to determine the motion speed of the reference point based on the position of the reference point in multiple consecutive frames of images. The second calculation unit is used to determine the movement speed of the elevator door based on the movement speed of the reference point.

11. The elevator fault early warning device according to claim 8, characterized in that, The second detection module includes: The first processing unit is used to detect the image data and identify the joints of the passenger; The second processing unit is used to input the coordinates of key points in multiple consecutive frames of images into the behavior recognition model and output the abnormal behavior of the passenger.

12. The elevator fault early warning device according to claim 8, characterized in that, The second detection module includes: A unit is established to establish an initial vector of abnormal behavior of length M, where M is the number of types of abnormal behavior, and each element of the initial vector of abnormal behavior represents an abnormal behavior. The update unit is used to count the number of each type of abnormal behavior of the passenger in the time period T2 before the current time, and update the value of the corresponding element in the abnormal behavior initial vector according to the number of each type of abnormal behavior of the passenger to obtain the abnormal behavior vector A.

13. The elevator fault early warning device according to claim 8, characterized in that, The fusion module is specifically used to perform any of the following: The abnormal behavior vector A is directly combined with the velocity vector V to obtain the fused vector P; The abnormal behavior vector A is reduced in dimensionality and then combined with the velocity vector V to obtain the fused vector P; The velocity vector V is reduced in dimension and then combined with the abnormal behavior vector A to obtain the fused vector P.

14. The elevator fault early warning device according to any one of claims 8-13, characterized in that, It also includes a training module for training the fault detection model, the training module specifically being used for: Establish an initial model for fault detection; The historical image data captured by the camera is detected to obtain the edge of the elevator door. Based on the edge, the movement speed of the elevator door at 2N position points during K round trips in the historical time period is determined. The historical speed vector V1 is formed by using the 2N*K movement speeds. Detect historical image data captured by the camera, identify abnormal behavior of passengers, and determine the historical abnormal behavior vector A1 within the historical time period based on the abnormal behavior of passengers. The historical abnormal behavior vector A1 and the historical velocity vector V1 are fused to obtain the historical fusion vector P1; The elevator's historical fault types within the historical time period are obtained, and the initial fault detection model is trained using the historical fault types and the historical fusion vector P1 to obtain the fault detection model.

Images