Elevator door opening and closing state detection method, device and equipment based on deep learning

Abstract

The present invention discloses a method, device and equipment for detecting the opening and closing door status of an elevator based on deep learning. First, a target elevator image sequence corresponding to a designated area of ​​an elevator door is obtained; then, each frame of the elevator door area image in the target elevator image sequence is classified and processed to obtain a probability data sequence corresponding to the opening and closing door status of the elevator; then, based on the probability data sequence corresponding to the opening and closing door status of the elevator, the opening and closing door status timing information representing the opening and closing door status of the elevator door in the time period corresponding to the target elevator image sequence is generated; finally, the opening and closing door status of the elevator is detected according to the opening and closing door status timing information to obtain the opening and closing door status detection result. The above implementation method can realize real-time and accurate detection of the opening and closing door status of the elevator.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device and equipment for detecting the opening and closing state of an elevator door based on deep learning. Background Art

[0002] Elevators are an important tool in modern urban life. While they bring convenience to people's lives, they also pose safety issues. Therefore, it is necessary to quickly and accurately detect the opening and closing status of elevator doors.

[0003] In the related art, based on the deep learning segmentation model, the change of the edge position of the elevator is judged to detect the opening and closing state of the elevator door. However, the detection method based on the deep learning segmentation model needs to be improved. Summary of the invention

[0004] The embodiments of this specification are intended to solve at least one of the technical problems in the related art to a certain extent. To this end, the embodiments of this specification propose a method, device and equipment for detecting the state of an elevator door opening and closing based on deep learning.

[0005] The embodiment of this specification provides a method for detecting the state of an elevator door opening and closing based on deep learning, and the method includes:

[0006] Acquire a target elevator image sequence; wherein the target elevator image sequence includes a plurality of frames of elevator door area images corresponding to a designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object does not overlap with the elevator door opening and closing area in the elevator door opening and closing direction when the entry and exit object passes through the elevator door opening and closing area;

[0007] Classify each frame of the elevator door area image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state;

[0008] Generate door opening and closing state timing information based on the probability data sequence corresponding to the elevator door opening and closing state; wherein the door opening and closing state timing information is used to represent the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence;

[0009] The elevator door opening and closing state is detected according to the door opening and closing state timing information to obtain the door opening and closing state detection result.

[0010] In one embodiment, the generating of the door opening and closing state timing information based on the probability data sequence corresponding to the door opening and closing state of the elevator includes:

[0011] Filtering the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence;

[0012] The filtered probability sequence is binarized to obtain the switch door state timing information.

[0013] In one embodiment, the binarization processing of the filtered probability sequence to obtain the switch state timing information includes:

[0014] Binarizing the filtered probability sequence to obtain initial state sequence information;

[0015] If there is a false detection element in the initial state sequence information, the value of the false detection element is updated using a preset value to obtain the door opening and closing state timing information; wherein, at the moment corresponding to the false detection element, the elevator door opening and closing state does not change but the value of the false detection element suddenly changes; the preset value is the value of the adjacent element of the false detection element.

[0016] In one embodiment, the step of detecting the elevator door opening and closing state according to the door opening and closing state timing information to obtain the door opening and closing state detection result includes:

[0017] If the elevator is in a moving state, and it is determined according to the door opening and closing state timing information that the elevator is in a non-closed state within a preset time period, a detection result of abnormal door opening and closing state of the elevator is obtained.

[0018] In one embodiment, the probability data in the probability data sequence is output by a target binary classification model; the training process of the target binary classification model includes:

[0019] Acquire a first state image obtained by photographing an elevator door in a first state, and a second state image obtained by photographing an elevator door in a second state; wherein the first state corresponds to a first elevator door opening width range, and the second state corresponds to a second elevator door opening width range; within the first elevator door opening width range, an image acquisition device in the elevator car can photograph the outside of the elevator, and within the second elevator door opening width range, an image acquisition device in the elevator car cannot photograph the outside of the elevator;

[0020] Intercepting the first state image to obtain a first state image sample corresponding to the designated area of ​​the elevator door;

[0021] Intercepting the second state image to obtain a second state image sample corresponding to the designated area of ​​the elevator door;

[0022] Constructing a training sample set based on the first state image samples and the second state image samples;

[0023] The initial binary classification model is trained using the training sample set to obtain the target binary classification model.

[0024] In one embodiment, the target binary classification model is used as a base model in a specified elevator operating environment; the method further includes:

[0025] Determine a target elevator under the specified elevator operating environment;

[0026] Acquire a video file captured by an image acquisition device installed in the target elevator;

[0027] Constructing a supplementary sample set for training the basic model based on the video file;

[0028] The basic model is further trained using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

[0029] In one embodiment, constructing a supplementary sample set for training the basic model based on the video file includes:

[0030] Determine a first starting time corresponding to the elevator being in the first state and a second starting time corresponding to the elevator being in the second state in the video file;

[0031] Based on the first starting time and the second starting time, extract a first state video segment and a second state video segment from the video file;

[0032] The supplementary sample set is constructed based on the first state video segment and the second state video segment.

[0033] In one embodiment, the designated elevator operating environment includes a new operating environment after the elevator operating environment changes, and the elevator operating environment change includes any of the following situations:

[0034] The internal environment of the elevator car changes, including adding any one of stickers, hanging lights, protection plates, and colored silk;

[0035] The external environment of the elevator car changes, including adding any one of stickers, hanging lights, protection plates, and colored silk;

[0036] The target binary classification model trained for the first operating environment is used in a second operating environment, where the second operating environment is different from the first operating environment; and the supplementary sample set is a video file collected in the second operating environment.

[0037] The embodiment of this specification provides an elevator door opening and closing state detection device based on deep learning, and the device includes:

[0038] An image sequence acquisition module is used to acquire a target elevator image sequence; wherein the target elevator image sequence includes a plurality of frames of elevator door area images corresponding to a designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object does not overlap with the elevator door opening and closing area in the elevator door opening and closing direction when the entry and exit object passes through the elevator door opening and closing area;

[0039] A regional image classification module is used to classify each frame of the elevator door regional image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state;

[0040] A state time sequence generation module, used to generate door opening and closing state time sequence information based on the probability data sequence corresponding to the elevator door opening and closing state; wherein the door opening and closing state time sequence information is used to represent the door opening and closing state of the elevator door within the time period corresponding to the target elevator image sequence;

[0041] The door status detection module is used to detect the door opening and closing status of the elevator according to the door opening and closing status timing information to obtain the door opening and closing status detection result.

[0042] In one embodiment, the state timing generation module is further used to filter the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence; and binarize the filtered probability sequence to obtain the door opening and closing state timing information.

[0043] In one embodiment, the state timing generation module is also used to binarize the filtered probability sequence to obtain initial state sequence information; if there is a false detection element in the initial state sequence information, the value of the false detection element is updated using a preset value to obtain the door opening and closing state timing information; wherein, at the moment corresponding to the false detection element, the elevator door opening and closing state does not change but the value of the false detection element suddenly changes; the preset value is the value of the adjacent element of the false detection element.

[0044] In one embodiment, the door status detection module is also used to determine that the elevator is in a non-closed state within a preset time period if the elevator is in a moving state and based on the door opening and closing state timing information, obtain a detection result of abnormal elevator door opening and closing state.

[0045] In one embodiment, the probability data in the probability data sequence is output by a target binary classification model; the device also includes a classification model training module, which is used to obtain a first state image obtained by photographing an elevator door in a first state, and a second state image obtained by photographing an elevator door in a second state; the first state corresponds to a first elevator door opening width range, and the second state corresponds to a second elevator door opening width range; within the first elevator door opening width range, the image acquisition device in the elevator car can photograph the outside of the elevator, and within the second elevator door opening width range, the image acquisition device in the elevator car cannot photograph the outside of the elevator; the first state image is intercepted to obtain a first state image sample corresponding to the designated area of ​​the elevator door; the second state image is intercepted to obtain a second state image sample corresponding to the designated area of ​​the elevator door; a training sample set is constructed based on the first state image sample and the second state image sample; the initial binary classification model is trained using the training sample set to obtain the target binary classification model.

[0046] In one embodiment, the target binary classification model is used as a basic model in a specified elevator operating environment; the device also includes a classification model training module, which is used to determine the target elevator in the specified elevator operating environment; obtain a video file captured by an image acquisition device installed in the target elevator; construct a supplementary sample set for training the basic model based on the video file; and continue to train the basic model using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

[0047] An embodiment of the present specification provides a computer device, which includes: a memory, and one or more processors communicatively connected to the memory; the memory stores instructions executable by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the steps of the method described in any of the above embodiments.

[0048] The embodiments of this specification provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method described in any of the above embodiments are implemented.

[0049] An embodiment of the present specification provides a computer program product, wherein the computer program product includes instructions, and when the instructions are executed by a processor of a computer device, the computer device is enabled to perform the steps of the method described in any one of the above embodiments.

[0050] An embodiment of the present specification provides a heterogeneous chip, including an embedded neural network processor NPU, a central processing unit CPU, a memory, and a computer program stored in the memory and configured to be executed by the central processing unit CPU and the embedded neural network processor NPU. When the central processing unit CPU and the embedded neural network processor NPU execute the computer program, they implement the method described in any of the above embodiments.

[0051] In the above-mentioned implementation of the specification, first, a target elevator image sequence corresponding to a designated area of ​​an elevator door is obtained; then, each frame of the elevator door area image in the target elevator image sequence is classified and processed to obtain a probability data sequence corresponding to the elevator door opening and closing state; then, based on the probability data sequence corresponding to the elevator door opening and closing state, the door opening and closing state timing information representing the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence is generated; finally, the elevator door opening and closing state is detected according to the door opening and closing state timing information to obtain the door opening and closing state detection result. End-to-end reasoning and real-time and accurate detection of the elevator door opening and closing state are achieved; further, by detecting the door opening and closing state of each frame image in the image sequence and outputting the door opening and closing state timing information, the interference of factors such as the internal environment of the elevator car, the external environment of the elevator car, and people entering and exiting the elevator on the detection results can be reduced, thereby improving the generalization and robustness of the detection method. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1a A schematic diagram of a scene of an elevator door opening and closing state detection method provided in an embodiment of this specification;

[0053] Figure 1b A schematic diagram of a scene of an elevator door opening and closing state detection method provided in another embodiment of this specification;

[0054] Figure 1c A schematic diagram of a flow chart of an elevator door opening and closing state detection method provided in one embodiment of this specification;

[0055] Figure 1d A schematic diagram for determining a designated area of ​​an elevator door provided for an implementation of this specification;

[0056] Figure 1e A schematic diagram of a curve corresponding to a probability data sequence provided in an embodiment of this specification;

[0057] Figure 1f A schematic diagram of a curve corresponding to a new probability data sequence provided in an embodiment of this specification;

[0058] Figure 1g A schematic diagram of a curve corresponding to a probability sequence after filtering provided in an embodiment of this specification;

[0059] Figure 1h A schematic diagram of a curve showing the existence of misdetected elements provided in an embodiment of this specification;

[0060] Figure 1i A schematic diagram of a curve corresponding to the timing information of the switch state after the element correction provided in the implementation mode of this specification;

[0061] Figure 2 A schematic diagram of a process for generating the timing information of the door opening and closing states provided in the implementation mode of this specification;

[0062] Figure 3 A schematic diagram of a process for determining the timing information of the door opening and closing states provided in an embodiment of this specification;

[0063] Figure 4a A flowchart of a target binary classification model training process provided in an embodiment of this specification;

[0064] Figure 4b A schematic diagram of the structure of a target binary classification model provided in the implementation mode of this specification;

[0065] Figure 4c A schematic diagram of the structure of a common separable convolution provided for the implementation of this specification;

[0066] Figure 4d A schematic diagram of the structure of a Depthwise Convlution (DW Conv) layer provided in an embodiment of this specification;

[0067] Figure 4e A schematic diagram of the improved structure of Pointwise Convolution (PW Conv, point-by-point convolution layer) provided for the implementation mode of this specification;

[0068] Figure 5 A schematic diagram of a process for obtaining a target model suitable for a specified elevator operating environment provided in an embodiment of this specification;

[0069] Figure 6 A schematic diagram of a process for constructing a supplementary sample set provided in an embodiment of this specification;

[0070] Figure 7 A schematic diagram of a flow chart of an elevator door opening and closing state detection method provided in another embodiment of this specification;

[0071] Figure 8 A schematic diagram of an elevator door opening and closing state detection device provided in an embodiment of this specification;

[0072] Fig. 9 An internal structural diagram of a computer device provided for an embodiment of this specification. DETAILED DESCRIPTION

[0073] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0074] Elevators are an important tool in modern urban life and are widely installed in office areas, shopping malls and residential buildings. While elevators bring many conveniences to people's lives, they also pose safety issues. When an elevator has an abnormal situation, such as abnormal speed, abnormal door opening and closing, abnormal entrapment, etc., it is necessary to quickly and accurately detect the elevator door opening and closing status. Therefore, an efficient and stable door opening and closing detection method is crucial.

[0075] Elevator door opening and closing detection technology needs to detect the elevator opening and closing status in a timely manner to effectively avoid accidents caused by elevator door failures. While improving elevator operating efficiency and reducing maintenance costs, it also brings convenience and safety to people's lives.

[0076] At present, there are several directions for detecting the switch status of elevators. The first is to use traditional physical mechanical equipment to detect the switch status of elevator doors. Traditional physical mechanical equipment is mainly used in old elevators. The door machine structure uses touch to feedback the switch status of the elevator door during operation, but the failure rate of traditional physical mechanical equipment is high. The second is to use sensor-type devices (Hall switches) to detect the switch status of elevator doors. Sensor-type devices can determine the switch status of elevator doors by detecting magnetic fields, but the installation of sensor-type devices requires that sensor-type devices must cut the edge of the magnetic field, and the distance must be less than 2cm. After long-term use, the weakening of the magnetic field will also cause the failure of sensor-type equipment detection. Sensor-type devices can be acceleration sensors. From starting to stopping, the elevator must have an acceleration and a deceleration process. During the movement of the elevator, the normal door is in a closed state. If the elevator door is in an open state, the elevator is in a fault state. However, the two are unrelated in the fault state. In addition, the acceleration sensor is affected by the movement of people in the elevator, the vibration of the elevator, etc., which will cause misjudgment, so the acceleration sensor can be used as an auxiliary method to detect the switch status of the elevator.

[0077] In related technologies, traditional image processing methods (such as the Canny edge detection method) can be used to detect the two vertical lines of the elevator door. When the elevator door is in the closed state, there is only one vertical line. During the door opening or closing process, the distance between the two lines gradually increases or decreases. The frame difference method or the background difference method can be used to obtain the first frame image as the basic image and record the switch status using the elevator camera. Subtract the subsequent images from the basic image, and use certain logic to determine whether the status of the subsequent images is consistent with the basic image, thereby determining the switch status of the elevator door.

[0078] In related technologies, the optical flow method can be used to set the regions of interest on both sides of the elevator door and calculate the optical flow features, and judge the door opening and closing status of the elevator door by movement. The template matching method can be used to define the region of interest and extract the features when the door is closed, and then the current features of the region of interest and the features when the door is closed are compared for similarity during the operation of the elevator. Finally, the similarity score is compared with the preset threshold to detect the door opening and closing status of the elevator.

[0079] In the related technology, you can manually set up markers, stick markers in areas where the elevator door is not easily disturbed, and then use the target detection method to detect the location of the marker to determine the door opening and closing. Then use the traditional Kernelized Correlation Filter (KCF) to detect and track the marker to determine the elevator door opening and closing status. You can also use deep learning methods to detect the elevator door opening and closing status. This method can improve accuracy and stability to a certain extent. In order to improve accuracy, multiple deep learning detection models will be cascaded, and the accuracy can usually reach 98+%. The deep learning algorithm based on segmentation obtains the edge line of the elevator door, determines the change in the edge position, and thus determines the elevator door opening and closing status. Based on the deep learning door opening and closing classification model, each image is directly classified to achieve end-to-end reasoning, and the accuracy can reach 99+%.

[0080] However, traditional image processing methods are affected by lighting, people entering and exiting, etc., which may cause false detection. Moreover, when the brightness of the elevator door is similar to that outside the elevator, the two vertical lines of the elevator door cannot be detected due to low recognition accuracy. The frame difference method or background difference method is greatly affected by motion, color, and lighting, and the operation is unstable, resulting in low recognition accuracy. The optical flow method has a large amount of calculation and slow operation speed. At the same time, it is greatly affected by the delineation of the region of interest, lighting, and image texture, resulting in low recognition accuracy of this method. The template matching method is limited by the feature extraction algorithm and the similarity comparison method, and the correctness of the result cannot be stably guaranteed under a fixed threshold. The KCF algorithm has low accuracy and is greatly affected by the environment. In addition, the method of setting markers cannot be replaced at will, and must be permanently saved after being affixed to the door. High-precision detection models require large computing power and poor real-time performance. When building the model, the data annotation workload is large, and the cost of continuous upgrading and updating in the later stage is high.

[0081] The segmentation-based deep learning algorithm requires a huge workload to annotate data during training, and it is also difficult to obtain a high-precision model. In addition, model inference must be placed on a server with high computing power. Considering the time-consuming data transmission and model inference, the real-time detection of the elevator door opening and closing status cannot be guaranteed. The deep learning-based door opening and closing classification model is difficult to handle the interference after people enter the elevator, and there are also limitations on generalization, which will reduce the recognition accuracy of new elevators. Non-end-to-end recognition methods require a large number of manually set rules and thresholds to be added in the later stage, and have poor universality. Traditional detection algorithms have poor anti-interference capabilities.

[0082] Based on this, the implementation method of this specification provides an elevator door opening and closing state detection method based on deep learning. First, obtain the target elevator image sequence corresponding to the designated area of ​​the elevator door; then, classify and process each frame of the elevator door area image in the target elevator image sequence to obtain the probability data sequence corresponding to the elevator door opening and closing state; then, based on the probability data sequence corresponding to the elevator door opening and closing state, generate the door opening and closing state timing information representing the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence; finally, detect the elevator door opening and closing state according to the door opening and closing state timing information to obtain the door opening and closing state detection result. It can reduce the artificial setting logic and realize end-to-end reasoning. Through the above implementation method, not only can the real-time and accurate detection of the elevator door opening and closing state be realized. Moreover, by detecting the door opening and closing state of each frame image in the image sequence and outputting the door opening and closing state timing information, it can reduce the interference of factors such as the internal environment of the elevator car, the external environment of the elevator car, and people entering and exiting the elevator on the detection results, thereby improving the generalization and robustness of the detection method.

[0083] The method provided in the embodiments of this specification can be applied to Figure 1aIn the application scenario, the camera includes an image acquisition unit and a model inference unit. The model inference unit includes a binary classification model and a post-processing unit. The binary classification model is deployed on the NPU core (Neural-network Process Units), and the post-processing unit is deployed on the CPU core (Central Processing Unit). The control center includes data management, display management, model management, and user management. The control center is mainly responsible for binary classification model version management and model update release and rollback. The display platform includes content management and display layout.

[0084] Specifically, the target elevator image sequence is collected by the acquisition unit of the camera installed inside the elevator. The image acquisition unit inputs the target elevator image sequence into the model inference unit inside the camera, calls the door opening and closing binary classification model and the post-processing unit, and obtains the door opening and closing state timing information representing the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence. Then, the door opening and closing state timing information is uploaded to the control center through the network. The control center uses the door opening and closing state timing information and business logic (such as the door is open for a long time; the elevator is occupied but still in a closed state for a long time; the elevator is in the ascending and descending state and the door is open) to obtain the door opening and closing state detection results to be displayed and push them to the display platform for the staff to view.

[0085] Please note that Figure 1b , elevator 1 is installed with camera 1, elevator 2 is installed with camera 2, and elevator N is installed with camera N. N elevators collect the target elevator image sequence through the cameras installed inside, and perform binary classification and post-processing on the target elevator image sequence to obtain the door opening and closing status timing information. The camera uploads the door opening and closing status timing information to the control center server of the elevator alarm platform through the network. The control center server detects the elevator door opening and closing status to obtain the door opening and closing status detection result. The control center uploads the door opening and closing status detection result to the NVR (Network Video Recorder) through the network for storage and transmits it to the monitoring display screen for display for staff to view. You can also use a mobile device to remotely access the control center through the network to obtain the door opening and closing status detection result.

[0086] This specification provides a method for detecting the state of an elevator door opening and closing based on deep learning. Figure 1c , the method may include the following steps:

[0087] S110: Acquire a target elevator image sequence.

[0088] The target elevator image sequence includes several frames of elevator door area images corresponding to the designated area of ​​the elevator door. The designated area of ​​the elevator door is the area where the object does not overlap with the elevator door opening and closing area in the direction of the elevator door opening and closing when the object passes through the elevator door opening and closing area. The elevator door opening and closing area can be the space occupied by the elevator door during the opening and closing process. This area is usually composed of the space on both sides of the elevator door and within a certain distance in the front and rear directions. For example, please refer to Figure 1d The area where the elevator door opening and closing area 102 and the entry and exit object 104 do not overlap in the elevator door opening and closing direction is the elevator door designated area 106.

[0089] In some cases, by setting the designated area of ​​the elevator door as the area where the entry and exit object and the elevator door opening and closing area do not overlap in the opening and closing direction of the elevator door when the entry and exit object passes through the elevator door opening and closing area, the occlusion of the elevator door caused by the entry and exit object when entering and exiting the elevator can be reduced, and the image data processed by the model can be reduced, thereby speeding up the inference detection of the elevator door opening and closing status.

[0090] Specifically, the designated area of ​​the elevator door can be photographed by an image acquisition device to obtain a target elevator image sequence. The captured target elevator image sequence can be stored locally in the image acquisition device, so that the elevator door area image can be subsequently subjected to binary classification processing by a target binary classification model deployed in the image acquisition device, thereby achieving real-time detection of the elevator door opening and closing state. In other embodiments, the server locally stores several initial elevator image sequences, and the target elevator image sequence can be directly obtained from the initial elevator image sequence.

[0091] For example, the image acquisition device may be at least one of a camera, a surveillance camera, an industrial camera, and a fisheye camera. The staff may mark the area to be monitored by the elevator, i.e., the designated area of ​​the elevator door, through the interactive interface of the monitoring platform. The marked area may be the upper 1 / 4 of the elevator door.

[0092] S120, classify each frame of the elevator door area image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state.

[0093] The classification result of the elevator door area image may be an elevator open state or an elevator closed state. The elevator open state may be the state of the elevator door when the image acquisition device in the elevator car can capture the outside of the elevator. The elevator closed state may be the state of the elevator door when the image acquisition device in the elevator car cannot capture the outside of the elevator. The probability data sequence may be a sequence formed by arranging the corresponding probability data of each frame of the elevator door area image being in the elevator open state or the corresponding probability data of each frame of the elevator door area image being in the elevator closed state based on the time sequence.

[0094] Specifically, the target elevator image sequence includes several frames of elevator door area images. By using the target binary classification model to classify each frame of the elevator door area image, the probability value of the elevator being in the open state and the probability value of the elevator being in the closed state corresponding to each frame of the elevator door area image can be obtained. In some embodiments, based on the time sequence, the probability values ​​of the elevator being in the open state corresponding to each frame of the elevator door area image are arranged to form a probability data sequence corresponding to the elevator door opening and closing state. In other embodiments, based on the time sequence, the probability values ​​of the elevator being in the closed state corresponding to each frame of the elevator door area image are arranged to form a probability data sequence corresponding to the elevator door opening and closing state. A corresponding curve graph can be generated according to the probability data sequence. For example, please refer to Figure 1e , curve 108 is a curve graph corresponding to the probability data sequence.

[0095] Exemplarily, the target elevator image sequence includes three frames of elevator door area images A1, A2, and A3 corresponding to the designated area of ​​the elevator door. The elevator door area image A1 is input into the target binary classification model for classification processing, and the probability value of the elevator door area image A1 being in the elevator open state is 0.8, and the probability value of the elevator door area image A1 being in the elevator closed state is 0.2. The elevator door area image A2 is input into the target binary classification model for classification processing, and the probability value of the elevator door area image A2 being in the elevator open state is 0.7, and the probability value of the elevator door area image A2 being in the elevator closed state is 0.3. The elevator door area image A3 is input into the target binary classification model for classification processing, and the probability value of the elevator door area image A3 being in the elevator open state is 0.6, and the probability value of the elevator door area image A3 being in the elevator closed state is 0.4. The probability data sequence corresponding to the elevator door opening and closing state can be [0.8, 0.7, 0.6] according to the probability value of the elevator door area image A1 being in the elevator opening state is 0.8, the probability value of the elevator door area image A2 being in the elevator opening state is 0.7, and the probability value of the elevator door area image A3 being in the elevator opening state is 0.6. The probability data sequence corresponding to the elevator door opening and closing state can also be [0.2, 0.3, 0.4] according to the probability value of the elevator door area image A1 being in the elevator closing state is 0.2, the probability value of the elevator door area image A2 being in the elevator closing state is 0.3, and the probability value of the elevator door area image A3 being in the elevator closing state is 0.4.

[0096] S130. Generate door opening and closing state timing information based on the probability data sequence corresponding to the elevator door opening and closing state.

[0097] The door opening and closing state timing information is used to indicate the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence. The elevator opening state can be indicated by 1, and the elevator closing state can be indicated by 0.

[0098] In some cases, the door opening and closing status timing information can be used to monitor the elevator door opening and closing status so as to subsequently obtain the elevator fault condition.

[0099] Specifically, since the image acquisition device can acquire multiple frames of elevator door area images within a specified time period (for example, within one second), it will cause image redundancy and increase the consumption of computing resources. Therefore, the probability data sequence can be sampled, and a specified number of elements can be extracted from the probability data sequence corresponding to the elevator door opening and closing state of the elevator door area image included in the specified time period, and the extracted elements are arranged in chronological order to form a new probability data sequence. The target elevator image sequence is processed through the above operations to obtain a probability data sequence corresponding to the new elevator door opening and closing state. Based on the probability data sequence corresponding to the new elevator door opening and closing state, the door opening and closing state timing information is generated. The probability data sequence corresponding to the new elevator door opening and closing state can generate a corresponding curve graph. For example, please refer to Figure 1f , curve 110 is a curve graph corresponding to the new probability data sequence.

[0100] For example, 30 frames of elevator door area images can be collected in one second. It is also possible to extract 5 to 10 frames in one second, that is, probability data corresponding to the elevator door opening and closing state within a time accuracy of 1 / 6s to 1 / 3s.

[0101] S140, detecting the elevator door opening and closing state according to the door opening and closing state timing information to obtain the door opening and closing state detection result.

[0102] The door opening and closing state detection result may be that the elevator door opening and closing state is abnormal, or may be that the elevator door opening and closing state is normal.

[0103] In some cases, combining the door opening and closing status timing information with the actual situation can detect the elevator fault status, which helps maintenance personnel to perform timely maintenance and repairs on the elevator, thereby improving the safety of the elevator.

[0104] Specifically, the elevator door state and the conversion process of the elevator door state can be determined by finding the transition point between the element value 0 and the element value 1. In combination with the actual situation, the elevator door state is detected according to the door state timing information to obtain the door state detection result.

[0105] In some embodiments, after the user calls the elevator, when the elevator responds to the user's request and runs to the user's floor and is in a leveling state, the elevator door needs to go through a normal door opening process before the user enters the elevator. Or when the user presses a floor button inside the elevator, the elevator responds to the user's request and reaches the user's selected floor and is in a leveling state, the elevator door needs to go through a normal door opening process before the user walks out of the elevator. The length of time the elevator door is open may vary depending on the specific situation. Generally speaking, when no one touches the elevator door, the normal opening time of the elevator door is usually between 3 and 10 seconds, depending on the design of the elevator and the requirements of the manufacturer. For example, the elevator door opening time can be set to 3 seconds, and the corresponding door opening and closing state timing information within the time period of the elevator door opening time can be 0000011111111111.

[0106] In actual situations, if it is detected that the corresponding door opening and closing state timing information during the time period of the elevator door opening duration in the actual door opening process is 000011111111111 or 000000111111111, it can be considered that the elevator is normal, and the detection result of the elevator door opening and closing state is normal. Because the node of 0 and 1 conversion is the critical point of the elevator opening state and the elevator closing state conversion, it can be classified as the elevator opening state or the closing state. If it is detected that the corresponding door opening and closing state timing information during the time period of the elevator door opening duration in the actual door opening process is 0000000000000111, it can be considered that the elevator fault is that the elevator door cannot be opened, and the detection result of the elevator door opening and closing state is abnormal.

[0107] In other embodiments, the elevator is in a leveling state, and after the user enters the elevator, the elevator door needs to go through a normal closing process.

[0108] The duration of the elevator door closing can vary depending on the specific situation. Generally speaking, when no one touches the elevator door, the normal closing time of the elevator door is usually between 3 and 10 seconds, depending on the design of the elevator and the requirements of the manufacturer. For example, the elevator door closing time can be set to 3 seconds, and the corresponding door opening and closing state timing information within the time period of the elevator door closing time can be 111111111100000.

[0109] In actual situations, if it is detected that the corresponding door opening and closing state timing information within the time period of the elevator door closing duration during the actual door closing process is 1111111111110000 or 111111111000000, it can be considered that the elevator is normal, and a detection result of a normal elevator door opening and closing state is obtained. If it is detected that the corresponding door opening and closing state timing information within the time period of the elevator door closing duration during the actual door closing process is 1111111111111111, it can be considered that the elevator fault is that the elevator door cannot be closed, and a detection result of an abnormal elevator door opening and closing state is obtained.

[0110] In some other embodiments, by detecting each frame of the elevator door area image, if the detection result is that there is someone in the elevator, the elevator is in the elevator closed state, and the speed sensor installed on the top of the elevator detects that the elevator is in a stationary state, then the timer starts. If the timer duration exceeds 60 seconds, a pop-up alarm is sent to the monitoring platform that someone is trapped.

[0111] In the above-mentioned elevator door opening and closing state detection method, first, the target elevator image sequence corresponding to the designated area of ​​the elevator door is obtained; then, each frame of the elevator door area image in the target elevator image sequence is classified and processed to obtain the probability data sequence corresponding to the elevator door opening and closing state; then, based on the probability data sequence corresponding to the elevator door opening and closing state, the door opening and closing state timing information representing the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence is generated; finally, the elevator door opening and closing state is detected according to the door opening and closing state timing information to obtain the door opening and closing state detection result. It can reduce the artificial setting logic and realize end-to-end reasoning. Through the above-mentioned implementation method, not only can the real-time and accurate detection of the elevator door opening and closing state be realized. Moreover, by detecting the door opening and closing state of each frame image in the image sequence and outputting the door opening and closing state timing information, it can reduce the interference of factors such as the internal environment of the elevator car, the external environment of the elevator car, and people entering and exiting the elevator on the detection results, thereby improving the generalization and robustness of the detection method.

[0112] In some embodiments, see Figure 2 , generating the door opening and closing state timing information based on the probability data sequence corresponding to the door opening and closing state of the elevator may include the following steps:

[0113] S210, filtering the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence.

[0114] Filtering can be used to reduce or eliminate noise, oscillation or irregularity in data. By filtering data, changes or mutations can be reduced and data can be made more continuous and predictable. Filtering can be achieved by filtering algorithms, which can be moving average filtering, median filtering, or Gaussian filtering.

[0115] In some cases, filtering the probability data sequence corresponding to the elevator door opening and closing status can make the probability data more regular and easier to interpret, which is helpful for data analysis.

[0116] Specifically, by filtering the probability data sequence corresponding to the elevator door opening and closing state through the filter function, the protruding part in the probability data sequence corresponding to the elevator door opening and closing state can be smoothed to obtain a filtered probability sequence. Figure 1g, curve 112 is a curve diagram corresponding to the probability sequence after filtering.

[0117] For example, a sliding average window is used to filter the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence. The window size can be adjusted according to the frame rate of the camera.

[0118] S220, binarizing the filtered probability sequence to obtain the switch state timing information.

[0119] Among them, binarization processing can be a method of converting continuous data into discrete data, by setting a threshold, setting values ​​greater than the threshold to a fixed value, and setting values ​​less than the threshold to another fixed value.

[0120] In some cases, binarization simplifies the representation and processing of data to reduce storage space and computational cost. Binarization can make features easier to distinguish and more suitable for data analysis.

[0121] Specifically, the threshold of the binarization process is set to D. If the probability data sequence corresponding to the elevator door opening and closing state is composed of the corresponding probability data of each frame of the elevator door area image in the elevator opening state, the filtered probability sequence corresponding to the elevator door opening and closing state is binarized. When the filtered probability data corresponding to the elevator door opening and closing state is greater than or equal to the threshold D, the data is set to the fixed value 1 corresponding to the elevator opening state, and when the filtered probability data corresponding to the elevator door opening and closing state is less than the threshold D, the data is set to the fixed value 0 corresponding to the elevator closing state. If the probability data sequence corresponding to the elevator door opening and closing state is composed of the corresponding probability data of each frame of the elevator door area image in the elevator closing state, the filtered probability sequence corresponding to the elevator door opening and closing state is binarized. When the filtered probability data corresponding to the elevator door opening and closing state is greater than the threshold D, the data is set to the fixed value 0 corresponding to the elevator closing state, and when the filtered probability data corresponding to the elevator door opening and closing state is less than or equal to the threshold D, the data is set to the fixed value 1 corresponding to the elevator opening state.

[0122] Exemplarily, the threshold of the binarization process is set to 0.5. If the probability data sequence corresponding to the elevator door opening and closing state is composed of the corresponding probability data of each frame of the elevator door area image in the elevator opening state, the filtered probability sequence can be [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0]. The filtered probability sequence is binarized, and the filtered probability data greater than or equal to 0.5 is set to a fixed value 1 corresponding to the elevator opening state, and the filtered probability data less than 0.5 is set to a fixed value 0 corresponding to the elevator closing state. Therefore, the filtered probability sequence [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1, 0.0] can be converted to [1, 1, 1, 1, 1, 0, 0, 0, 0, 0].

[0123] If the probability data sequence corresponding to the elevator door opening and closing state is a sequence composed of the corresponding probability data of each frame of the elevator door area image in the elevator closing state, the filtered probability sequence can be [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0]. The filtered probability sequence is binarized, and the filtered probability data greater than 0.5 is set to a fixed value of 0 corresponding to the elevator opening state, and the filtered probability data less than or equal to 0.5 is set to a fixed value of 1 corresponding to the elevator closing state. Therefore, the filtered probability sequence [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] can be converted to [1, 1, 1, 1, 1, 0, 0, 0, 0, 0].

[0124] In the above-mentioned elevator door opening and closing status detection method, the probability data sequence corresponding to the elevator door opening and closing status is filtered to obtain a filtered probability sequence, and the filtered probability sequence is binarized to obtain the door opening and closing status timing information, so as to subsequently detect the elevator door opening and closing status and obtain the door opening and closing status detection result.

[0125] In some embodiments, see Figure 3 , binarizing the filtered probability sequence to obtain the switch state timing information, which may include the following steps:

[0126] S310, binarizing the filtered probability sequence to obtain initial state sequence information.

[0127] S320: If there are false detection elements in the initial state sequence information, the values ​​of the false detection elements are updated using preset values ​​to obtain the door opening and closing state timing information.

[0128] Among them, at the moment corresponding to the false detection element, the elevator door opening and closing state does not change but the value of the false detection element suddenly changes, and the preset value is the value of the adjacent element of the false detection element.

[0129] In some cases, since the elevator door's elevator open state and elevator closed state are completed within a time period, the elevator door open and close state will not be suddenly switched multiple times within the time corresponding to a single frame of the elevator door area image or the time corresponding to several frames of the elevator door area image (the time is extremely short). Even if a failure occurs, it will be a stable state change process. Therefore, when there is a sudden change in the initial state sequence information but the elevator door open and close state has not changed, it needs to be corrected.

[0130] Specifically, the probability sequence after filtering is binarized to obtain the initial state sequence information. If there is an element in the initial state sequence information where the elevator door opening and closing state does not change but the value changes suddenly, that is, a false detection element, the value of the adjacent element of the false detection element is set to a preset value. The value of the false detection element is updated using the preset value to obtain the door opening and closing state timing information. For example, please refer to Figure 1h , it can be considered that there are falsely detected elements in region 114, and it is updated to the value of the adjacent element, and we get Figure 1h The gate state timing information after the elements in the correction corresponds to the curve 116. Figure 1i , curve 116 may be a curve graph corresponding to the timing information of the switch door state.

[0131] In some implementations, if the initial state sequence is 000000000010000, there is an element that is 1 and the values ​​of the adjacent elements of the element are all different from the values ​​of the element, it can be considered that the duration of the elevator switching from the elevator closed state to the elevator open state and then to the elevator closed state is very short, and it is considered that the target binary classification model has caused a false detection. The mutation element 1 in the initial state sequence 0000000000010000 is updated to the value of its adjacent element 0, and the detection sequence is 000000000000000, so as to achieve the correction of the initial state sequence 000000000010000.

[0132] In the above elevator door opening and closing state detection method, the probability sequence after filtering is binarized to obtain the initial state sequence information. If there is a false detection element in the initial state sequence information, the value of the false detection element is updated with a preset value to obtain the door opening and closing state timing information. By correcting the false detection element, the accuracy and reliability of the door opening and closing state timing information can be improved.

[0133] In some embodiments, the door opening and closing status of an elevator is detected according to the door opening and closing status timing information to obtain the door opening and closing status detection result, which may include: if the elevator is in a moving state, and it is determined according to the door opening and closing status timing information that the elevator is in a non-closed state within a preset time length, a detection result of an abnormal elevator door opening and closing status is obtained.

[0134] The preset time length can be set according to the specific situation. For example, the preset time length can be 2 seconds when the elevator is in the elevator start state.

[0135] Specifically, the speed sensor is used to detect whether the elevator is in motion. When the elevator is in motion, the value of the element in the door opening and closing state timing information should always be 0. However, based on the actual situation, in the door opening and closing state timing information obtained after classification, filtering and binarization of the target elevator image sequence, there is a case where the element value is continuously 1, and the duration of continuous 1 reaches a preset duration. It can be determined that the elevator is in a non-closed state within the preset duration, and the detection result of abnormal elevator door opening and closing state is obtained.

[0136] For example, when the elevator is in motion, the door opening and closing state timing information should be 0000000000. However, the door opening and closing state timing information actually detected is 0011111111, and it is considered that the elevator door is in an open state during the movement of the elevator. It can be determined that the elevator is in a non-closed state within the preset time length, and the detection result of the elevator door opening and closing state abnormality is obtained.

[0137] In the above elevator door opening and closing state detection method, if the elevator is in a moving state, and it is determined that the elevator is in a non-closed state within a preset time period according to the door opening and closing state timing information, the abnormal door opening and closing state detection result is obtained, which helps maintenance personnel to perform timely maintenance and repair of the elevator, thereby improving the safety of the elevator.

[0138] In some embodiments, see Figure 4a , the probability data in the probability data sequence is the output of the target binary classification model. The training process of the target binary classification model can include the following steps:

[0139] S410: Acquire a first state image obtained by photographing the elevator door in the first state, and a second state image obtained by photographing the elevator door in the second state.

[0140] The first state may be the elevator open state in the above embodiment. The second state may be the elevator closed state in the above embodiment. The first state corresponds to the first elevator door opening width range, and the second state corresponds to the second elevator door opening width range. Within the first elevator door opening width range, the image acquisition device in the elevator car can capture the outside of the elevator. Within the second elevator door opening width range, the image acquisition device in the elevator car cannot capture the outside of the elevator.

[0141] Specifically, when the image acquisition device in the elevator car can capture the exterior of the elevator through the elevator door opening width range, the elevator door is in the first state, and the elevator door opening width range can be defined as the first elevator door opening width range. The elevator door in the first state is captured by the image acquisition device in the elevator car to obtain a first state image. When the image acquisition device in the elevator car cannot capture the exterior of the elevator through the elevator door opening width range, the elevator door is in the second state, and the elevator door opening width range can be defined as the second elevator door opening width range. The elevator door in the second state is captured by the image acquisition device in the elevator car to obtain a second state image.

[0142] Exemplarily, in order to improve the accuracy and generalization ability of the target binary classification model, approximately 600,000 elevator images of elevators of different types and with different camera installation angles in the first state and the second state may be collected.

[0143] S420: intercept the first state image to obtain a first state image sample corresponding to the designated area of ​​the elevator door.

[0144] S430: intercept the second state image to obtain a second state image sample corresponding to the designated area of ​​the elevator door.

[0145] S440: Construct a training sample set based on the first state image samples and the second state image samples.

[0146] Specifically, the first state image is intercepted, and the area where the entry and exit object and the elevator door opening and closing area do not overlap in the elevator door opening and closing direction is retained when the entry and exit object passes through the elevator door opening and closing area, so as to obtain the first state image sample. The second state image is intercepted, and the area where the entry and exit object and the elevator door opening and closing area do not overlap in the elevator door opening and closing direction is retained when the entry and exit object passes through the elevator door opening and closing area, so as to obtain the second state image sample. A training sample set is constructed based on the first state image sample and the second state image sample.

[0147] Exemplarily, the first state image sample or the second state image sample may be an RBG three-channel image with a height of 28 and a width of 112. The collected elevator images are divided into a training set and a test set in a ratio of 9:1.

[0148] S450: Train the initial binary classification model using the training sample set to obtain a target binary classification model.

[0149] Specifically, the training sample set includes several frames of first state image samples or second state image samples. The first state image samples correspond to the first label, and the second state image samples correspond to the second label. The first state image samples are used as the input of the initial binary classification model to obtain the probability value of the elevator being in the open state and the probability value of the elevator being in the closed state corresponding to the first state image samples. Based on the obtained probability value of the elevator being in the open state and the probability value of the elevator being in the closed state corresponding to the first state image samples and the first label, the model loss value is determined, and the initial binary classification model is updated based on the model loss value. The second state image samples are used as the input of the initial binary classification model to obtain the probability value of the elevator being in the open state and the probability value of the elevator being in the closed state corresponding to the second state image samples. Based on the obtained probability value of the elevator being in the open state and the probability value of the elevator being in the closed state corresponding to the second state image samples and the second label, the model loss value is determined, and the initial binary classification model is updated based on the model loss value. By analogy, the updated initial binary classification model is continuously trained, and when the model training stop condition is reached, the target binary classification model can be obtained. The model training stop condition can be that the model loss value tends to converge, or that the number of training rounds reaches a preset number of rounds.

[0150] In some embodiments, the label may be a real probability data sequence corresponding to the training sample set. The training sample set is used as the input of the initial binary classification model to obtain the probability data sequence. The model loss value is determined based on the obtained probability data sequence and the label, and the initial binary classification model is updated based on the model loss value. When the model training stop condition is reached, the target binary classification model can be obtained. Exemplarily, the performance of the target binary classification model can be evaluated using ten-fold cross validation.

[0151] In the above-mentioned elevator door opening and closing state detection method, by obtaining the target binary classification model, the probability data sequence corresponding to the elevator door opening and closing state can be obtained, so as to subsequently detect the elevator door opening and closing state and obtain the door opening and closing state detection result.

[0152] In some implementations, the target binary classification model can be generated based on a lightweight Lcnet (lightweight communication network). The lightweight Lcnet occupies little storage, has high accuracy, fast inference speed, and is easy to quantize and deploy to edge hardware.

[0153] See also Figure 4bThe structure of the target binary classification model consists of a 3×3 convolution layer, three ordinary separable convolution layers (Separable Convolution), four separable convolution layers with Rep (Reparameter) reparameterization technology, two separable convolution layers with shotcut, a GAP (Global-Average-Pooling) global average pooling layer, a 1x1 convolution layer and a FC (Fully Connect) fully connected layer.

[0154] The 3×28×112 elevator door area image is input into the target binary classification model for classification processing. The elevator door area image first passes through a 3×3 convolution layer, then passes through three ordinary separable convolution layers, passes through four layers of separable convolution with Rep re-parameterization technology, and then passes through two layers of separable convolution layers with shotcut. Finally, it passes through a GAP global average pooling layer, a 1x1 convolution layer, and a FC fully connected layer to obtain 2×1×1 probability data.

[0155] See also Figure 4c , the ordinary separable convolution consists of a layer of Depthwise Convolution and a layer of Pointwise Convolution. It can be seen that in Depthwise Convolution, the convolution kernel size is n. n is usually 3, 5 or 7.

[0156] See also Figure 4d During the model training process, the Depthwise Convlution (DW Conv) layer changes from the original convolution kernel size n to 1, 3, and 5 branches, so that the network can have different receptive fields and obtain features of different scales, thereby improving the recognition accuracy of the model. During model reasoning, the reparameterization technology merges the branches of three scales into one branch, which can maintain the accuracy of multiple branches and obtain the efficiency of a single branch.

[0157] See also Figure 4e , the ordinary structure has only one layer. In order to make the depth-separable convolution have stronger fitting ability, two layers of PW convolution can be used. At the same time, in order to control the model efficiency without affecting it, the first layer of PW convolution compresses the feature map in the channel dimension, and the second layer of PW convolution restores the feature map channel by enlarging it. The above method can improve the model performance.

[0158] In lightweight convolutional neural networks, the element-wise addition operation brought by the residual structure will affect the speed of the model, so the residual structure is only used in the last stage and a shortcut is added to the block.

[0159] The GAP (Global-Average-Pooling) layer is usually directly connected to the classification layer, but in a lightweight network, the features extracted after the GAP layer will not be further fused and processed. If a 1x1 convolutional layer (equivalent to the FC layer) is used after the GAP layer, the features after the GAP layer will not pass directly through the classification layer, but will be fused first, and the fused features will be classified. This can improve the accuracy without affecting the model inference speed.

[0160] In some embodiments, see Figure 5 , the target binary classification model is used as the basic model in the specified elevator operation environment; the following steps may be included:

[0161] S510. Determine a target elevator in a specified elevator operating environment.

[0162] S520: Acquire a video file captured by an image capture device installed in the target elevator.

[0163] S530: construct a supplementary sample set for training a basic model based on the video file.

[0164] S540. Continue to train the basic model using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

[0165] The designated elevator operating environment may be a new operating environment after the elevator operating environment has changed, such as a residential area where a new elevator is installed. For example, at night, the light brightness inside the elevator is too high or the ambient light outside the elevator is dark.

[0166] In some cases, the accuracy of the target binary classification model will decrease in specific scenarios. Therefore, it is necessary to build a supplementary sample set for specific scenarios. Use the supplementary sample set to continue training the basic model to obtain a target model suitable for the specified elevator operating environment and improve the accuracy of elevator door opening and closing status detection. For example, in a community where a new elevator is installed, the elevators in the community are likely to be the same, so it is sufficient to collect a complete door opening and closing video file for the same type of elevator.

[0167] Specifically, an elevator in a specified elevator operating environment can be determined as a target elevator in an artificial manner. A video file captured by an image acquisition device in the target elevator can be obtained from a storage unit of the image acquisition device installed in the target elevator. A video file captured by an image acquisition device installed in the target elevator can also be obtained from a server. The video file is framed to obtain several frames of elevator door images included in the video file. Several elevator door images are intercepted to generate elevator door area images. The elevator door area images corresponding to the video file can be used to construct a supplementary sample set for training a basic model, i.e., a target binary classification model. The basic model is further trained using the supplementary sample set, and the parameters of the basic model are updated to obtain a target model suitable for the specified elevator operating environment.

[0168] It should be noted that the supplementary sample set can be added to the original sample set to tune the basic model. The tuning of the basic model is controlled by a code-free visual interactive platform. After the basic model is updated, a target model suitable for the specified elevator operating environment is obtained. The control center can send the target model for the specified elevator operating environment to the image acquisition device inside the selected target elevator.

[0169] In the above elevator door opening and closing state detection method, a supplementary sample set of the basic model is constructed, and the basic model is further trained using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment. The generalization, interpretability and accuracy of the model can be improved. The supplementary sample set can include a variety of application scenarios that may often appear in actual applications. Using the supplementary sample set to continue training the basic model can improve the robustness of the model so that it can still maintain good performance when facing new and unknown situations.

[0170] In some embodiments, see Figure 6 , constructing a supplementary sample set for training a basic model based on video files may include the following steps:

[0171] S610: Determine a first starting time corresponding to when the elevator is in the first state and a second starting time corresponding to when the elevator is in the second state in the video file.

[0172] S620: Capture a first state video segment and a second state video segment from the video file based on the first starting time and the second starting time.

[0173] S630: Construct a supplementary sample set based on the first-state video segment and the second-state video segment.

[0174] In some cases, the time that the elevator is in the first state (elevator open state) and the second state (elevator closed state) is usually a few seconds. If someone blocks the elevator door, the time that the elevator is in the first state (elevator open state) will increase. Therefore, it is sufficient to capture a few minutes of door opening and closing video files when the elevator is in the first state (elevator open state) or the second state (elevator closed state).

[0175] Specifically, after determining the video file, open the video file corresponding to the target elevator, and through the video playback and preview functions, you can see the switch state of the elevator. Based on the switch state of the elevator, the first starting time corresponding to the elevator being in the first state and the second starting time corresponding to the elevator being in the second state in the video file can be determined. Starting from the first starting time, by dragging and clicking the video file with the mouse, a first state video segment of the elevator in the first state can be intercepted from the video file. Starting from the second starting time, by dragging and clicking the video file with the mouse, a second state video segment of the elevator in the second state can be intercepted from the video file. The first state video segment and the second state video segment are subjected to frame extraction and interception operations to generate an elevator door area image. The elevator door area image corresponding to the first state video segment and the second state video segment can be used to construct a supplementary sample set for training the basic model, i.e., the target binary classification model.

[0176] It should be noted that the first state video segment and the second state video segment can be the smallest video clip unit. Therefore, more video clip units or image frames in the video clip units can be selected as supplementary samples, so that the basic model can better learn unconventional scenes.

[0177] It should be noted that this implementation requires human participation, so the target model is a semi-automatic self-learning model. Human participation is necessary for supplementing the sample set. By calibrating the data by humans, accurate identification of all application environments and scenarios can be achieved.

[0178] In the above elevator door opening and closing state detection method, adding a supplementary sample set can increase the diversity of data, reduce the excessive dependence of the basic model on specific samples, and thus reduce the risk of overfitting. The supplementary sample set can contain a variety of application scenarios that may often appear in actual applications. Using the supplementary sample set to continue training the basic model can improve the robustness of the model, so that it can still maintain good performance when facing new and unknown situations.

[0179] In some implementations, the designated elevator operating environment includes a new operating environment after the elevator operating environment changes. The elevator operating environment change includes any of the following situations:

[0180] The internal environment of the elevator car changes, including adding any one of stickers, hanging lights, protection panels, and colored silk.

[0181] The external environment of the elevator car changes, including adding any one of stickers, hanging lights, protection plates, and colored silk.

[0182] The target binary classification model trained for the first operating environment is used in the second operating environment, which is different from the first operating environment; the supplementary sample set is a video file collected in the second operating environment.

[0183] The first operating environment may be a conventional environment, such as no decorative items inside the elevator car and no decorative items outside the elevator car, which will not interfere with or block the state recognition of the elevator door. The second operating environment may be an unconventional environment. For example, if stickers, hanging lights, protective plates, and colored silk are added to the interior of the elevator car and the exterior of the elevator car, the state recognition of the elevator door may be interfered with or blocked.

[0184] In some cases, when the elevator interior environment is in an unconventional environment due to stickers, hanging lights, protective plates, or colored silk, or when the elevator exterior environment is in an unconventional environment due to stickers, hanging lights, protective plates, or colored silk, the accuracy of the basic model will decrease. Also, because the elevator interior scene and environment are relatively fixed, the elevator exterior environment will change after the elevator door opens, but the difference between different floors is also small, so based on elevators where the recognition accuracy needs to be further improved, the target model obtained can flow back data.

[0185] Specifically, the target binary classification model trained for the first operating environment is deployed in the image acquisition device inside the elevator in the second operating environment. Since the second operating environment is different from the first operating environment, it is necessary to collect video files in the second operating environment, and a supplementary sample set can be constructed based on the video files.

[0186] In the above elevator door opening and closing state detection method, various application scenarios that may often appear in actual applications are used to construct a supplementary sample set. The supplementary sample set can improve the robustness of the model so that it can still maintain good performance when facing new and unknown situations.

[0187] The embodiments of this specification also provide a heterogeneous chip, including an embedded neural network processor NPU, a central processing unit CPU, a memory, and a computer program stored in the memory and configured to be executed by the central processing unit CPU and the embedded neural network processor NPU. When the central processing unit CPU and the embedded neural network processor NPU execute the computer program, they implement any of the above-mentioned implementation methods.

[0188] Among them, the binary classification model included in the model reasoning unit is deployed on the NPU core (Neural-network Process Units, embedded neural network processor), and the post-processing unit included in the model reasoning unit is deployed on the CPU core (Central Processing Unit, central processing unit).

[0189] The present specification also provides a method for detecting the state of an elevator door opening and closing based on deep learning. For example, see Figure 7 The elevator door opening and closing state detection method may include the following steps:

[0190] S702: Acquire a target elevator image sequence.

[0191] Among them, the target elevator image sequence includes several frames of elevator door area images corresponding to the designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object and the elevator door opening and closing area do not overlap in the opening and closing direction of the elevator door when the entry and exit object passes through the elevator door opening and closing area.

[0192] S704: Classify each frame of the elevator door area image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state.

[0193] S706: Filter the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence.

[0194] S708: Binarize the filtered probability sequence to obtain initial state sequence information.

[0195] S710: If there are false detection elements in the initial state sequence information, use a preset value to update the value of the false detection element to obtain the door opening and closing state timing information.

[0196] Among them, at the moment corresponding to the false detection element, the elevator door opening and closing state has not changed, but the value of the false detection element has suddenly changed; the preset value is the value of the adjacent element of the false detection element. The door opening and closing state timing information is used to indicate the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence;

[0197] S712: If the elevator is in motion, and it is determined that the elevator is in a non-closed state within a preset time period based on the door opening and closing state timing information, a detection result of an abnormal elevator door opening and closing state is obtained.

[0198] This specification provides an elevator door opening and closing state detection device 800 based on deep learning. Figure 8The elevator door opening and closing state detection device 800 based on deep learning includes: an image sequence acquisition module 810, a regional image classification module 820, a state timing generation module 830, and a door state detection module 840.

[0199] The image sequence acquisition module 810 is used to acquire a target elevator image sequence; wherein the target elevator image sequence includes a plurality of frames of elevator door area images corresponding to a designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object does not overlap with the elevator door opening and closing area in the elevator door opening and closing direction when the entry and exit object passes through the elevator door opening and closing area;

[0200] The regional image classification module 820 is used to classify each frame of the elevator door regional image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state;

[0201] The state time sequence generation module 830 is used to generate the door opening and closing state time sequence information based on the probability data sequence corresponding to the door opening and closing state of the elevator; wherein the door opening and closing state time sequence information is used to represent the door opening and closing state of the elevator door in the time period corresponding to the target elevator image sequence;

[0202] The door state detection module 840 is used to detect the elevator door opening and closing state according to the door opening and closing state timing information to obtain the door opening and closing state detection result.

[0203] In one embodiment, the state timing generation module is further used to filter the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence; and binarize the filtered probability sequence to obtain the door opening and closing state timing information.

[0204] In some embodiments, the state timing generation module is also used to binarize the filtered probability sequence to obtain initial state sequence information; if there is a false detection element in the initial state sequence information, the value of the false detection element is updated using a preset value to obtain the door opening and closing state timing information; wherein, at the moment corresponding to the false detection element, the elevator door opening and closing state does not change but the value of the false detection element suddenly changes; the preset value is the value of the adjacent element of the false detection element.

[0205] In some embodiments, the door status detection module is also used to, if the elevator is in a moving state, determine that the elevator is in a non-closed state within a preset time period based on the door opening and closing state timing information, to obtain a detection result of abnormal elevator door opening and closing state.

[0206] In some embodiments, the probability data in the probability data sequence is output by a target binary classification model; the device also includes a classification model training module, which is used to obtain a first state image obtained by photographing an elevator door in a first state, and a second state image obtained by photographing an elevator door in a second state; the first state corresponds to a first elevator door opening width range, and the second state corresponds to a second elevator door opening width range; within the first elevator door opening width range, the image acquisition device in the elevator car can photograph the outside of the elevator, and within the second elevator door opening width range, the image acquisition device in the elevator car cannot photograph the outside of the elevator; the first state image is intercepted to obtain a first state image sample corresponding to the specified area of ​​the elevator door; the second state image is intercepted to obtain a second state image sample corresponding to the specified area of ​​the elevator door; a training sample set is constructed based on the first state image sample and the second state image sample; the initial binary classification model is trained using the training sample set to obtain the target binary classification model.

[0207] In some embodiments, the target binary classification model is used as a basic model in a specified elevator operating environment; the device also includes a classification model training module, which is used to determine a target elevator in the specified elevator operating environment; obtain a video file captured by an image acquisition device installed in the target elevator; construct a supplementary sample set for training the basic model based on the video file; and continue to train the basic model using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

[0208] For a detailed description of the elevator door opening and closing state detection device based on deep learning, please refer to the above description of the elevator door opening and closing state detection method based on deep learning, which will not be repeated here.

[0209] An embodiment of the present specification provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method steps in the above embodiment are implemented.

[0210] An embodiment of the present specification provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method in any of the above embodiments are implemented.

[0211] One embodiment of the present specification provides a computer program product, which includes instructions. When the instructions are executed by a processor of a computer device, the computer device can perform the steps of the method of any of the above embodiments.

[0212] In some embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a method for detecting the state of an elevator door switch based on deep learning is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0213] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a portion of the structure related to the solution disclosed in this specification, and does not constitute a limitation on the computer device to which the solution disclosed in this specification is applied. Specifically, the computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0214] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.

Claims

1. A method for detecting the state of elevator door opening and closing based on deep learning. It is characterized in that The method comprises: Acquire a target elevator image sequence; wherein the target elevator image sequence includes a plurality of frames of elevator door area images corresponding to a designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object and the elevator door opening and closing area do not overlap in the elevator door opening and closing direction when the entry and exit object passes through the elevator door opening and closing area; the elevator door opening and closing area is the space range occupied by the elevator door during the opening and closing process; The target binary classification model is used to classify each frame of the elevator door area image in the target elevator image sequence to obtain a probability data sequence corresponding to the elevator door opening and closing state; wherein the classification result of the elevator door area image is the elevator opening state or the elevator closing state; the target binary classification model is also used as a basic model under the specified elevator operating environment; the elevator operating environment change includes: changes in the internal and external environment of the elevator car, including adding any one of stickers, hanging lights, protective panels, and colored silk; The probability data sequence corresponding to the elevator door opening and closing state is filtered to obtain a filtered probability sequence; the filtered probability sequence is binarized to obtain initial state sequence information; if there is a false detection element in the initial state sequence information, the value of the false detection element is updated by a preset value to obtain the door opening and closing state timing information; wherein, at the moment corresponding to the false detection element, the elevator door opening and closing state is not converted but the value of the false detection element changes suddenly; the preset value is the value of the adjacent element of the false detection element; wherein, the door opening and closing state timing information is used to indicate the door opening and closing state of the elevator door within the time period corresponding to the target elevator image sequence; the elevator opening state is represented by 1, the elevator closing state is represented by 0, and the node where the element value 0 and the element value 1 are converted is the critical point of the conversion between the elevator opening state and the elevator closing state; The elevator door opening and closing state is detected according to the door opening and closing state timing information to obtain the door opening and closing state detection result.

2. The method according to claim 1, It is characterized in that The step of detecting the elevator door opening and closing state according to the door opening and closing state timing information to obtain the door opening and closing state detection result includes: If the elevator is in a moving state, and it is determined according to the door opening and closing state timing information that the elevator is in a non-closed state within a preset time period, a detection result of abnormal door opening and closing state of the elevator is obtained.

3. The method according to claim 1, It is characterized in that The training process of the target binary classification model includes: Acquire a first state image obtained by photographing an elevator door in a first state, and a second state image obtained by photographing an elevator door in a second state; wherein the first state corresponds to a first elevator door opening width range, and the second state corresponds to a second elevator door opening width range; within the first elevator door opening width range, an image acquisition device in the elevator car can photograph the outside of the elevator, and within the second elevator door opening width range, an image acquisition device in the elevator car cannot photograph the outside of the elevator; Intercepting the first state image to obtain a first state image sample corresponding to the designated area of ​​the elevator door; Intercepting the second state image to obtain a second state image sample corresponding to the designated area of ​​the elevator door; Constructing a training sample set based on the first state image samples and the second state image samples; The initial binary classification model is trained using the training sample set to obtain the target binary classification model.

4. The method according to claim 3, It is characterized in that The target binary classification model is used as a basic model in a specified elevator operation environment; the method further includes: Determine a target elevator under the specified elevator operating environment; Acquire a video file captured by an image acquisition device installed in the target elevator; Constructing a supplementary sample set for training the basic model based on the video file; The basic model is further trained using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

5. The method according to claim 4, It is characterized in that The constructing a supplementary sample set for training the basic model based on the video file includes: Determine a first starting time corresponding to the elevator being in the first state and a second starting time corresponding to the elevator being in the second state in the video file; Based on the first starting time and the second starting time, extract a first state video segment and a second state video segment from the video file; The supplementary sample set is constructed based on the first state video segment and the second state video segment.

6. The method according to claim 4, It is characterized in that The designated elevator operating environment includes a new operating environment after the elevator operating environment changes, and the elevator operating environment changes include: The target binary classification model trained for the first operating environment is used in a second operating environment, where the second operating environment is different from the first operating environment; and the supplementary sample set is a video file collected in the second operating environment.

7. An elevator door opening and closing state detection device based on deep learning, It is characterized in that The device comprises: An image sequence acquisition module is used to acquire a target elevator image sequence; wherein the target elevator image sequence includes a plurality of frames of elevator door area images corresponding to a designated area of ​​the elevator door; the designated area of ​​the elevator door is an area where the entry and exit object and the elevator door opening and closing area do not overlap in the elevator door opening and closing direction when the entry and exit object passes through the elevator door opening and closing area; the elevator door opening and closing area is the space range occupied by the elevator door during the opening and closing process; The regional image classification module is used to classify each frame of the elevator door regional image in the target elevator image sequence using the target binary classification model to obtain a probability data sequence corresponding to the elevator door opening and closing state; wherein the classification result of the elevator door regional image is the elevator opening state or the elevator closing state; the target binary classification model is also used as a basic model under the specified elevator operating environment; the elevator operating environment change includes: changes in the internal and external environment of the elevator car, including adding any one of stickers, hanging lights, protective panels, and colored silk; A state timing generation module is used to filter the probability data sequence corresponding to the elevator door opening and closing state to obtain a filtered probability sequence; binarize the filtered probability sequence to obtain initial state sequence information; if there is a false detection element in the initial state sequence information, use a preset value to update the value of the false detection element to obtain the door opening and closing state timing information; wherein, at the moment corresponding to the false detection element, the elevator door opening and closing state does not change but the value of the false detection element changes suddenly; the preset value is the value of the adjacent element of the false detection element; wherein, the door opening and closing state timing information is used to indicate the door opening and closing state of the elevator door within the time period corresponding to the target elevator image sequence; the elevator opening state is represented by 1, the elevator closing state is represented by 0, and the node where the element value 0 and the element value 1 are converted is the critical point of the conversion between the elevator opening state and the elevator closing state; The door status detection module is used to detect the door opening and closing status of the elevator according to the door opening and closing status timing information to obtain the door opening and closing status detection result.

8. The device according to claim 7, It is characterized in that The door status detection module is also used to obtain a detection result of abnormal elevator door opening and closing status if the elevator is in a moving state and determines that the elevator is in a non-closed state within a preset time period based on the door opening and closing status timing information.

9. The device according to claim 7, It is characterized in that The device also includes a classification model training module, which is used to obtain a first state image obtained by photographing an elevator door in a first state, and a second state image obtained by photographing an elevator door in a second state; the first state corresponds to a first elevator door opening width range, and the second state corresponds to a second elevator door opening width range; within the first elevator door opening width range, the image acquisition device in the elevator car can photograph the outside of the elevator, and within the second elevator door opening width range, the image acquisition device in the elevator car cannot photograph the outside of the elevator; Intercepting the first state image to obtain a first state image sample corresponding to the designated area of ​​the elevator door; The second state image is intercepted to obtain a second state image sample corresponding to the designated area of ​​the elevator door; a training sample set is constructed based on the first state image sample and the second state image sample; The initial binary classification model is trained using the training sample set to obtain the target binary classification model.

10. The device according to claim 9, It is characterized in that The target binary classification model is used as a basic model in a specified elevator operating environment; the device also includes a classification model training module for determining a target elevator in the specified elevator operating environment; Acquire a video file captured by an image acquisition device installed in the target elevator; A supplementary sample set for training the basic model is constructed based on the video file; and the basic model is further trained using the supplementary sample set to obtain a target model suitable for the specified elevator operating environment.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program. It is characterized in that When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

13. A heterogeneous chip, It is characterized in that It includes an embedded neural network processor NPU, a central processing unit CPU, a memory, and a computer program stored in the memory and configured to be executed by the central processing unit CPU and the embedded neural network processor NPU. When the central processing unit CPU and the embedded neural network processor NPU execute the computer program, they implement the method described in any one of claims 1 to 6.

Images