A method and system for detecting abnormal behavior of elevator cars based on machine vision

Abstract

The present disclosure belongs to the field of data processing technology and provides a method and system for detecting abnormal behavior in an elevator car based on machine vision, comprising the following steps: acquiring video data in the elevator car; performing pedestrian detection based on the acquired video images to obtain pedestrian detection results; constructing a video sequence to be identified based on the obtained pedestrian detection results, and identifying abnormal behavior in the elevator car based on the constructed video sequence and an abnormal behavior recognition model; wherein the abnormal behavior detection model uses a deep separation convolution algorithm to perform classification and recognition of the video sequence.

Description

Technical Field

[0001] The present disclosure belongs to the field of data processing technology, and particularly relates to a method and system for detecting abnormal behavior of an elevator car based on machine vision. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the rapid development of China's economy and the gradual advancement of urbanization, more and more high-rise buildings such as shopping malls, residential areas, hotels and office buildings have begun to install elevators in large quantities, and elevators have been widely used. Elevators can effectively solve the problem of people going up and down stairs, but they also have some safety hazards. Its safety and reliability have gradually attracted people's attention in recent years.

[0004] The elevator car is a relatively closed space. When criminals commit violence or robbery in the elevator car, it is usually difficult for outsiders to detect the abnormal behavior in the elevator car in time. In addition, when passengers in the elevator destroy the elevator, it is easy to cause the elevator to malfunction. Summary of the Invention

[0005] In order to solve the above problems, the present disclosure proposes a method and system for detecting abnormal behavior in an elevator car based on machine vision, which performs safety monitoring of passengers in the elevator car, detects the behavior of passengers in the elevator car in real time, improves the safety inside the elevator car, and reduces abnormal accidents.

[0006] According to some embodiments, a first solution of the present disclosure provides a method for detecting abnormal behavior of an elevator car based on machine vision, which adopts the following technical solutions:

[0007] A method for detecting abnormal behavior of an elevator car based on machine vision comprises the following steps:

[0008] Obtain video data inside the elevator car;

[0009] Perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result;

[0010] Constructing a video sequence to be identified based on the obtained pedestrian detection results, and identifying abnormal behaviors in the elevator car based on the constructed video sequence and the abnormal behavior recognition model;

[0011] The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence.

[0012] As a further technical limitation, in the process of acquiring the video image inside the elevator car, a video surveillance image containing a pedestrian target inside the elevator car is collected.

[0013] Furthermore, before identifying abnormal behavior in the elevator car, a pedestrian detection dataset is constructed based on the obtained video surveillance images, and pedestrians in the elevator car are detected and identified through single-stage target detection.

[0014] Furthermore, when a pedestrian is detected and identified in the elevator car, the channels of the input video frame and the output feature map of the single-stage target detection are spliced ​​for each frame of the image to obtain and save the fused feature tensor of each frame of the image.

[0015] Furthermore, if no pedestrian is detected and recognized in 10 consecutive frames, the saved fusion feature tensor is deleted.

[0016] Furthermore, the single-stage target detection adopts the yolov5s model, and a 3×1×1 convolution layer and a 2x upsampling module are added after the third output of the Neck layer in the yolov5s model to construct the output layer.

[0017] As a further technical limitation, in the process of identifying abnormal behavior in the elevator car, 3DMobilenet-v2 is used to construct an abnormal behavior recognition model to identify whether the video sequence contains abnormal behavior. The model input is the video feature sequence extracted from each video, and the model outputs the abnormal behavior prediction result for each video.

[0018] According to some embodiments, a second solution of the present disclosure provides an elevator car abnormal behavior detection system based on machine vision, which adopts the following technical solutions:

[0019] A machine vision-based elevator car abnormal behavior detection system, comprising:

[0020] an acquisition module configured to acquire video data in the elevator car; perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result;

[0021] The behavior detection model is configured to construct a video sequence to be identified based on the obtained pedestrian detection results, and identify abnormal behavior in the elevator car based on the constructed video sequence and the abnormal behavior recognition model;

[0022] The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence.

[0023] According to some embodiments, a third solution of the present disclosure provides a computer-readable storage medium, which adopts the following technical solution:

[0024] A computer-readable storage medium stores a program thereon, which, when executed by a processor, implements the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in the first aspect of the present disclosure.

[0025] According to some embodiments, a fourth solution of the present disclosure provides an electronic device, which adopts the following technical solution:

[0026] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in the first aspect of the present disclosure are implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present disclosure uses the yolov5s algorithm to detect pedestrians and determine whether there are pedestrians in the elevator. The abnormal behavior recognition model is only used to identify the input video sequence when there are pedestrians in the elevator car, which effectively reduces the computational complexity of the hardware equipment. The fighting behavior recognition model is constructed using 3D mobilenet-v2, and the 3D convolution in the model is decomposed to reduce the number of model parameters and make it easier to optimize during training. In addition, the feature map output by the yolov5s model and the input video are used together as the input data of the abnormal behavior recognition model, which enhances the feature expression ability of the input data, solves the problem of low feature extraction ability of the lightweight model 3Dmobilenet-v2, and makes the recognition results of the abnormal behavior recognition model more accurate. The technology adopted by the present disclosure is fast, accurate, and real-time, and is suitable for applications and engineering. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which constitute a part of the present disclosure, are used to provide a further understanding of the present disclosure. The exemplary embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation to the present disclosure.

[0030] Figure 1 is a flow chart of a method for detecting abnormal behavior of an elevator car based on machine vision in the first embodiment of the present disclosure;

[0031] Figure 2 1 is a schematic diagram of the overall workflow of the method for detecting abnormal behavior of an elevator car based on machine vision in the first embodiment of the present disclosure;

[0032] Figure 3 This is a structural block diagram of the elevator car abnormal behavior detection system based on machine vision in the second embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0036] In the absence of conflict, the embodiments of the present disclosure and the features thereof may be combined with each other.

[0037] Example 1

[0038] Embodiment 1 of the present disclosure introduces a method for detecting abnormal behavior of an elevator car based on machine vision.

[0039] like Figure 1 A method for detecting abnormal behavior of an elevator car based on machine vision is shown, comprising the following steps:

[0040] Obtain video data inside the elevator car;

[0041] Perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result;

[0042] Constructing a video sequence to be identified based on the obtained pedestrian detection results, and identifying abnormal behaviors in the elevator car based on the constructed video sequence and the abnormal behavior recognition model;

[0043] The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence.

[0044] As one or more implementation methods, when a pedestrian is detected in the elevator car, the input video frame and the output feature map of the current yolov5s model are channel-spliced ​​to obtain a fused feature tensor, and the fused feature tensor extracted from each frame image is saved.

[0045] In one or more implementations, if no pedestrian is detected for 10 consecutive frames, all stored fused feature tensors are deleted. When the number of stored fused feature tensors reaches 64, all fused feature tensors are spliced ​​in the temporal direction to construct a video feature sequence for abnormal behavior recognition.

[0046] As one or more implementations, the feature output layer of yolov5 is constructed by adding a 3×1×1 convolution layer and a 2x upsampling module after the third output of the Neck layer in the yolov5s model.

[0047] As one or more implementation methods, segments containing abnormal behavior and segments not containing abnormal behavior are selected from elevator car monitoring videos to form an elevator abnormal behavior dataset, wherein 1000 segments of normal videos and 1000 segments of abnormal behavior videos are included, and each video segment is 150 frames.

[0048] As one or more implementation methods, each video clip is sampled at equal intervals, and the sampled video clips are 64 frames. The resolution of the video frames in each video segment is scaled to 896×896 and input into the yolov5s model. The feature map output by the feature map output branch is spliced ​​with the input video frame scaled to 224×224 resolution in the channel direction to obtain a 6×224×224 fused feature tensor F. Finally, the fused feature tensor F generated by each frame image in each video segment is spliced ​​in the time sequence direction to obtain a video feature sequence of size 64×6×224×224. The video feature sequence extracted from each video segment is saved for subsequent abnormal behavior recognition model training.

[0049] As one or more implementation methods, 3D Mobilenet-v2 is used to build an abnormal behavior recognition model to identify whether a video sequence contains abnormal behavior. The model input is the video feature sequence extracted from each video, and the model outputs the abnormal behavior prediction result for each video.

[0050] In one or more implementations, the 3D convolution kernel of size 3×3×3 in 3D Mobilenet-v2 is decomposed into a 1×3×3 spatial convolution kernel and a 3×1×1 temporal convolution kernel. The 1×3×3 kernel is used to extract spatial feature information from the feature tensor, and the 3×1×1 kernel is used to extract temporal feature information from the feature tensor, to approximate the 3D convolution operation. Compared to the 3DMobilenet-v2 model, the improved 3D Mobilenet-v2 model has fewer parameters and is easier to optimize during training.

[0051] In this embodiment, the abnormal behavior detection method in the elevator car is divided into three stages, namely, the detection of people in the elevator car, the generation of the video sequence to be identified, and the abnormal behavior identification stage. Figure 2 Provide a detailed introduction.

[0052] (1) Pedestrian detection in the elevator car

[0053] The video captured by the surveillance camera inside the elevator car is obtained. Then, one frame of image is extracted every 25 frames of the collected video. The extracted images are calibrated using the LabelImg tool to construct a pedestrian detection dataset.

[0054] Use yolov5s to build a pedestrian detection model. Add a 3×1×1 convolutional layer and a 2x upsampling module after the third output of the Neck layer in yolov5s. Construct an image feature output layer so that the yolov5 model outputs both the detection results and the image feature map. Use the constructed dataset to train the yolov5s model to obtain a pedestrian detection model.

[0055] Use FFmpeg to read surveillance camera video and parse the images by skipping frames. For 25fps video, detect every 3 frames and output pedestrian detection results and feature maps extracted from each frame.

[0056] (2) Video sequence generation stage to be identified

[0057] The feature map output by the pedestrian detection model is concatenated with the input video frame in the channel direction to generate a 6×224×224 fused feature tensor F. This fused feature tensor F extracted from each frame is saved. When the number of saved fused feature tensors reaches 64, all fused feature tensors are concatenated in the temporal direction to generate a video feature sequence of size 64×6×224×224. This video feature sequence is used for subsequent abnormal behavior identification. During the feature tensor saving process, if no pedestrian target is detected in the corresponding image of 10 consecutively saved feature tensors, all currently saved fused feature tensors are deleted.

[0058] (3) Abnormal behavior identification stage

[0059] An elevator abnormal behavior dataset was constructed by selecting segments containing both abnormal and non-abnormal behavior from elevator car surveillance videos. The dataset consisted of 1,000 normal and 1,000 abnormal behavior segments, each containing 192 frames. Each segment was sampled at equal intervals, extracting one frame every three frames, resulting in a total of 64 frames. Each frame in the segment was input into the constructed pedestrian detection model. The feature map output by the model was concatenated with the input video frame along the channel direction to produce a 6×224×224 fused feature tensor F. The fused feature tensors F generated for each frame in each segment were then concatenated along the temporal direction to produce a video feature sequence of size 64×6×224×224. The extracted feature sequence for each video segment was saved for subsequent training of the abnormal behavior recognition model.

[0060] An abnormal behavior recognition model was built using 3D Mobilenet-v2 to identify abnormal behavior in video sequences. It takes as input a sequence of extracted video features for each video and outputs an abnormal behavior prediction result for each video. To improve computational efficiency, the 3D convolution kernel of size 3×3×3 in 3D Mobilenet-v2 is decomposed into a 1×3×3 spatial convolution kernel and a 3×1×1 temporal convolution kernel. The 1×3×3 kernel extracts spatial features from the feature tensor, while the 3×1×1 kernel extracts temporal features from the feature tensor, approximating the 3D convolution operation. Compared to the 3D Mobilenet-v2 model, the improved 3D Mobilenet-v2 model has fewer parameters and is easier to optimize during training.

[0061] This embodiment uses the yolov5s algorithm to detect pedestrians and determine whether there are pedestrians in the elevator. The abnormal behavior recognition model is only used to identify the input video sequence when there are pedestrians in the elevator car, which effectively reduces the computational complexity of the hardware equipment. The fighting behavior recognition model is constructed using 3D mobilenet-v2, and the 3D convolution in the model is decomposed, so that the number of model parameters is reduced and it is easy to optimize during training. In addition, the feature map output by the yolov5s model is used together with the input video as the input data of the abnormal behavior recognition model, which enhances the feature expression ability of the input data and solves the problem of low feature extraction ability of the lightweight model 3Dmobilenet-v2, making the recognition results of the abnormal behavior recognition model more accurate; the technology used is fast, accurate, and real-time, and is suitable for applications and engineering.

[0062] Example 2

[0063] A second embodiment of the present disclosure introduces a system for detecting abnormal behavior of an elevator car based on machine vision.

[0064] like Figure 3 The system for detecting abnormal behavior of an elevator car based on machine vision is shown, comprising:

[0065] an acquisition module configured to acquire video data in the elevator car; perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result;

[0066] The behavior detection model is configured to construct a video sequence to be identified based on the obtained pedestrian detection results, and identify abnormal behavior in the elevator car based on the constructed video sequence and the abnormal behavior recognition model;

[0067] The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence.

[0068] The detailed steps are the same as those of the machine vision-based elevator car abnormal behavior detection method provided in Example 1 and will not be repeated here.

[0069] Example 3

[0070] A third embodiment of the present disclosure provides a computer-readable storage medium.

[0071] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in the first embodiment of the present disclosure.

[0072] The detailed steps are the same as those of the machine vision-based elevator car abnormal behavior detection method provided in Example 1 and will not be repeated here.

[0073] Example 4

[0074] A fourth embodiment of the present disclosure provides an electronic device.

[0075] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in the first embodiment of the present disclosure are implemented.

[0076] The detailed steps are the same as those of the machine vision-based elevator car abnormal behavior detection method provided in Example 1 and will not be repeated here.

[0077] The foregoing description is merely a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Those skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A method for detecting abnormal behavior of an elevator car based on machine vision, characterized in that: The following steps are involved: Obtain video data inside the elevator car; Perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result; Constructing a video sequence to be identified based on the obtained pedestrian detection results, and identifying abnormal behaviors in the elevator car based on the constructed video sequence and the abnormal behavior recognition model; The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence; In the process of acquiring the video image inside the elevator car, a video surveillance image containing a pedestrian target inside the elevator car is collected; Before identifying abnormal behavior in the elevator car, a pedestrian detection dataset is constructed based on the obtained video surveillance images, and pedestrian detection and identification in the elevator car are performed through single-stage target detection; When a pedestrian is detected in the elevator car, the input video frame and the output feature map of the single-stage target detection are performed on each frame of the image to obtain and save a fused feature tensor for each frame of the image; In the process of identifying abnormal behavior in the elevator car, an abnormal behavior recognition model is constructed using 3D Mobilenet-v2. The 3D convolution kernel of size 3×3×3 in the 3D Mobilenet-v2 is decomposed into a 1×3×3 spatial convolution kernel and a 3×1×1 temporal convolution kernel. The 1×3×3 kernel is used to extract spatial feature information from the feature tensor, and the 3×1×1 kernel is used to extract temporal feature information from the feature tensor. The model identifies whether a video sequence contains abnormal behavior. The model input is the video feature sequence extracted from each video segment, and the model outputs the abnormal behavior prediction result for each video segment.

2. A method for detecting abnormal behavior of an elevator car based on machine vision as claimed in claim 1, characterized in that: If no pedestrian is detected and recognized in 10 consecutive frames, the saved fusion feature tensor is deleted.

3. The method for detecting abnormal behavior of an elevator car based on machine vision as claimed in claim 1, characterized in that: The single-stage target detection adopts the yolov5s model, and a 3×1×1 convolution layer and a 2x upsampling module are added after the third output of the Neck layer in the yolov5s model to construct the output layer.

4. A machine vision-based elevator car abnormal behavior detection system, characterized in that: A method for detecting abnormal behavior of an elevator car based on machine vision according to any one of claims 1 to 3 is adopted, comprising: an acquisition module configured to acquire video data in the elevator car; perform pedestrian detection based on the acquired video image to obtain a pedestrian detection result; The behavior detection model is configured to construct a video sequence to be identified based on the obtained pedestrian detection results, and identify abnormal behavior in the elevator car based on the constructed video sequence and the abnormal behavior recognition model; The abnormal behavior detection model uses a deep separable convolution algorithm to classify and identify the video sequence.

5. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in any one of claims 1 to 3 are implemented.

6. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for detecting abnormal behavior of an elevator car based on machine vision as described in any one of claims 1 to 3 are implemented.

Images