An elevator door space foreign object detection method based on deep learning and time-of-flight

Abstract

The present invention discloses a method for detecting foreign objects in the elevator door space based on deep learning and time-of-flight, including: creating a dataset of elevator door space foreign object images and performing data augmentation; building an improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT, and training the model with the dataset; using a time-of-flight technology camera to generate a depth image of the elevator door space, and performing threshold segmentation on the depth image of the elevator door space using the K-means clustering threshold segmentation method; performing detection of foreign objects on the segmented image using the bar segmentation localization detection algorithm; and finally making a comprehensive judgment on the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation localization detection algorithm on the time-of-flight technology camera. The present invention can perform high-precision detection of foreign objects in the elevator door space, thereby effectively ensuring the safe operation of the elevator.

Description

Technical Field

[0001] The present invention belongs to the technical fields of artificial intelligence and elevator technology, and particularly relates to a method for detecting foreign objects in the elevator door space based on deep learning and time of flight. Background Art

[0002] As a transportation tool between floors in high-rise buildings, elevators play an essential role in the vertical transportation of people and goods in modern times. The extensive use of elevators has brought great convenience to people's lives, and at the same time, people have higher requirements for the performance of elevators, such as safety, stability, and intelligence. According to statistics, more than 80% of elevator failures are often closely related to the elevator door system, and one of the main failures of the elevator door system is the pinching of people and objects by the elevator door. Therefore, the detection of foreign objects in the elevator door space is of great significance for the safe and stable operation of the entire elevator.

[0003] There are mainly two types of detection methods for foreign objects in the elevator door space in the prior art, namely contact safety edges and infrared light curtains. Chinese Patent (CN201920894934.2) "Elevator Door Contact Protection Device and Elevator Door System", the main principle of its contact safety edge is that the force in the opposite direction when the elevator door pinches a rigid object touches a micro switch, and the micro switch returns a signal to reopen the door to the elevator control system. After receiving the signal, the elevator door will reopen. Chinese Patent (CN201620970810.4) "An Intelligent Infrared Light Curtain for Elevators" introduces an infrared light curtain based on an optoelectronic protection device, which forms a dense infrared detection light beam network in the door space through infrared beam transceiver devices installed on both sides of the door, thereby detecting the intrusion of foreign objects in the door space.

[0004] The main deficiencies of the above prior art are as follows: The contact safety edge has a low detection rate for flexible objects, so it is difficult to detect relatively soft flexible objects such as the fingers of young children, which is likely to cause greater safety hazards. The infrared light curtain detects whether there is an object through infrared beams, and it is difficult to detect objects in the gap between two infrared beams. Summary of the Invention

[0005] The purpose of the present invention is to overcome the defects of the prior art and provide a method for detecting foreign objects in the elevator door space based on deep learning and time of flight. Brightness data augmentation is adopted in the self-built image dataset of foreign objects in the elevator door space, and an improved YOLOv5s target detection model based on the SMU activation function and the novel attention convolution CoT is built. On the basis of the improved YOLOv5s target detection model for foreign objects, the time of flight technology depth image is used to perform secondary detection on foreign objects in the elevator door space. This method can detect foreign objects in the elevator door space with high precision and effectively ensure the safe operation of the elevator.

[0006] To solve the above technical problems, the present invention adopts the following technical solutions.

[0007] A method for detecting foreign objects in the elevator door space based on deep learning and time of flight, comprising the following steps:

[0008] Step 1, collect and create a dataset of foreign object images in the elevator door space, and perform brightness data augmentation; the brightness adjustment formula for RGB images in its data augmentation is:

[0009] g(i,j) = af(i,j) + b

[0010] where f(i,j) and g(i,j) are the pictures before and after adjustment respectively, a is the contrast, and b is the saturation.

[0011] Step 2, build an improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT, and use the dataset obtained in Step 1 to train the improved YOLOv5s object detection model;

[0012] Specifically, improve all the activation functions in the basic CBL module in YOLOv5s to the SMU activation function, and add a spatial attention convolution CoT structure to the backbone feature extraction network Backbone of the YOLOv5s object detection model, and improve the 3×3 convolution structure in the Res unit X of the C3 module to the CoT structure, thereby obtaining the improved YOLOv5s object detection model;

[0013] Step 3, use a time of flight technology camera to generate a depth image of the elevator door space, and perform threshold segmentation on the depth image of the elevator door space by the K-means clustering threshold segmentation method; use a bar segmentation and positioning detection algorithm to detect foreign objects on the segmented image; finally, comprehensively judge the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation and positioning detection algorithm on a time of flight technology camera. Specifically, it includes:

[0014] Step 3-1, adopt the K-means clustering method to cluster the pixels of the ground area and the non-ground area of the depth image of the elevator door space to obtain a segmentation threshold; then use the obtained segmentation threshold to perform image threshold segmentation on the depth image of the elevator door space;

[0015] Step 3-2, use the designed bar segmentation and positioning method to detect the depth image of the elevator door space after threshold segmentation; first divide the ground area of the depth image of the elevator door space after threshold segmentation into detection areas; in the bar segmentation and positioning detection algorithm, divide the detection areas horizontally and vertically into several equal parts, calculate the average pixel value in each divided area, and if it is lower than the set threshold, it is judged that there is a foreign object in this area; its expression is as follows:

[0016]

[0017] Among them, the result of 1 represents that there is a foreign object in this area, and 0 represents that there is no foreign object in this area; p(i, j) is the pixel value of the bar-shaped area, i is the horizontal coordinate of the pixel of the bar-shaped area, j is the vertical coordinate of the pixel of the bar-shaped area, and T is the set threshold; finally, count the number of areas with foreign objects in the horizontally and vertically divided areas, and mark the areas with foreign objects to obtain the specific position and size of the detected foreign objects.

[0018] Step 3-3 comprehensively judges the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation and positioning detection algorithm on a time-of-flight technology camera. Only when the improved YOLOv5s object detection model network first detects a foreign object, and then a foreign object is detected based on the depth image of the time-of-flight technology, will the detection algorithm process output the result as having a foreign object.

[0019] Furthermore, calculate the average value of the gray values of all pixels in the depth image of the elevator door space. When there is no foreign object in the elevator door space, its average value is t; and the x-axis coordinate value of each pixel point can be obtained by subtracting the average value from the gray value of the current point; the coordinate point of a pixel point with a gray value of v in the clustering coordinate system is (v - a, v); by this method, all points in the image have their specific positions in the clustering coordinate system, and these points satisfy the relationship:

[0020] y = x + t

[0021] where y is the vertical coordinate, x is the horizontal coordinate, and t is the average value.

[0022] Put these points into the K-means clustering algorithm to achieve clustering; before clustering, set the number of clusters to 2; the centroid with a larger value is the threshold for threshold segmentation.

[0023] Furthermore, the threshold segmentation of the depth image of the elevator door space is to distinguish the car floor and the non-car floor in the depth image of the elevator door space through the setting of the threshold, so as to detect foreign objects in the depth image of the elevator door space; the final effect of the threshold segmentation of the elevator door space is to divide the car floor area higher than the segmentation threshold into an area with a gray value of 255, and divide the car floor area lower than the segmentation threshold into an area with a gray value of 0, and finally form the depth image of the elevator door space after threshold segmentation.

[0024] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0025] 1. The present invention constructs an improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT, enhancing the weight of the model for the detected target area in the global image. It can detect the vast majority of foreign objects within the visible range, without problems such as dead zones. The problem of missed detection is greatly reduced, ensuring the safety of elevator passengers.

[0026] 2. The present invention enriches the dataset by collecting images of foreign objects in the elevator door space of different elevators and enhancing the brightness data, effectively ensuring the completeness and scientificity of the training dataset.

[0027] 3. Based on the detection of foreign objects by the improved YOLOv5s object detection model, the present invention uses the depth image of the time-of-flight technology to perform secondary detection on the foreign objects in the elevator door space, reducing the false detection rate of the ground pattern. The detection method of the present invention can detect foreign objects in the elevator door space with high precision and plays an important role in the safe operation of the elevator. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flowchart of the method of an embodiment of the present invention.

[0029] Figure 2 It is a CBL network diagram combining SMU of an embodiment of the present invention.

[0030] Figure 3 It is an improved C3 structure diagram of an embodiment of the present invention.

[0031] Figure 4 It is a detection flowchart of the bar segmentation and positioning method of an embodiment of the present invention.

[0032] Figure 5 It is a time-of-flight technology enhancement flowchart of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] A method for detecting foreign objects in the elevator door space based on deep learning and time-of-flight in the present invention adopts brightness data enhancement in the self-built image dataset of foreign objects in the elevator door space, enriching the dataset. An improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT is constructed to enhance the weight of the model for the detected target area in the global image. Based on the detection of foreign objects by the improved YOLOv5s object detection model, the depth image of the time-of-flight technology is used to perform secondary detection on the foreign objects in the elevator door space, reducing the false detection rate of the ground pattern. The detection method of the present invention can detect foreign objects in the elevator door space with high precision and has an important role in the safe operation of the elevator.

[0034] The following further describes the present invention in detail with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart of a method for an embodiment of the present invention. A method for detecting foreign objects in the elevator door space based on deep learning and time-of-flight includes the following steps:

[0036] Step 1: Collect and create a dataset of foreign object images in the elevator door space, and perform brightness data augmentation;

[0037] 1-1: Install cameras at the centers of the door lintels of different elevators, with the viewing angle facing directly below the elevator door space, and take sample pictures including arms, human bodies, books, wooden sticks of different specifications, etc.; Crop the pictures to the size of 640×150 of the elevator door space to prevent interference from non-detection areas; By adjusting the brightness of the collected pictures, enrich the pictures in the dataset to make the training model more accurate.

[0038] 1-2: The main means of collecting the dataset for detecting foreign objects in the elevator door space is to install cameras between the door lintels or door seams of different elevators to take pictures of the elevator door space, and take pictures at different positions of different foreign objects in the elevator door space to improve the richness of the data. In practical applications, the camera is fixed at a specific position in the center of the door lintel, which is convenient for the device to collect image information of the elevator door space and also convenient for later installation and maintenance.

[0039] 1-3: To conform to the actual application scenario, cameras are installed on the door lintels of different elevators in this dataset to take pictures of different foreign objects such as arms, water bottles, and wooden sticks at multiple positions in the elevator door space. In practical applications, the detection of foreign objects in the elevator door space does not require a specific identification function for the types of foreign objects, but only needs to detect foreign objects when there are foreign objects in the elevator door space and transmit this information to the door machine control system to prevent the door from closing. Therefore, we collectively refer to all invading foreign objects as "object".

[0040] 1-4: There is only one lighting environment in the same elevator at the same time, and collecting samples in multiple lighting environments is a great challenge. In order to better simulate the elevator door space environment with multiple lights, improve the accuracy of the target detection algorithm, and meet the requirements of actual use, specific data augmentation must be performed on the collected picture dataset. The brightness adjustment formula for RGB images is:

[0041] g(i,j) = af(i,j) + b

[0042] where f(i,j) and g(i,j) are the pictures before and after adjustment respectively, where i is the pixel abscissa of the dataset image, j is the pixel ordinate of the dataset image, a is the contrast, and b is the saturation.

[0043] After image collection under different conditions, the specific specifications of the statistical dataset are as follows: the number of training set pictures is 1347, and the number of validation set pictures is 148.

[0044] Step 2: Build an improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT, and use the dataset obtained in Step 1 to train the improved YOLOv5s object detection model;

[0045] 2-1: Use the SMU activation function as the activation function of the improved YOLOv5s object detection model. The specific method is as follows: Improve the activation function in the basic CBL module in the YOLOv5s object detection model to the SMU activation function. The CBL module is one of the main components of the YOLOv5s object detection model and is the main convolutional module. It includes convolution, batch normalization, and an activation function. In the present invention, by replacing the activation function in the CBL module in the YOLOv5s object detection model, all activation functions of the YOLOv5s object detection model are replaced to obtain the improved YOLOv5s object detection model. Specifically, as Figure 2 shown.

[0046] 2-2: Add a spatial attention convolution CoT structure to the backbone feature extraction network Backbone of the YOLOv5s object detection model, and improve the 3×3 convolution structure of the C3 module to the CoT structure. The convolutional neural network is mainly constructed on 3×3 and 1×1 convolutional kernels, and these convolutional kernels are used for feature extraction of pictures. However, limited by the receptive field size of the convolutional kernel, the convolutional neural network ignores the global internal connections of the picture when extracting features. In the present invention, by introducing the attention mechanism CoT structure. As Figure 3 shown, replace the 3×3 convolution operation in Res unit X of the C3 structure. The weight of the YOLOv5s object detection model for the detection area is increased, thereby improving the detection accuracy.

[0047] Step 3: Use a time-of-flight technology camera to generate an elevator door spatial depth image, perform threshold segmentation on the depth image of the elevator door space by the K-means clustering threshold segmentation method, and detect foreign objects on the segmented image using a bar segmentation and positioning detection algorithm. Finally, comprehensively judge the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation and positioning detection algorithm on a time-of-flight technology camera. The time-of-flight technology refers to the technology of using emitted modulated light to perform three-dimensional modeling on the shooting area and form a depth image.

[0048] 3-1: When performing threshold segmentation on the depth image of the elevator door space, the K-means clustering method is used to cluster the pixels of the ground area and non-ground area of the depth image of the elevator door space. The segmentation threshold is obtained through clustering, and then the obtained segmentation threshold is used to perform image threshold segmentation on the depth image of the elevator door space.

[0049] By calculating the average value of the gray values of all pixels in the depth image of the elevator door space, when there is no foreign object in the elevator door space, the average value is t. And the x-axis coordinate value of each pixel point can be obtained by subtracting the average value from the gray value of the current point. Therefore, the coordinate point of a pixel point with a gray value of v in the clustering coordinate system is (v - a, v). Through this method, all points in the image have their specific positions in the clustering coordinate system. These points satisfy the relationship:

[0050] y = x + t

[0051] where y is the ordinate, x is the abscissa, and t is the average value.

[0052] Then these points are input into the K-means clustering algorithm to achieve clustering. Before clustering, the number of clusters is set to 2. The centroid with a larger value is the threshold for threshold segmentation.

[0053] The threshold segmentation of the depth image of the elevator door space separates the car floor and non-car floor in the depth image of the elevator door space through the setting of the threshold, so as to detect foreign objects in the depth image of the elevator door space. The final effect of the threshold segmentation of the elevator door space is to divide the car floor area higher than the segmentation threshold into an area with a gray value of 255, and divide the car floor area lower than the segmentation threshold into an area with a gray value of 0. Finally, the depth image of the elevator door space after threshold segmentation is formed.

[0054] 3-2: Use the designed bar segmentation and positioning method to detect the depth image of the elevator door space after threshold segmentation; first, divide the ground area of the depth image of the elevator door space after threshold segmentation into detection areas. In the bar segmentation and positioning detection algorithm, the detection area is divided into several equal parts horizontally and vertically, and the average pixel value in each divided area is calculated. If it is lower than the set threshold, it is judged that there is a foreign object in this area; the specific expression is as follows:

[0055]

[0056] where the result 1 represents that there is a foreign object in this area, and 0 represents that there is no foreign object in this area. p(i, j) is the pixel value of the bar area, i is the pixel abscissa of the bar area, j is the pixel ordinate of the bar area, and T is the set threshold; finally, count the number of areas with foreign objects in the horizontally and vertically divided areas, and mark the areas with foreign objects to obtain the specific position and size of the detected foreign object.

[0057] The bar segmentation method divides the detection area into multiple bar areas for detection. The bar segmentation positioning detection algorithm first evenly divides the detection area into several bar areas with a width of m in the horizontal axis direction, and calculates the average value of the pixel gray values of each bar area. If the average value of the bar area is less than the set value, it means that there is a foreign object in this area. Mark all bar areas and count the specific positions of the leftmost and rightmost bar areas to form the width of the foreign object on the horizontal axis of the detection area. Similarly, in the vertical axis direction, the detection area is also evenly divided into several bar areas with a width of n, calculate whether there is a foreign object in these areas, and mark the position and width of the foreign object on the vertical axis of the detection area. Through the bar area positions of the foreign object on the horizontal and vertical axes, the specific position of the foreign object can be marked. The specific process is as Figure 4 shown.

[0058] The larger the number of bar areas, the more accurate it is. At the same time, the computational cost consumed also increases greatly. In the selection of the width of the bar area in the present invention, a width of 5 pixels is adopted in both cases.

[0059] 3-3: Comprehensively judge the detection results of the improved YOLOv5 object detection model on a common camera and the detection results of the bar segmentation positioning detection algorithm on a time-of-flight technology camera. In the detection process as Figure 5 shown, generally four situations are divided to represent the combination of the detection results of the improved YOLOv5 object detection model on a common camera and the detection results of the bar segmentation positioning detection algorithm on a time-of-flight technology camera in the output. The specific combination is shown in Table 1 below

[0060] Table 1

[0061]

[0062] In the above table, "11" represents the situation where the improved YOLOv5s object detection model first detects foreign objects, then the depth image based on the time-of-flight technology detects foreign objects, and finally it is determined that there are foreign objects in the elevator door space area. "10" represents that the improved YOLOv5s object detection model detects foreign objects while the depth image detects no foreign objects. In this case, the improved YOLOv5s object detection model is affected by light and shadow stains and outputs that there are foreign objects, but the depth image detection is not affected and outputs a detection result of no foreign objects. The final determination result is no foreign objects. "01" represents that the improved YOLOv5s object detection model detects no foreign objects while the depth image detects foreign objects. The depth image based on the time-of-flight technology is susceptible to the influence of tiny suspended particles such as dust close to the acquisition camera, which will form large black dots in the detection area of the depth image, resulting in misjudgment. Therefore, in this case, the output result to be determined is no foreign objects. Finally, in the case of "00", the neural network of the improved YOLOv5s object detection model and the depth image detection algorithm both fail to detect foreign objects, and the final determination output result is no foreign objects. To sum up, only when the improved YOLOv5s object detection model network first detects foreign objects, and then the depth image based on the time-of-flight technology detects foreign objects, that is, in the case of "11", will the detection algorithm process represent the output result as having foreign objects.

Claims

1. An elevator door space foreign object detection method based on deep learning and time of flight, characterized in that, It includes the following steps: Step 1: Collect an image dataset of foreign objects in the elevator door space and perform brightness data augmentation; Step 2: Build an improved YOLOv5s object detection model based on the SMU activation function and the novel attention convolution CoT, and use the dataset obtained in Step 1 to train the improved YOLOv5s object detection model; Step 3: Use a time-of-flight technology camera to generate a depth image of the elevator door space, and perform threshold segmentation on the depth image of the elevator door space by the K-means clustering threshold segmentation method; detect foreign objects on the segmented image using the bar segmentation and positioning detection algorithm; finally, comprehensively judge the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation and positioning detection algorithm on the time-of-flight technology camera; In Step 2, all activation functions in the basic CBL module in YOLOv5s are improved to the SMU activation function; and a spatial attention convolution CoT structure is added to the backbone feature extraction network Backbone of the YOLOv5s object detection model, and the 3×3 convolution structure in the Res unit X of the C3 module is improved to the CoT structure; Step 3 includes: Step 3-1: Adopt the K-means clustering method to cluster the pixels of the ground area and the non-ground area of the depth image of the elevator door space to obtain a segmentation threshold; then perform image threshold segmentation on the depth image of the elevator door space with the obtained segmentation threshold; Step 3-2: Use the designed bar segmentation and positioning method to detect the depth image of the elevator door space after threshold segmentation; first divide the ground area of the depth image of the elevator door space after threshold segmentation into detection areas; in the bar segmentation and positioning detection algorithm, divide the detection area horizontally and vertically into several equal parts, calculate the average pixel value in each divided area, and if it is lower than the set threshold, it is judged that there is a foreign object in this area; its expression is as follows: Among them, result 1 represents that there is a foreign object in this area, 0 represents that there is no foreign object in this area; p(i, j) is the pixel value of the bar area, i is the pixel abscissa of the bar area, j is the pixel ordinate of the bar area, and T is the set threshold; finally, count the number of areas with foreign objects in the horizontally and vertically divided areas, and mark the areas with foreign objects to obtain the specific position and size of the detected foreign object; Step 3-3: Comprehensively judge the detection results of the improved YOLOv5 object detection model on an ordinary camera and the detection results of the bar segmentation and positioning detection algorithm on the time-of-flight technology camera. Only when the improved YOLOv5s object detection model network first detects a foreign object, and then a foreign object is detected based on the depth image of the time-of-flight technology, will the detection algorithm process output the result as having a foreign object.

2. The method for detecting foreign objects in the elevator door space based on deep learning and time of flight according to claim 1, wherein, For the data augmentation of the collected picture dataset, the brightness adjustment formula for RGB images is: g(i, j) = af(i, j) + b Where f(i, j) and g(i, jj) are the pictures before and after adjustment respectively, a is the contrast, and b is the saturation.

3. The method for detecting foreign objects in the elevator door space based on deep learning and time-of-flight according to claim 1, wherein Average the gray values of all pixels in the depth image of the elevator door space. When there is no foreign object in the elevator door space, the average value is t. The x-axis coordinate value of each pixel point can be obtained by subtracting the average value from the gray value of the current point. The coordinate point of a pixel point with a gray value of v in the clustering coordinate system is (v - a, v). By this method, all points in the image have their specific positions in the clustering coordinate system, and these points satisfy the relationship: y = x + t where y is the ordinate, x is the abscissa, and t is the average value; Input these points into the K-means clustering algorithm to achieve clustering. Before clustering, set the number of clusters to 2. The centroid with a larger value is the threshold for threshold segmentation.

4. A method for detecting foreign objects in the elevator door space based on deep learning and time-of-flight, as claimed in claim 1, wherein The threshold segmentation of the depth image of the elevator door space is to distinguish the car floor and non-car floor in the depth image of the elevator door space by setting the threshold, so as to detect foreign objects in the depth image of the elevator door space. The final effect of the threshold segmentation of the elevator door space is to divide the car floor area higher than the segmentation threshold into an area with a gray value of 255, and divide the car floor area lower than the segmentation threshold into an area with a gray value of 0, and finally form the depth image of the elevator door space after threshold segmentation.

Images