A safety detection method for unmanned overhead crane operation area based on visual perception

Abstract

The present invention discloses a safety detection method for an unmanned overhead crane operation area based on visual perception, and relates to the field of industrial production technology. The detection method obtains a dedicated data set by annotating pictures of actual scenes in a steel plant, and further fine-tunes a target detection model using the data set, thereby optimizing the recognition effect of the target detection model on workers in a special scene such as a steel plant. In addition, the present invention combines an overlap judgment algorithm with a target detection model, so that the target detection program can determine whether the identified person is in the working area, thereby further determining whether it is the right time to prevent steel coils, effectively preventing the safety hazards caused by the unmanned overhead crane lowering steel coils when there are workers working in the working area, and also providing a safer working environment for workers in the steel plant.

Description

Technical Field

[0001] The present invention relates to the field of industrial production technology, and in particular to a method for detecting safety of an unmanned overhead crane operation area based on visual perception. Background Art

[0002] With the continuous development of intelligent technology, intelligent cranes, namely unmanned overhead cranes, have been increasingly used in steel mills. The unmanned overhead crane system is based on the traditional manual overhead crane system. It uses sensor technology, information technology and network technology, and adds a basic layer, management layer and control layer to the manual overhead crane system to realize the intelligent and unmanned warehouse integrated management system of the overhead crane. At present, the unmanned overhead crane system is relatively mature and can effectively improve the operating efficiency of steel mills and greatly reduce labor costs. However, the unmanned design also creates some safety hazards. For example, when the unmanned overhead crane transports the steel coil to the working area, it cannot judge the personnel situation in the working area. In the process of lowering the steel coil, it may cause harm to the workers working in the working area, thus leading to serious safety accidents. At the same time, with the vigorous development of the field of computer vision, there are now a large number of advanced and mature deep learning-based image and video recognition technologies that can be used for personnel detection. However, these methods usually use human images in a variety of scenarios to train the models, and the effect on personnel detection tasks in specific scenarios such as steel mills is average, so it cannot meet the various needs of actual applications in steel mill operations; in addition, whether the timing for the unmanned overhead crane to lower the steel coil to the work area is appropriate also depends on whether the detected personnel overlap with the work area. Therefore, when detecting the personnel, it is equally important to judge their relative position relationship with the work area. Summary of the invention

[0003] In view of the deficiencies in the prior art, the present invention provides a method for safety detection of an unmanned overhead crane operating area based on visual perception, which solves the problems mentioned in the above background technology.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions: a safety detection method for an unmanned overhead crane operation area based on visual perception, the safety detection method comprising the following steps:

[0005] S1: Frame extraction is performed on the collected videos of the working area inside the steel plant and its vicinity to obtain a dedicated dataset for optimizing the target detection model;

[0006] S2: Based on the optimization of the target recognition model Yolo, on the basis of the Yolo model that has been loaded with pre-trained weights, fine-tuning is performed using the collected dedicated data set to enhance its application effect in the actual scenario of the steel plant;

[0007] S3: Design an area division program to delineate the work area with a polygon on a picture extracted from the video, and save the slopes and vertices of each side of the work area;

[0008] S4: Add an overlap judgment module to the target detection model;

[0009] S5: When the unmanned overhead crane is about to place the steel coil in the working area, the camera extracts several pictures from the real-time video stream and transmits them to the server, which then detects personnel in the working area.

[0010] Optionally, the S1 further comprises the following steps:

[0011] S11: analyzing and screening the monitoring video data of the steel plant, and intercepting video clips containing a large number of people and a large number of different postures of people;

[0012] S12: performing frame extraction processing on the intercepted video clips to obtain image data in a specific scene of the steel plant;

[0013] S13: Use a marking tool to mark the acquired image data and generate a marking file in a YOLO format. In order to make the subsequent overlapping judgment more accurate, the marking box is as close to the outer contour of the staff as possible.

[0014] Optionally, S2 includes the following steps:

[0015] S21: The target detection model of the present invention is improved based on the yolov4 model, and CSPDarkNet53 is used as the backbone network for feature extraction. DarkNet53 is composed of Resblock_body modules, and the Resblock_body module is composed of a stack of downsampling and multiple residual structures. CSPDarkNet53 modifies the activation function from LeakyRelu to Mish based on DarkNet53. The formula of Mish is as follows:

[0016] Mish = x × tanh (ln (1 + e x )),

[0017] In addition, CSPDarkNet53 also modifies the Resblock_body in DarkNet53 to the CSPnet structure;

[0018] S22: In the feature pyramid part, the target detection model uses both the SPP structure and the PANet structure.

[0019] Optionally, S2 further includes the following steps:

[0020] S23: In the feature utilization part, the model extracts three feature layers for target detection. Based on the voc dataset, the shapes of the output layers are (19, 19, 75), (38, 38, 75), and (19, 19, 1024).

[0021] S24: The loss function is calculated as follows:

[0022]

[0023] S25: Load the weights pre-trained on the voc dataset into the yolo network model;

[0024] S26: Use the new data set obtained by annotating the steel plant pictures to fine-tune the YOLO model after loading the pre-trained weights until the model converges and saves the parameters, which can be used for personnel detection.

[0025] Optionally, the S4 further comprises the following steps:

[0026] S31: extract a frame of image from the surveillance video of the steel plant;

[0027] S32: Use the pylab library of Python to select points on the extracted image in the order of polygon vertices with the mouse, so as to obtain the vertex coordinates of the polygon corresponding to the working area;

[0028] S33: The slope and intercept of each edge of the polygon are calculated using these vertex coordinates, and the slope, intercept and vertex coordinates are saved in a txt file to facilitate the subsequent target detection program to read the scope of the working area.

[0029] Optionally, S4 includes the following steps:

[0030] S41: adding a working area reading module to the target detection program, specifying a txt file corresponding to the working area, and reading the slope and intercept of each side of the polygon corresponding to the working area and the coordinates of each vertex from the txt file;

[0031] S42: The lower two vertices of the Bounding Box of the target detection are used to represent the foot position of the detected person, thereby converting the judgment of the overlap between the person and the work area into the judgment of whether at least one of the two vertices below the Bounding Box is in the polygon corresponding to the work area. Therefore, a function for determining the position relationship between the judgment point and the polygon should be added to the target detection program.

[0032] Optionally, the S4 further comprises the following steps:

[0033] S43: Design an overlap judgment function, and input the two lower vertices of the Bounding Box into the function in S42 respectively. As long as one point is judged to be within the polygon corresponding to the working area, it is considered that the detected person corresponding to the Bounding Box is within the working area.

[0034] Optionally, the S5 further comprises the following steps:

[0035] S51: Deploy the target detection program on the server and use the flask framework to build a pipeline for front-end and back-end communication;

[0036] S52: When the unmanned overhead crane is about to place the steel coil in the working area, the camera end uses the capture tool to capture the steel plant picture from the real-time video stream, captures one frame every 6 frames and transmits it to the server end, and captures a total of 10 frames. Then the regional personnel target detection program on the server end detects these 10 frames of pictures once, and for each detected person in each frame of the picture, it is determined whether it overlaps with the working area. For all 10 frames of pictures, as long as a person is detected in the working area, it is determined that there is someone in the current working area, and a signal is sent to the unmanned overhead crane to prevent the placement of the steel coil this time.

[0037] The present invention provides a method for detecting the safety of an unmanned overhead crane operation area based on visual perception. It has the following beneficial effects:

[0038] The method for detecting the safety of the unmanned overhead crane operation area based on visual perception obtains a dedicated data set by annotating pictures of actual scenes in the steel plant, and further fine-tunes the target detection model with the data set, thereby optimizing the recognition effect of the target detection model on the staff in the special scene of the steel plant. In addition, the present invention combines the overlapping judgment algorithm with the target detection model, so that the target detection program can determine whether the identified person is in the working area, thereby further determining whether it is the right time to prevent the steel coils, effectively preventing the safety hazards caused by the unmanned overhead crane lowering the steel coils when there are workers working in the working area, and also providing a safer working environment for the workers in the steel plant. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 The present invention provides a technical solution: a method for detecting the safety of an unmanned overhead crane operation area based on visual perception, comprising the following steps:

[0042] S1: Frame extraction is performed on the collected videos of the working area inside the steel plant and its vicinity to obtain a dedicated dataset for optimizing the target detection model;

[0043] Specifically, S1 includes the following steps:

[0044] S11: analyzing and screening the monitoring video data of the steel plant, and intercepting video clips containing a large number of people and a large number of different postures of people;

[0045] S12: performing frame extraction processing on the intercepted video clips to obtain image data in a specific scene of the steel plant;

[0046] S13: Use a marking tool to mark the acquired image data and generate a marking file in a YOLO format. In order to make the subsequent overlapping judgment more accurate, the marking box is as close to the outer contour of the staff as possible.

[0047] S2: Based on the optimization of the target recognition model YOLO, on the basis of the YOLO model that has been loaded with pre-trained weights, fine-tuning is performed using the collected dedicated data set to enhance its application effect in the actual scenario of the steel plant.

[0048] Specifically, S2 includes the following steps:

[0049] S21: The target detection model of the present invention is improved based on the yolov4 model, in which we use CSPDarkNet53 as the backbone network for feature extraction. DarkNet53 is composed of Resblock_body modules, and the Resblock_body module is composed of a downsampling and a stack of multiple residual structures. CSPDarkNet53 modifies the activation function from LeakyRelu to Mish based on DarkNet53. The formula of Mish is as follows:

[0050] Mish = x × tanh (ln (1 + e x )),

[0051] In addition, CSPDarkNet53 also modifies the Resblock_body in DarkNet53 to the CSPnet structure;

[0052] S22: In the feature pyramid part, the target detection model uses both the SPP structure and the PANet structure. After performing three convolutions on the last feature layer of CSPDarkNet53, SPP uses four different scales of maximum pooling for processing. The pooling kernel sizes of the maximum pooling are (13*13), (9*9), (5*5), and (1*1), respectively. This structure can greatly increase the receptive field and separate the most significant contextual features. PANet is an instance segmentation algorithm. One of its very important characteristics is the repeated extraction of features. The target detection model used in the present invention mainly uses the PANet structure on three effective feature layers;

[0053] S23: In the feature utilization part, the model extracts three feature layers for target detection. The shapes of the three feature layers are (76, 76, 256), (38, 38, 512), and (19, 19, 1024). When based on the voc data set, the shapes of the output layers are (19, 19, 75), (38, 38, 75), and (19, 19, 1024). In actual application, the actual shapes of the output layers of the present invention are (19, 19, 8), (38, 38, 8), and (76, 76, 8), because the voc data set contains 20 classes, and the present invention only needs to detect the person class when applied;

[0054] S24: The loss function is calculated as follows:

[0055]

[0056] S25: Load the weights pre-trained on the voc dataset into the yolo network model;

[0057] S26: Use the new data set obtained by annotating the steel plant pictures to fine-tune the YOLO model after loading the pre-trained weights until the model converges and saves the parameters, which can be used for personnel detection.

[0058] S3: Design an area division program to use a polygon to delineate the work area on a picture extracted from the video, and save the slopes and vertices of each edge of the work area to facilitate subsequent judgment of the overlap between personnel and the work area.

[0059] Specifically, S3 includes the following steps:

[0060] S31: extract a frame of image from the surveillance video of the steel plant;

[0061] S32: Use the pylab library of Python to select points on the extracted image in the order of polygon vertices with the mouse, so as to obtain the vertex coordinates of the polygon corresponding to the working area;

[0062] S33: The slope and intercept of each edge of the polygon are calculated using these vertex coordinates, and the slope, intercept and vertex coordinates are saved in a txt file to facilitate the subsequent target detection program to read the scope of the working area.

[0063] S4: Add an overlap judgment module to the target detection model.

[0064] Specifically, S4 includes the following steps:

[0065] S41: adding a working area reading module to the target detection program, specifying a txt file corresponding to the working area, and reading the slope and intercept of each side of the polygon corresponding to the working area and the coordinates of each vertex from the txt file;

[0066] S42: Use the two vertices below the Bounding Box of the target detection to represent the foot position of the detected person, so as to transform the judgment of the overlap between the person and the work area into the judgment of whether at least one of the two vertices below the Bounding Box is in the polygon corresponding to the work area. Therefore, a function for judging the positional relationship between the point and the polygon should be added to the target detection program. In addition, considering that the calculation method of the positional relationship between the point and the polygon is related to the shape of the polygon itself, it is also necessary to design a special judgment function for work areas of different shapes;

[0067] S43: Design an overlap judgment function, and input the two lower vertices of the Bounding Box into the function in S42 respectively. As long as one point is judged to be within the polygon corresponding to the working area, it is considered that the detected person corresponding to the Bounding Box is within the working area.

[0068] S5: When the unmanned overhead crane is about to place the steel coil in the working area, the camera extracts several pictures from the real-time video stream and transmits them to the server, which then detects personnel in the working area.

[0069] Specifically, S5 includes the following steps:

[0070] S51: Deploy the target detection program on the server and use the flask framework to build a pipeline for front-end and back-end communication;

[0071] S52: When the unmanned overhead crane is about to place the steel coil in the working area, the camera end uses the capture tool to capture the steel plant picture from the real-time video stream, captures one frame every 6 frames and transmits it to the server end, and captures a total of 10 frames. Then the regional personnel target detection program on the server end detects these 10 frames of pictures once, and for each detected person in each frame of the picture, it is determined whether it overlaps with the working area. For all 10 frames of pictures, as long as a person is detected in the working area, it is determined that there is someone in the current working area, and a signal is sent to the unmanned overhead crane to prevent the placement of the steel coil this time.

[0072] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting safety of an unmanned overhead crane operation area based on visual perception, characterized in that: The safety detection method comprises the following steps: S1: Frame extraction is performed on the collected videos of the working area inside the steel plant and its vicinity to obtain a dedicated dataset for optimizing the target detection model; S2: Optimize the target recognition model Yolo, and fine-tune the Yolo model with pre-trained weights using the collected dedicated dataset; S3: Design an area division program to delineate the work area with a polygon on a picture extracted from the video, and save the slopes and vertices of each side of the work area; S4: Add an overlap judgment module to the target detection model; S5: When the unmanned overhead crane is about to place the steel coil in the working area, the camera extracts several pictures from the real-time video stream and transmits them to the server, which then detects personnel in the working area. The S4 comprises the following steps: S41: adding a working area reading module to the target detection program, specifying a txt file corresponding to the working area, and reading the slope and intercept of each side of the polygon corresponding to the working area and the coordinates of each vertex from the txt file; S42: The two lower vertices of the bounding box of the target detection are used to represent the foot position of the detected person, so as to transform the judgment of the overlap between the person and the working area into the judgment of whether at least one of the two lower vertices of the bounding box is in the polygon corresponding to the working area. Therefore, a function for judging the position relationship between the point and the polygon is added to the target detection program; S43: Design an overlap judgment function, and input the two lower vertices of the Bounding Box into the function in S42 respectively. As long as one point is judged to be within the polygon corresponding to the working area, it is considered that the detected person corresponding to the Bounding Box is within the working area.

2. The method for detecting safety of an unmanned overhead crane operation area based on visual perception according to claim 1, characterized in that: The S1 further comprises the following steps: S11: analyzing and screening the monitoring video data of the steel plant, and intercepting video clips containing a large number of people and a large number of different postures of people; S12: performing frame extraction processing on the intercepted video clips to obtain image data in a specific scene of the steel plant; S13: Use a marking tool to mark the acquired image data and generate a marking file in a YOLO format. In order to make the subsequent overlapping judgment more accurate, the marking box is as close to the outer contour of the staff as possible.

3. The method for detecting safety of an unmanned overhead crane operation area based on visual perception according to claim 1 is characterized in that: The S2 comprises the following steps: S21: The target detection model of the present invention is improved based on the yolov4 model, and CSPDarkNet53 is used as the backbone network for feature extraction. DarkNet53 is composed of Resblock_body modules, and the Resblock_body module is composed of a stack of one downsampling and multiple residual structures. CSPDarkNet53 modifies the activation function from LeakyRelu to Mish on the basis of DarkNet53; in addition, CSPDarkNet53 also modifies the Resblock_body in DarkNet53 to the CSPnet structure; S22: In the feature pyramid part, the target detection model uses both the SPP structure and the PANet structure.

4. The method for detecting safety of an unmanned overhead crane operation area based on visual perception according to claim 3 is characterized in that: The S2 further comprises the following steps: S23: In the feature utilization part, the model extracts three feature layers for object detection; S25: Load the weights pre-trained on the voc dataset into the yolo network model; S26: Use the new data set obtained by annotating the steel plant pictures to fine-tune the YOLO model after loading the pre-trained weights until the model converges and saves the parameters, which can be used for personnel detection.

5. The method for detecting safety of an unmanned overhead crane operation area based on visual perception according to claim 1, characterized in that: The S4 further comprises the following steps: S31: extract a frame of image from the surveillance video of the steel plant; S32: Use the pylab library of Python to select points on the extracted image in the order of polygon vertices with the mouse, so as to obtain the vertex coordinates of the polygon corresponding to the working area; S33: The slope and intercept of each edge of the polygon are calculated using these vertex coordinates, and the slope, intercept and vertex coordinates are saved in a txt file to facilitate the subsequent target detection program to read the scope of the working area.

6. The method for detecting safety of an unmanned overhead crane operation area based on visual perception according to claim 1, characterized in that: The S5 further comprises the following steps: S51: Deploy the target detection program on the server and use the flask framework to build a pipeline for front-end and back-end communication; S52: When the unmanned overhead crane is about to place the steel coil in the working area, the camera end uses the capture tool to capture the steel plant picture from the real-time video stream, captures one frame every 6 frames and transmits it to the server end, and captures a total of 10 frames. Then the regional personnel target detection program on the server end detects these 10 frames of pictures once, and for each detected person in each frame of the picture, it is determined whether it overlaps with the working area. For all 10 frames of pictures, as long as a person is detected in the working area, it is determined that there is someone in the current working area, and a signal is sent to the unmanned overhead crane to prevent the placement of the steel coil this time.

Images