Hook anti-drop device detection method based on deep learning

Abstract

The present invention discloses a method for detecting hook anti-dropping devices based on deep learning, specifically: obtaining a hook image containing a hook from a monitoring video and manually annotating it as sample data, constructing a neural network model and training it, inputting the monitoring video obtained in real time into the trained neural network model, obtaining a hook target and an anti-dropping device target, calculating the intersection and union ratio of the hook target and the anti-dropping device target. If the N temporary calculation results of the hook are that the anti-dropping device is not installed temporarily, then the detection result of the hook is that the anti-dropping device is not installed, and the detection result will be output; the detection of the group to be tested is completed. The beneficial effect of the present invention is that the present invention can realize real-time detection of whether the anti-dropping device is installed on the crane hook during operation through the target detection technology, which not only meets the management requirements, but also can timely discover the hook without the anti-dropping device and issue a warning to avoid accidents.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method capable of detecting an anti-slip device of a crane hook in a business scenario. Background Art

[0002] With the rapid development of deep learning technology, object detection, as an important branch of computer vision, has been widely used in security, industry, unmanned driving and other fields. In some outdoor construction sites, cranes are used to lift large heavy objects and are used very frequently. In order to prevent the objects from falling off the hook, the hooks in operation must be equipped with anti-falling devices, which can largely avoid accidents.

[0003] "3 Management Content and Requirements" - "3.4 Lifting operators should comply with the following regulations" - "3.4.4 Lifting operations are prohibited when the brakes, safety devices fail, the hook anti-loosening device is damaged, the wire rope is damaged to the scrap standard, and other lifting equipment and facilities are in an incomplete state. The internal management of the oil field requires real-time monitoring of whether the cranes in operation are equipped with anti-loosening devices to minimize the probability of accidents. Therefore, it is thought of using target detection technology to detect some hooks without anti-loosening devices and issue early warnings to ensure life safety and meet management requirements at the same time. Summary of the invention

[0004] In order to realize real-time detection of whether a crane hook has been installed with an anti-drop device during operation and meet management requirements, the present invention provides a hook anti-drop device detection method based on deep learning.

[0005] In order to achieve the above-mentioned purpose of the invention, the present invention provides a detection method for a hook anti-dropping device based on deep learning, characterized in that the detection method is:

[0006] Step 1: Use monitoring equipment to obtain monitoring video of the hook position during crane operation at the construction site;

[0007] Step 2: Obtain a hook image containing the hook from the surveillance video of step 1;

[0008] Step 3: Manually annotate the hook image obtained in step 2 as sample data; wherein, the manual annotation is to annotate the hook and the anti-drop device in the sample data respectively, 75% of the annotated sample data is used as training samples, and 25% of the annotated sample data is used as test samples;

[0009] Step 4: Construct a neural network model, and use the training samples and test samples to train the neural network model to obtain a trained neural network model;

[0010] Step 5: Use the monitoring equipment at the construction site to obtain real-time monitoring video, and obtain N consecutive frames of images of the hook to be tested at intervals as a group to be tested;

[0011] Step 6: Input the hook images to be tested that belong to the same test group into the trained neural network model, obtain the hook target and the anti-dropping device target, and extract the coordinates of the hook target and the anti-dropping device target;

[0012] Step 7: Calculate the intersection and union ratio of the hook target and the anti-dropping device target in the same frame of the hook image to be tested; if the intersection and union ratio of the hook target and any other anti-dropping device target is less than 0.05, the hook target is not matched with the corresponding anti-dropping device target, and the hook target is counted as not having the anti-dropping device installed, otherwise it is counted as having the anti-dropping device installed;

[0013] Step 8: Calculate the intersection and union ratio of the hook target and any other anti-dropping device target in the N frames of hook images to be tested, and obtain corresponding N provisional calculation results for each hook. If the N provisional calculation results of the hook are all that the anti-dropping device is not installed temporarily, the detection result of the hook is that the anti-dropping device is not installed, and the detection result (including the target object and coordinate position) is output; the detection of the group to be tested is completed;

[0014] Step 9: Repeat the actions in step 5 to obtain the next group to be tested, and continue to repeat steps 6-8.

[0015] Among them, the monitoring equipment in step 1 and step 5 is a monitoring equipment installed on a utility pole or street lamp in the construction site, which can obtain images around the crane during construction site operations; the horizontal distance between the monitoring equipment and the crane position is within 100 meters.

[0016] In step 6, the hook target and the anti-slip device target output by the neural network model are bounding boxes with a confidence level greater than 0.3.

[0017] The neural network model constructed in step 4 adopts the YOLOv4 network model structure, and the YOLOv4 network model only uses 4-fold and 8-fold down-sampling feature maps.

[0018] In the step 5, N consecutive frames of hook images to be tested are obtained from the monitoring video every N frames; the test group includes N frames of hook images to be tested, and N is 45.

[0019] The sample data in step 3 also includes filling sample data, which is a simulated picture under different weather conditions by changing the brightness, hue and saturation based on the actually obtained hook picture.

[0020] The sample data in step 3 also includes pictures obtained by copying based on the actually obtained hook pictures.

[0021] The beneficial effect of the present invention is that the present invention can detect in real time whether the crane hook is equipped with an anti-slip device during operation through target detection technology, which not only meets the management requirements, but also can timely find the hook without the anti-slip device and issue an early warning to avoid accidents. In addition, the present invention performs data enhancement on the training samples of the model based on the actual situation of the hook and the anti-slip device. In order to speed up the recognition efficiency, the feature maps of 4 times and 8 times downsampling in YOLOv4 are used, and through a specific early warning scheme, the position information of the hook can be immediately warned and output. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a flow chart of Example 1 of the present invention.

[0023] Figure 2 Schematic diagram of labeling sample data in Example 1 of the present invention.

[0024] Figure 3 It is a schematic diagram of obtaining sample data enhancement effect by color transformation in Embodiment 2 of the present invention.

[0025] Figure 4 It is a schematic diagram of obtaining sample data enhancement effect by copying method in Example 3 of the present invention. DETAILED DESCRIPTION

[0026] The present invention uses YOLOv4 as a detector and deploys it in the background server, which is integrated with the front-end camera to form a hook anti-drop device detection system. For the detected hook, if the corresponding anti-drop device is not detected within a sufficient number of frames, an early warning is immediately issued and the location information of the hook without the anti-drop device is output to ensure that rectification is discovered in the first place.

[0027] Before designing the present invention, the technical problems that need to be solved are: first, the features of the pictures taken by the front-end camera change greatly with the weather, and pictures with a single feature are not conducive to model training; second, in the scene picture, whether from an absolute or relative perspective, the anti-slip device is a small target type, and the amount of information itself is small, which brings great challenges to detection. In order to alleviate the impact of the above problems on the YOLOv4 detection algorithm, the present invention actually proposes three improvement methods, which are as follows:

[0028] 1. Data enhancement: Detectors with excellent performance require the support of massive data. A large number of hook images are collected from major construction sites as training samples. Because the data involves confidentiality, they cannot be made public. The images are selected arbitrarily during the collection process. In the first stage, the diversity of image features has been favored. In order to enrich the image features and make the trained model more robust to weather and lighting conditions, the present invention carefully utilizes the color enhancement method, starting from brightness, hue, and saturation, to simulate images under different weather conditions. In order to improve the problem of missed detection due to the small anti-slip device, the present invention uses the copying method to increase the number of samples in each image without introducing new images, and the detection performance is significantly improved.

[0029] 2. Selection and utilization of feature maps; target detection must not only accurately locate the position of the target, but also identify the category of the target. Generally speaking, the semantic information contained in the deep network is relatively rich, but the position information is seriously lost. Shallow networks are the opposite. Therefore, too deep networks are not always used for detection tasks, and the introduction of feature pyramids (FPN) alleviates the contradiction between the two aspects, proving that shallow feature maps can also be used for detection. Because small targets are small in size, information will be continuously lost during the downsampling process, so it is more appropriate to use shallow layers to detect small targets. In YOLOv4, feature maps of 8, 16 and 32 times downsampling are used to detect small, medium and large targets. However, the task anti-slip device targeted by the present invention is too small, and the feature map of 8 times downsampling is far from enough, so the feature map of 4 times downsampling is also added, and because the targets involved are only two types of small targets, hooks and anti-slip devices, this invention only uses 4 times and 8 times downsampling feature maps, and the feature maps of the other two scales are omitted, which also reduces unnecessary computational overhead to a certain extent and speeds up the reasoning time.

[0030] 3. Early warning scheme: The anti-slip device detection frame is included in the hook detection frame. If the anti-slip device is not included in the hook detection frame in the prediction stage, the early warning will be started. In order to prevent false alarms due to missed detection or occlusion of the hook itself, a tolerance of 45 frames is set. When a hook that may not have an anti-slip device appears, recording will begin. If there is no frame with an anti-slip device matching it within 45 frames, an early warning will be issued immediately, and the current hook position information will be output.

[0031] In order to clearly illustrate the technical features of this solution, this solution is described below through a specific implementation method.

[0032] Example 1

[0033] See also Figure 1 and Figure 2 , the embodiment of the present invention provides a hook anti-dropping device detection method based on deep learning, the detection method is specifically:

[0034] Step 1: Use monitoring equipment to obtain monitoring video of the hook position during crane operation at the construction site;

[0035] Step 2: Obtain a hook image containing the hook from the surveillance video of step 1;

[0036] Step 3: Manually annotate the hook image obtained in step 2 as sample data; wherein, the manual annotation is to annotate the hook and the anti-drop device in the sample data respectively, 75% of the annotated sample data is used as training samples, and 25% of the annotated sample data is used as test samples;

[0037] Step 4: Construct a neural network model, and use the training samples and test samples to train the neural network model to obtain a trained neural network model;

[0038] Step 5: Use the monitoring equipment at the construction site to obtain real-time monitoring video, and obtain 45 consecutive frames of images of the hook to be tested at intervals as a group to be tested;

[0039] Step 6: Input the hook images to be tested that belong to the same test group into the trained neural network model, obtain the hook target and the anti-dropping device target, and extract the coordinates of the hook target and the anti-dropping device target;

[0040] Step 7: Calculate the intersection and union ratio of the hook target and the anti-dropping device target in the same frame of the hook image to be tested; if the intersection and union ratio of the hook target and any other anti-dropping device target is less than 0.05, the hook target is not matched with the corresponding anti-dropping device target, and the hook target is counted as not having the anti-dropping device installed, otherwise it is counted as having the anti-dropping device installed;

[0041] Step 8: Calculate the intersection and union ratio of the hook target and any other anti-dropping device target in the 45 frames of hook images to be tested, and obtain N corresponding provisional calculation results for each hook. If the 45 provisional calculation results of the hook are all that the anti-dropping device is not installed temporarily, then the detection result of the hook is that the anti-dropping device is not installed, and the detection result will be output; the detection of the group to be tested is completed;

[0042] Step 9: Repeat the actions in step 5 to obtain the next group to be tested, and continue to repeat steps 6-8.

[0043] Among them, the monitoring equipment in step 1 and step 5 is a monitoring equipment installed on a utility pole or street lamp in the construction site, which can obtain images around the crane during construction site operations; the horizontal distance between the monitoring equipment and the crane position is controlled within 100 meters.

[0044] In step 1 and step 5, the hook image is obtained from the surveillance video by adopting the method of grouping frame extraction.

[0045] In step 6, the hook target and the anti-slip device target output by the neural network model are bounding boxes with a confidence level greater than 0.3.

[0046] The neural network model constructed in step 4 adopts the YOLOv4 network model structure, and the YOLOv4 network model only uses 4x and 8x down-sampled feature maps.

[0047] In step 5, 45 consecutive frames of images of the hook to be tested are acquired from the monitoring video every 45 frames.

[0048] Example 2

[0049] See also Figure 3 The embodiment of the present invention provides a method for detecting a hook anti-dropping device based on deep learning. The specific method is the same as that of Example 1, except that: on the basis of Example 1, the sample data in step 3 also includes filling sample data, and the filling sample data is based on the actual obtained hook picture and simulates pictures under different weather conditions by changing the brightness, hue, and saturation.

[0050] Example 3

[0051] See also Figure 4 The embodiment of the present invention provides a hook anti-dropping device detection method based on deep learning. The specific method is the same as that of Example 1, except that: based on Example 1 or 2, the sample data in step 3 also includes a picture obtained by copying based on the actual hook picture.

[0052] Experimental example

[0053] This experiment was conducted under the conditions of Intel Core i7 CPU, 4G memory and NVIDIA GeForce 2080Ti independent graphics card, using the construction site scene dataset collected by myself for training and testing. The training of this model was loaded with the weight file after ImageNet pre-training, and iterated 70 times. Two test scenes were selected to obtain 3684 frames of image testing. The specific results are shown in the table below.

[0054] <![CDATA[ Algorithm Type ]]> <![CDATA[ Number of test scenarios ]]> <![CDATA[ Video frame rate ]]> <![CDATA[ Correct identification ]]> <![CDATA[ Error identification ]]> <![CDATA[ Accuracy ]]> Hook anti-drop 2 3684 3612 72 98％

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A hook anti-drop device detection method based on deep learning, characterized in that: The detection method is: Step 1: Use monitoring equipment to obtain monitoring video of the hook position during crane operation at the construction site; Step 2: Obtain a hook image containing the hook from the surveillance video of step 1; Step 3: Manually annotate the hook image obtained in step 2 as sample data; wherein, the manual annotation is to annotate the hook and the anti-drop device in the sample data respectively, 75% of the annotated sample data is used as training samples, and 25% of the annotated sample data is used as test samples; Step 4: Construct a neural network model, and use the training samples and test samples to train the neural network model to obtain a trained neural network model; Step 5: Use the monitoring equipment at the construction site to obtain real-time monitoring video, and obtain N consecutive frames of images of the hook to be tested at intervals as a group to be tested; Step 6: Input the hook images to be tested that belong to the same test group into the trained neural network model, obtain the hook target and the anti-dropping device target, and extract the coordinates of the hook target and the anti-dropping device target; Step 7: Calculate the intersection and union ratio of the hook target and the anti-dropping device target in the same frame of the hook image to be tested; if the intersection and union ratio of the hook target and any other anti-dropping device target is less than 0.05, the hook target is not matched with the corresponding anti-dropping device target, and the hook target is counted as not having the anti-dropping device installed, otherwise it is counted as having the anti-dropping device installed; Step 8: Calculate the intersection and union ratio of the hook target and any other anti-dropping device target in the N frames of hook images to be tested, and obtain corresponding N provisional calculation results for each hook. If the N provisional calculation results of the hook are all that the anti-dropping device is not installed temporarily, the detection result of the hook is that the anti-dropping device is not installed, and the detection result will be output; the detection of the group to be tested is completed; Step 9: Repeat the actions in step 5 to obtain the next group to be tested, and continue to repeat steps 6-8.

2. The detection method according to claim 1, characterized in that: The monitoring equipment in step 1 and step 5 is a monitoring equipment installed on a utility pole or street lamp in the construction site, and can obtain images around the crane during construction site operations.

3. The detection method according to claim 2, characterized in that: The horizontal distance between the monitoring equipment and the crane position is within 100 meters.

4. The detection method according to any one of claims 1 to 3, characterized in that: In step 6, the hook target and the anti-slip device target output by the neural network model are bounding boxes with a confidence level greater than 0.

3.

5. The detection method according to any one of claims 1 to 3, characterized in that: The neural network model constructed in step 4 adopts the YOLOv4 network model structure, and the YOLOv4 network model only uses 4-fold and 8-fold down-sampling feature maps.

6. The detection method according to any one of claims 1 to 3, characterized in that: In step 5, N consecutive frames of images of the hook to be tested are acquired every N frames from the monitoring video.

7. The detection method according to any one of claims 1 to 3, characterized in that: The group to be tested includes N frames of hook images to be tested, where N is 45.

8. The detection method according to any one of claims 1 to 3, characterized in that: The sample data in step 3 also includes filling sample data, which is a simulated picture under different weather conditions by changing the brightness, hue and saturation based on the actually obtained hook picture.

9. The detection method according to claim 8, characterized in that: The sample data in step 3 also includes pictures obtained by copying based on the actually obtained hook pictures.

Images