Crane key component defect detection method based on YOLO

By improving the model structure of YOLO neural network, the problem of feature loss and large calculation amount of detection of few samples and small targets in complex backgrounds is solved, and efficient and accurate detection of defects of key components of cranes is achieved, which improves detection accuracy and speed.

CN118982528APending Publication Date: 2024-11-19HARBIN UNIV OF SCI & TECH

Patent Information

Application Number
CN202411082122.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing target defect detection methods have problems such as feature loss, large convolutional layer depth, large number of feature map channels and large calculation amount in complex backgrounds, making it difficult to achieve efficient and accurate defect detection.

Method used

The defect detection method for key components of cranes based on YOLO neural network has been improved by improving the model structure, including replacing the coupling head with a decoupling head, performing channel pruning, combining convolution and attention mechanisms, and using a lightweight upsampling operator CARAFE, The adaptive spatial feature fusion module is introduced to optimize the detection algorithm to improve detection accuracy and speed.

Benefits of technology

It realizes high-precision, fast and lightweight target defect detection in complex backgrounds, reduces feature loss and calculation amount, enhances the ability to identify small targets, and improves the generalization ability and detection efficiency of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982528A_ABST
    Figure CN118982528A_ABST
Patent Text Reader

Abstract

The invention discloses a crane key component defect detection method based on YOLO, and belongs to the field of target defect detection methods. An existing target defect detection method has the problems of feature loss, large convolutional layer depth, large number of feature map channels and large calculation amount in the detection process of few samples and small targets under a complex background. The method comprises the following steps: collecting and processing a data set; improving a YOLO model structure, and replacing an original coupling head with a decoupling head; performing channel pruning; a detection algorithm is improved in cooperation with convolution and an attention mechanism, and a better regression function of EIoU Loss is selected as a bounding box regression loss function; a lightweight general up-sampling operator CARAFE is used for replacing nearest neighbor interpolation so as to improve the up-sampling method; training the improved YOLO model structure by using part of data; and detecting the defects of the key components of the crane by using the trained model. The method is suitable for small-sample and small-target defect identification under a complex background, and has the advantages of light model weight, high identification accuracy and high speed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a component defect detection method, and in particular to a method for detecting defects of key components of a crane based on YOLO. Background Art

[0002] Due to the harsh conditions of the industrial environment, the long-term operation of cranes, the stretching and bending of wire ropes, the corrosion of seawater and oil pollution and other factors, various problems such as wire rope drawing and breaking, stair corrosion and cracking, and screw corrosion often occur. These problems may have immeasurable consequences for the safe production of the people and enterprises.

[0003] Therefore, regularly checking the condition of equipment and promptly replacing and maintaining old and damaged parts can not only ensure that work proceeds in an orderly manner, but also protect the personal safety of staff.

[0004] For fault detection, the traditional detection method is to observe with human eyes and make direct judgments when obvious faults are found. This method has obvious defects: first, it is difficult to judge small defects. The construction site environment is harsh, such as the wire rope is covered with oil or the human handrail is slightly cracked, these problems are easily overlooked; second, the human eye cannot detect efficiently for a long time, and fatigue will inevitably occur over time, and the accuracy of judgment will plummet; third, additional labor costs are added. Therefore, in order to minimize the occurrence of accidents, each department will arrange special personnel to label and record some easily consumable objects. When the number of operations reaches the specified limit, they will be replaced regardless of their status. Although this method avoids accidents to the greatest extent, it also brings huge economic losses. Therefore, an efficient and reliable automatic fault detection system is needed to solve the above problems. This system can accurately detect and identify various defects of key components of cranes, and can work efficiently for a long time without being affected by harsh environments. This not only reduces labor costs, but also can detect potential problems in a timely manner, thereby avoiding accidents and reducing economic losses caused by unnecessary replacements.

[0005] In recent years, with the continuous development of science and technology, many scholars have begun to adopt more advanced methods for defect monitoring. For example, the use of physical sensor methods is to use electromagnetic detection, acoustic detection, etc. to qualitatively detect the damage of wire rope notches, but this method is difficult to quantitatively analyze defects and difficult to distinguish categories. In addition, the weak anti-noise ability and high price are also one of the reasons why it is difficult to popularize. There are also high-speed cameras to collect images of wire rope and nut surfaces, and then pre-process the images, and use machine learning models to realize the defect classification of wire ropes. Although machine vision methods are low-cost and highly intelligent, it is difficult for machine vision to handle unquantifiable object defects. This is because the defects in robot vision are defined by personal standards. In addition, the method has poor robustness and cannot be used for all types of defects. Some features are difficult to distinguish and extract manually. With the rapid development of deep learning technology, features that are difficult to quantify and extract by machine vision can also be easily solved by deep learning technology. Generally speaking, target detection algorithms are divided into two categories: single-stage algorithms and two-stage algorithms. Two-stage target detection algorithms are represented by RCNN, Fast-RCNN, and Faster-RCNN. This type of method has high detection accuracy, but slow detection speed. The single-stage target detection algorithm SSD and YOLO series are representative. This type of algorithm realizes end-to-end detection. The accuracy of early single-stage target detection algorithms is poor. However, with the continuous development and improvement of the YOLO series, the detection accuracy of YOLOv4 has also surpassed the two-stage target detection algorithm in some specific scenarios. Therefore, the research on the defect detection method of key parts of cranes based on YOLO neural network has high value and prospects, which can greatly improve the safety and operation efficiency of cranes.

[0006] The detection methods for small samples and small targets in complex backgrounds have problems such as feature loss, large convolution layer depth, large number of feature map channels, and large amount of calculation in the recognition process. In recent years, deep learning has gradually become a research hotspot. Deep learning originated from the study of artificial neural networks as a new research field in machine learning. The present invention is based on the improvement of the YOLO series machine learning model, and studies and designs a targeted target defect detection method that can combine accuracy, speed, and lightness. Summary of the invention

[0007] The purpose of the present invention is to solve the problems of feature loss, large convolution layer depth, large number of feature map channels and large amount of calculation in the detection and recognition process of few samples and small targets in complex backgrounds in the existing target defect detection methods, and to propose a defect detection method for key components of cranes based on YOLO.

[0008] The above purpose is achieved through the following technical solutions:

[0009] A method for detecting defects of key components of a crane based on YOLO is implemented by the following steps:

[0010] Step 1: Collect and process data sets for model training;

[0011] The dataset is collected from real damaged pictures of construction site cranes, some of which are from the Internet as original pictures, and the original pictures are flipped, scaled, and cut; then, the dataset damage is annotated using marking software, including wire rope drawing, cracking, human corrosion, and wear defect types, and a txt file of the corresponding format is generated for the annotated pictures;

[0012] Step 2: Improve the YOLO model structure based on YOLOv6 or YOLOv7, specifically:

[0013] Replace the original coupling head with a decoupling head;

[0014] Perform channel pruning to make the model lightweight;

[0015] Step 3: Improve the detection algorithm with convolution and attention mechanism, and select a regression function with better EIoU Loss as the bounding box regression loss function;

[0016] Step 4: Use the lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to improve the upsampling method;

[0017] Step 5: Introduce an adaptive spatial feature fusion module into the feature fusion structure to optimize the Head part by adaptively fusing the information of shallow features and deep features of the image;

[0018] Step 6: Use the image features extracted from part of the data set in step 5 to train the YOLO model structure improved by the methods in steps 2, 3, 4 and 5;

[0019] Step seven, using the trained model obtained in step six to detect defects in key components of the crane.

[0020] Furthermore, the step of replacing the original coupled head with the decoupled head in step 2 is specifically: in the target detection task, on the Decoupled Head structure, the original coupled head is replaced with the decoupled head.

[0021] Furthermore, the step of performing channel pruning to make the model lightweight as described in step 2 is to sort each channel according to its importance during the training process, and then delete the channels with the lowest order from the network. Channel pruning is specifically divided into: calculating channel importance, pruning operation, and fine-tuning the network; wherein, channel importance is to calculate a weight value for each channel, and according to the channel importance sorting result, the number of channels to be retained is determined, and the channels with the lowest order are deleted from the network; in this process, the network structure and parameters need to be readjusted, and the model needs to be fine-tuned after the pruning operation.

[0022] Furthermore, the steps of improving the detection algorithm by coordinating convolution and attention mechanisms as described in step three are specifically as follows: including two modules, CAM and SAM; firstly performing channel attention, and performing pooling operations on the input feature map through global maximum pooling and global average pooling based on width and height respectively; then, inputting the pooled results into the multi-layer perceptron for processing respectively; the output features of the MLP are subjected to element-by-element addition operation, and then processed by the Sigmoid activation function to generate a channel attention feature map; this feature map is element-by-element multiplied with the input feature map to obtain the input features required by the spatial attention module; the spatial attention is to perform maximum pooling and average pooling on the basis of Channal, and the two results are concat and then convolved to obtain a Channal, and a spatial attention feature map is generated after Sigmoid, and this feature map is multiplied with the input features of the module.

[0023] Furthermore, the steps of improving the upsampling method described in step 4 are specifically: using a lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to obtain a higher quality upsampling feature map, the CARAFE operator includes two modules, namely an upsampling kernel prediction module and a feature recombination module; wherein the upsampling kernel prediction module predicts the upsampling kernels corresponding to feature points at different positions by analyzing and encoding the input feature map, and the feature recombination module uses the upsampling kernels predicted by the upsampling kernel prediction module to complete the upsampling work.

[0024] Furthermore, the step of introducing an adaptive spatial feature fusion module in the feature fusion structure to optimize the Head part described in step 5 is specifically: introducing an ASFF module in the detection head part to suppress the information conflict between features at different levels and improve the fusion effect of PANet.

[0025] The beneficial effects of the present invention are:

[0026] According to the current status of relevant research on the existing YOLO neural network, the present invention designs a targeted, high-precision, fast and lightweight detection method for defect detection methods of small samples and small targets under complex backgrounds. First, according to data enhancement means, the scale of the training set is expanded to improve the generalization ability of the model. By introducing the attention module, the original border loss function is improved to reduce the loss of features and the impact of the border loss function on accuracy. And according to the Decoupled Head structure proposed by YOLOX, an efficient coupled prediction head is proposed to reduce the depth of the convolution layer and the number of channels of the feature map, so as to achieve the effect of reducing the amount of calculation. In addition, through optimization means such as network pruning, we strive to achieve accuracy while ensuring the lightweight of the model.

[0027] The present invention also introduces the ASFF module into the feature fusion network, which adaptively fuses the information of shallow features and deep features of the image, enhances the model's detection ability for targets of different scales, effectively solves the problem of missed detection or false detection, and strengthens the scale invariance of features. It realizes the adaptive adjustment and fusion of feature space weights to improve the recognition effect of distant small targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flow chart of the method of the present invention;

[0029] Figure 2 It is a diagram of the ASFF module structure involved in the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific implementation method one:

[0032] A method for detecting defects of key crane components based on YOLO in this embodiment is characterized in that: Figure 1 As shown, the method is implemented by the following steps:

[0033] Step 1: Collect and process data sets for model training;

[0034] The dataset is collected from real damaged pictures of cranes on construction sites, some of which are from the Internet as original pictures. In order to increase the sample size of the dataset, the original pictures are flipped, scaled, cut, and other operations are performed. After that, the dataset damage is annotated using marking software, including wire rope wire drawing, cracking, human corrosion, and wear and tear. The annotated pictures are then converted into txt files of the corresponding format.

[0035] Step 2: Improve the YOLO model structure based on YOLOv6 or YOLOv7, specifically:

[0036] The original coupling head is replaced with a more efficient decoupling head to more efficiently capture feature information; this method can effectively capture global or local information;

[0037] Perform certain channel pruning to make the model more lightweight;

[0038] Step 3: Improve the detection algorithm by combining convolution and attention mechanisms, and focus on the image from the perspectives of space and channels, so that the model pays more attention to the defects of key components of the crane and improves the detection accuracy of defects;

[0039] Bounding box prediction is one of the most important tasks in target detection. In target detection, if you want to frame the detected target, you need to predict the location data of the bounding box. The original version uses the square loss, that is:

[0040] L local =(xx * ) 2 +(yy * ) 2 +(ww * ) 2 +(hh * ) 2

[0041] Where x, y, w, and h represent the coordinates of the upper left corner of the predicted box and the length and width of the box respectively. The advantages of this method are simplicity, low computational effort, and ease of optimization. However, this method has certain problems. Since the bounding box prediction needs to focus on the overlapping area between the box and the real box, and the ratio of the overlapping area to the area of ​​the union of the two is as large as possible, the square loss alone cannot measure this well. In the future, the secondary bounding box regression function will be replaced, and a regression function with a better EIoU Loss will be selected as the bounding box regression loss function.

[0042] Step 4: Use the lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to improve the upsampling method;

[0043] Step 5: In order to improve the model's ability to detect small objects in the image, the adaptive spatial feature fusion (ASFF) module is introduced into the feature fusion structure to optimize the Head part. By adaptively fusing the information of shallow features and deep features of the image, the model's ability to detect objects of different scales is enhanced, effectively solving the problem of missed detection or false detection, and strengthening the scale invariance of the feature.

[0044] Step 6: Use the image features extracted from part of the data set in step 5 to train the YOLO model structure improved by the methods in steps 2, 3, 4 and 5;

[0045] Step seven, using the trained model obtained in step six to detect defects in key components of the crane. Specific implementation method 2:

[0047] Different from the first specific implementation, in the present implementation, a method for detecting defects of key crane components based on YOLO, the step of replacing the original coupling head with a more efficient decoupling head described in step 2 is specifically: in the target detection task, the classification task and the regression task are two relatively independent tasks. The classification task pays more attention to the difference between each target, and the regression task pays more attention to the boundary characteristics of the target contour. In the target detection algorithm, the use of a coupled prediction head that simultaneously completes the classification task and the regression task prediction work will not only reduce the detection accuracy of the model, but also affect the convergence speed of the model. Therefore, YOLOX proposed a Decoupled Head structure, which can separate the classification task and the regression task, and use two parallel branches to predict the two respectively, but the Decoupled Head structure is too complicated, resulting in a substantial increase in the model volume and the amount of calculation. Therefore, on the Decoupled Head structure, the original coupling head is replaced with a more efficient decoupling head, and a coupling head that can balance the detection accuracy, model volume and detection speed is designed. Specific implementation method three:

[0049] Different from the second specific implementation method, the present implementation method for detecting defects of key components of a crane based on YOLO, the step of performing channel pruning in step 2 to make the model lightweight, channel pruning (ChannelPruning) is a neural network compression method, which aims to reduce the size of the model, improve the inference speed and reduce the computational complexity by reducing the number of useless channels in the network. Its basic idea is to sort the importance of each channel during the training process, and then delete the channels with the lower ranking from the network to achieve the effect of pruning. Channel pruning is specifically divided into: calculating channel importance, pruning operation, and fine-tuning the network; wherein, channel importance is to calculate a weight value for each channel for sorting. According to the ranking result of channel importance, the number of channels to be retained is determined, and the channels with the lower ranking are deleted from the network; in this process, the network structure and parameters need to be readjusted to ensure that the accuracy of the model is not affected. After the pruning operation, the model needs to be fine-tuned to restore the accuracy before pruning. This is also an optimization method for the model. Specific implementation method four:

[0051] Different from the third specific implementation mode, in the present implementation mode, a method for detecting defects of key components of a crane based on YOLO, the steps of improving the detection algorithm by coordinating the convolution and attention mechanism described in step three are as follows: CBAM (Convolutional Block Attention Module) is a simple and effective attention module for feedforward convolutional neural networks, that is, to let the network focus on a key part, which includes two modules: CAM (Channel Attention Module) and SAM (Spartial Attention Module); firstly, channel attention is performed, and the input feature map is pooled by global max pooling and global average pooling based on width and height. pooling) respectively for pooling operations; then, the pooled results are input into the multi-layer perceptron (MLP) for processing; the output features of the MLP are added element by element, and then processed by the Sigmoid activation function to generate a channel attention feature map; this feature map is multiplied element by element with the input feature map to obtain the input features required by the spatial attention module; spatial attention is to perform maximum pooling and average pooling on the basis of Channal, concatenate the two results and then convolve to obtain a Channal, generate a spatial attention feature map after Sigmoid, and multiply this feature map with the input features of the module. Specific implementation method five:

[0053] Different from the fourth specific implementation mode, in the present implementation mode, a method for detecting defects of key components of a crane based on YOLO, the steps of the improved upsampling method described in step 4 are specifically as follows: the upsampling method used in YOLO is the nearest neighbor interpolation sampling, and the upsampling kernel used is determined only by the spatial position of the pixel point, and the receptive field size is 1×1. The nearest neighbor interpolation upsampling does not utilize the rich semantic information extracted from the feature map, but simply upsamples all feature points on average according to the spatial position, ignoring the possible influence of the surrounding feature points, which will reduce the quality of the upsampled feature map to a certain extent. Therefore, the present invention proposes to use a lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to obtain a higher quality upsampled feature map, and the CARAFE operator includes two modules, namely, an upsampling kernel prediction module and a feature reorganization module; wherein, the upsampling kernel prediction module predicts the upsampling kernels corresponding to the feature points at different positions by analyzing and encoding the input feature map, and the feature reorganization module uses the upsampling kernels predicted by the upsampling kernel prediction module to complete the upsampling work. Specific implementation method six:

[0055] Different from the fifth specific implementation mode, the present implementation mode is a method for detecting defects of key crane components based on YOLO. The step of introducing an adaptive spatial feature fusion module in the feature fusion structure to optimize the Head part as described in step five is specifically: the traditional YOLOv7 model combines top-down and bottom-up feature fusion functions on the basis of the FPN (Feature pyramid networks) network, and can adaptively adjust the spatial fusion weights of features at each scale. However, multi-scale fusion will cause feature information conflicts between different levels, that is, inconsistency. The inconsistency between different feature scales is the main limitation of the single-stage detector based on FPN. The ASFF module can spatially filter the conflicting information to suppress the inconsistency, and can improve the scale invariance of the feature. Therefore, the present invention introduces the ASFF module in the detection head part to suppress the information conflict between features at different levels, improve the fusion effect of PANet, and then improve the target detection capability. The ASFF module structure is as follows: Figure 2 shown.

[0056] The embodiments of the present invention disclose preferred embodiments, but are not limited thereto. A person skilled in the art can easily understand the spirit of the present invention based on the above embodiments and make different extensions and changes. However, as long as they do not deviate from the spirit of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for detecting defects in key crane components based on YOLO. Features: The method is implemented by the following steps: Step 1: Collect and process data sets for model training; The dataset is collected from real damaged pictures of construction site cranes, some of which are from the Internet as original pictures, and the original pictures are flipped, scaled, and cut; then, the dataset damage is annotated using marking software, including wire rope drawing, cracking, human corrosion, and wear defect types, and a txt file of the corresponding format is generated for the annotated pictures; Step 2: Improve the YOLO model structure based on YOLOv6 or YOLOv7, specifically: Replace the original coupling head with a decoupling head; Perform channel pruning to make the model lightweight; Step 3: Improve the detection algorithm with convolution and attention mechanism, and select a regression function with better EIoU Loss as the bounding box regression loss function; Step 4: Use the lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to improve the upsampling method; Step 5: Introduce an adaptive spatial feature fusion module into the feature fusion structure to optimize the Head part by adaptively fusing the information of shallow features and deep features of the image; Step 6: Use the image features extracted from part of the data set in step 5 to train the YOLO model structure improved by the methods in steps 2, 3, 4 and 5; Step seven, using the trained model obtained in step six to detect defects in key components of the crane.

2. According to claim 1, a method for detecting defects of key components of a crane based on YOLO, Features: The step of replacing the original coupling head with the decoupling head in step 2 is specifically: in the target detection task, on the Decoupled Head structure, the original coupling head is replaced with the decoupling head.

3. According to claim 2, a method for detecting defects of key components of a crane based on YOLO, Features: The step of performing channel pruning to make the model lightweight as described in step 2 is to sort each channel according to its importance during the training process, and then delete the channels with the lowest ranking from the network. Channel pruning is specifically divided into: calculating channel importance, pruning operation, and fine-tuning the network; among them, channel importance is to calculate a weight value for each channel, and according to the channel importance sorting result, the number of channels to be retained is determined, and the channels with the lowest ranking are deleted from the network; in this process, the network structure and parameters need to be readjusted, and the model needs to be fine-tuned after the pruning operation.

4. According to claim 3, a method for detecting defects of key components of a crane based on YOLO, Features: The steps of improving the detection algorithm by coordinating convolution and attention mechanisms as described in step 3 are as follows: including two modules, CAM and SAM; firstly, performing channel attention, and performing pooling operations on the input feature map through global maximum pooling and global average pooling based on width and height respectively; then, inputting the pooled results into the multi-layer perceptron for processing respectively; the output features of the MLP are subjected to element-by-element addition operation, and then processed by the Sigmoid activation function to generate a channel attention feature map; this feature map is subjected to element-by-element multiplication operation with the input feature map to obtain the input features required by the spatial attention module; Spatial attention performs maximum pooling and average pooling on the basis of Channal. The two results are concat and then convolved to obtain a Channal. After Sigmoid, a spatial attention feature map is generated, which is multiplied by the input feature of the module.

5. According to claim 4, a method for detecting defects of key components of a crane based on YOLO, Features: The steps of improving the upsampling method described in step 4 are specifically: using a lightweight universal upsampling operator CARAFE to replace the nearest neighbor interpolation to obtain a higher quality upsampling feature map, the CARAFE operator includes two modules, namely an upsampling kernel prediction module and a feature recombination module; wherein the upsampling kernel prediction module predicts the upsampling kernels corresponding to feature points at different positions by analyzing and encoding the input feature map, and the feature recombination module uses the upsampling kernels predicted by the upsampling kernel prediction module to complete the upsampling work.

6. According to claim 5, a method for detecting defects of key components of a crane based on YOLO, Features: The step of introducing an adaptive spatial feature fusion module into the feature fusion structure to optimize the Head part described in step 5 is specifically: introducing an ASFF module into the detection head part to suppress the information conflict between features at different levels and improve the fusion effect of PANet.

Citation Information

Patent Citations

  • YOLOX-based steel surface defect detection method, system, equipment and medium

    CN117392052A

  • Steel surface defect detection algorithm based on improved YOLOv8 model

    CN117745697A

  • Deep learning-based face feature point detection method

    WO2022151535A1

Cited By

  • Intelligent detection method and device for state of cotter pin of tower crane

    CN119181034A

  • Light-weight photovoltaic cell EL image defect detection method

    CN120182733A

  • A lightweight method for detecting defects in EL images of photovoltaic cells

    CN120182733B