Intelligent detection method and device for state of cotter pin of tower crane

Abstract

The invention provides an intelligent detection method and device for the state of a cotter pin of a tower crane. The method comprises the following steps: inputting an image of the cotter pin of the tower crane to be detected into a target detection model obtained by training improved YOLOv8; performing multi-level feature extraction processing on the tower crane cotter image through a backbone network in the target detection model, and determining a plurality of different levels of target feature maps; performing fusion processing on the plurality of target feature layers through a neck network using an adaptive feature pyramid neural network structure in the target detection model, and determining a plurality of target fusion feature maps; and analyzing the plurality of target fusion feature maps through a head network in the target detection model, and generating a state detection result of the tower crane cotter pin. Therefore, according to the scheme, through the image recognition technology, the detection of the tower crane cotter pin is more convenient and faster, and the target detection model is obtained based on the improved YOLOv8 training, so that the detection efficiency and the accuracy of the detection result are improved.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an intelligent detection method and device for the status of tower crane cotter pins. Background Technology

[0002] In tower cranes, cotter pins are a critical fastener widely used to connect and secure various components. Their integrity and reliability directly affect the safety of the tower crane. However, cotter pins are prone to deformation, wear, and breakage during long-term use, leading to connection failure and threatening the normal operation of the tower crane. Therefore, detecting cotter pin defects is crucial for ensuring the safety of tower cranes.

[0003] Traditional methods for detecting cotter pin defects primarily rely on manual visual inspection, where operators use their experience to observe and judge surface defects, deformation, and connection status. While simple, this method has low accuracy and efficiency, and is easily affected by the operator's subjective factors, posing safety hazards, especially when working at heights.

[0004] In addition, existing technologies also employ ultrasonic testing for cotter pin defect detection. This method can detect internal cracks or defects in the cotter pin and determine the presence of internal abnormalities through reflected signals. While this method is effective for detecting internal defects, its effectiveness in detecting surface defects, wear, and other cosmetic issues in cotter pins is limited, and the testing cost is high.

[0005] Existing technologies also include X-ray inspection, which can identify the internal structural integrity of cotter pins and determine the presence of defects such as cracks through imaging analysis. This method is effective for inspecting metallic materials, but it is complex to operate and the equipment is expensive, making it unsuitable for on-site inspection. Summary of the Invention

[0006] In view of this, the purpose of this application is to provide an intelligent detection method and device for the status of tower crane cotter pins. By using image recognition technology, the detection of tower crane cotter pins becomes more convenient and faster. Furthermore, a target detection model is trained based on an improved YOLOv8, thereby improving detection efficiency and the accuracy of detection results.

[0007] This application provides an intelligent detection method for the status of cotter pins on tower cranes, the intelligent detection method comprising: The image of the cotter pin of the tower crane to be inspected, captured by a drone, is input into a pre-trained target detection model; wherein the target detection model is determined by training an improved YOLOv8. The target detection model uses a backbone network to perform multi-level feature extraction on the tower crane cotter pin image, thereby determining multiple target feature maps at different levels. The target C2F module in the backbone network is a C2F module obtained by incorporating a large convolutional kernel into the original C2F module. The target C2F module includes a dilated convolutional block, an SE module, and a feedforward neural network. The target detection model uses a neck network with an adaptive feature pyramid neural network structure to fuse multiple target feature layers, thereby determining multiple target fusion feature maps. The head network in the target detection model analyzes the fused feature maps of multiple targets to generate the status detection results of the tower crane cotter pins.

[0008] Optionally, the target detection model can be constructed through the following steps: Based on the inspection images of tower crane cotter pins, at least one batch of training sample sets is constructed; wherein each inspection image in the training sample set has a real state label. For each batch of training sample sets, the inspection images in the training sample set are sequentially input into the improved YOLOv8. The composite convolution module and the target C2F module in the backbone network are used to perform layer-by-layer feature extraction to obtain multiple sample feature maps at different levels for each inspection image. For each inspection image, the neck network of the improved YOLOv8, which uses an adaptive feature pyramid neural network structure, performs bottom-up feature fusion processing on multiple sample feature maps of the inspection image to generate multiple sample fused feature maps of the inspection image. The improved YOLOv8 head network was used to analyze the fused feature maps of multiple samples of the inspection image to determine the predicted cotter pin status of the inspection image. Based on the predicted cotter pin status and the true status label of each inspection image in the training sample set, the total loss function value of the improved YOLOv8 is determined. The improved YOLOv8 is then trained based on the total loss function value until the total loss function of the improved YOLOv8 converges, thus obtaining the object detection model.

[0009] Optionally, determining the improved YOLOv8 total loss function value based on the predicted cotter pin state and the true state label of each inspection image in the training sample set includes: For each inspection image, the angle loss value, distance loss value, and shape loss value of the inspection image are determined based on the predicted cotter pin status and the actual status label of the inspection image. Based on the angle loss value, distance loss value, and shape loss value of the inspection image, determine the prediction loss function value of the inspection image; Based on the predicted loss function values ​​of all inspected images, determine the total loss function value of the improved YOLOv8.

[0010] Optionally, training the improved YOLOv8 based on the total loss function value includes: Based on the total loss function value, determine whether the total loss function corresponding to the improved YOLOv8 has converged; If convergence is achieved, the target detection model is obtained; If convergence is not achieved, update the network parameters of the improved YOLOv8 and train the improved YOLOv8 using the training sample set until the total loss function corresponding to the improved YOLOv8 converges.

[0011] Optionally, the formula used to update the network parameters of the improved YOLOv8 is:

[0012]

[0013] in, For learning rate, That is, the score center of category j, where t is the number of iterations. Indicates the score center of the current iteration (t-th iteration). Let represent the score center of the next iteration (t+1th iteration), IF be the conditional expression, i be the initial value of the iteration, and M be the final value of the iteration. Let i be the score center of class i.

[0014] Optionally, the target C2F module, when performing feature extraction processing, includes: By working in parallel with multiple dilated convolutional blocks, multi-scale feature extraction is performed on the initial feature map to obtain the first feature map. The first feature map is enhanced by nonlinear mapping using the SE module to generate a second feature map. The second feature map is subjected to nonlinear transformation and regularization by a feedforward neural network to obtain a sample feature map that is input to the adaptive feature pyramid neural network, or a third feature map that is input to the next neural network layer in the backbone neural network.

[0015] Optionally, after analyzing the fused feature maps of multiple targets through the head network in the target detection model to generate the status detection result of the tower crane cotter pin, the intelligent detection method further includes: When the status detection result of the tower crane cotter pin shows that the cotter pin is missing or the degree of looseness exceeds the preset safety threshold; Issue early warnings and generate test reports.

[0016] This application embodiment also provides an intelligent detection device for the status of tower crane cotter pins, the intelligent detection device comprising: An input module is used to input images of the cotter pins of the tower crane to be inspected, captured by a drone, into a pre-trained target detection model; wherein the target detection model is determined by training an improved YOLOv8. The extraction module is used to perform multi-level feature extraction processing on the tower crane cotter pin image through the backbone network in the target detection model to determine multiple target feature maps at different levels; wherein, the target C2F module in the backbone network is a C2F module obtained by incorporating a large convolutional kernel into the original C2F module, and the target C2F module includes a dilated convolutional block, an SE module, and a feedforward neural network. The fusion module is used to fuse multiple target feature layers through the neck network of the adaptive feature pyramid neural network structure in the target detection model to determine multiple target fusion feature maps. The analysis module is used to analyze the fused feature maps of multiple targets through the head network in the target detection model to generate the status detection results of the tower crane cotter pins.

[0017] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the intelligent detection method described above are performed.

[0018] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the intelligent detection method described above.

[0019] This application provides an intelligent detection method and apparatus for the cotter pin status of a tower crane. The intelligent detection method includes: inputting an image of the cotter pin of the tower crane to be detected, captured by a drone, into a pre-trained target detection model; wherein the target detection model is determined by training an improved YOLOv8; performing multi-level feature extraction processing on the tower crane cotter pin image through the backbone network of the target detection model to determine multiple target feature maps at different levels; wherein the target C2F module in the backbone network is a C2F module obtained by incorporating a large convolutional kernel into the original C2F module, and the target C2F module includes dilated convolutional blocks, SE modules, and feedforward neural networks; fusing multiple target feature layers through the neck network of the target detection model using an adaptive feature pyramid neural network structure to determine multiple target fused feature maps; and analyzing the multiple target fused feature maps through the head network of the target detection model to generate the tower crane cotter pin status detection result. This method of using drones to inspect tower crane cotter pins reduces the safety risks of manual climbing. Drone photography is convenient and quick, significantly reducing inspection time and improving efficiency. Employing deep learning algorithms for intelligent image recognition reduces the probability of human error and improves inspection accuracy. The automated inspection process enables real-time monitoring of tower crane cotter pins, facilitating the timely detection and handling of potential safety hazards.

[0020] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating an intelligent detection method for the status of cotter pins on a tower crane, provided in an embodiment of this application; Figure 2 A schematic diagram of an improved YOLOv8 neural network provided in this application; Figure 3 A schematic diagram of the structure of a target C2F module provided in this application; Figure 4 This application provides an example of a tower inspection planning path. Figure 5An example of a crane boom inspection planning path provided in this application; Figure 6 This is one example of a practical inspection path for tower cranes provided in this application; Figure 7 This application provides a second example of a practical inspection route for tower cranes. Figure 8 This application provides a schematic diagram of the cotter pin status of a tower crane. Figure 9 This application provides a performance test result based on the original YOLOv8 neural network in an embodiment of the present application. Figure 10 This application provides a performance test result based on an improved YOLOv8 neural network, as exemplified in this application. Figure 11 One of the structural schematic diagrams of an intelligent detection device for the status of cotter pins of a tower crane provided in an embodiment of this application; Figure 12 This is a second schematic diagram of the structure of an intelligent detection device for the status of a tower crane cotter pin provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0024] In tower cranes, cotter pins are a critical fastener widely used to connect and secure various components. Their integrity and reliability directly affect the safety of the tower crane. However, cotter pins are prone to deformation, wear, and breakage during long-term use, leading to connection failure and threatening the normal operation of the tower crane. Therefore, detecting cotter pin defects is crucial for ensuring the safety of tower cranes.

[0025] Traditional methods for detecting cotter pin defects primarily rely on manual visual inspection, where operators use their experience to observe and judge surface defects, deformation, and connection status. While simple, this method has low accuracy and efficiency, and is easily affected by the operator's subjective factors, posing safety hazards, especially when working at heights.

[0026] In addition, existing technologies also employ ultrasonic testing for cotter pin defect detection. This method can detect internal cracks or defects in the cotter pin and determine the presence of internal abnormalities through reflected signals. While this method is effective for detecting internal defects, its effectiveness in detecting surface defects, wear, and other cosmetic issues in cotter pins is limited, and the testing cost is high.

[0027] Existing technologies also include X-ray inspection, which can identify the internal structural integrity of cotter pins and determine the presence of defects such as cracks through imaging analysis. This method is effective for inspecting metallic materials, but it is complex to operate and the equipment is expensive, making it unsuitable for on-site inspection.

[0028] Based on this, embodiments of this application provide an intelligent detection method and device for the status of tower crane cotter pins, so as to improve detection safety, improve detection efficiency, improve the accuracy of detection results, and reduce detection costs.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating an intelligent detection method for the cotter pin status of a tower crane, provided as an embodiment of this application. Figure 1 As shown in the embodiments of this application, the intelligent detection method includes: S101. Input the image of the cotter pin of the tower crane to be inspected, taken by the drone, into the pre-trained target detection model.

[0030] S102. The tower crane cotter pin image is processed by multi-level feature extraction through the backbone network in the target detection model to determine multiple target feature maps at different levels.

[0031] S103. The target detection model uses the neck network of the adaptive feature pyramid neural network structure to fuse multiple target feature layers and determine multiple target fusion feature maps.

[0032] S104. Analyze the fused feature maps of multiple targets through the head network in the target detection model to generate the status detection results of the tower crane cotter pin.

[0033] For step S101, in this step, the image of the cotter pin of the tower crane to be tested is acquired by the UAV according to the set shooting strategy, and the acquired image of the cotter pin of the tower crane to be tested is used as the model input and input into the pre-trained target detection model.

[0034] Here, the target detection model is determined by training the improved YOLOv8; the target detection model is used to analyze the input image and determine the cotter pin state.

[0035] The original YOLOv8 neural network consists of three parts: backbone, neck, and head.

[0036] The improved YOLOv8 is a neural network formed by modifying the original backbone, neck, and head.

[0037] For example, the improvements adopted in this application include: (1) Based on the original YOLOv8, the C2f module in its feature extraction network is improved and named C2f-DSF (Target C2F Module). The Target C2F Module mainly consists of three parts: Dilated Reparam Block, Squceze-and-Excitation Block (SE block), and Feed-Forward Network (FFN). The Dilated Reparam Block captures features from different scales through convolutional layers with different kernel sizes and dilation rates to enhance the feature representation capability of the model. Through reparameterization, multiple parallel convolutional layers are used during training, and then merged into one convolutional layer during inference to improve inference efficiency. The efficient SE Block is used to communicate between channels and enhance feature representation through nonlinear mapping. The Feed-Forward Network (FFN) enhances the nonlinear expression capability of the model and performs regularization through GELU activation and GRN units, enabling the model to learn more complex data representations, thereby improving the model's depth and representation capability, and thus enhancing its overall performance.

[0038] For example, dilated convolutional blocks use convolutional layers with different kernel sizes and dilation rates: small kernels help capture detailed features, while large kernels capture sparsely distributed features. The bottom layer uses a 7×7 kernel with kernel sizes of 5, 3, and 3, and dilation rates of 1, 2, and 3 to ensure a certain level of limitation while still capturing details. The higher layers use a 13×13 kernel with kernel sizes of 5, 7, 3, 3, and 3, and dilation rates of 1, 2, 3, 4, and 5 to capture global details.

[0039] (2) An Adaptive Feather Pyramid Network (AFPN) is used to detect targets of different sizes. In the bottom-up feature extraction process of the neck network, AFPN initially initiates the fusion process by combining two low-level features with different resolutions. Later, high-level features are gradually incorporated into the fusion process, and finally the top-level features of the neck network are fused. This fusion method can dynamically generate feature pyramids of different scales, ensuring that targets of different scales are effectively detected, thereby avoiding large semantic gaps between non-adjacent levels.

[0040] (3) Use SIoU loss function instead of CIoU loss function. SIoU loss function addresses the problem of orientation mismatch between predicted box and ground truth box. It adds angle loss, distance loss and shape loss to the IoU loss function, which solves the problems of false detection, multiple detection and missed detection to a certain extent. It can effectively reduce the total degree of freedom of loss and improve the convergence speed and detection effect of the model.

[0041] In this way, the target detection model trained based on the improved YOLOv8 can more accurately determine the state of the tower crane cotter pin.

[0042] For an example, please refer to Figure 2 , Figure 3 , Figure 2 A schematic diagram of an improved YOLOv8 neural network provided in this application; Figure 3 This is a schematic diagram of the structure of a target C2F module provided in this application. Figure 2 As shown, this paper presents the structural design of an improved YOLOv8 neural network obtained by improving the original YOLOv8. It consists of three parts: Backbone, Adaptive Feature Pyramid Neural Network (AFPN) (this scheme uses AFPN to replace Neck), and Head. Backbone: Includes CBS and C2f-DSF modules for feature extraction.

[0043] CBS (Complex Convolutional Module) typically represents a combination of convolution, batch normalization (BN), and Swish activation layers, enhancing the network's expressive power.

[0044] C2f-DSF (Target C2F Module) is a module that incorporates a large convolutional kernel DSF into the original C2F module. For details, please refer to [link / reference needed]. Figure 3 .

[0045] The SPPF (Spatial Pyramid Pooling-Fast) module is located in the last layer of the backbone and is used to further extract multi-scale features.

[0046] ASFF_3 is used for adaptive fusion between multiple resolution feature layers, which improves the representational power of features by adjusting the weights of features in each layer.

[0047] AFPN: Through cross-layer connections, it achieves adaptive feature enhancement, which facilitates subsequent object detection.

[0048] Head: Used for actual target detection. The final target classification and localization output is achieved by connecting to the detection head layer output from the ASFF_3 module.

[0049] This architecture extracts multi-scale features through the backbone, performs multi-scale feature fusion in the AFPN, and finally outputs the target detection in the head. The overall network design enhances detection accuracy and real-time performance, making it suitable for efficient target detection tasks.

[0050] It should be noted that in convolutional neural networks (CNNs), the size of the convolution kernel (also known as a filter or window) is usually represented by the length and width of the kernel.

[0051] Generally speaking: Small convolution kernels: Convolution kernels with a size of 1×1, 3×3, or 5×5 are considered "regular" or "small" convolution kernels.

[0052] Large convolutional kernel: usually refers to a convolutional kernel larger than 5×5, such as 7×7, 9×9, 11×11 or larger.

[0053] like Figure 3 The diagram shows the structural design of an improved C2F module (target C2F module). The following is a detailed description of the structure: Convolutional layer (Conv): The network input passes through two ordinary convolutional layers and can be used for feature extraction or dimensionality reduction.

[0054] Dilated Reparam Block: This module uses dilated convolution and reparameterization to capture a wider range of features by expanding the receptive field of the convolution kernel, while optimizing the number of parameters in the network structure.

[0055] Depthwise Convolution (DW Conv): Marked as "DW conv" next to the Dilated Reparam Block, it refers to the depthwise convolution operation, which is usually used to reduce the number of parameters while enhancing feature extraction capabilities.

[0056] SE Block: This module is the Squeeze-and-Excitation block. It compresses features through global average pooling (Avg-pool) and then readjusts the channel weights through ReLU and Sigmoid activation layers, thereby improving the model's feature representation capability.

[0057] FFN (Feedforward Neural Network): Here, FFN further enhances features through the GELU activation function and batch normalization (BN), and controls the weight allocation of different features.

[0058] Drop Path: Used to regularize the model and prevent overfitting. It randomly discards some information in certain paths to improve the model's generalization ability.

[0059] To improve the accuracy of tower crane cotter pin condition detection results, the quality of the cotter pin images of the tower crane under inspection is also required. The higher the quality of the cotter pin images, the higher the accuracy of the detection results.

[0060] This solution uses drones to acquire images of the cotter pins on the tower crane to be inspected. Specifically, the drone flies along a pre-defined path, at a set speed and shooting interval, capturing images of the cotter pins. During flight, the drone ensures complete coverage of the cotter pins based on the set flight speed and shooting intervals.

[0061] Here, the path defined includes both the planned path and the actual path. For an example, please refer to [link / reference]. Figure 4 , 5 6, 7 Figure 4 This application provides an example of a tower inspection planning path. Figure 5 An example of a crane boom inspection planning path provided in this application; Figure 6 This is one example of a practical inspection path for tower cranes provided in this application; Figure 7 This application provides a second example of a practical inspection path for tower cranes; such as... Figures 4-7 As shown, the ox-plowing-style inspection path can quickly and comprehensively acquire images of complex structural surfaces such as the tower body and jib, enabling rapid and accurate acquisition of images of the cotter pins of the tower crane under inspection in practical applications. Here, d represents the pin diameter, and 1-8 indicate the camera's shooting direction. For example, it can be set to take a group of photos of a standard tower crane section every 45 degrees, achieving comprehensive, blind-spot-free imaging.

[0062] For step S102, the backbone network of the target detection model is used to perform multi-level feature extraction on the input image, extracting features at different levels in the image to obtain multiple target feature maps.

[0063] For example, such as Figure 2 As shown, this application can extract three target feature maps.

[0064] Here, the target C2F module in the backbone network is a C2F module obtained by incorporating a large convolutional kernel into the original C2F module. The target C2F module includes dilated convolutional blocks, SE modules, and feedforward neural networks.

[0065] Regarding step S103, in this step, the neck network adopts an adaptive feature pyramid neural network structure. The neck neural network in the target detection model fuses multiple target feature layers to determine multiple target fusion feature maps, so as to comprehensively analyze features at various scales.

[0066] For examples, please continue to refer to them. Figure 2 This application can obtain three target fusion feature maps.

[0067] Regarding step S104, the status detection result of the tower crane cotter pin may include one of three detection results: cotter pin normal, cotter pin detached, and cotter pin loose. Among them, when the detection result is cotter pin loose, a looseness value will also be output.

[0068] It should be noted that for a normal cotter pin, the head, pin shaft, and tail features are all present. The distance between the head and the pin shaft is slightly larger than the pin shaft diameter, but remains within the normal range (e.g., 1.2 times the diameter d). When the distance between the head and the pin shaft exceeds the normal range of the pin shaft diameter, the cotter pin is loose. When the tail feature also disappears, it indicates severe looseness. When both the head and tail features disappear, and only the pin shaft remains, the cotter pin is missing.

[0069] For an example, please refer to Figure 8 , Figure 8 This application provides a schematic diagram of the cotter pin status of a tower crane, wherein... Figure 8 The image in the middle left shows a normal cotter pin. Figure 8 The image in the middle shows a loose cotter pin. Figure 8 The image on the right shows a missing cotter pin.

[0070] In one embodiment provided in this application, after analyzing the fused feature maps of multiple targets through the head network in the target detection model to generate the status detection result of the tower crane cotter pin, the intelligent detection method further includes: when the status detection result of the tower crane cotter pin shows that the cotter pin is missing or the degree of looseness exceeds a preset safety threshold; issuing an early warning and generating a detection report.

[0071] Here, if the inspection results show that the cotter pin is missing or loose beyond the safety threshold, an immediate warning will be issued. This warning can be used to remind operators or maintenance personnel to pay attention to cotter pin safety issues. In addition, a detailed inspection report will be generated, recording the inspection results and potential safety hazards for subsequent maintenance and inspection.

[0072] Therefore, through automated status detection and early warning mechanisms, potential security risks can be effectively identified, timely alerts can be provided, and detection data support can be offered.

[0073] The above method is based on the object detection model, so how to construct the object detection model is also a key technical solution of this application.

[0074] In one embodiment provided in this application, the target detection model is constructed through the following steps: S201. Based on the inspection images of the tower crane cotter pins, construct at least one batch of training sample sets.

[0075] S202. For each batch of training sample sets, the inspection images in the training sample set are sequentially input into the improved YOLOv8. The composite convolution module and the target C2F module in the backbone network are used to perform layer-by-layer feature extraction processing to obtain multiple sample feature maps of different levels for each inspection image.

[0076] S203. For each inspection image, the neck network of the improved YOLOv8 using the adaptive feature pyramid neural network structure performs bottom-up feature fusion processing on multiple sample feature maps of the inspection image to generate multiple sample fused feature maps of the inspection image.

[0077] S204. The head network in the improved YOLOv8 is used to analyze the fusion feature map of multiple samples of the inspection image to determine the prediction result of the cotter pin state of the inspection image.

[0078] S205. Based on the predicted cotter pin status and the true status label of each inspection image in the training sample set, determine the total loss function value of the improved YOLOv8.

[0079] S206. The improved YOLOv8 is trained based on the total loss function value until the total loss function of the improved YOLOv8 converges, thus obtaining the target detection model.

[0080] For step S201, multiple inspection images of tower crane cotter pins are acquired, and at least one batch of training sample sets is created.

[0081] Each inspection image in the training sample set has a real-state label; for example, the status of the cotter pin is marked (such as normal, missing, or loose (or degree of looseness)).

[0082] In step S202, each batch of training images is sequentially input into the improved YOLOv8 model. The backbone network of this model includes a composite convolutional module and a target C2F module. The composite convolutional module and the target C2F module extract features from the images layer by layer to capture the features at different levels of the images, resulting in multiple sample feature maps for each inspection image.

[0083] For example, for each inspection image, three sample feature maps of that inspection image can be obtained.

[0084] In one embodiment provided in this application, the target C2F module, when performing feature extraction processing, includes: S2021. By working in parallel with multiple dilated convolutional blocks, multi-scale feature extraction processing is performed on the initial feature map to obtain the first feature map.

[0085] S2022. The first feature map is enhanced by nonlinear mapping using the SE module to generate a second feature map.

[0086] S2023. The second feature map is subjected to nonlinear transformation and regularization through a feedforward neural network to obtain a sample feature map that is input to the adaptive feature pyramid neural network, or a third feature map that is input to the next neural network layer in the backbone neural network.

[0087] In step S2021, the initial input feature map is processed in parallel using multiple dilated convolutional blocks to obtain feature information at different scales. Dilated convolution can expand the receptive field without increasing computational cost, enabling the model to capture more contextual information and helping to identify objects with inconsistent sizes or shapes.

[0088] It should be noted that, through the reparameterization technique, multiple parallel convolutional layers are used during training, while they are merged into a single convolutional layer during inference.

[0089] In step S2022, the first feature map is input into the SE module. The SE (Squeeze-and-Excitation) module communicates between channels, enhances the feature representation through nonlinear mapping, calculates the importance coefficient of each channel, enhances important feature channels, and suppresses unimportant channels. The resulting second feature map, after the channel weights are redistributed, has a stronger feature representation capability.

[0090] In step S2023, the second feature map is input into the feedforward neural network for nonlinear transformation, and regularization is applied to improve the model's generalization ability. The feature map generated in this process (i.e., the third feature map) can be used as input to the adaptive feature pyramid network (AFPN structure in Neck) or to the next layer of the neural network in the backbone network.

[0091] In step S203, multi-level feature fusion is performed using the Adaptive Feature Pyramid Network (AFPN) in YOLOv8. The bottom-up path of AFPN fuses features from various levels into a comprehensive feature map, which is used to represent the overall information of the image.

[0092] For example, the fusion process is initially initiated by combining two low-level features with different resolutions; later, high-level features are gradually incorporated into the fusion process, and finally the top-level features of the neck network are fused.

[0093] For step S204, the improved YOLOv8 head network analyzes the fused feature map and predicts the state of the cotter pin in the image based on the knowledge learned by the model.

[0094] For step S205, the prediction results of the cotter pin state of each training image by the improved YOLOv8 are compared with the true state label, the prediction error is calculated, and the total loss function value is obtained.

[0095] Here, the SIoU loss function is used instead of the CIoU loss function.

[0096] In one embodiment provided in this application, the step of determining the improved YOLOv8 total loss function value based on the cotter pin state prediction result and the true state label of each inspection image in the training sample set includes: for each inspection image, determining the angle loss value, distance loss value, and shape loss value of the inspection image based on the cotter pin state prediction result and the true state label of the inspection image; determining the prediction loss function value of the inspection image based on the angle loss value, distance loss value, and shape loss value of the inspection image; and determining the improved YOLOv8 total loss function value based on the prediction loss function values ​​of all inspection images.

[0097] The specific formula and process for calculating the total loss function value are shown below: Angular loss The calculation is shown in equation (1): (1) Distance loss The calculation is shown in equation (2): (2) Shape loss The calculation is shown in equation (3): (3) The IoU loss is calculated as shown in equation (4): (4) The final SIoU loss function formula is shown in equation (5): (5) In the formula, The height difference between the center points of the ground truth bounding box and the predicted bounding box. This is the straight-line distance between the center points of the ground truth bounding box and the predicted bounding box. The time-priority distance value is γ = 2 - Λ; , cw and ch represent the width and height of the true bounding box and the minimum bounding rectangle of the bounding box, respectively. , θ is the attention value controlling the shape loss, and the setting range of θ is [2,6]; A is the ground truth bounding box, and B is the predicted bounding box. gt represents the target bounding box. and This represents the normalized squared distance in the x and y directions. and This represents the coordinates of the center point of the actual bounding box. and These are the coordinates of the center point of the prediction box. and These represent the width and height of the actual bounding box, respectively. and It is a normalized difference measure of width and height. It is calculated by dividing the absolute difference between the width and height of the predicted bounding box and the ground truth bounding box by the maximum of these two values ​​(max(w, ...). Normalization is performed. This ensures that the difference values ​​are within the range of [0, 1], reducing the impact of scale differences.

[0098] In step S206, based on the total loss function value, the parameters of the improved YOLOv8 neural network are adjusted until the loss function converges, indicating that the model has reached a high level of accuracy, and training is stopped, ultimately obtaining the object detection model.

[0099] The improved YOLOv8 neural network is updated during training as training samples are added in batches.

[0100] In one embodiment provided in this application, training the improved YOLOv8 based on the total loss function value includes: determining whether the total loss function corresponding to the improved YOLOv8 has converged based on the total loss function value; if it has converged, obtaining the object detection model; if it has not converged, updating the network parameters of the improved YOLOv8, and training the improved YOLOv8 using the training sample set until the total loss function corresponding to the improved YOLOv8 converges.

[0101] Here, the formula used to update the network parameters of the improved YOLOv8 is:

[0102]

[0103] in, For learning rate, That is, the score center of category j, where t is the number of iterations. Indicates the score center of the current iteration (t-th iteration). Let i represent the score center of the next iteration (t+1th iteration), where i is the initial value of the iteration and M is the final value of the iteration. Let i be the score center of class i, and IF be the conditional expression, where IF(condition) = 1 if i is true and 0 otherwise. Update the score center when it is the same as j.

[0104] In this way, a relatively accurate target detection model can be obtained through the above methods.

[0105] In addition, for an explanation of the beneficial effects of this solution, please refer to [link / reference needed]. Figure 9 , Figure 10 , Figure 9 This application provides a performance test result based on the original YOLOv8 neural network in an embodiment of the present application. Figure 10 This application provides an embodiment of a performance test based on an improved YOLOv8 neural network. Figure 9 and Figure 10 It can be seen that this solution can effectively improve the accuracy of cotter pin inspection results.

[0106] This method of using drones to inspect tower crane cotter pins reduces the safety risks of manual climbing. Drone photography is convenient and quick, significantly reducing inspection time and improving efficiency. Employing deep learning algorithms for intelligent image recognition reduces the probability of human error and improves inspection accuracy. The automated inspection process enables real-time monitoring of tower crane cotter pins, facilitating the timely detection and handling of potential safety hazards.

[0107] Based on the same inventive concept, this application also provides an intelligent detection device corresponding to the intelligent detection method. Since the principle of the device in this application is similar to that of the intelligent detection method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0108] Please see Figure 11 , Figure 12 , Figure 11 This is one of the structural schematic diagrams of an intelligent detection device for the cotter pin status of a tower crane provided in an embodiment of this application. Figure 12 This is a second schematic diagram of an intelligent detection device for the cotter pin status of a tower crane, provided as an embodiment of this application. Figure 11 As shown, the intelligent detection device 1100 includes: The input module 1110 is used to input the image of the cotter pin of the tower crane to be inspected, taken by a drone, into a pre-trained target detection model; wherein the target detection model is determined by training an improved YOLOv8. The extraction module 1120 is used to perform multi-level feature extraction processing on the tower crane cotter pin image through the backbone network in the target detection model to determine multiple target feature maps at different levels; wherein, the target C2F module in the backbone network is a C2F module obtained by incorporating a large convolutional kernel into the original C2F module, and the target C2F module includes a dilated convolutional block, an SE module, and a feedforward neural network; The fusion module 1130 is used to perform fusion processing on multiple target feature layers through the neck network of the adaptive feature pyramid neural network structure in the target detection model to determine multiple target fusion feature maps; The analysis module 1140 is used to analyze the fused feature maps of multiple targets through the head network in the target detection model to generate the status detection results of the tower crane cotter pin.

[0109] like Figure 12 As shown, optionally, the intelligent detection device 1100 further includes a construction module 1150, which is used for: Based on the inspection images of tower crane cotter pins, at least one batch of training sample sets is constructed; wherein each inspection image in the training sample set has a real state label. For each batch of training sample sets, the inspection images in the training sample set are sequentially input into the improved YOLOv8. The composite convolution module and the target C2F module in the backbone network are used to perform layer-by-layer feature extraction to obtain multiple sample feature maps at different levels for each inspection image. For each inspection image, the neck network of the improved YOLOv8, which uses an adaptive feature pyramid neural network structure, performs bottom-up feature fusion processing on multiple sample feature maps of the inspection image to generate multiple sample fused feature maps of the inspection image. The improved YOLOv8 head network was used to analyze the fused feature maps of multiple samples of the inspection image to determine the predicted cotter pin status of the inspection image. Based on the predicted cotter pin status and the true status label of each inspection image in the training sample set, the total loss function value of the improved YOLOv8 is determined. The improved YOLOv8 is then trained based on the total loss function value until the total loss function of the improved YOLOv8 converges, thus obtaining the object detection model.

[0110] Optionally, when determining the improved total loss function value of YOLOv8 based on the cotter pin state prediction result and the true state label of each inspection image in the training sample set, the construction module 1150 is used to: For each inspection image, the angle loss value, distance loss value, and shape loss value of the inspection image are determined based on the predicted cotter pin status and the actual status label of the inspection image. Based on the angle loss value, distance loss value, and shape loss value of the inspection image, determine the prediction loss function value of the inspection image; Based on the predicted loss function values ​​of all inspected images, determine the total loss function value of the improved YOLOv8.

[0111] Optionally, when the construction module 1150 is used to train the improved YOLOv8 based on the total loss function value, the construction module 1150 is used to: Based on the total loss function value, determine whether the total loss function corresponding to the improved YOLOv8 has converged; If convergence is achieved, the target detection model is obtained; If convergence is not achieved, update the network parameters of the improved YOLOv8 and train the improved YOLOv8 using the training sample set until the total loss function corresponding to the improved YOLOv8 converges.

[0112] Optionally, the formula used to update the network parameters of the improved YOLOv8 is:

[0113]

[0114] in, For learning rate, That is, the score center of category j, where t is the number of iterations. Indicates the score center of the current iteration (t-th iteration). Let represent the score center of the next iteration (t+1th iteration), IF be the conditional expression, i be the initial value of the iteration, and M be the final value of the iteration. Let i be the score center of class i.

[0115] Optionally, when the construction module 1150 is used for feature extraction processing via the target C2F module, the construction module 1150 is used to: By working in parallel with multiple dilated convolutional blocks, multi-scale feature extraction is performed on the initial feature map to obtain the first feature map. The first feature map is enhanced by nonlinear mapping using the SE module to generate a second feature map. The second feature map is subjected to nonlinear transformation and regularization by a feedforward neural network to obtain a sample feature map that is input to the adaptive feature pyramid neural network, or a third feature map that is input to the next neural network layer in the backbone neural network.

[0116] Optionally, the intelligent detection device 1100 further includes an early warning module 1160. The early warning module 1160: after analyzing the fusion feature maps of multiple targets through the head network in the target detection model and generating the status detection result of the tower crane cotter pin, when the status detection result of the tower crane cotter pin shows that the cotter pin is missing or the degree of looseness exceeds the preset safety threshold; Issue early warnings and generate test reports.

[0117] Please see Figure 13 , Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 13 As shown, the electronic device 1300 includes a processor 1310, a memory 1320, and a bus 1330.

[0118] The memory 1320 stores machine-readable instructions executable by the processor 1310. When the electronic device 1300 is running, the processor 1310 communicates with the memory 1320 via the bus 1330. When the machine-readable instructions are executed by the processor 1310, they can perform the operations described above. Figures 1 to 10 The steps in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0119] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figures 1 to 10 The steps in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0120] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0121] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0122] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0123] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0124] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0125] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An intelligent detection method for the state of a tower crane cotter pin, characterized in that: The intelligent detection method comprises: Input the image of the tower crane cotter pin to be detected taken by the drone into a pre-trained target detection model; wherein the target detection model is determined by training the improved YOLOv8; The main network in the target detection model performs multi-level feature extraction processing on the tower crane cotter pin image to determine target feature maps of multiple different levels; wherein the target C2F module in the main network is a C2F module obtained by integrating a large convolution kernel into the original C2F module, and the target C2F module includes an expansion convolution block, an SE module and a feedforward neural network; A plurality of target feature layers are fused by using a neck network of an adaptive feature pyramid neural network structure in the target detection model to determine a plurality of target fusion feature maps; The head network in the target detection model is used to analyze multiple target fusion feature maps to generate a state detection result of the tower crane cotter pin.

2. The intelligent detection method according to claim 1, characterized in that: The target detection model is constructed by following the steps below: According to the inspection images of the tower crane cotter pins, at least one batch of training sample sets is constructed; wherein each inspection image in the training sample set has a real state label; For each batch of training sample sets, the inspection images in the training sample sets are input into the improved YOLOv8 in turn, and the composite convolution module and the target C2F module in the backbone network are used to perform layer-by-layer feature extraction processing to obtain sample feature maps of multiple different levels for each inspection image. For each inspection image, the neck network using the adaptive feature pyramid neural network structure in the improved YOLOv8 performs bottom-up feature fusion processing on multiple sample feature maps of the inspection image to generate multiple sample fusion feature maps of the inspection image; The multiple sample fusion feature maps of the inspection image are analyzed by the improved head network in YOLOv8 to determine the prediction result of the cotter pin state of the inspection image; Determine the total loss function value of the improved YOLOv8 based on the cotter pin state prediction result and the true state label of each inspection image in the training sample set; The improved YOLOv8 is trained based on the total loss function value until the total loss function of the improved YOLOv8 converges to obtain a target detection model.

3. The intelligent detection method according to claim 2, characterized in that: The method of determining the total loss function value of the improved YOLOv8 according to the cotter pin state prediction result and the true state label of each inspection image in the training sample set includes: For each inspection image, determine the angle loss value, distance loss value and shape loss value of the inspection image according to the cotter pin state prediction result and the true state label of the inspection image; Determine the prediction loss function value of the inspection image according to the angle loss value, distance loss value and shape loss value of the inspection image; According to the predicted loss function values ​​of all inspection images, the total loss function value of the improved YOLOv8 is determined.

4. The intelligent detection method according to claim 2, characterized in that: The step of training the improved YOLOv8 based on the total loss function value includes: Based on the total loss function value, determine whether the total loss function corresponding to the improved YOLOv8 converges; If converged, the target detection model is obtained; If it does not converge, update the network parameters of the improved YOLOv8, and use the training sample set to train the improved YOLOv8 until the total loss function corresponding to the improved YOLOv8 converges.

5. The intelligent detection method according to claim 4, characterized in that: The formula used to update the network parameters of the improved YOLOv8 is: in, is the learning rate, That is, the score center of category j, t is the number of iterations, represents the score center of the current iteration (tth time), Indicates the score center of the next iteration (t+1th), IF is the conditional expression, i is the initial value of the iteration, and M is the final value of the iteration. is the score center of class i.

6. The intelligent detection method according to claim 2, characterized in that: When the target C2F module performs feature extraction processing, it includes: By using multiple dilated convolution blocks to work in parallel, multi-scale feature extraction is performed on the initial feature map to obtain a first feature map; Performing nonlinear mapping enhancement processing on the first feature map through the SE module to generate a second feature map; The second feature map is subjected to nonlinear transformation and regularization processing through a feedforward neural network to obtain a sample feature map input to an adaptive feature pyramid neural network, or a third feature map input to a next neural network layer in a backbone neural network.

7. The intelligent detection method according to claim 1, characterized in that: After analyzing multiple target fusion feature maps through the head network in the target detection model to generate a state detection result of the tower crane cotter pin, the intelligent detection method further includes: When the state detection result of the tower crane cotter pin shows that the cotter pin is missing or the degree of looseness exceeds a preset safety threshold; Provide early warning and generate test reports.

8. An intelligent detection device for the cotter pin state of a tower crane, characterized in that: The intelligent detection device comprises: An input module is used to input the image of the tower crane cotter pin to be detected taken by the drone into a pre-trained target detection model; wherein the target detection model is determined by training the improved YOLOv8; An extraction module is used to perform multi-level feature extraction processing on the tower crane cotter pin image through the backbone network in the target detection model to determine target feature maps of multiple different levels; wherein the target C2F module in the backbone network is a C2F module obtained by integrating a large convolution kernel into the original C2F module, and the target C2F module includes an expansion convolution block, an SE module and a feedforward neural network; A fusion module is used to perform fusion processing on multiple target feature layers through a neck network using an adaptive feature pyramid neural network structure in the target detection model to determine multiple target fusion feature maps; The analysis module is used to analyze multiple target fusion feature maps through the head network in the target detection model to generate a state detection result of the tower crane cotter pin.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the intelligent detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the intelligent detection method according to any one of claims 1 to 7 are executed.

Images