Hoisting equipment wheel rail fault detection method and device, electronic equipment and storage medium

Through the improved YOLOv8 model, combined with Swin Transformer and asymptotic feature pyramid network, the problem of poor handling of small and medium-sized targets and complex scenarios of wheel and rail fault detection of lifting equipment is solved, improving the accuracy and robustness of fault diagnosis, and avoiding features Information loss enables more robust fault detection.

CN119091110APending Publication Date: 2024-12-06BEIJING MATERIALS HANDLING TECH INST CO LTD

Patent Information

Application Number
CN202410519937.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-04-28
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing wheel and rail fault detection methods for lifting equipment are not robust enough to handle small targets, low-quality images or complex scenes, and the semantic information of deep features is easily lost or degraded during the detection process, affecting the accuracy and robustness of fault diagnosis.

Method used

Adopting the improved YOLOv8 model, combining the Swin Transformer module and the asymptotic feature pyramid network, replace the backbone network and Neck network in the original model, and train the optimization model through transfer learning to improve the global information extraction ability and avoid feature information during transmission and interaction. loss or degradation.

Benefits of technology

It effectively improves the accuracy and robustness of wheel and rail fault diagnosis of lifting equipment, improves the processing ability of small targets and complex scenarios, avoids the loss of deep feature information, and enhances the robustness of fault detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091110A_ABST
    Figure CN119091110A_ABST
Patent Text Reader

Abstract

The invention provides a hoisting equipment wheel rail fault detection method and device, electronic equipment and a storage medium. The method comprises the steps that a to-be-detected image of a hoisting equipment wheel rail is acquired; inputting the to-be-detected image into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model comprises a Swin Transformer module and an asymptotic feature pyramid network, and is obtained by performing training optimization according to a railway wheel track fault data set and a hoisting equipment wheel track fault data set. According to the method, an improved YOLOv8 model is obtained through improvement by combining a Swin Transformer module and an asymptotic feature pyramid network, and fault detection is performed on the to-be-detected image of the hoisting equipment wheel rail by using the improved YOLOv8 model, so that the global information extraction capability is effectively improved, the feature information is prevented from being lost or degraded in the transmission and interaction process, and the fault detection efficiency is improved. And the accuracy and the robustness of the wheel track fault diagnosis of the hoisting equipment are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method, device, electronic equipment and storage medium for detecting wheel-rail faults of lifting equipment. Background Art

[0002] Lifting equipment plays a vital role in modern society, providing necessary support and guarantee for human production activities. Lifting equipment usually operates in working environments such as high temperature, low temperature, high humidity, and corrosive gases. Long-term operation in such an environment can easily cause the equipment to be damaged, corroded, and worn, which in turn causes failures. Therefore, it is of great significance to diagnose the faults of lifting equipment.

[0003] Machine vision has important advantages in fault diagnosis. It uses image and video data, with the help of deep learning and image processing technology, to achieve fast and accurate equipment fault detection and diagnosis. Object detection algorithms based on deep learning occupy an important position in the field of computer vision, and these algorithms can generally be divided into two categories according to their design frameworks: two-stage frameworks and single-stage frameworks.

[0004] The core idea of ​​the two-stage framework is to decompose the object detection task into two key stages: first, coarse classification and region proposal, and then fine position adjustment. Therefore, the two-stage framework shows unique advantages in dealing with the problem of imbalanced fault categories of special mechanical equipment.

[0005] The single-stage framework uses an end-to-end approach to directly predict the category and location of the target through a single neural network model. Compared with the two-stage framework, the single-stage framework is usually faster and more straightforward.

[0006] The YOLO model is a representative single-stage target detection algorithm. However, the YOLO model may have problems of missed detection or inaccurate positioning when dealing with small objects or target-dense areas in scenes.

[0007] The recently proposed YOLOv8 model uses a smaller model structure, more convolution kernels, faster convolution operations, and a simpler model architecture, making the inference speed faster than previous versions of the YOLO model.

[0008] However, the YOLOv8 model is not robust enough for small objects, low-quality images or complex scenes, and it is difficult to directly apply it to fault detection of lifting equipment.

[0009] In addition, the YOLOv8 model introduces a path aggregation network feature pyramid structure, which aggregates feature maps of different levels through horizontal and vertical paths, thereby enhancing the model's ability to detect objects of different scales. By better utilizing the information within the network and establishing connections between feature maps of different levels, information flow is promoted and feature expression at the semantic level is strengthened. However, in this process of propagation and interaction, semantic information from deep features may be lost or degraded.

[0010] Therefore, how to solve the problem that the existing lifting equipment wheel-rail fault detection methods are not robust enough for handling small targets, low-quality images or complex scenes of lifting equipment, and the semantic information of deep features is easily lost or degraded during the detection process, is an important issue that needs to be urgently solved in the field of machine vision technology. Summary of the invention

[0011] The present invention provides a method, device, electronic device and storage medium for detecting wheel-rail faults of lifting equipment, so as to overcome the defects that the existing methods for detecting wheel-rail faults of lifting equipment are not robust enough in processing small targets, low-quality images or complex scenes of lifting equipment, and the semantic information of deep features is easily lost or degraded during the detection process. The method can effectively improve the global information extraction capability and avoid the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of wheel-rail fault diagnosis of lifting equipment.

[0012] On the one hand, the present invention provides a method for detecting wheel-rail faults of lifting equipment, comprising: obtaining an image to be detected of a lifting equipment wheel-rail; inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, and is obtained by training and optimizing according to a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0013] Furthermore, the lifting equipment wheel-rail fault detection method also includes: based on the Swim Transformer module and the asymptotic feature pyramid network, improving the original YOLOv8 model to obtain the improved YOLOv8 model.

[0014] Furthermore, the original YOLOv8 model includes a backbone network, a Neck network and a prediction network, and the backbone network includes a convolution module, a C2f module and a fast spatial pyramid pooling module; accordingly, the original YOLOv8 model is improved based on the SwimTransformer module and the progressive feature pyramid network, including: replacing the C2f module in the backbone network with the Swim Transformer module; and replacing the Neck network with the asymptotic feature pyramid network.

[0015] Furthermore, the image to be detected is input into a pre-trained improved YOLOv8 model to obtain a fault detection result, including: based on the improved backbone network, extracting shallow feature vectors and deep feature vectors in the image to be detected; based on the asymptotic feature pyramid network, performing a downsampling operation on the shallow feature vector to obtain deep features, and performing an upsampling operation on the deep feature vector to obtain shallow features; using an adaptive spatial feature fusion method to fuse the deep features and the shallow features layer by layer to obtain multi-scale features; based on the prediction network, predicting the multi-scale features to obtain a fault detection result corresponding to the image to be detected; wherein the fault detection result includes a fault location and a fault category.

[0016] Furthermore, the improved YOLOv8 model is trained by transfer learning, specifically including: constructing the railway wheel-rail fault data set and the lifting equipment wheel-rail fault data set; using the railway wheel-rail fault data set to pre-train the improved YOLOv8 model to obtain a pre-trained YOLOv8 model with optimal configuration parameters; based on the pre-trained YOLOv8 model, using the lifting equipment wheel-rail fault data set to fine-tune the model parameters of the pre-trained YOLOv8 model to obtain a pre-trained improved YOLOv8 model.

[0017] Furthermore, the step of constructing a railway wheel-rail fault data set includes: collecting a preset number of railway wheel-rail fault images; labeling the railway wheel-rail fault images according to fault categories, and performing enhancement operations on the railway wheel-rail fault images to obtain the railway wheel-rail fault data set.

[0018] Furthermore, the enhancement operation includes one or more combinations of denoising, missing value filling, flipping, scaling, color gamut conversion and random cropping.

[0019] In a second aspect, the present invention also provides a lifting equipment wheel-rail fault detection device, comprising: an image acquisition module, used to acquire the image to be detected of the lifting equipment wheel-rail; a fault detection module, used to input the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model includes a SwinTransformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing according to a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0020] In a third aspect, the present invention further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for detecting wheel-rail faults of a lifting equipment as described in any one of the above is implemented.

[0021] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for detecting wheel-rail faults of a lifting equipment as described in any one of the above.

[0022] The present invention provides a method for detecting wheel-rail faults of lifting equipment, which obtains a to-be-detected image of the wheel-rail of the lifting equipment and inputs the to-be-detected image into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing according to a railway wheel-rail fault data set and a lifting equipment wheel-rail fault data set. The method improves the improved YOLOv8 model by combining the Swin Transformer module and the asymptotic feature pyramid network, and uses the improved YOLOv8 model to perform fault detection on the to-be-detected image of the wheel-rail of the lifting equipment, effectively improving the global information extraction capability, and avoiding the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of the wheel-rail fault diagnosis of the lifting equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0024] Figure 1 It is a flow chart of a method for detecting wheel-rail faults of a lifting equipment provided by an embodiment of the present invention;

[0025] Figure 2 It is a schematic diagram of the structure of the original YOLOv8 model provided by the prior art;

[0026] Figure 3 It is a structural diagram of the Swin Transformer module provided by the prior art;

[0027] Figure 4 It is a structural diagram of a path aggregation network feature pyramid network provided by the prior art;

[0028] Figure 5 is a schematic diagram of the structure of an asymptotic feature pyramid network provided by an embodiment of the present invention;

[0029] Figure 6 Schematic diagram of feature extraction of asymptotic feature pyramid network provided by an embodiment of the present invention;

[0030] Figure 7 Schematic diagram of the complete structure of the improved YOLOv8 model provided by an embodiment of the present invention;

[0031] Figure 8 is a schematic diagram of the detection results of the improved YOLOv8 model provided in an embodiment of the present invention;

[0032] Fig. 9 It is a structural schematic diagram of a lifting equipment wheel-rail fault detection device provided by the present invention;

[0033] Fig.10 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] It should be noted that the YOLO model is a representative single-stage target detection algorithm. The recently proposed YOLOv8 model uses a smaller model structure, more convolution kernels, faster convolution operations, and a simpler model architecture, making the inference speed faster than previous versions of the YOLO model.

[0036] However, the YOLOv8 model is not robust enough for handling small targets, low-quality images or complex scenes, and it is difficult to directly apply it to fault detection of special equipment such as lifting equipment.

[0037] In addition, the YOLOv8 model introduces a path aggregation network feature pyramid structure, which aggregates feature maps of different levels through horizontal and vertical paths, thereby enhancing the model's ability to detect objects of different scales. By better utilizing the information within the network and establishing connections between feature maps of different levels, information flow is promoted and feature expression at the semantic level is strengthened. However, in this process of propagation and interaction, semantic information from deep features may be lost or degraded.

[0038] In view of this, the present invention proposes a method for detecting wheel-rail faults of a lifting equipment, specifically, Figure 1 A schematic flow chart of a method for detecting wheel-rail faults of a lifting equipment provided in an embodiment of the present invention is shown.

[0039] like Figure 1 As shown, the method includes:

[0040] S110, acquiring an image of a wheel and rail of a lifting equipment to be inspected;

[0041] S120, inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result;

[0042] The improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing based on a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0043] It is understandable that in order to perform wheel-rail fault detection of a lifting equipment, first, in step S110, it is necessary to obtain an image of the wheel-rail of the lifting equipment to be detected, which can be obtained by an image acquisition device (such as a camera).

[0044] In a specific embodiment, the lifting device is particularly a crane.

[0045] Then, in step S120, the image to be detected is input into the pre-trained improved YOLOv8 model to obtain an output fault detection result, which includes the fault location and fault category of the wheel rail of the lifting equipment.

[0046] It should be emphasized that the improved YOLOv8 model applicable to this embodiment is obtained by improving the original YOLOv8 model.

[0047] Specifically, based on the original YOLOv8 model, the C2f module in the backbone network of the original YOLOv8 model is replaced with a Swim Transformer module to improve the ability to pay attention to the detected target, and a progressive feature pyramid network is used to strengthen the multi-scale feature fusion process to avoid the loss or degradation of feature information during transmission and interaction.

[0048] The process of improving the original YOLOv8 model will be described in detail in the following examples and will not be described in detail here.

[0049] It should also be noted that the improved YOLOV8 model is pre-trained. Specifically, it is obtained by training and optimizing based on the railway wheel-rail fault dataset and the lifting equipment wheel-rail fault dataset.

[0050] Among them, the railway wheel-rail fault dataset and the lifting equipment wheel-rail fault dataset are composed of railway wheel-rail fault images / lifting equipment wheel-rail fault images and their corresponding label annotations.

[0051] In this embodiment, the fault detection result is obtained by obtaining the image to be detected of the wheel and rail of the lifting equipment, and inputting the image to be detected into the pre-trained improved YOLOv8 model; wherein the improved YOLOv8 model includes a SwinTransformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing according to the railway wheel and rail fault data set and the lifting equipment wheel and rail fault data set. This method improves the improved YOLOv8 model by combining the Swin Transformer module and the asymptotic feature pyramid network, and uses the improved YOLOv8 model to perform fault detection on the image to be detected of the wheel and rail of the lifting equipment, which effectively improves the global information extraction capability and avoids the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of the wheel and rail fault diagnosis of the lifting equipment.

[0052] On the basis of the above embodiments, further, the process of improving the original YOLOv8 model to obtain the improved YOLOv8 model will be described in detail below.

[0053] It is understandable that the original YOLOv8 model uses deep learning technology to achieve fast and accurate detection and positioning of multiple targets in an image through a single neural network model. Since the backbone network refers to the simple and lightweight CSPDarkNet-53 network, the original YOLOv8 model is fast and can achieve real-time target detection, which is suitable for scenarios that require efficient processing of large numbers of images.

[0054] Figure 2 A schematic diagram of the structure of the original YOLOv8 model provided by the prior art is shown.

[0055] like Figure 2 As shown, the original YOLOv8 model includes a backbone network (Backbone), a Neck network (Neck), and a prediction network (Head).

[0056] Among them, the backbone network includes 5 convolution modules (ConvModule), 4 C2f modules (CSPLayer_2Conv) and 1 fast spatial pyramid pooling module (Spatial Pyramid Pooling–Fast, SPPF).

[0057] The Neck network consists of 3 convolution modules, 4 C2f modules, 2 upsampling modules (Upsample) and 4 connection layers (Concat).

[0058] The prediction network consists of 6 convolution modules and 6 2D convolution layers (Conv2d). Each 2D convolution layer outputs the current training loss, namely Bbox_loss (rectangular box loss) and Cls_loss (classification loss).

[0059] As can be seen from the above, the original YOLOv8 model contains multiple CNN (Convolutional Neural Network) layers, and the CNN layer is only good at capturing local information.

[0060] Compared with CNN, Transformer is not restricted in local interactions, which is mainly closely related to its self-attention mechanism that can be calculated in parallel. Due to its excellent sequence modeling ability and the characteristics of automatically focusing on fault features, Transformer has become a potential tool in fault diagnosis. Although the Transformer model has a certain degree of performance improvement in fault diagnosis compared with traditional deep learning methods, it still has a large number of model parameters and lacks the ability to model spatial information.

[0061] Fortunately, Swin Transformer replaces the standard multi-head self-attention module in the Transformer block with a shifted window-based module, while other layers remain unchanged. Swin Transformer brings higher efficiency by limiting the self-attention calculation to a local window, ensuring the flexibility of modeling at various scales.

[0062] Swin Transformer is a backbone network for visual tasks that uses a fully autonomous attention mechanism. Figure 3 The figure shows a structural diagram of a Swin Transformer module provided by the prior art.

[0063] like Figure 3 As shown in the figure, the structure of the Swin Transformer module mainly includes an image block segmentation layer (PatchParttion), a linear embedding layer (Linear Embedding), an image block merging layer (Patch Merging) and a Swin Transformer encoder (Swin Transformer Encoder).

[0064] The Swin Transformer module first divides the input image into non-overlapping small blocks through the image block segmentation layer, and the features of each block are set as the concatenation of the original pixel RGB values. Then, the features are projected to any dimension through the linear embedding layer, that is, the feature map becomes a query vector, a key vector, and a value vector.

[0065] Next, it passes through the Transformer encoder. The multi-head self-attention mechanism is the core of the Transformer encoder. It can assign different weights according to the importance of specific image areas, allowing the network to focus on key information and match the extracted features with the detected targets. After scaling and Softmax normalization, the similarity value obtained by the dot product of the key vector and the query vector is multiplied by the value vector to obtain the semantic weight. All semantic weights are weighted and summed to obtain the self-attention feature.

[0066] Finally, an informative feature map is obtained through multi-layer perceptron processing.

[0067] It should be noted that global computation will lead to quadratic complexity related to the number of blocks and is not suitable for dense prediction.

[0068] For effective modeling, the Swin Transformer module calculates self-attention within a local window according to Figure 3 It can be seen that the encoder includes a normalization layer, a window multi-head self-attention module (Windows Multi-Head Self-Attention, W-MSA), a shifted window multi-head self-attention module (Shifted Windows Multi-Head Self-Attention) and a multi-layer perception machine (Multi-layer perception).

[0069] The window multi-head self-attention module divides the input image into non-overlapping small windows and performs self-attention calculations in different small windows. Although the window multi-head self-attention module can reduce the complexity of self-attention calculations, it is difficult to extract the deep semantic information of the image due to the lack of information interaction between small windows. Therefore, a moving window multi-head self-attention module is also designed to solve this problem.

[0070] The moving window multi-head self-attention module connects adjacent but non-overlapping small windows of the previous layer to increase the receptive field and capture the deep semantic information of the image. Therefore, the Swin Transformer module needs to use an even number of encoders in succession to alternate between the window multi-head self-attention module and the moving window multi-head self-attention module.

[0071] The Swin Transformer module encodes geometric relationships in the self-attention computation by introducing an additional parameter bias term. The relative position bias encodes the relative spatial configuration of visual elements and shows criticality in various visual tasks, especially for dense recognition tasks such as object detection.

[0072] As the network goes deeper, in order to produce a hierarchical representation, it is necessary to merge layers to reduce the number of features, and then apply the Swin Transformer encoder to transform its features.

[0073] The original YOLOv8 model introduced a path aggregation network feature pyramid structure. Figure 4 The schematic diagram of the structure of the path aggregation network feature pyramid network provided by the prior art is shown. Figure 4 As shown in Figure 1, the deep features at the top of the feature pyramid need to propagate through multiple intermediate scales and interact with the features of these scales before being fused with the shallow features at the bottom. In this propagation and interaction process, the semantic information from the deep features may be lost or degraded.

[0074] Specifically, the path aggregation network transfers and fuses deep feature information through upsampling to obtain feature maps for prediction and realize the transfer of semantic information from high dimensions to low dimensions. Feature information is assigned to different levels according to the size of the network, small features are assigned to shallow layers, and large features are assigned to deep layers. As a result, the original YOLOv8 model can accurately detect objects of various sizes, shapes, and categories.

[0075] For object detection tasks, truly useful features must contain detailed information and semantic information about the object, and these features should be extracted by sufficiently deep neural networks. Although the original YOLOv8 model is simple and effective, the bottom-up path of the path aggregation network brings the opposite problem, that is, detailed information from deep features may be lost or degraded during propagation and interaction.

[0076] HRNet maintains shallow features throughout the feature extraction process and repeatedly fuses shallow and deep features to generate richer shallow features. Inspired by the HRNet network architecture, this embodiment proposes a progressive feature pyramid network to address the above limitations. Figure 5 The structure diagram of the asymptotic feature pyramid network provided by the embodiment of the present invention is shown, and Figure 6 A schematic diagram of feature extraction of an asymptotic feature pyramid network provided by an embodiment of the present invention is shown.

[0077] like Figure 6 As shown in the figure, in the bottom-up feature extraction process in the backbone, the fusion process is started by combining two shallow features with different resolutions in the first stage. When proceeding to the later stages, the deep features are gradually merged into the fusion process and finally fused to form the top-level features. This fusion method can avoid the semantic gap between non-adjacent levels. In this process, the semantic information of shallow features is fused with deep features, and the detail information of deep features is fused with shallow features. Since they interact directly, the information loss or degradation in multi-level transmission is avoided.

[0078] In summary, the original YOLOv8 model includes a backbone network, a Neck network and a prediction network. The backbone network includes a convolution module, a C2f module and a fast spatial pyramid pooling module. Based on the Swim Transformer module and the progressive feature pyramid network, the original YOLOv8 model is improved, including: replacing the C2f module in the backbone network with the Swim Transformer module; and replacing the Neck network with the asymptotic feature pyramid network.

[0079] Specifically, Figure 7 A complete structural diagram of the improved YOLOv8 model provided in an embodiment of the present invention is shown.

[0080] like Figure 7 As shown in the figure, in the improved YOLOv8 model, the C2f module in the backbone network of the original YOLOv8 model is replaced by the Swim Transformer module; at the same time, the Neck network of the original YOLOv8 model is replaced by the progressive feature pyramid network.

[0081] In addition, in the entire feature fusion process, since there may be contradictions between different objects at a certain position between the levels, element-by-element summation is not an effective method. To solve this problem, this embodiment uses ASFF (Adaptively Spatial Feature Fusion) to assign different spatial weights to features at different levels in the multi-level feature fusion process, enhance the importance of key levels, and reduce the impact of contradictory information from different objects.

[0082] In this embodiment, the C2f module in the backbone network is replaced by the Swim Transformer module, and the Neck network is replaced by the asymptotic feature pyramid network to obtain an improved YOLOv8 model. Then, the image to be detected is input into the pre-trained improved YOLOv8 model to obtain the fault detection result. This method improves the improved YOLOv8 model by combining the SwinTransformer module and the asymptotic feature pyramid network, and uses the improved YOLOv8 model to perform fault detection on the image to be detected of the wheel and rail of the lifting equipment, which effectively improves the global information extraction capability and avoids the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of the wheel and rail fault diagnosis of the lifting equipment.

[0083] On the basis of the above embodiments, the process of using the improved YOLOv8 model to perform fault detection on the image to be detected will be described in detail below.

[0084] The image to be detected is input into the pre-trained improved YOLOv8 model to obtain the fault detection result, including: based on the improved backbone network, extracting the shallow feature vector and the deep feature vector in the image to be detected; based on the asymptotic feature pyramid network, performing a downsampling operation on the shallow feature vector to obtain the deep feature, and performing an upsampling operation on the deep feature vector to obtain the shallow feature; using an adaptive spatial feature fusion method to fuse the deep features and the shallow features layer by layer to obtain multi-scale features; based on the prediction network, predicting the multi-scale features to obtain the fault detection result corresponding to the image to be detected; wherein the fault detection result includes the fault location and the fault category.

[0085] Based on the improved backbone network, shallow feature vectors and deep feature vectors in the image to be detected are extracted. Specifically, the image to be detected is sequentially input into the Conv module, Conv module, Swin Transformer module, Conv module, Swin Transformer module, Conv module, Swin Transformer module, Conv module, Swin Transformer module, and SPPF module of the backbone network in the improved YOLOv8 model for feature extraction to obtain shallow feature vectors (e.g. Figure 7 A set of feature vectors output by the second Swin Transformer module from top to bottom) and deep feature vectors (e.g. Figure 7 The feature vector output by the SPPF module in .

[0086] Further, the feature vectors are aligned, that is, a downsampling operation is performed on the shallow feature vector to obtain the deep feature, and at the same time, an upsampling operation is performed on the deep feature vector to obtain the shallow feature.

[0087] Then, an adaptive spatial feature fusion method is used to fuse deep features and shallow features layer by layer to obtain a set of multi-scale features.

[0088] Finally, based on the prediction network, the multi-scale features are predicted to obtain the fault detection results corresponding to the image to be detected.

[0089] The fault detection results include the fault location and fault category. The fault category includes but is not limited to track tread peeling, rail gnawing, track cracks, wheel tread peeling, fastener breakage, and bolt breakage.

[0090] In this embodiment, shallow feature vectors and deep feature vectors in the image to be detected are extracted based on the improved backbone network, and deep features are obtained by downsampling the shallow feature vectors based on the asymptotic feature pyramid network, and shallow features are obtained by upsampling the deep feature vectors, and then the deep features and shallow features are fused layer by layer using an adaptive spatial feature fusion method to obtain multi-scale features, and then the multi-scale features are predicted based on the prediction network to obtain the fault detection result corresponding to the image to be detected. This method improves the improved YOLOv8 model by combining the Swin Transformer module and the asymptotic feature pyramid network, and uses the improved YOLOv8 model to perform fault detection on the image to be detected of the wheel and rail of the lifting equipment, which effectively improves the global information extraction capability and avoids the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of the wheel and rail fault diagnosis of the lifting equipment.

[0091] On the basis of the above embodiments, the training process of the improved YOLOv8 model will be described in detail below.

[0092] The improved YOLOv8 model is trained by transfer learning, which specifically includes: constructing a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset; using the railway wheel-rail fault dataset to pre-train the improved YOLOv8 model to obtain a pre-trained YOLOv8 model with optimal configuration parameters; based on the pre-trained YOLOv8 model, the model parameters of the pre-trained YOLOv8 model are fine-tuned using the lifting equipment wheel-rail fault dataset to obtain a pre-trained improved YOLOv8 model.

[0093] It is understandable that the training of the improved YOLOv8 model requires a large number of fault samples to ensure training performance. However, due to the limited number of fault samples of lifting equipment, it is difficult to obtain good detection results by training directly from scratch.

[0094] Transfer learning is a technique that applies acquired knowledge in a known field to a target field. It can migrate a trained network model from a large dataset to a new dataset, and reuse the network model parameters and weights on the new dataset. In view of the large-scale lack of crane wheel-rail fault image samples, a transfer learning method is introduced to improve model performance. The characteristics of crane wheel-rail faults are similar to those of railway wheel-rail faults, so the collected railway wheel-rail fault dataset is used for pre-training.

[0095] A railway wheel-rail fault dataset is constructed. Specifically, a preset number of railway wheel-rail fault images collected from high-speed railways and other places are sorted out. Based on the collected railway wheel-rail fault images, an original data feature table is established, and the features of the data in the original data feature table are annotated. Before training the model, the railway wheel-rail fault images are labeled according to the corresponding categories, and the feature values ​​of each column in the data feature table are summarized to obtain a processed summary data table, which is then divided into a training set and a test set.

[0096] Among them, the preset number can be set according to actual needs and is not specifically limited here.

[0097] During the training stage, the acquired railway wheel-rail fault images are subjected to data enhancement operations such as denoising, missing value filling, flipping (flipping the original image left and right), scaling (scaling the size of the original image), color gamut transformation (changing the brightness, saturation, and hue of the original image), and random cropping, so as to achieve the effect of expanding the training samples.

[0098] It should be noted that the process of constructing the lifting equipment wheel-rail fault dataset is the same as the process of constructing the railway wheel-rail fault dataset, which will not be repeated here.

[0099] Then, the improved YOLOv8 model is pre-trained using the enhanced railway wheel-rail fault dataset to obtain the optimal weight of the improved YOLOv8 model, so as to obtain a pre-trained YOLOv8 model with optimal configuration parameters.

[0100] Furthermore, the pre-trained YOLOv8 model is used as the starting point of transfer learning, the lifting equipment wheel-rail failure is selected as the target task, and the constructed lifting equipment wheel-rail failure dataset is used to fine-tune the model parameters of the pre-trained YOLOv8 model through limited training to adapt to the characteristics of the target domain, and the optimal model weights suitable for the target task are obtained, thereby obtaining a trained improved YOLOv8 model.

[0101] The trained improved YOLOv8 model can be directly used to perform fault detection on the image to be detected.

[0102] In this embodiment, by constructing a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset, and using the railway wheel-rail fault dataset to pre-train an improved YOLOv8 model, a pre-trained YOLOv8 model with optimal configuration parameters is obtained, and then based on the pre-trained YOLOv8 model, the model parameters of the pre-trained YOLOv8 model are fine-tuned using the lifting equipment wheel-rail fault dataset to obtain a pre-trained improved YOLOv8 model, so that the image to be detected is input into the pre-trained improved YOLOv8 model to obtain the fault detection result. This method further accelerates the model convergence speed by training and optimizing the improved YOLOv8 model in a transfer learning manner, and improves the accuracy and robustness of lifting equipment wheel-rail fault diagnosis under small sample conditions.

[0103] In some embodiments, Figure 8 A schematic diagram of the detection results of the improved YOLOv8 model provided in an embodiment of the present invention is shown.

[0104] like Figure 8 As shown in the figure, the fault detection results output by the improved YOLOv8 model will display the fault location, fault category and detection accuracy in the image to be detected. The detection accuracy value in the fault detection result is proportional to the detection effect.

[0105] In some other embodiments, the advantages of the lifting equipment wheel-rail fault detection method provided by the embodiments of the present invention will be described.

[0106] (1) The embodiment of the present invention improves on the single-stage framework YOLOv8 by combining the Swin Transformer module with the network structure of the original YOLOv8 model to enhance the global information extraction capability of the model.

[0107] (2) The embodiment of the present invention proposes an improved YOLOv8 model based on a progressive feature pyramid network to achieve direct feature fusion across non-adjacent layers and avoid the loss or degradation of feature information during transmission and interaction.

[0108] (3) Considering the similarities between the characteristics of hoisting equipment wheel-rail faults and railway wheel-rail faults, a transfer learning strategy is adopted. The improved YOLOv8 model is pre-trained using the railway wheel-rail fault dataset, and then the trained model is transferred to the hoisting equipment wheel-rail fault dataset, which further accelerates the convergence speed of the proposed algorithm and improves the accuracy and robustness of hoisting equipment wheel-rail fault diagnosis in small sample conditions.

[0109] Fig. 9 The schematic diagram of the structure of the wheel-rail fault detection device for lifting equipment provided by the embodiment of the present invention is shown.

[0110] like Fig. 9As shown, the device comprises:

[0111] An image acquisition module 910 is used to acquire the image to be detected of the wheel rail of the lifting equipment;

[0112] A fault detection module 920 is used to input the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result;

[0113] The improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing based on a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0114] In this embodiment, the image to be detected of the lifting equipment wheel and rail is obtained by the image acquisition module 910, and the fault detection module 920 inputs the image to be detected into the pre-trained improved YOLOv8 model to obtain the fault detection result; wherein, the improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing according to a railway wheel and rail fault data set and a lifting equipment wheel and rail fault data set. The device improves the improved YOLOv8 model by combining the Swin Transformer module and the asymptotic feature pyramid network, and uses the improved YOLOv8 model to perform fault detection on the image to be detected of the lifting equipment wheel and rail, effectively improving the global information extraction capability, and avoiding the loss or degradation of feature information during transmission and interaction, thereby improving the accuracy and robustness of the lifting equipment wheel and rail fault diagnosis.

[0115] It should be noted that the lifting equipment wheel-rail fault detection device provided in this embodiment and the lifting equipment wheel-rail fault detection method described above can correspond to each other and will not be described in detail here.

[0116] Fig.10 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.10As shown, the electronic device may include: a processor 1010, a communication interface 1020, a memory 1030 and a communication bus 1040, wherein the processor 1010, the communication interface 1020 and the memory 1030 communicate with each other through the communication bus 1040. The processor 1010 may call the logic instructions in the memory 1030 to execute the method for detecting wheel-rail faults of a lifting equipment, the method comprising: obtaining an image to be detected of a lifting equipment wheel-rail; inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model includes a SwinTransformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing according to a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0117] In addition, the logic instructions in the above-mentioned memory 1030 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the lifting equipment wheel-rail fault detection method provided by the above-mentioned methods, the method comprising: obtaining an image to be detected of the lifting equipment wheel-rail; inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; wherein the improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, and is obtained by training and optimizing according to a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting wheel-rail faults of lifting equipment, characterized in that: include: Acquire the image of the wheel and rail of the lifting equipment to be inspected; Inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; The improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing based on a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

2. The method for detecting wheel-rail faults of lifting equipment according to claim 1, characterized in that: Also includes: Based on the Swim Transformer module and the asymptotic feature pyramid network, the original YOLOv8 model is improved to obtain the improved YOLOv8 model.

3. The method for detecting wheel-rail faults of lifting equipment according to claim 2, characterized in that: The original YOLOv8 model includes a backbone network, a Neck network and a prediction network, and the backbone network includes a convolution module, a C2f module and a fast spatial pyramid pooling module; Accordingly, the original YOLOv8 model is improved based on the Swim Transformer module and the progressive feature pyramid network, including: Use the Swim Transformer module to replace the C2f module in the backbone network; And, the Neck network is replaced by the asymptotic feature pyramid network.

4. The method for detecting wheel-rail faults of lifting equipment according to claim 3, characterized in that: The step of inputting the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result includes: Based on the improved backbone network, extracting shallow feature vectors and deep feature vectors in the image to be detected; Based on the asymptotic feature pyramid network, performing a downsampling operation on the shallow feature vector to obtain a deep feature, and performing an upsampling operation on the deep feature vector to obtain a shallow feature; Adopting an adaptive spatial feature fusion method to fuse the deep features and the shallow features layer by layer to obtain multi-scale features; Predicting the multi-scale features based on the prediction network to obtain a fault detection result corresponding to the image to be detected; The fault detection result includes the fault location and the fault category.

5. The method for detecting wheel-rail faults of lifting equipment according to claim 1, characterized in that: The improved YOLOv8 model is trained by transfer learning, specifically including: Constructing the railway wheel-rail fault data set and the lifting equipment wheel-rail fault data set; Pre-training the improved YOLOv8 model using the railway wheel-rail fault dataset to obtain a pre-trained YOLOv8 model with optimal configuration parameters; Based on the pre-trained YOLOv8 model, the model parameters of the pre-trained YOLOv8 model are fine-tuned using the lifting equipment wheel-rail fault dataset to obtain a pre-trained improved YOLOv8 model.

6. The method for detecting wheel-rail faults of lifting equipment according to claim 5, characterized in that: The steps of constructing a railway wheel-rail fault data set include: Collecting a preset number of railway wheel-rail fault images; The railway wheel-rail fault images are labeled according to fault categories, and an enhancement operation is performed on the railway wheel-rail fault images to obtain the railway wheel-rail fault dataset.

7. The method for detecting wheel-rail faults of lifting equipment according to claim 6, characterized in that: The enhancement operation includes one or more combinations of denoising, missing value filling, flipping, scaling, color gamut conversion, and random cropping.

8. A wheel-rail fault detection device for a lifting equipment, characterized in that: include: An image acquisition module, used to acquire the image to be inspected of the wheel rail of the lifting equipment; A fault detection module is used to input the image to be detected into a pre-trained improved YOLOv8 model to obtain a fault detection result; The improved YOLOv8 model includes a Swin Transformer module and an asymptotic feature pyramid network, which is obtained by training and optimizing based on a railway wheel-rail fault dataset and a lifting equipment wheel-rail fault dataset.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the lifting equipment wheel-rail fault detection method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting wheel-rail faults of a lifting equipment according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Tower crane crack image detection method

    CN112241950A

  • Intelligent defect detection method for hoisting equipment

    CN117557556A

  • Apple maturity detection method based on Swin Transform

    CN117893823A

  • Safety rule violation detection in a construction or constructed site

    WO2024074327A1

Cited By

  • Capacitive equipment fault diagnosis method of improved YOLO model

    CN119832259A