Method for identifying looseness of bolts at bottom of maglev train based on deep learning

Abstract

The invention provides a maglev train bottom bolt looseness identification method based on deep learning. The method comprises the following steps: acquiring a bottom image of a maglev train by adopting imaging equipment, and carrying out bolt target detection on an image in a bottom image data set through an improved YOLOv8 network to obtain a bolt image; performing classification processing on the bolt image through a lightweight network RepViT fused with a lightweight ViT design principle; color space conversion, image binaryzation, morphological processing and connected domain screening processing are carried out on the bolt image to extract an anti-loosening line, and whether the vehicle bottom bolt in the bolt image is loosened or not is judged based on the bolt anti-loosening line according to loosening characteristics of different types of bolts under multiple visual angles. According to the method, the detection precision is improved by means of improving a target detection network structure, refining bolt loosening characteristics, providing adaptive detection methods for different types of bolts and the like; by designing a lightweight classification network mode, the calculation amount and time consumption are reduced, and the detection efficiency is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and in particular to a method for identifying loosening of bolts on the underframe of a maglev train based on deep learning. BACKGROUND

[0002] Maglev trains have gradually become the main direction of future urban rail transit development due to their advantages of high speed, stability and low noise, and their safety and reliability are the inevitable requirements for people's travel. With the rapid growth of railway operation mileage and the scale of maglev trains, the workload of manual operation has increased significantly.

[0003] Bolts, as key fasteners connecting various components of the underframe of a maglev train, their pre-tightening force and structural stability are important guarantees for the safety of maglev train operation. Regular inspection and tightening of the underframe bolts of the train are important for train maintenance and are a key measure to ensure safe train operation and extend the service life of the train. Due to the complex structure of the underframe of the maglev train, there is a visual obstruction during inspection, the bolt position is difficult to observe, and the number and types of bolts are numerous and complex. How to achieve efficient and accurate loosening detection has become a key problem to be solved.

[0004] Currently, the existing bolt loosening detection mainly relies on manual inspection and sensor-based detection methods. Manual inspection mainly relies on inspection workers to visually detect whether the marking line is misaligned or measure the bolt torque to determine whether the bolt is loose. This method has the problems of large detection workload, long time consumption and low efficiency, and the inspection time is often at night, which is affected by the working environment, and the inspection workers are prone to visual fatigue, causing false detection, missed detection and other problems. The sensor-based detection method often requires the installation of a large number of sensors and auxiliary equipment, which is greatly affected by the environment, has high detection cost and is complex to operate, and the detection effect is not good when the number and types of bolts are numerous and densely distributed. With the significant increase in manual operation workload, the risk of bolt loosening detection failure is also increasing.

[0005] The detection method based on deep learning and digital image processing provides a new perspective for solving the bolt loosening detection problem with its high detection accuracy, non-contact and high efficiency, and has become the main research direction for the detection of bolt loosening on the underframe of a maglev train.

[0006] The visual-based maglev train bolt loosening detection method in the prior art includes two-stage detection of target detection-loosening identification. In the target detection stage, the precise positioning and identification of the train bolt are realized through deep learning algorithms such as Faster R-CNN and YOLO series, the positioning effect is optimized by improving the network structure, and the identification accuracy is improved. In the loosening identification stage, the detection methods mainly include image processing, deep learning and 3D point cloud-based detection methods. The loosening detection based on image processing captures the bolt loosening features through edge detection, Hough transform and other methods, establishes the loosening judgment standard, and has fast detection speed. The identification method based on deep learning can autonomously learn the target features in the bolt image, and realize more automatic and intelligent detection, but the feature extraction capability of the deep learning algorithm is often related to the network layers and structure, and while realizing high-precision detection, it also faces the challenges of huge calculation amount and time consumption. The detection based on 3D point cloud reconstructs the 3D point cloud of the bolt through two-dimensional bolt images under different perspectives, and quantifies the bolt loosening angle, but the detection based on three-dimensional point cloud is not as good as the two-dimensional visual detection method in terms of universality and use cost, and needs to be further developed.

[0007] The disadvantages of the above-mentioned visual-based maglev train bolt loosening detection scheme include that the method often only targets a single type or specific perspective of the bolt, and it is difficult to meet the characteristics of the maglev train bolt being difficult to observe and numerous under real working conditions, and an intelligent, automatic and lightweight bolt loosening detection system needs to be constructed. SUMMARY

[0008] Embodiments of the present application provide a maglev train bolt loosening identification method based on deep learning to effectively identify the loosening condition of the train bolt of the maglev train.

[0009] To achieve the above-mentioned purpose, the present application adopts the following technical solutions.

[0010] A maglev train bolt loosening identification method based on deep learning, comprising:

[0011] An imaging device is used to collect the train bottom image of the maglev train, and after preprocessing, the train bottom image dataset is formed;

[0012] A YOLOv8 network is improved to detect the bolt target in the image in the train bottom image dataset, and a bolt image is obtained;

[0013] A lightweight network RepViT that fuses the lightweight ViT design principle is used to classify and process the bolt image;

[0014] The bolt image is subjected to color space conversion, image binarization, morphological processing and connected domain screening processing to extract the anti-loose line, the bolt anti-loose line is extracted, and whether the underbody bolt in the bolt image is loose is judged based on the bolt anti-loose line according to the loosening characteristics of bolts of different categories under multiple perspectives.

[0015] Preferably, the underbody image of the maglev train is collected by the imaging device, and after the underbody image is preprocessed, an underbody image data set is formed, comprising:

[0016] The underbody image of the maglev train is collected by the imaging device on site, the underbody image contains multi-scale, multi-space orientation and multi-type bolt targets, all underbody images form an underbody image data set, the bolts in the underbody images in the underbody image data set are subjected to data label processing by using an image labeling tool, the position information of the bolts in each underbody image is labeled, the underbody images in the underbody image data set are subjected to gray scale conversion, random area masking, Gaussian noise introduction, dynamic brightness adjustment, Gaussian filter disturbance and affine space transformation preprocessing by using an imgaug image enhancement library, and the preprocessed and enhanced underbody image data set is used as an input for model training.

[0017] Preferably, the underbody image in the underbody image data set is subjected to bolt target detection by improving the YOLOv8 network to obtain a bolt image, comprising:

[0018] The improved YOLOv8 network including adding a feature fusion line, adding a Dropout layer and replacing a loss function is constructed, in the training process of the improved YOLOv8 network, the size of the input underbody image is 640*640, the training process adopts a stochastic gradient descent (SGD) optimization algorithm, the YOLOv8 network model is deployed on a PyTorch framework, an anchor box is denoted as B=[x,y,w,h], a target box is denoted as B gt =[x gt ,y gt ,w gt ,h gt ], and a dynamic method is used to calculate the IoU loss in the class prediction loss of the YOLOv8 network.

[0019] L WioU =R WIoU L IoU

[0020]

[0021] Wherein, W g , H g are the sizes of the minimum bounding box, L IoU is the loss term of IoU, L WIoU is the total loss of WIoU, and RWIoU R is a penalty term for WloU WIoU ∈[1,e),L IoU is amplified, L IoU ∈[0,1],R WIoU is reduced, and the attention to the center point distance is reduced when the anchor frame overlaps well with the target frame;

[0022] After the iterative training is completed, a trained YOLOv8 network is obtained.

[0023] The bottom image of the maglev train is input into the trained improved YOLOv8 network to position and identify the bolt, and the boundary box information of the bolt is output. After cutting and removing the fuzzy sample, the bolt image is output.

[0024] Preferably, the RepViT that fuses the lightweight ViT design principle classifies the bolt image, including:

[0025] According to the part features of the bolt, a-d four-level classification standards are established. According to the change of the observed pitch angle of the bolt, the a-class bolts are subdivided into: pitch angle = 90°; pitch angle = 0°; pitch angle ∈(0°, 90°) three categories, and the b-class bolts are subdivided into: pitch angle = 90°; pitch angle = 0°; pitch angle ∈(0°, 90°) three categories, a total of eight categories of bolt images of a1, a2, a3, b1, b2, b3, c and d.

[0026] The bolt image is enhanced by using gray scale enhancement, rectangular discarding, Gaussian noise, up-down flipping, left-right flipping, brightness enhancement, Gaussian blur, average value blur and affine transformation image enhancement method. The enhanced bolt image is input into the lightweight network RepViT that fuses the lightweight ViT design principle for image classification, wherein the image size is 224*224, the AdamW optimizer is used, the cosine annealing learning rate scheduling strategy is combined, the data enhancement includes Mixup, AutoAugment and random erasing, and the generalization ability is improved through label smoothing. The algorithm deployment is completed based on the PyTorch framework, and the RepViT network outputs the classification result of the bolt.

[0027] Preferably, the bolt image is subjected to color space conversion, image binarization, morphological processing and connected domain screening processing, and the bolt anti-loose line is extracted, including:

[0028] The bolt image is converted from an RGB space to an HSV space, the bolt image in the HSV space is binarized through an HSV threshold, a bolt marker line image is obtained by extracting a marker line part in the binarized bolt image, and the bolt marker line image is subjected to erosion operation and dilation operation, and then small connected domains in the bolt marker line image are removed according to a set connected domain area range, and a bolt lock wire is extracted.

[0029] Preferably, the bolt lock wire is used to judge whether the underbody bolt in the bolt image is loose according to the loosening characteristics of the bolt under different categories of bolts in multiple views, and the method comprises the following steps:

[0030] For the a1, b1 and b3 bolt images, the bolt lock wire is used to draw minimum circumscribed rectangles of all connected domains in the bolt marker line image, a rectangular coordinate system is established on the marker line image, a rotation angle of the x-axis positive direction is rotated to a rotation angle coinciding with a long side of the minimum circumscribed rectangle as a rotation angle of the connected domain, the rotation angle is in a range of [0°, 180°], and an angle difference abs between rotation angles of the minimum circumscribed rectangles is calculated. An angle threshold is set, if the angle difference is greater than the angle threshold, it is determined that the bolt is loose, otherwise, it is determined that the bolt is fastened.

[0031] For the a2 bolt image, the bolt lock wire is used to dilate the bolt marker line image by a certain size of a convolution kernel for a certain number of times, the number of connected domains in the dilated bolt marker line image is calculated, if the number of connected domains is equal to 1, it is determined that the bolt is fastened, and if the number of connected domains is greater than 1, it is determined that the bolt is loose.

[0032] For the a3 bolt image, the bolt lock wire is used to calculate the angle difference between rotation angles of the minimum circumscribed rectangles, an angle threshold is set, if the angle difference is less than or equal to the angle threshold, it is determined that the bolt is fastened, if the angle difference is greater than the angle threshold, the same number of dilations is performed by a certain size of a convolution kernel, the number of connected domains in the dilated bolt marker line image is calculated, if the number of connected domains is equal to 1, it is determined that the bolt is fastened, and if the number of connected domains is greater than 1, it is determined that the bolt is loose.

[0033] For the c and d bolts, the bolt lock wire is used to draw minimum circumscribed rectangles of connected domains greater than an area threshold in the bolt marker line image, the angle difference between rotation angles of the minimum circumscribed rectangles is calculated, if the angle difference is less than or equal to an angle threshold, it is determined that the bolt is fastened, if the angle difference is greater than the angle threshold, the minimum circumscribed rectangles of all connected domains are drawn, the aspect ratio of the minimum circumscribed rectangle is calculated, an aspect ratio threshold is set, if the aspect ratio of the minimum circumscribed rectangle is greater than the aspect ratio threshold, it is determined that the bolt is fastened, and if the aspect ratio of the minimum circumscribed rectangle is less than the aspect ratio threshold, it is determined that the bolt is loose.

[0034] For the b2 type bolt, the minimum circumscribed rectangle of the connected domain whose bolt anti-looseness line is greater than the area threshold is drawn, when the number of minimum circumscribed rectangles is less than or equal to 1, it is determined that the bolt is loose, when the number of minimum circumscribed rectangles is greater than 1, a rectangular coordinate system is established on the bolt marking line image, the smallest x coordinate value of the four corner points of each minimum circumscribed rectangle is recorded, and the corner point distance between the rectangles is calculated, a distance threshold is set, if the corner point distance is less than or equal to the distance threshold, it is determined that the bolt is fastened, if the corner point distance is greater than the distance threshold, it is determined that the bolt is loose.

[0035] As can be seen from the technical solutions provided by the above embodiments of the present application, the present application proposes a three-stage bolt loosening detection method based on deep learning and digital image processing technology, which improves the detection accuracy by improving the target detection network structure, refining the bolt loosening features, and proposing adaptive detection methods for different types of bolts; the calculation amount and time consumption are reduced and the detection efficiency is improved by designing a lightweight classification network and using image processing technology.

[0036] Additional aspects and advantages of the present application will be described in the following description, which will become apparent from the following description, or will be learned by practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below, and obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0038] Figure 1 A processing flow chart of a kind of magnetic levitation train bolt loosening identification method based on deep learning provided by the embodiment of the present application;

[0039] Figure 2 A magnetic levitation train undercarriage image schematic diagram provided by the embodiment of the present application;

[0040] Figure 3 A target detection data set enhancement schematic diagram provided by the embodiment of the present application;

[0041] Figure 4 An improved YOLOv8 network structure diagram provided by the embodiment of the present application;

[0042] Figure 5 An IoU schematic diagram provided by the embodiment of the present application;

[0043] Figure 6 A bolt classification schematic diagram provided by the embodiment of the present application;

[0044] Figure 7 A RepViT network structure diagram provided for an embodiment of the present application;

[0045] Figure 8 A classification data set enhancement schematic diagram provided for an embodiment of the present application;

[0046] Figure 9 An artificial simulation bolt negative sample schematic diagram provided for an embodiment of the present application;

[0047] Figure 10 A loosening line extraction flowchart provided for an embodiment of the present application;

[0048] Figure 11 A rectangular coordinate system schematic diagram provided for an embodiment of the present application;

[0049] Figure 12 An angle method loosening identification process schematic and flowchart provided for an embodiment of the present application;

[0050] Figure 13 An expansion method loosening identification process schematic and flowchart provided for an embodiment of the present application;

[0051] Figure 14 An angle method + expansion method loosening identification process schematic and flowchart provided for an embodiment of the present application;

[0052] Figure 15 An angle method + rectangular method identification process schematic and flowchart provided for an embodiment of the present application;

[0053] Figure 16 An angle point distance method identification process schematic and flowchart provided for an embodiment of the present application. DETAILED DESCRIPTION

[0054] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be interpreted as a limitation on the present application.

[0055] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0056] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0057] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0058] This invention proposes a three-stage method for identifying loose bolts on the undercarriage of a maglev train based on deep learning and digital image processing: target detection, image classification, and loosening identification. This method improves bolt positioning accuracy by refining the target detection network; designs a lightweight image classification network to reduce computational parameters and improve classification efficiency; and designs an adaptive loosening detection method based on the loosening characteristics of different types of bolts under multiple perspectives, utilizing digital image processing technology. Addressing the difficulty of existing technologies in capturing bolt loosening characteristics in complex environments, this invention adds an image classification task to the two-stage detection process, refining the loosening characteristics of different types of bolts under multiple perspectives, and improving the adaptability and robustness of the detection method.

[0059] The flowchart of a deep learning-based method for identifying loose bolts on the undercarriage of a maglev train, provided in this embodiment of the invention, is as follows: Figure 1 As shown, the processing steps include the following:

[0060] Step S10: Acquire images of the undercarriage of the maglev train using imaging equipment to form an undercarriage image dataset. Perform bolt labeling and data augmentation on the undercarriage image dataset to obtain an augmented undercarriage image dataset.

[0061] A schematic diagram of the undercarriage of a maglev train provided in this embodiment of the invention is shown below. Figure 2 As shown. In this embodiment of the invention, the images of the undercarriage of the maglev train were all captured on-site using professional-grade imaging equipment. Each undercarriage image contains bolt targets of multiple scales, spatial orientations, and types, forming an undercarriage image dataset to meet the morphological diversity requirements under actual working conditions. Deep learning object detection is a supervised learning method. Before model training, Labelimg (an image annotation tool) was used to perform data labeling on the bolts in the undercarriage images in the undercarriage image dataset, annotating the positional information of the bolts in each undercarriage image.

[0062] To further expand the sample size of the object detection dataset, enabling the model to learn more features from the image dataset and improve its robustness and generalization ability, and to address potential issues such as thermal noise interference caused by device thermal effects and motion blur caused by focus deviations during camera imaging, the imgaug image enhancement library is used to perform preprocessing steps on the vehicle undercarriage images in the dataset. These steps include grayscale conversion, random region occlusion, Gaussian noise introduction, dynamic brightness adjustment, Gaussian filtering perturbation, and affine space transformation. Figure 3 This is a schematic diagram illustrating an object detection dataset enhancement method provided in an embodiment of the present invention. The enhanced vehicle undercarriage image dataset is used as input for model training.

[0063] Step S20: Perform bolt target detection and localization processing on the enhanced vehicle undercarriage image dataset using an improved YOLOv8 network, output the bounding box of the bolt, and output the bolt image after cropping and removing blurred samples.

[0064] Figure 4 This invention provides an improved YOLOv8 network structure diagram. The improved YOLOv8 network is used to locate and identify bolt targets in vehicle undercarriage images. The improvement is mainly achieved by adding feature fusion lines, adding a Dropout layer, and replacing the loss function. The improved YOLOv8 network, based on the original, inputs the downsampling results of the 5th and 7th Conv modules in the Backbone into the Neck part. In the C2f module of the Neck part, which outputs the detection feature map, a Dropout layer is introduced, and the loss function CIoU is replaced with Wise-IoU, effectively improving the target detection accuracy.

[0065] like Figure 4As shown, a feature fusion line is added: after feature extraction from the Backbone part and feature fusion from the Neck part of the input image, a detection feature map is obtained. The network's deep semantic information is richer, but some original image information is lost. Therefore, the downsampling results of the Conv modules at layers 5 and 7 of the network structure are input into the Neck part. This allows it to contain rich deep semantic information and shallow spatial information, further achieving feature fusion and improving the target detection network's ability to recognize bolts.

[0066] Dropout Layer: Dropout is a classic regularization technique. Its core idea is to randomly discard some neuron outputs with a certain probability during the training phase to prevent the model from overfitting the training data. This random discarding mechanism can significantly reduce the co-adaptation between neurons, forcing the network to learn more generalizable features. The C2f module, as a key feature transformation and fusion module in YOLOv8, directly affects the performance of the detection head. Introducing a Dropout layer into the C2f module of the aforementioned detection feature map allows for the random discarding of some feature channels during the training phase, enhancing the model's robustness to features. This enables the network to extract key features more stably when faced with different inputs, reducing over-reliance on certain specific features.

[0067] Improving the Loss Function: Bounding box loss is a core component of object detection, and the choice of loss function significantly impacts model performance. In object detection tasks, IoU is widely used to identify the similarity between the model's prediction and the ground truth object annotations. It is calculated by taking the intersection and union of the model's prediction and the ground truth annotation regions, dividing the intersection by the union, and obtaining a ratio between 0 and 1. A higher IoU value indicates a higher similarity between the model's prediction and the ground truth annotations. If the IoU value is greater than a preset threshold, the predicted bounding box is considered to have successfully matched the ground truth bounding box. During training, IoU is often used as part of the loss function. Let the anchor box be B = [x, y, w, h], and the target box be B... gt =[x gt y gt w gt h gt ], Figure 5 This invention provides an IoU diagram, where the IoU expression is:

[0068]

[0069] YOLOv8 uses Complete-IoU in L IoUBuilding upon previous work, the vanishing gradient problem during backpropagation was addressed, and consideration of aspect ratio consistency was added. However, since low-quality samples are unavoidable in training data, geometric metrics such as distance and aspect ratio exacerbate the penalty for low-quality samples, reducing the model's generalization performance. Therefore, Wise-IoU is introduced into the loss function calculation, and a dynamic method is used to calculate the IoU loss in the category prediction loss.

[0070] L WIoU =R WIoU L IoU

[0071]

[0072] Among them W g H g L is the size of the minimum bounding box. IoU For the loss term of IoU, L WIoU R represents the total loss of WIoU. WIoU For WIoU's penalty item, R WIoU ∈[1, e), the loss term L of the ordinary mass anchor frame IoU Significantly magnified. L IoU ∈[0,1], making the R of the high-quality anchor frame WIoU It significantly reduces the focus on the distance to the center point, and when the anchor frame and the target frame overlap well, it reduces the focus on the distance to the center point.

[0073] During training, the network input image size was 640*640. The training process employed the SGD (Stochastic Gradient Descent) optimization algorithm with a learning rate momentum of 0.937. A pre-trained model was loaded into the network during training, with an initial learning rate of 0.01 and 300 training epochs. The algorithm model was deployed on the PyTorch framework. Images of the underside of a maglev train were input into the trained, improved YOLOv8 network to locate and identify bolts, outputting the bolt bounding box information. After cropping and removing blurry samples, the bolt images were output.

[0074] Step S30: The bolt images are classified using the lightweight network RepViT, which incorporates lightweight ViT design principles.

[0075] To further refine the loosening characteristics of bolts, this invention adds an image classification task to the two-stage process. First, a four-level classification standard (ad) is established based on the location characteristics of the bolts. Then, based on the changes in the observed elevation angle, bolts of category a are further subdivided into three categories: elevation angle = 90°; elevation angle = 0°; and elevation angle ∈ (0°, 90°). Bolts of category b are further subdivided into three categories: depression angle = 90°; depression angle = 0°; and depression angle ∈ (0°, 90°). The bolt images are divided into a total of eight categories, resulting in bolt images of eight categories: a1, a2, a3, b1, b2, b3, c, and d. Figure 6 This is a schematic diagram illustrating the classification of bolt categories provided in an embodiment of the present invention.

[0076] Nine image augmentation methods, including grayscale enhancement and rectangular discarding, were employed to enhance the bolt dataset. The enhanced dataset was then input into the innovative lightweight network RepViT, which incorporates lightweight ViT design principles, for image classification. The image size was 224*224. The AdamW optimizer was used, combined with a cosine annealing learning rate scheduling strategy. Data augmentation included Mixup, AutoAugment, and random erasure. Label smoothing was used to improve generalization ability. The algorithm was deployed using the PyTorch framework. After iterative training, the trained RepViT network was obtained. Bolt images were then input into the trained RepViT network, which output the bolt classification results.

[0077] Considering the complexity and computational time of deep learning networks, this invention employs the lightweight RepViT network, which is based on a lightweight CNN and innovatively integrates the design principles of lightweight ViT, for image classification. The RepViT network achieves efficient feature extraction through the following core components:

[0078] ① The RepViT block with reparameterized structure transforms the multi-branch structure during training into a single convolutional layer during inference, effectively reducing computational complexity and memory access overhead.

[0079] ② Early convolutional stem structure, which replaces the traditional patchify operation with two 3×3 stride convolutions, enhances the ability to express spatial features while reducing the initial stage delay;

[0080] ③ The independent downsampling module, combined with the RepViT block, deepens the network layers and mitigates information loss during the downsampling process through hierarchical feature processing;

[0081] ④ A minimalist classifier design is adopted, which combines global average pooling with linear layers to optimize the classification efficiency of high-resolution features;

[0082] ⑤ The phase ratio optimization strategy allocates network depth in a ratio of 1:1:7:1 to concentrate computing resources and enhance the ability to process high-dimensional features. Figure 7This is a RepViT network structure diagram provided for an embodiment of the present invention.

[0083] To improve the generalization ability of the classification network, image enhancement methods such as grayscale enhancement, rectangular discarding, Gaussian noise, vertical flipping, horizontal flipping, brightness enhancement, Gaussian blur, mean blur, and affine transformation are used to enhance the bolt dataset. Figure 8 This is a schematic diagram of a classification dataset enhancement provided in an embodiment of the present invention, such as... Figure 8 As shown. The image size in the input network of the augmented dataset is 224*224. The AdamW optimizer (initial learning rate 0.02) is used in combination with the cosine annealing learning rate scheduling strategy. Data augmentation includes Mixup (α = 0.8), AutoAugment, and random erasure (probability 0.2). Label smoothing (ε = 0.1) is used to improve generalization ability. The algorithm is deployed based on the PyTorch framework.

[0084] Step S40: Perform color space conversion, image binarization, morphological processing and connected component filtering on the classified bolt images, and then extract the anti-loosening lines from the bolt images based on the loosening characteristics of different types of bolts under multiple perspectives.

[0085] Most of the bolt images captured on-site showed fastened bolts. To address the severe imbalance between normal and loose bolt samples at the bottom of the maglev train, balance the number of positive and negative samples, and enhance the accuracy and robustness of the loose bolt identification model, an artificial simulation experimental environment was built. Using mechanical fasteners such as hexagonal bolts, socket head cap screws, Phillips head bolts, and round-hole square plates, red anti-loosening marking lines were drawn with industrial-grade paint pens to accurately represent bolt loosening characteristics. The results included bolt images of different types, viewing angles, and degrees of looseness. Figure 9 This is a schematic diagram of an artificially simulated negative bolt provided in an embodiment of the present invention.

[0086] In traditional manual visual inspection, maintenance workers draw anti-loosening lines on bolts and judge the bolt's tightness by observing whether the anti-loosening lines shift. This invention is a bolt loosening detection based on computer vision, and the construction of the recognition method is also based on this. The bolt anti-loosening lines are extracted by color space conversion, image binarization, morphological processing, and connected component filtering. Then, a suitable bolt loosening recognition method is constructed based on the loosening characteristics of different types of bolts under multiple perspectives.

[0087] Converting from RGB (Red, Green, Blue) space to HSV (Hue-Saturation-Value) space: Bolt anti-loosening lines are usually marked in red, which is clearly different from the bolt background and its environment. The original bolt image is an RGB image. The RGB color space is directly related to the working principle of pixels on computer display devices, while HSV images are more in line with human visual perception. Before image processing, it is first converted from RGB space to HSV space.

[0088] Image binarization: Due to the different environments in which different bolts are located, the images of bolts and their anti-loosening lines suffer from variations in lighting and clarity. Furthermore, anti-loosening lines drawn at different times will have different HSV thresholds due to factors such as time changes and paint peeling. When setting the red HSV color range, the HSV threshold needs to be dynamically adjusted in batches to perform binarization on the bolt images. By examining the HSV color range of the anti-loosening lines and narrowing the HSV threshold, the extraction effect of the marker lines can be further improved.

[0089] Morphological processing and connected component filtering: When extracting bolt anti-loosening lines by filtering the HSV color range, interference from the surrounding environment and paint splatter can occur. Based on image binarization, a closing operation is first performed on the extracted marker lines. Erosion removes minor noise around the marker lines, and dilation fills in voids within the marker lines, optimizing the extraction results. Even after morphological processing such as erosion and dilation, small areas of connected component noise still exist in the image. Therefore, a range of connected component areas to be retained is further defined to remove these small noise particles. Figure 10 This is a flowchart of a process for extracting anti-loosening lines, provided as an embodiment of the present invention.

[0090] The type of bolt and the viewing angle have a great influence on the identification of bolt loosening. Therefore, based on the image characteristics of different types of bolts and their anti-loosening lines, this invention designs a bolt loosening detection method adapted to digital image processing technology.

[0091] The observation pitch angles of bolts of types a1, b1, and b3 are between (0° and 90°), allowing for relatively complete observation of the anti-loosening lines. These lines are composed of long, narrow line segments and can be approximated as rectangles. When the bolts are tight, the rotation angles of the bounding rectangles of their anti-loosening lines are approximately the same, while when the bolts are loose, the rotation angles of the bounding rectangles differ significantly. Based on these characteristics, the minimum bounding rectangles of all connected components in the bolt marking line image are first drawn. Since manually drawing the anti-loosening lines may result in paint splattering or missing parts, forming smaller connected components, an area threshold is set before drawing the minimum bounding rectangles of all connected components to eliminate interference from these smaller components. Only when the area of ​​a connected component exceeds the area threshold is the minimum bounding rectangle drawn. Secondly, a rectangular coordinate system is established on the bolt marking line image, such as... Figure 11 As shown, the rotation angle of the connected component is defined as the angle at which the component rotates in the positive x-axis direction to coincide with the long side of the smallest bounding rectangle. The rotation angle ranges from [0°, 180°]. The angle difference between the rotation angles of each smallest bounding rectangle is calculated. An angle threshold is set; if the angle difference is greater than the threshold, the bolt is considered loose; otherwise, it is considered tight. Figure 12 This invention provides a schematic diagram and flowchart of an angle-based loosening identification process.

[0092] For type A2 bolts, the observation elevation angle is 0°. Their marking lines consist of multiple connected components of varying lengths. When the bolt is tightened, these connected components are tightly packed; when the bolt is loose, they are sparsely packed. Based on these characteristics, the bolt marking line image is dilated a certain number of times using a convolution kernel of a specific size. If the bolt is tightened, tightly packed connected components will connect to each other after dilation, forming a single connected component. Sparsely packed connected components, however, will not connect after dilation. Therefore, the image is first dilated, and then the number of connected components is determined. If the number of connected components is equal to 1, the bolt is considered tightened; if the number of connected components is greater than 1, the bolt is considered loose. Figure 13 This invention provides a schematic diagram and flowchart of a loosening identification process using the expansion method, as an embodiment of the present invention.

[0093] The observation elevation angle for type A3 bolts is [0°, 90°]. Slightly different from the marking lines of types A1 and A2 bolts, their bolt marking line images consist of multiple connected regions of varying lengths. In some images of tightened bolts, the rotation angles of the minimum bounding rectangles of each connected region are similar, while in others, the connected regions are closely arranged. In images of loose bolts, the rotation angles of the minimum bounding rectangles of each connected region differ significantly, and the connected regions are scattered. Therefore, for the loosening identification of type A3 bolts, an angle method and a dilation method are combined. An area threshold is set; only when the area of ​​a connected region exceeds the area threshold is the minimum bounding rectangle drawn. The difference in rotation angles between the minimum bounding rectangles is calculated. If the angle difference is less than or equal to the angle threshold, the bolt is considered tightened. If the angle difference is greater than the angle threshold, the next step is performed: dilation is performed using a convolution kernel of a certain size for the same number of times. The number of connected regions after dilation is calculated. If the number of connected regions is equal to 1, the bolt is considered tightened; if the number of connected regions is greater than 1, the bolt is considered loose. Figure 14 This invention provides a schematic diagram and flowchart of a loosening identification process using the angle method and the expansion method, as provided in an embodiment of the invention.

[0094] Bolts of types c and d are socket head cap screws and Phillips head screws, respectively. Due to the internal holes in the bolts, the marking lines are not continuous straight segments. The marking lines for some fastened bolts consist of shorter and longer line segments. When the shorter and longer segments are approximated as rectangles, calculating the rotation angle of the long side of the rectangle can result in a significant angle difference. Therefore, the angle method is only applicable to some fastened bolts. For bolt marking lines composed of different connected domains, there exists a unique minimum bounding rectangle enclosing all connected domains. When bolts of types c and d are fastened, their minimum bounding rectangle is elongated, with a large aspect ratio. When they are loose, the marking lines are more dispersed, and the aspect ratio of their minimum bounding rectangle is smaller. This method is called the rectangle method. Based on these characteristics, bolts of types c and d are identified as loose using a combination of the angle and rectangle methods. First, the minimum bounding rectangle of the connected domains larger than the area threshold is drawn. The rotation angle difference of the rectangles is calculated. If the angle difference is less than or equal to the angle threshold, the bolt is considered fastened; otherwise, the next step is performed. Draw the minimum bounding rectangle of all connected components, calculate the aspect ratio of the rectangle, set an aspect ratio threshold, and determine the component as tight if the aspect ratio is greater than the threshold; otherwise, determine the component as loose. Figure 15 This invention provides a schematic diagram and flowchart of an angle method + rectangle method recognition process.

[0095] For type B2 bolts, the observation angle is 0°. From this angle, the bolt marking line consists of two rectangles. When the bolt is tightened, the two marking lines are approximately on the same straight line, and the vertical distance between the rectangles is relatively close. When the bolt is loose and the loosening angle is (0°, 90°]∪[270°, 360°), two rectangles can be observed. At this point, the marking lines are no longer on the same straight line, and the vertical distance between the rectangles is greater. When the loosening angle is (90°, 270°), only one rectangle can be observed. Based on these characteristics, type B2 bolts are identified as loose by calculating the distance between the corner points of the rectangles. First, the smallest bounding rectangle of the connected component with an area greater than the area threshold is drawn. When the number of rectangles is less than or equal to 1, it is directly determined to be loose; when the number of rectangles is greater than 1, the corner point distance is judged. A system is established as follows... Figure 11 The rectangular coordinate system shown records the minimum x-coordinate value among the four corner points of each rectangle, and calculates the corner distance between rectangles. A distance threshold is set; if the corner distance is less than or equal to the threshold, the rectangle is considered secure; if the corner distance is greater than the threshold, the rectangle is considered loose. Figure 16 This invention provides a schematic diagram and flowchart of a corner distance method for identification.

[0096] After a three-stage detection process—target detection, image classification, and loosening identification—the bolt tightness status is output. This invention improves the target detection network by subdividing bolt images into eight categories. Based on the geometric features of various bolt anti-loosening lines under different observation perspectives, it proposes multiple identification methods and their combinations, including the angle method, dilation method, rectangle method, and corner distance method. This constructs a multi-view, multi-category adaptive bolt loosening identification model, effectively improving the accuracy and robustness of loosening detection. Simultaneously, by employing a lightweight image classification network and a loosening identification method based on digital image processing technology, the detection speed is significantly improved while meeting actual detection accuracy requirements, and the number of computational parameters and time consumption are reduced.

[0097] In summary, compared to the current mainstream two-stage detection methods, the embodiments of this invention innovatively incorporate image classification tasks. By refining the loosening characteristics of various types of bolts under different observation perspectives, an adaptive loosening detection method is designed, effectively solving the problem of limited research types and perspectives in existing methods, and improving the universality and robustness of the detection method. This provides a reference for improving the efficiency and accuracy of railway train inspection, enhancing the adaptability and robustness of the detection system, reducing manual maintenance workload, ensuring the safe operation of maglev trains, and realizing intelligent railway operation and maintenance. It also has significant reference value for the identification of loosening of key fasteners in multiple fields such as rail transit, aerospace, and bridge construction.

[0098] Furthermore, this invention optimizes the YOLOv8 network structure by adding feature fusion lines, adding Dropout layers, and replacing the loss function, thereby improving the accuracy of bolt positioning and recognition. It also designs adaptive detection methods for the loosening characteristics of different types of bolts, which greatly improves the accuracy of bolt loosening recognition.

[0099] In terms of improving detection efficiency, this invention adopts a lightweight classification network, which effectively reduces the requirements for hardware devices, the number of model parameters and the computation time while ensuring classification accuracy. At the same time, it designs an image processing-based looseness detection method, which significantly improves detection efficiency compared with deep learning detection methods.

[0100] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0101] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0102] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0103] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying loose bolts on the undercarriage of a maglev train based on deep learning, characterized in that, include: Imaging equipment was used to collect images of the undercarriage of the maglev train. After preprocessing, the images were compiled into a dataset. Bolt images were obtained by detecting bolt targets in images from a vehicle undercarriage image dataset using an improved YOLOv8 network. The bolt images are classified using RepViT, a lightweight network that incorporates lightweight ViT design principles. The bolt image is subjected to color space conversion, image binarization, morphological processing and connected component filtering to extract the anti-loosening line. Based on the bolt anti-loosening line, the loosening characteristics of different types of bolts under multiple views are used to determine whether the bottom bolts in the bolt image are loose.

2. The method according to claim 1, characterized in that, The method involves using imaging equipment to acquire images of the undercarriage of the maglev train, and then preprocessing these images to create an undercarriage image dataset, which includes: Imaging equipment was used to collect images of the undercarriage of the maglev train on-site. These images contained bolt targets of multiple scales, spatial orientations, and types. All the undercarriage images formed an undercarriage image dataset. Image annotation tools were used to label the bolts in the undercarriage images of the dataset, annotating the position information of the bolts in each undercarriage image. The imgaug image enhancement library was used to preprocess the undercarriage images in the dataset by performing grayscale conversion, random region occlusion, Gaussian noise introduction, dynamic brightness adjustment, Gaussian filtering perturbation, and affine space transformation. The preprocessed and enhanced undercarriage image dataset was used as the input for model training.

3. The method according to claim 1, characterized in that, The method of detecting bolt targets in vehicle undercarriage images from a dataset using an improved YOLOv8 network to obtain bolt images includes: An improved YOLOv8 network was constructed, incorporating the addition of feature fusion lines, a Dropout layer, and a replacement loss function. During training, the input image of the undercarriage of a vehicle was 640*640 pixels. The training process employed the Stochastic Gradient Descent (SGD) optimization algorithm. The YOLOv8 network model was deployed on the PyTorch framework. The anchor box is denoted as B = [x, y, w, h], and the target box as B0. gt =[x gt y gt w gt h gt A dynamic method was used to calculate the IoU loss in the category prediction loss of the YOLOv8 network; L WIoU =R WIoU L IoU Among them W g H g L is the size of the minimum bounding box. IoU For the loss term of IoU, L WIoU R represents the total loss of WIoU. WIoU For WIoU's penalty item, R WIoU ∈[1, e), the loss term L of the ordinary mass anchor frame IoU Magnification, L IoU ∈[0,1], making the R of the high-quality anchor frame WIoU Reduce the focus on the distance to the center point when the anchor frame overlaps well with the target frame; After iterative training, the trained YOLOv8 network is obtained; The image of the undercarriage of the maglev train is input into the trained improved YOLOv8 network to locate and identify bolts, output the bounding box information of the bolts, and output the bolt image after cropping and removing blurry samples.

4. The method according to claim 1, 2, or 3, characterized in that, The described classification process for the bolt images using RepViT, which incorporates lightweight ViT design principles, includes: Based on the location characteristics of the bolts, a four-level classification standard (ad) was established. Then, based on the changes in the observed elevation angle of the bolts, bolts of category a were further subdivided into three categories: elevation angle = 90°; elevation angle = 0°; and elevation angle ∈ (0°, 90°). Bolts of category b were further subdivided into three categories: depression angle = 90°; depression angle = 0°; and depression angle ∈ (0°, 90°). A total of eight categories of bolt images were obtained: a1, a2, a3, b1, b2, b3, c, and d. Bolt images were enhanced using grayscale enhancement, rectangular discarding, Gaussian noise reduction, vertical flipping, horizontal flipping, brightness enhancement, Gaussian blurring, average blurring, and affine transformation image enhancement methods. The enhanced bolt images were then input into the lightweight network RepViT, which incorporates lightweight ViT design principles, for image classification. The image size was 224*224. The AdamW optimizer was used, combined with a cosine annealing learning rate scheduling strategy. Data augmentation included Mixup, AutoAugment, and random erasure. Label smoothing was used to improve generalization ability. The algorithm was deployed using the PyTorch framework, and the RepViT network output the bolt classification results.

5. The method according to claim 4, characterized in that, The process of performing color space conversion, image binarization, morphological processing, and connected component filtering on the bolt image to extract the bolt anti-loosening line includes: The bolt image is converted from RGB space to HSV space. The bolt image in HSV space is then binarized using an HSV threshold to obtain a binarized bolt image. The marking line portion in the binarized bolt image is extracted to obtain a bolt marking line image. After performing erosion and dilation operations on the bolt marking line image, small connected components in the bolt marking line image are removed according to a set connected component area range to extract the bolt anti-loosening line.

6. The method according to claim 5, characterized in that, The method described above, which uses a bolt anti-loosening line to determine whether the undercarriage bolts in a bolt image are loose based on the loosening characteristics of different types of bolts under multiple viewing angles, includes: For bolt images of types a1, b1, and b3, the minimum bounding rectangle of all connected components in the bolt marking line image is drawn based on the bolt anti-loosening line. A rectangular coordinate system is established on the marking line image. The rotation angle of the connected component is taken as the rotation angle in the positive x-axis direction until it coincides with the long side of the minimum bounding rectangle. The rotation angle range is [0°, 180°]. The angle difference abs between the rotation angles of each minimum bounding rectangle is calculated. An angle threshold is set. If the angle difference is greater than the angle threshold, the bolt is judged to be loose; otherwise, the bolt is judged to be tight. For bolt images of type a2, the bolt marking line image is expanded a certain number of times based on the bolt anti-loosening line through a convolution kernel of a certain size. The number of connected components in the expanded bolt marking line image is calculated. If the number of connected components is equal to 1, the bolt is determined to be tight; if the number of connected components is greater than 1, the bolt is determined to be loose. For bolt images of type A3, the angle difference between the rotation angles of each minimum bounding rectangle is calculated based on the bolt anti-loosening line. An angle threshold is set. If the angle difference is less than or equal to the angle threshold, the bolt is determined to be tight. If the angle difference is greater than the angle threshold, the bolt is dilated by a convolution kernel of a certain size for the same number of times. The number of connected components in the dilated bolt marking line image is calculated. If the number of connected components is equal to 1, the bolt is determined to be tight. If the number of connected components is greater than 1, the bolt is determined to be loose. For bolts of types c and d, the minimum bounding rectangle of the connected components in the bolt marking line image that are larger than the area threshold is drawn based on the bolt anti-loosening line. The angle difference of the rotation angle between each minimum bounding rectangle is calculated. If the angle difference is less than or equal to the angle threshold, the bolt is determined to be tight. If the angle difference is greater than the angle threshold, the minimum bounding rectangle of all connected components is drawn, and the aspect ratio of the minimum bounding rectangle is calculated. An aspect ratio threshold is set. If the aspect ratio of the minimum bounding rectangle is greater than the aspect ratio threshold, the bolt is determined to be tight. If the aspect ratio of the minimum bounding rectangle is less than the aspect ratio threshold, the bolt is determined to be loose. For type b2 bolts, the minimum bounding rectangle of the connected region with an area greater than the bolt anti-loosening line is drawn. When the number of minimum bounding rectangles is less than or equal to 1, the bolt is judged to be loose. When the number of minimum bounding rectangles is greater than 1, a rectangular coordinate system is established on the bolt marking line image, the minimum x-coordinate value of the four corner points of each minimum bounding rectangle is recorded, and the corner point distance between the rectangles is calculated. A distance threshold is set. If the corner point distance is less than or equal to the distance threshold, the bolt is judged to be tight; if the corner point distance is greater than the distance threshold, the bolt is judged to be loose.

Images