Bolt looseness detection method, device and equipment and storage medium

Abstract

The embodiment of the invention relates to the field of train safety detection, and discloses a bolt looseness detection method, device and equipment and a storage medium, and the method comprises the steps: obtaining a train image, and at least part of the train image comprises an area where a bolt is located; inputting the train image into a positioning identification model to identify an area where the bolt is located, and determining area boundary information of the area where the bolt is located; segmenting the train image according to the area boundary information of the area where the bolt is located, and generating an image of the area where the bolt is located; and inputting the image of the area where the bolt is located into the bolt looseness detection model for bolt looseness detection. According to the method, the positioning recognition model is utilized to position and segment the area where the bolt is located for the input train image, and then the bolt looseness detection model is utilized to accurately detect the looseness and falling of the bolt for the segmented image, so that the area where the bolt is located is determined firstly, and then the bolt looseness condition in the area where the bolt is located is detected. The efficiency and the detection precision are improved.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of train safety detection, and in particular to a bolt loosening detection method, device, equipment and storage medium. Background Art

[0002] When the train is running at high speed, the loosening or falling off of the axle box cover bolts may cause serious safety accidents. The current detection method mainly relies on manual detection, which is inefficient and has safety hazards. In recent years, with the development of computer vision technology, deep learning has made significant progress in the field of target detection, especially small target detection. For the fault detection of train bolts, C. Li et al. proposed a training mechanism that performs a self-update mechanism during the training process, which can accurately distinguish the type of bolts, but it cannot distinguish multiple fault types and has a low accuracy rate. Z. Wang et al. used resnet to confirm the number of bolts and then determine whether there are any bolts missing, but it cannot determine whether the bolts are loose, which may bring hidden dangers. However, automatic detection may miss many loose bolts. Q. Geng et al. studied an automatic visual inspection system based on spatial scale and proposed a method for checking the status of rail side plate bolts using a normalization method. J. Sun et al. proposed using a convolutional neural network (CNN) to calculate the distance between the bolt cap and the mounting surface, and compared it with the reference value to determine whether the bolt is loose. This method can effectively detect the loosening and loss of bolts, but it is difficult for this detection method to detect tiny looseness. In summary, the existing calculation methods for train bolt detection have technical problems such as low detection accuracy, low efficiency, and low recognition accuracy in complex backgrounds. Summary of the invention

[0003] The purpose of the present invention is to at least provide a method, device, equipment and storage medium for detecting loose bolts, which can at least solve the technical problems of low bolt detection accuracy and low efficiency, and at least achieve fast and accurate detection of loose bolts, thereby improving the safety and detection efficiency of train operation.

[0004] In order to solve the above technical problems, at least one embodiment of the present application provides a method for detecting bolt loosening, comprising:

[0005] Acquire a train image, wherein at least a portion of the train image includes an area where the bolt is located;

[0006] Inputting the train image into a positioning and recognition model to identify the area where the bolts are located, and determining the area boundary information of the area where the bolts are located;

[0007] Segmenting the train image according to the region boundary information of the region where the bolts are located to generate an image of the region where the bolts are located;

[0008] The image of the area where the bolt is located is input into a bolt loosening detection model to perform bolt loosening detection.

[0009] At least one embodiment of the present application further provides a bolt loosening detection device, comprising:

[0010] An acquisition module, used for acquiring a train image, wherein at least a part of the train image includes an area where the bolts are located;

[0011] An identification module, used for inputting the train image into a positioning identification model to identify the area where the bolts are located, and determining the area boundary information of the area where the bolts are located;

[0012] A segmentation module, used to segment the train image according to the regional boundary information of the area where the bolts are located, and generate an image of the area where the bolts are located;

[0013] The detection module is used to input the image of the area where the bolt is located into the bolt loosening detection model to perform bolt loosening detection.

[0014] At least one embodiment of the present application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned bolt loosening detection method.

[0015] At least one embodiment of the present application further provides a computer-readable storage medium storing a computer program, wherein the computer program implements the above-mentioned bolt loosening detection method when executed by a processor.

[0016] The bolt loosening detection method provided in the embodiment of the present application first uses a positioning and recognition model to locate and segment the area where the bolts are located (for example, the axle box cover and the shock absorber) of the input train image, and then uses a bolt loosening detection model in the segmented image to accurately detect the loosening and falling off of the bolts, thereby first determining the area where the bolts are located, and then detecting the looseness of the bolts in the area where the bolts are located, so as to improve efficiency and detection accuracy.

[0017] In some optional embodiments, the step of inputting the train image into the positioning recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located includes:

[0018] Scanning the train image through a region proposal network to obtain multiple candidate regions;

[0019] Extracting features from each candidate region to obtain feature information of each candidate region;

[0020] The feature information of each candidate area is classified and a bounding box is regressed to determine the area boundary of the bolt area.

[0021] By obtaining multiple candidate regions through the region proposal network, classifying the features of the multiple candidate regions and regressing the bounding boxes, the regional boundaries of the bolt area can be accurately determined.

[0022] In some optional embodiments, the step of inputting the image of the area where the bolt is located into a bolt loosening detection model to perform bolt loosening detection includes:

[0023] Extracting feature information of the image of the area where the bolt is located by a convolutional neural network to obtain feature maps at each depth;

[0024] Fuse the feature maps at each depth to obtain a fused feature map;

[0025] Based on the fused feature map, the prediction head of the bolt loosening detection model outputs the bolt loosening category probability, bounding box coordinates and confidence of the corresponding position, wherein the bolt loosening category includes at least two of the following: the bolt is not loose, the bolt is loose and the bolt falls off;

[0026] The bolt loosening detection result is determined according to the bolt loosening category probability, the bounding box coordinates and the confidence level.

[0027] In this way, the model-based prediction head outputs the loose bolt category probability, bounding box coordinates, and confidence level of the corresponding position. That is, through feature fusion and the use of multiple prediction heads, it can capture multi-scale information in the image and output accurate detection results to ensure the accuracy of the detection results.

[0028] In some optional embodiments, the step of determining the bolt loosening detection result according to the bolt loosening category probability, bounding box coordinates and confidence level includes:

[0029] Decoding the bounding box coordinates to obtain image coordinates of the bounding box on the image of the area where the bolt is located;

[0030] Calculating the degree of overlap between the bounding box and the adjacent bounding box according to the image coordinates of the bounding box on the image of the area where the bolt is located;

[0031] The bounding boxes whose overlap exceeds a preset threshold are screened by a non-maximum suppression method, and the bounding boxes with the highest confidence are retained;

[0032] A bolt loosening detection result is determined based on the retained bounding box and the corresponding bolt loosening category probability and confidence.

[0033] In this way, the accuracy and reliability of the detection results are further improved through bounding box decoding and non-maximum suppression.

[0034] In some optional embodiments, before the step of inputting the train image into the positioning recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located, the step further includes:

[0035] Acquire a sample train image including an area where a bolt is located and annotating the area where the bolt is located;

[0036] The Faster R-CNN model is trained based on the sample train images to obtain a positioning recognition model.

[0037] By training the Faster R-CNN model, a positioning and recognition model is obtained. Faster R-CNN can handle multi-target and multi-category target detection tasks well, and has high efficiency when processing high-resolution images; that is, in this embodiment, the train image is input into the positioning and recognition model, which can quickly and accurately realize the identification of the areas where multiple bolts are located.

[0038] In some optional embodiments, before the step of inputting the image of the area where the bolt is located into the bolt loosening detection model for bolt loosening detection, the step further includes:

[0039] Acquire an image of a region where sample bolts are located, including loose bolts and corresponding annotations of loose bolt categories;

[0040] An initial target recognition model is constructed based on the YOLO algorithm, and the image of the area where the sample bolts are located is input into the initial target recognition model for optimization training to obtain a bolt loosening detection model.

[0041] The bolt loosening detection model is constructed through the YOLO algorithm. The YOLO algorithm can process images in real time or near real time, and can quickly and accurately detect the looseness of bolts.

[0042] In some optional embodiments, before the step of fusing the feature maps of each depth to obtain a fused feature map, the step further includes:

[0043] The channel and spatial weights of the feature map are adjusted through an attention mechanism module.

[0044] In order to improve the detection performance, the attention mechanism is introduced in this embodiment to adjust the channel and spatial weights of the feature map. The attention mechanism module enhances the model's attention to important features by adaptively adjusting the channel and spatial weights of the feature map. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0046] Figure 1 is a flow chart of a bolt loosening detection method provided by an embodiment of the present application;

[0047] Figure 2 is a flow chart of a bolt loosening detection method provided by another embodiment of the present application;

[0048] Figure 3 is a flowchart of key area positioning and segmentation provided by another embodiment of the present application;

[0049] Figure 4 is a flow chart of bolt loosening and falling-off detection provided by another embodiment of the present application;

[0050] Figure 5 is a schematic diagram of a bolt loosening detection device provided by another embodiment of the present application;

[0051] Figure 6 is a schematic diagram of an optical camera device of an acquisition module provided by another embodiment of the present application;

[0052] Figure 7 It is a structural schematic diagram of an electronic device provided by another example of the present application. DETAILED DESCRIPTION

[0053] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present application can also be implemented. The division of the following embodiments is for the convenience of description, and the specific implementation of the present application should not be construed as any limitation, and the various embodiments can be combined and referenced with each other under the premise of no contradiction.

[0054] The present invention proposes a method for detecting bolt loosening. The implementation details of the method for detecting bolt loosening of this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for implementing this solution.

[0055] Embodiment 1:

[0056] The bolt loosening detection method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 1 As shown, including:

[0057] Step 111 : acquiring a train image, wherein the train image at least partially includes an area where the bolts are located.

[0058] Specifically, a plurality of train images are acquired through a visual shooting system, wherein at least part of the plurality of train images include a region where bolts are located; further, the region where bolts are located may be a region where bolts are concentrated, such as a train axle box cover, a shock absorber region, etc. In this embodiment, the region where bolts are located is taken as an example of a train axle box cover and a shock absorber region.

[0059] In some examples, the train images are obtained through a train database.

[0060] Step 112: input the train image into a positioning and recognition model to identify the area where the bolts are located, and determine the area boundary information of the area where the bolts are located.

[0061] Specifically, a positioning and recognition model is pre-trained, and the train image is input into the positioning and recognition model to identify the area where the bolts are located, and the area boundary information of the area where the bolts are located is output.

[0062] In some examples, a suitable deep learning model or target detection model is selected and trained with a sample data set to obtain a positioning and recognition model. The sample data set includes a sample train image of the area where the bolt is located, and the location and boundary information of the area where the bolt is located is marked in the sample train image; the sample train image is input into the deep learning model or target detection model for training. During the training process, the parameters and structure of the model are adjusted according to the difference between the output of the model and the labeled data to improve the recognition accuracy of the model; and the model is optimized through cross-validation, regularization, data enhancement and other technologies to improve its generalization ability and robustness.

[0063] Using the trained positioning and recognition model, the train image to be recognized is input into the positioning and recognition model to identify the position of the bolt area and determine the regional boundary information of the bolt area, that is, determine the regional boundary information of the axle box cover and the shock absorber.

[0064] Step 113 , segmenting the train image according to the region boundary information of the region where the bolts are located, and generating an image of the region where the bolts are located.

[0065] Specifically, according to the determined regional boundary information of the bolt area, the precise boundary of the bolt area is defined in the image, which can be achieved by drawing polygons, rectangles or other shapes on the image, which are closely surrounding the bolt area. According to the determined boundary, the train image is segmented, and the partial area defined by the boundary, that is, the bolt area, is extracted from the train image, and the segmented image area is saved as a new image file, that is, the bolt area image is obtained.

[0066] By segmenting the train image, an image of the area where the bolts are located is generated, and then the image of the area where the bolts are located is used for bolt loosening detection to further improve the efficiency and accuracy of bolt loosening detection.

[0067] Step 114: input the image of the area where the bolts are located into a bolt loosening detection model to perform bolt loosening detection.

[0068] Specifically, a bolt loosening detection model is pre-trained, and an image of the area where the bolts are located is input into the bolt loosening detection model to perform bolt loosening detection, and output bolt loosening detection information.

[0069] In some examples, a suitable deep learning model or target detection model is selected and trained with a sample data set to obtain a bolt loosening detection model. The sample data set is an image of the area where the bolts are located, and the bolt loosening conditions are marked in the image of the area where the bolts are located; further, the bolt loosening conditions include three conditions: the bolts are not loose, the bolts are loose, and the bolts fall off; the image of the area where the bolts are located is input into the deep learning model or target detection model for training. During the training process, the parameters and structure of the model are adjusted according to the difference between the output of the model and the labeled data to improve the recognition accuracy of the model; and the model is optimized through cross-validation, regularization, data enhancement and other technologies to improve its generalization ability and robustness.

[0070] Using the trained bolt loosening detection model, the image of the area where the bolts to be identified are located is input into the bolt loosening detection model to identify the looseness of the bolts.

[0071] In this embodiment, the positioning and recognition model is first used to locate and segment the area where the bolts are located (for example, the axle box cover and the shock absorber) of the input train image, and then the bolt loosening detection model is used in the segmented image to accurately detect the loosening and falling off of the bolts, so as to first determine the area where the bolts are located, and then detect the looseness of the bolts in the area where the bolts are located, so as to improve efficiency and detection accuracy.

[0072] In some embodiments, the step of inputting the train image into the positioning recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located includes:

[0073] Scanning the train image through a region proposal network to obtain multiple candidate regions;

[0074] Extracting features from each candidate region to obtain feature information of each candidate region;

[0075] The feature information of each candidate area is classified and a bounding box is regressed to determine the area boundary of the bolt area.

[0076] Specifically, the train image is scanned by a region proposal network to obtain multiple candidate regions; wherein the region proposal network (RPN) is used to generate region proposals that may contain target objects, and these region proposals can efficiently cover objects that may exist in the image, providing a basis for subsequent accurate detection. Feature extraction will be performed on each candidate region, and further, a convolutional neural network (CNN) is used to extract image features to obtain feature information of each candidate region; wherein the convolutional neural network (CNN) can automatically learn and extract key features in the image, such as edges, textures, etc. After multiple layers of convolution and pooling operations, CNN outputs a feature map containing image feature information; classification and bounding box regression are performed based on the feature information of each candidate region, wherein classification is used to determine whether the candidate region includes the area where the bolts are located (i.e., the train axle box cover or shock absorber area), and bounding box regression is to adjust the bounding box to more accurately locate the bolts. Further, according to the classification results, the train axle box cover area or shock absorber area is screened out; according to the bounding box information obtained by bounding box regression, the precise position and boundary of the train axle box cover area are determined.

[0077] In some embodiments, before the train image is input into the positioning recognition model, the train image is also preprocessed to make it meet the input specifications of the model. Furthermore, the operation of preprocessing the train image includes at least one of scaling, cropping, and normalizing the image.

[0078] In this embodiment, the train image is identified and located based on the positioning recognition model, and the regional boundary of the area where the bolts are located can be accurately output, which is convenient for subsequent judgment on the looseness of the bolts in the area where the bolts are located.

[0079] In some embodiments, the step of inputting the image of the area where the bolt is located into a bolt loosening detection model to perform bolt loosening detection includes:

[0080] Extracting feature information of the image of the area where the bolt is located by a convolutional neural network to obtain feature maps at each depth;

[0081] Fuse the feature maps at each depth to obtain a fused feature map;

[0082] Based on the fused feature map, the prediction head of the bolt loosening detection model outputs the bolt loosening category probability, bounding box coordinates and confidence of the corresponding position, wherein the bolt loosening category includes at least two of the following: the bolt is not loose, the bolt is loose and the bolt falls off;

[0083] The bolt loosening detection result is determined according to the bolt loosening category probability, the bounding box coordinates and the confidence level.

[0084] Specifically, the image of the area where the bolt is located is input into the bolt loosening detection model. The model uses convolutional neural networks to extract image features. These networks gradually extract features from low-level to high-level features from the image through structures such as convolutional layers, activation functions, and pooling layers to obtain feature maps of each depth, where the feature maps of each depth contain information of different levels; the feature maps of each depth are fused, and further, it can be achieved through feature pyramid networks (FPN), path aggregation networks (PAN) or other feature fusion technologies. The fused feature maps contain richer information, which is helpful for the subsequent detection of bolt looseness. On the fused feature maps, a prediction head is set to output the detection results of bolt loosening; the prediction head is usually implemented through a fully connected layer and a softmax function, which converts the output into a probability distribution; the prediction head also outputs the bounding box coordinates of the bolt in the image, which represent the position and size of the bolt. Each prediction result will have a confidence score, indicating the reliability of the prediction result. The higher the confidence score, the more reliable the prediction result.

[0085] Each prediction head is connected to a feature map of a specific scale. This means that each prediction head generates predictions based on the feature map it is connected to. Since feature maps have different scales and resolutions, each prediction head is able to capture and detect objects of different sizes. For each prediction head, the class probability, bounding box coordinates, and confidence score are output. The bolt loosening class probability refers to the probability value of the bolt not loosening or the bolt loosening or bolt falling off class at this location. It is usually normalized by the Softmax function or the Sigmoid function so that the sum of the probability values ​​is 1. The bounding box coordinates represent the position of the target object in the image predicted by the model. In YOLOv8, the bounding box is usually represented by four values: x_min, y_min, x_max, y_max, which correspond to the x-coordinate, y-coordinate of the upper left corner of the bounding box and the x-coordinate, y-coordinate of the lower right corner. These coordinate values ​​are decoded according to the size of the feature map and the output of the model, and need to be mapped back to the size of the original image. The confidence score indicates how certain the model is about the prediction result. It reflects the possibility that the model believes that the predicted target object does exist. Unlike the category probability, the confidence score is independent of the category. In YOLOv8, the confidence score is usually obtained by calculating the intersection over union (IoU) between the predicted bounding box and the true bounding box, which can also be understood as the model's evaluation of the accuracy of the predicted bounding box.

[0086] The bolt loosening detection result is determined according to the bolt loosening category probability, the bounding box coordinates and the confidence level.

[0087] In this embodiment, the bolt loosening condition in the image is accurately detected by the bolt loosening detection model, thereby ensuring the accuracy of the bolt loosening condition detection.

[0088] In some embodiments, the step of determining the bolt loosening detection result according to the bolt loosening category probability, bounding box coordinates and confidence level includes:

[0089] Decoding the bounding box coordinates to obtain image coordinates of the bounding box on the image of the area where the bolt is located;

[0090] Calculating the overlap of adjacent bounding boxes according to the image coordinates of the bounding boxes on the image of the area where the bolts are located;

[0091] The bounding boxes whose overlap exceeds a preset threshold are screened by a non-maximum suppression method, and the bounding boxes with the highest confidence are retained;

[0092] A bolt loosening detection result is determined based on the retained bounding box and the corresponding bolt loosening category probability and confidence.

[0093] Specifically, the bounding box coordinates output by the prediction head are offsets relative to the size of the feature map, so they need to be decoded into coordinates on the actual image. This usually involves multiplying the offset by the scaling factor of the feature map to the original image, and adding the corresponding offset (such as the coordinates of the upper left corner of the grid) to obtain the exact position of the bounding box on the original image. Further, in target detection, since the model may predict multiple overlapping bounding boxes on the same target, non-maximum suppression (NMS) is needed to filter out these redundant boxes. The basic idea of ​​NMS is: for each target category, select the bounding box with the highest confidence, and suppress (i.e. delete) the bounding box with an overlap exceeding a certain threshold and a lower confidence. In this way, each target ultimately retains only one most representative bounding box; the bolt loosening detection result is determined based on the retained bounding box and the corresponding bolt loosening category probability and confidence. In this embodiment, through feature fusion and the use of multiple prediction heads, multi-scale information in the image can be captured and accurate detection results can be output. Bounding box decoding and non-maximum suppression further improve the accuracy and reliability of the detection results.

[0094] In some embodiments, before the step of inputting the train image into the positioning recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located, the step further includes:

[0095] Acquire a sample train image including an area where a bolt is located and annotating the area where the bolt is located;

[0096] The Faster R-CNN model is trained based on the sample train images to obtain a positioning recognition model.

[0097] Specifically, Faster R-CNN is an object detection model that is further optimized based on R-CNN and Fast R-CNN. Its core idea is to quickly generate candidate regions by introducing a region proposal network (RPN) and share convolutional features with the detection network, thereby greatly improving the detection speed and accuracy. Among them, the Faster R-CNN model consists of three parts: a feature extraction network (such as VGG16, ResNet, etc.), a region proposal network (RPN), and a classification regression network (RoIHead). The Faster R-CNN model usually uses pre-trained deep convolutional neural networks (such as VGG, ResNet, etc.) as feature extraction networks. These networks have been trained on large-scale image classification tasks and have good feature extraction capabilities; RPN is one of the core components of Faster R-CNN and is used to generate candidate regions (Region of Interest, RoI); RPN generates a large number of candidate regions by sliding a small window on the feature map output by the feature extraction network and predicting the foreground / background score and position offset of the candidate region corresponding to each window; RoIHead is responsible for classifying and adjusting the position of the candidate regions generated by RPN. It first uses RoIPooling to map candidate regions of different sizes to a feature map of fixed size, and then uses a fully connected layer for classification and regression operations.

[0098] Specifically, the training of the positioning recognition model includes the following steps:

[0099] First, a sample data set including train images is obtained, wherein the train images include annotations of areas where bolts are located and the looseness of bolts in each area.

[0100] In this embodiment, the area where the bolts are located is the axle box cover and shock absorber area, and the bolt loosening conditions include three categories: bolts are not loose, bolts are loose, and bolts are fallen off. The positions of the axle box cover and the shock absorber, as well as the loosening conditions of the axle box cover bolts and the loosening conditions of the shock absorber bolts are marked in the train image.

[0101] In some examples, after obtaining the train images, the images need to be preprocessed to improve the diversity of the images and facilitate the subsequent training of the machine learning model based on the train images. Specifically, the preprocessing operations on the images include scaling, cropping, and normalizing the images.

[0102] Second, model training.

[0103] The Faster R-CNN model is trained using a sample dataset of train images with labeled axle box cover position information and shock absorber position information.

[0104] The preprocessed train image is input into the trained Faster R-CNN model. The region proposal network (RPN) generates a series of candidate regions (anchor boxes) on the feature map through a sliding window. The candidate regions are screened according to the intersection over union (IoU) between the anchor box and the true annotation box, and the candidate regions with higher IoU are retained as regions of interest (ROIs). The ROIs are mapped to the feature map, and the corresponding feature vectors are extracted. The feature vectors are classified (axle box cover or shock absorber) and regressed (accurate region boundaries) using a convolutional neural network (CNN). According to the classification results, the category (axle box cover or shock absorber) to which each ROI belongs is determined. According to the regression results, the bounding box of the ROI is adjusted to make it surround the target area more accurately. The model will learn how to identify the features of the train axle box cover and shock absorber, and generate accurate region boundary information.

[0105] Furthermore, in order to ensure the accuracy of the predictions of the trained model, a test data set is also used to evaluate the model and calculate indicators such as accuracy and recall. Based on the evaluation results, the model will be fine-tuned or retrained to improve the performance of the model.

[0106] In some embodiments, before the step of inputting the image of the area where the bolt is located into the bolt loosening detection model for bolt loosening detection, the step further includes:

[0107] Acquire an image of a region where sample bolts are located, including loose bolts and corresponding annotations of loose bolt categories;

[0108] An initial target recognition model is constructed based on the YOLO algorithm, and the image of the area where the sample bolts are located is input into the initial target recognition model for optimization training to obtain a bolt loosening detection model.

[0109] Specifically, YOLO (You Only Look Once) is a target detection algorithm based on deep neural networks, which is used to identify and locate multiple objects in real time in images or videos. The working principle of the YOLO algorithm includes: preprocessing, that is, a series of preprocessing is performed on the input image, such as normalization, scaling, etc., to ensure the standardization and uniformity of the input data; feature extraction, a convolutional neural network (CNN) is constructed through deep learning methods to extract features from the original image, and extract the contour, texture, color and other information of the object in the image. These features are used to assist the task of target detection and classification; feature mapping, the extracted features are mapped to different feature maps, and the grid on each feature map is responsible for detecting objects of different sizes and positions; target regression classification, the anchor box method is used to locate the object in the image, and the regression method and classification method are used to determine the position and category of the object. The regression method predicts the center point coordinates, width and height and other attributes of the object, while the classification method classifies the object into pre-defined categories.

[0110] In some embodiments, the YOLO v8 model is selected. YOLO v8 is the latest version in the YOLO (You Only Look Once) series and is a deep learning model for target detection and image segmentation. The YOLO v8 model structure is mainly divided into three parts: the backbone network (backbone), the feature enhancement network (neck) and the detection head (head). Among them, the backbone network: YOLO v8 uses Darknet53 or other advanced convolutional neural networks as the backbone network to extract features from the image. The backbone network gradually converts the image into a high-level feature representation by stacking convolutional layers and pooling layers. Feature enhancement network: The feature enhancement network is responsible for further fusing and enhancing the features extracted by the backbone network. YOLO v8 introduces a variety of feature fusion methods, such as FPN (Feature Pyramid Network) and PANet (Path Aggregation Network), to improve the model's detection ability for targets of different scales; FPN is used to fuse feature maps of different resolutions, while PAN strengthens the flow of semantic information, making positioning more accurate. At the same time, modules such as SPP (Spatial Pyramid Pooling) and SAM (Spatial Attention Module) are introduced to improve the receptive field and feature expression ability of the model. Detection head: The detection head is the core part of the YOLO v8 model, responsible for predicting the location and category of the target based on the feature map output by the feature enhancement network. YOLO v8 uses an anchor-free detection head to improve the accuracy of detection, especially when dealing with small-sized objects. At the same time, new loss functions and label assignment strategies are introduced to better balance positive and negative samples and improve the learning efficiency of the model during training.

[0111] The training of the YOLO model includes obtaining an image of the area where the sample bolts are located, including loose bolts and corresponding annotations of the categories of loose bolts. In this embodiment, the regional images of the axle box cover and the shock absorber after segmentation are used as an example for explanation, wherein the categories of loose bolts have been annotated in the regional images of the axle box cover and the shock absorber. The data set is divided into a training set, a validation set, and a test set. The training set is used to train the model, the validation set is used to adjust the model parameters and monitor the training process, and the test set is used to evaluate the performance of the model; the model is trained using the training set, and during the training process, the model will continuously learn the features in the image and the location information of the target; and the training process of the model is monitored through the validation set, and parameters such as the learning rate are adjusted to improve performance.

[0112] In some examples, YOLO v8 uses deep convolutional neural networks such as CSPDarknet53 to extract image features. These networks gradually extract low-level to high-level features from images through structures such as convolutional layers, activation functions, and pooling layers; fuse each feature, specifically, using feature pyramid networks (FPN) and path aggregation networks (PAN) to fuse features. FPN fuses feature maps of different resolutions through bottom-up paths and top-down paths, thereby enhancing the model's ability to detect objects of different scales; PAN, based on FPN, further strengthens the information transfer between features through additional bottom-up paths, making the model more accurate in locating objects. For feature maps of different scales, YOLOv8 uses multiple prediction heads to output prediction results. Each prediction head outputs the category probability, bounding box coordinates, and confidence scores of the corresponding position; the prediction head of YOLOv8 does not rely on predefined anchor boxes, but directly regresses the bounding box. This method reduces the model's dependence on the anchor box and improves the accuracy of small object detection. Among them, the bounding box output by the model is usually an offset relative to the image size. In order to obtain the actual bounding box coordinates, these offsets need to be decoded; in the decoded bounding box, there may be multiple overlapping boxes. The (non-maximum suppression) NMS algorithm compares the confidence scores of these boxes, retains the box with the highest confidence, and filters out other overlapping boxes; YOLOv8 ultimately outputs the category label of each target object (such as whether the bolt is loose), the confidence score (indicating the reliability of the prediction result), and the bounding box position (indicating the specific location of the target object in the image).

[0113] In some embodiments, the step of constructing an initial target recognition model based on the YOLO algorithm, inputting the image of the area where the sample bolts are located into the initial target recognition model for optimization training, and obtaining a bolt loosening detection model further includes:

[0114] The channel and spatial weights of the feature map are adjusted through an attention mechanism module.

[0115] Specifically, the CBAM attention mechanism module is used to adjust the channel and spatial weights of the feature map; CBAM (Convolutional Block Attention Module) attention mechanism: CBAM is an attention module used to improve the performance of convolutional neural networks. It enhances the expressiveness of the model by focusing on the more important parts of the input feature map. CBAM includes two parts: Channel Attention Module and SpatialAttention Module. Channel Attention Module: This module focuses on the weights of different channels in the feature map. By adjusting these weights, the model can pay more attention to channels that are useful for detection tasks and ignore those unimportant channels; specifically, the channel weights are adjusted through global average pooling and fully connected layers to generate a channel attention map, emphasizing important channels and suppressing unimportant channels. Spatial Attention Module: This module focuses on the weights of different spatial positions in the feature map. By adjusting these weights, the model can pay more attention to spatial locations that are useful for detection tasks and ignore those unimportant locations; specifically, combining maximum pooling and average pooling to generate a spatial attention map improves the model's visual task processing capabilities, emphasizing important parts of the image and suppressing unimportant parts.

[0116] The specific working principle of the channel attention module is as follows: (1) Input: define the input feature map as F, whose shape is H×W×C, where H is the height, W is the width, and C is the number of channels; (2) Global average pooling and global maximum pooling: perform global average pooling on the input feature map F in the spatial dimensions (H and W) to obtain a one-dimensional vector F with a length of C average Each element in this vector represents the average activation value of the corresponding channel in the entire space; similarly, the input feature map F is globally max-pooled in the spatial dimension to obtain a one-dimensional vector F with a length of C. max Each element in this vector represents the maximum activation value of the corresponding channel in the entire space; these two vectors F average and F max They capture the global information of each channel and provide input for the subsequent multi-layer perceptron; (3) Multi-layer perceptron (MLP): average and F maxThey are passed through two independent shared multi-layer perceptrons (MLPs) respectively. Each MLP usually contains two fully connected layers (FC layers), where the first FC layer is used for dimensionality reduction (usually reducing the number of channels to one-r of the original, r is the dimensionality reduction ratio), and the second FC layer is used for dimensionality increase (restoring to the original number of channels C). After each MLP, the ReLU activation function is applied to increase nonlinearity. The outputs of the two MLPs are weighted summed (usually by simple addition or weighted addition) to obtain a one-dimensional vector of length C, which is the channel attention map M. c (4) Weighting: Finally, the input feature map F is multiplied by M(c) to obtain the enhanced feature map.

[0117] The specific working principle of the spatial attention module is as follows: (1) Input: The input is the feature map F′ processed by the channel attention module, and its shape is H×W×C, where H is the height, W is the width, and C is the number of channels; (2) Inter-channel aggregation: The input feature map F′ is average pooled in the channel dimension to obtain a spatial description matrix F with a shape of H×W×1. average ′. Each element in this matrix represents the average activation value of the corresponding spatial position in all channels; similarly, the input feature map F′ is pooled in the channel dimension to obtain a spatial description matrix F with a shape of H×W×1 max ′. Each element in this matrix represents the maximum activation value of the corresponding spatial position on all channels; these two matrices F average′ and F max ′ jointly captures the global information of each spatial position (although aggregated in the channel dimension), providing input for the subsequent spatial attention map generation; (3) Convolution operation: Convolution operation: Convolution operation: Conversion of two spatial description matrices F average′ and F max ′ is concatenated in the channel dimension to obtain a matrix with a shape of H×W×2. ; This concatenated matrix is ​​convolved through a 7×7 convolution layer. The function of this convolution layer is to fuse the information of the two description matrices and learn a spatial attention map. The output of the convolution layer is a spatial attention map M(s) with a shape of H×W×1; (4) Weighting: The spatial attention map M(s) is normalized by the sigmoid activation function so that its value is between 0 and 1. In this way, each element in M(s) represents the weight or importance of the corresponding spatial position. Finally, the input feature map F′ is multiplied element by element with the spatial attention map M(s) (note that the multiplication is performed in both the spatial dimension and the channel dimension, but since M(s) is a single channel, the feature map of each channel is actually multiplied by the same spatial weight). In this way, each spatial position in the original feature map is reweighted to obtain the final enhanced feature map.

[0118] In some examples, combining ResBlock and CBAM attention mechanism modules in the YOLOv8 model can further enhance the model's feature extraction capabilities, thereby improving detection accuracy and robustness. ResBlock (ResidualBlock) is a network structure commonly used in deep learning. It solves the gradient vanishing and gradient exploding problems in the training process of deep neural networks by introducing residual connections. The basic idea of ​​ResBlock is to establish a direct connection between the input and output, so that the network can learn the identity mapping, making it easier to optimize the deep network.

[0119] To combine ResBlock and CBAM in the YOLOv8 model, follow these steps:

[0120] First: Introduce ResBlock: In the YOLOv8 network architecture, find a suitable location to insert the ResBlock module. This is usually after the convolutional layer, or in the part where the feature extraction capability needs to be enhanced.

[0121] Second: Add CBAM: After ResBlock, add the CBAM attention mechanism module. CBAM can be seamlessly integrated into any CNN architecture, including networks containing ResBlock.

[0122] Third: Adjust the network configuration: According to the added ResBlock and CBAM modules, adjust the network configuration of YOLOv8, including modifying the parameters of the network layer, adjusting the number of input and output channels, etc.

[0123] Fourth: Training and optimization: Use the adjusted network configuration for training, and further improve the performance of the model through techniques such as hyperparameter optimization and data augmentation.

[0124] In this embodiment, by combining the ResBlock and CBAM attention mechanism modules in the YOLOv8 model, the feature extraction capability of the model can be further enhanced, thereby improving the accuracy and robustness of detection. This requires careful adjustment of network architecture, hyperparameters, and data enhancement techniques to obtain the best performance.

[0125] Embodiment 2:

[0126] The bolt loosening detection method of this embodiment can be applied to electronic devices with communication, computing and data storage capabilities. The specific process can be as follows: Figure 2 As shown, including:

[0127] Step 101: key area positioning and segmentation.

[0128] Specifically, the input train image is processed using the Faster R-CNN model. Faster R-CNN generates candidate regions through the Region Proposal Network (RPN), and performs feature extraction and classification on these candidate regions.

[0129] In this embodiment, Faster R-CNN is used to locate and segment the train axle box cover and shock absorber area as an example for explanation.

[0130] like Figure 3 As shown, the specific steps of key area positioning and segmentation include:

[0131] Step 11: Feed the train images of any size into the Faster R-CNN model.

[0132] In this embodiment, the train images of any size are pre-processed by scaling, normalization, etc. to make them meet the input requirements of the model. The Faster R-CNN model is pre-trained based on the sample data set.

[0133] Step 12: Use RPN to generate candidate regions that may contain the axle box cover and shock absorber.

[0134] Specifically, the train image is scanned through a region proposal network to obtain multiple candidate regions; wherein the region proposal network (RPN) is used to generate region proposals that may contain target objects. These region proposals can efficiently cover objects that may exist in the image and provide a basis for subsequent accurate detection.

[0135] Furthermore, the RPN network generates a series of candidate regions (i.e., anchor boxes) on the feature map by means of a sliding window, and screens them according to the intersection over union (IoU) between the anchor box and the true annotation box, retaining the candidate regions with higher IoU as regions of interest (ROIs), i.e., candidate regions.

[0136] Step 13: Extract features from the candidate regions and use a convolutional neural network (CNN) to classify and regress these features to obtain accurate region boundaries.

[0137] Specifically, a convolutional neural network (CNN) is used to extract image features to obtain feature information of each candidate area. The convolutional neural network (CNN) can automatically learn and extract key features in the image, such as edges, textures, etc. After multiple layers of convolution and pooling operations, CNN outputs a feature map containing image feature information. Classification and bounding box regression are performed based on the feature information of each candidate area. Classification is used to determine whether the candidate area includes the area where the bolts are located (i.e., the train axle box cover or the shock absorber area), and bounding box regression is to adjust the bounding box to more accurately locate the bolts.

[0138] In one example, the characteristic information of each candidate region is extracted from the feature map and mapped to a higher-dimensional feature vector. A classifier (such as a softmax layer) is used to classify the feature vector of the candidate region to determine whether it contains a bolt. At the same time, a regressor is used to adjust the bounding box of the candidate region so that it more accurately surrounds the bolt area. The regressor predicts the offset of the bounding box to adjust its position and size. Based on the results of the target classification and box boundary regression, the most likely area for the bolt is determined and its precise boundary information is output. This usually includes the coordinates of the bounding box (such as the coordinates of the upper left and lower right corners) and the width and height of the bounding box.

[0139] Step 14: According to the classification results, the positions of the axle box cover and the shock absorber are determined, and image segmentation is performed to generate independent images of the axle box cover and the shock absorber.

[0140] Furthermore, based on the determined precise boundaries of the train axle box cover area and the shock absorber, independent images of the axle box cover and the shock absorber are cropped out from the train image.

[0141] In one example, in order to improve the accuracy of recognition, before segmenting the image, the identified bolt area is further processed, such as removing overlapping areas, merging adjacent areas, etc., to ensure the accuracy of the boundary of the train axle box cover area or the shock absorber area.

[0142] Step 102: Bolt loosening and falling off detection.

[0143] Specifically, the segmented images of the axle box cover and shock absorber are fed into the YOLOv8 model for further processing. YOLOv8 is an improved YOLO model released in 2023 with higher detection accuracy and speed. When the images from the first stage are fed in, they are resized to a fixed size (e.g. 640x640) required by the network. Images are usually normalized so that the pixel values ​​are between 0 and 1.

[0144] Among them, the YOLOv8 model is used to detect the loose bolts in the axle box cover and shock absorber images. Figure 4As shown in the figure, the specific steps for bolt loosening and falling off detection are as follows:

[0145] Step 21: Input the segmented axle box cover image into the YOLOv8 model.

[0146] Specifically, the segmented axle box cover image is scaled, normalized and other preprocessing operations are performed to make it meet the input requirements of the model. Among them, the YOLOv8 model is pre-trained based on the sample data set.

[0147] Step 22: The image is passed through a deep convolutional neural network (CSPDarknet53, etc.) to extract features. This part of the network is responsible for extracting features at different levels, from low-level edge features to high-level semantic features.

[0148] Specifically, YOLOv8 uses deep convolutional neural networks such as CSPDarknet53 to extract image features. These networks gradually extract low-level to high-level features from images through structures such as convolutional layers, activation functions, and pooling layers.

[0149] Step 23: YOLOv8 uses FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) structures for feature fusion at different scales. FPN is used to fuse feature maps of different resolutions, while PAN strengthens the flow of semantic information, making positioning more accurate.

[0150] Among them, the Feature Pyramid Network (FPN) fuses feature maps of different resolutions through bottom-up and top-down paths, thereby enhancing the model's ability to detect objects of different scales. Based on FPN, the Path Aggregation Network (PAN) further strengthens the information transfer between features through an additional bottom-up path, making the model more accurate in locating objects.

[0151] Step 24: After feature fusion, YOLOv8 uses multiple prediction heads to output the final prediction results. Each prediction head operates on feature maps of different scales and outputs the category probability, bounding box coordinates and confidence score of the corresponding position. The prediction head of YOLOv8 adopts an anchor-free approach, that is, it does not rely on predefined anchor boxes, but directly regresses the bounding box. This approach performs better in small object detection.

[0152] Specifically, on the fused feature map, a prediction head is set to output the detection results of bolt looseness; the prediction head is usually implemented through a fully connected layer and a softmax function, which converts the output into a probability distribution; the prediction head also outputs the bounding box coordinates of the bolt in the image, which represent the position and size of the bolt. Each prediction result will have a confidence score, indicating the reliability of the prediction result. The higher the confidence score, the more reliable the prediction result.

[0153] Each prediction head is connected to a feature map of a specific scale. This means that each prediction head generates predictions based on the feature map it is connected to. Since feature maps have different scales and resolutions, each prediction head is able to capture and detect objects of different sizes. For each prediction head, the class probability, bounding box coordinates, and confidence score are output. The bolt loosening class probability refers to the probability value of the bolt not loosening or the bolt loosening or bolt falling off class at this location. It is usually normalized by the Softmax function or the Sigmoid function so that the sum of the probability values ​​is 1. The bounding box coordinates represent the position of the target object in the image predicted by the model. In YOLOv8, the bounding box is usually represented by four values: x_min, y_min, x_max, y_max, which correspond to the x-coordinate, y-coordinate of the upper left corner of the bounding box and the x-coordinate, y-coordinate of the lower right corner. These coordinate values ​​are decoded according to the size of the feature map and the output of the model, and need to be mapped back to the size of the original image. The confidence score indicates how certain the model is about the prediction result. It reflects the possibility that the model believes that the predicted target object does exist. Unlike the category probability, the confidence score is independent of the category. In YOLOv8, the confidence score is usually obtained by calculating the intersection over union (IoU) between the predicted bounding box and the true bounding box, which can also be understood as the model's evaluation of the accuracy of the predicted bounding box.

[0154] Step 25: The bounding box output by the model is usually an offset relative to the image size and needs to be decoded into coordinates on the actual image. Then, non-maximum suppression (NMS) is used to filter out overlapping boxes and retain the prediction results with the highest confidence.

[0155] Specifically, the bounding box coordinates output by the prediction head are offsets relative to the size of the feature map, so they need to be decoded into coordinates on the actual image. This usually involves multiplying the offset by the scaling factor of the feature map to the original image, and adding the corresponding offset (such as the coordinates of the upper left corner of the grid) to obtain the exact position of the bounding box on the original image. Further, in target detection, since the model may predict multiple overlapping bounding boxes on the same target, non-maximum suppression (NMS) is needed to filter out these redundant boxes. The basic idea of ​​NMS is: for each target category, select the bounding box with the highest confidence, and suppress (i.e. delete) the bounding box with an overlap exceeding a certain threshold and a lower confidence. In this way, each target ultimately retains only one most representative bounding box; the bolt loosening detection result is determined based on the retained bounding box and the corresponding bolt loosening category probability and confidence. In this embodiment, through feature fusion and the use of multiple prediction heads, multi-scale information in the image can be captured and accurate detection results can be output. Bounding box decoding and non-maximum suppression further improve the accuracy and reliability of the detection results.

[0156] Step 26: YOLOv8 outputs the class label, confidence score, and bounding box location of each target object. These results can be used for further processing or displayed directly.

[0157] Specifically, YOLOv8 ultimately outputs the category label of each target object (such as whether the bolt is loose), the confidence score (indicating the reliability of the prediction result), and the bounding box position (indicating the specific location of the target object in the image).

[0158] Step 27: Input the segmented shock absorber image into the YOLOv8 model and repeat the above steps to detect the status of the bolts.

[0159] Similarly, for the segmented shock absorber image, the above steps are repeated to detect the looseness of the bolts in the shock absorber area.

[0160] In some embodiments, in order to improve the detection performance, a CBAM (Convolutional Block Attention Module) attention mechanism is introduced. CBAM consists of two parts: a channel attention module and a spatial attention module. The CBAM module enhances the model's attention to important features by adaptively adjusting the channel and spatial weights of the feature map.

[0161] Specifically, CBAM is an attention module used to improve the performance of convolutional neural networks. It enhances the expressiveness of the model by focusing on the more important parts of the input feature map. CBAM consists of two parts: Channel Attention Module and Spatial Attention Module. Channel Attention Module: This module focuses on the weights of different channels in the feature map. By adjusting these weights, the model can pay more attention to the channels that are useful for the detection task and ignore those unimportant channels; specifically, the channel weights are adjusted through global average pooling and fully connected layers to generate channel attention maps, emphasizing important channels and suppressing unimportant channels. Spatial Attention Module: This module focuses on the weights of different spatial positions in the feature map. By adjusting these weights, the model can pay more attention to the spatial positions that are useful for the detection task and ignore those unimportant positions; specifically, the spatial attention map is generated by combining maximum pooling and average pooling, which improves the model's visual task processing ability, emphasizes the important parts of the image and suppresses the unimportant parts.

[0162] The specific working principle of the channel attention module is as follows: (1) Input: define the input feature map as F, whose shape is H×W×C, where H is the height, W is the width, and C is the number of channels; (2) Global average pooling and global maximum pooling: perform global average pooling on the input feature map F in the spatial dimensions (H and W) to obtain a one-dimensional vector F with a length of C average Each element in this vector represents the average activation value of the corresponding channel in the entire space; similarly, the input feature map F is globally max-pooled in the spatial dimension to obtain a one-dimensional vector F with a length of C. max Each element in this vector represents the maximum activation value of the corresponding channel in the entire space; these two vectors F average and F max They capture the global information of each channel and provide input for the subsequent multi-layer perceptron; (3) Multi-layer perceptron (MLP): average and F maxThey are passed through two independent shared multi-layer perceptrons (MLPs) respectively. Each MLP usually contains two fully connected layers (FC layers), where the first FC layer is used for dimensionality reduction (usually reducing the number of channels to one-r of the original, r is the dimensionality reduction ratio), and the second FC layer is used for dimensionality increase (restoring to the original number of channels C). After each MLP, the ReLU activation function is applied to increase nonlinearity. The outputs of the two MLPs are weighted summed (usually by simple addition or weighted addition) to obtain a one-dimensional vector of length C, which is the channel attention map M. c (4) Weighting: Finally, the input feature map F is multiplied by M(c) to obtain the enhanced feature map.

[0163] The specific working principle of the spatial attention module is as follows: (1) Input: The input is the feature map F′ processed by the channel attention module, and its shape is H×W×C, where H is the height, W is the width, and C is the number of channels; (2) Inter-channel aggregation: The input feature map F′ is average pooled in the channel dimension to obtain a spatial description matrix F with a shape of H×W×1. average ′. Each element in this matrix represents the average activation value of the corresponding spatial position in all channels; similarly, the input feature map F′ is pooled in the channel dimension to obtain a spatial description matrix F with a shape of H×W×1 max ′. Each element in this matrix represents the maximum activation value of the corresponding spatial position on all channels; these two matrices F average′ and F max ′ jointly captures the global information of each spatial position (although aggregated in the channel dimension), providing input for the subsequent spatial attention map generation; (3) Convolution operation: Convolution operation: Convolution operation: Conversion of two spatial description matrices F average′ and F max ′ is concatenated in the channel dimension to obtain a matrix with a shape of H×W×2. ; This concatenated matrix is ​​convolved through a 7×7 convolution layer. The function of this convolution layer is to fuse the information of the two description matrices and learn a spatial attention map. The output of the convolution layer is a spatial attention map M(s) with a shape of H×W×1; (4) Weighting: The spatial attention map M(s) is normalized by the sigmoid activation function so that its value is between 0 and 1. In this way, each element in M(s) represents the weight or importance of the corresponding spatial position. Finally, the input feature map F′ is multiplied element by element with the spatial attention map M(s) (note that the multiplication is performed in both the spatial dimension and the channel dimension, but since M(s) is a single channel, the feature map of each channel is actually multiplied by the same spatial weight). In this way, each spatial position in the original feature map is reweighted to obtain the final enhanced feature map.

[0164] In some examples, combining ResBlock and CBAM attention mechanism modules in the YOLOv8 model can further enhance the model's feature extraction capabilities, thereby improving detection accuracy and robustness. ResBlock (ResidualBlock) is a network structure commonly used in deep learning. It solves the gradient vanishing and gradient exploding problems in the training process of deep neural networks by introducing residual connections. The basic idea of ​​ResBlock is to establish a direct connection between the input and output, so that the network can learn the identity mapping, making it easier to optimize the deep network.

[0165] To combine ResBlock and CBAM in the YOLOv8 model, follow these steps:

[0166] First: Introduce ResBlock: In the YOLOv8 network architecture, find a suitable location to insert the ResBlock module. This is usually after the convolutional layer, or in the part where the feature extraction capability needs to be enhanced.

[0167] Second: Add CBAM: After ResBlock, add the CBAM attention mechanism module. CBAM can be seamlessly integrated into any CNN architecture, including networks containing ResBlock.

[0168] Third: Adjust the network configuration: According to the added ResBlock and CBAM modules, adjust the network configuration of YOLOv8, including modifying the parameters of the network layer, adjusting the number of input and output channels, etc.

[0169] Fourth: Training and optimization: Use the adjusted network configuration for training, and further improve the performance of the model through techniques such as hyperparameter optimization and data augmentation.

[0170] In this embodiment, by combining the ResBlock and CBAM attention mechanism modules in the YOLOv8 model, the feature extraction capability of the model can be further enhanced, thereby improving the accuracy and robustness of detection. This requires careful adjustment of network architecture, hyperparameters, and data enhancement techniques to obtain the best performance.

[0171] Embodiment three:

[0172] Another embodiment of the present application relates to a bolt loosening detection device. The implementation details of the bolt loosening detection device of this embodiment are specifically described below. The following content is only for the convenience of understanding the implementation details provided, and is not necessary for the implementation of this solution. The schematic diagram of the bolt loosening detection of this embodiment can be as follows: Figure 5 As shown, it includes an acquisition module 801 , a recognition module 802 , a segmentation module 803 and a detection module 804 .

[0173] The acquisition module 801 is used to acquire a train image, wherein the train image at least partially includes the area where the bolts are located.

[0174] The identification module 802 is used to input the train image into the positioning and identification model to identify the area where the bolts are located, and determine the area boundary information of the area where the bolts are located.

[0175] The segmentation module 803 is used to segment the train image according to the region boundary information of the region where the bolts are located, so as to generate an image of the region where the bolts are located.

[0176] The detection module 804 is used to input the image of the area where the bolt is located into the bolt loosening detection model to perform bolt loosening detection.

[0177] Furthermore, if Figure 6 As shown, the acquisition module 801 includes an optical camera device 810, which is arranged on the ground inside the rail and on both sides of the rail to collect images of the bottom and both sides of the passing train.

[0178] For the specific definition of the bolt loosening detection device, please refer to the definition of the bolt loosening detection method above, which will not be repeated here. Each unit in the above-mentioned bolt loosening detection device can be implemented in whole or in part by software, hardware and their combination. The above-mentioned units can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above-mentioned units.

[0179] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.

[0180] Embodiment 4:

[0181] Another embodiment of the present application relates to an electronic device, such as Figure 7 As shown, it includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the bolt loosening detection method in the above-mentioned embodiments.

[0182] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are all well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0183] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0184] Embodiment five:

[0185] Another embodiment of the present application relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0186] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0187] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.

Claims

1. A method for detecting bolt loosening, characterized in that: include: Acquire a train image, wherein at least a portion of the train image includes an area where the bolt is located; Inputting the train image into a positioning and recognition model to identify the area where the bolts are located, and determining the area boundary information of the area where the bolts are located; Segmenting the train image according to the region boundary information of the region where the bolts are located to generate an image of the region where the bolts are located; The image of the area where the bolt is located is input into a bolt loosening detection model to perform bolt loosening detection.

2. The bolt loosening detection method according to claim 1, characterized in that: The step of inputting the train image into the positioning recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located comprises: Scanning the train image through a region proposal network to obtain multiple candidate regions; Extracting features from each candidate region to obtain feature information of each candidate region; The feature information of each candidate area is classified and a bounding box is regressed to determine the area boundary of the bolt area.

3. The bolt loosening detection method according to claim 1, characterized in that: The step of inputting the image of the area where the bolt is located into the bolt loosening detection model to perform bolt loosening detection comprises: Extracting feature information of the image of the area where the bolt is located by a convolutional neural network to obtain feature maps at each depth; Fuse the feature maps at each depth to obtain a fused feature map; Based on the fused feature map, the prediction head of the bolt loosening detection model outputs the bolt loosening category probability, bounding box coordinates and confidence of the corresponding position, wherein the bolt loosening category includes at least two of the bolt not loosened, the bolt loosened and the bolt falling off; The bolt loosening detection result is determined according to the bolt loosening category probability, the bounding box coordinates and the confidence level.

4. The bolt loosening detection method according to claim 3, characterized in that: The step of determining the bolt loosening detection result according to the bolt loosening category probability, the bounding box coordinates and the confidence level comprises: Decoding the bounding box coordinates to obtain image coordinates of the bounding box on the image of the area where the bolt is located; Calculating the degree of overlap between the bounding box and the adjacent bounding box according to the image coordinates of the bounding box on the image of the area where the bolt is located; The bounding boxes whose overlap exceeds a preset threshold are screened by a non-maximum suppression method, and the bounding boxes with the highest confidence are retained; A bolt loosening detection result is determined based on the retained bounding box and the corresponding bolt loosening category probability and confidence.

5. The bolt loosening detection method according to claim 1, characterized in that: Before the step of inputting the train image into the positioning and recognition model to identify the area where the bolts are located and determining the area boundary information of the area where the bolts are located, the step further includes: Acquire a sample train image including an area where a bolt is located and annotating the area where the bolt is located; The Faster R-CNN model is trained based on the sample train images to obtain a positioning recognition model.

6. The bolt loosening detection method according to claim 1, characterized in that: Before the step of inputting the image of the area where the bolt is located into the bolt loosening detection model for bolt loosening detection, the step further includes: Acquire an image of a region where sample bolts are located, including loose bolts and corresponding annotations of loose bolt categories; An initial target recognition model is constructed based on the YOLO algorithm, and the image of the area where the sample bolts are located is input into the initial target recognition model for optimization training to obtain a bolt loosening detection model.

7. The bolt loosening detection method according to claim 3, characterized in that: Before the step of fusing the feature maps of each depth to obtain a fused feature map, the following step is further included: The channel and spatial weights of the feature map are adjusted through an attention mechanism module.

8. A bolt loosening detection device, characterized in that: include: An acquisition module, used for acquiring a train image, wherein at least a part of the train image includes an area where the bolts are located; An identification module, used for inputting the train image into a positioning identification model to identify the area where the bolts are located, and determining the area boundary information of the area where the bolts are located; A segmentation module, used to segment the train image according to the regional boundary information of the area where the bolts are located, and generate an image of the area where the bolts are located; The detection module is used to input the image of the area where the bolt is located into the bolt loosening detection model to perform bolt loosening detection.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the bolt loosening detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the bolt loosening detection method according to any one of claims 1 to 7 is implemented.

Images