Bolt color marking line detection method based on image segmentation and classification fusion

Abstract

The invention discloses a bolt color marking line detection method based on image segmentation and classification fusion, and belongs to the field of track maintenance. According to the method, a color mark line area of a bolt head is accurately segmented by collecting a track bolt image and combining image preprocessing and mask extraction. On the basis, through deep semantic feature extraction and fusion, pixel-level masks and global context information are integrated and sent to a classification network to judge whether the bolt color graticules are aligned or not. And finally, the whole network structure is optimized through a joint training mechanism, so that the accuracy and robustness of the system in a complex environment are improved. According to the method, the bolt color marking line can be accurately extracted under the complex conditions of weak light, light reflection, shielding and the like, the fine offset with the alignment deviation smaller than 15 degrees can be effectively recognized, and the method can be widely applied to the fields of subway lines, urban rail vehicle maintenance, automatic inspection robots and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of track maintenance, and in particular to a bolt color mark line detection method based on image segmentation and classification fusion. Background Art

[0002] In urban rail transit systems, bolted connections are widely used in key areas such as track plates, switches, and fastener systems. They play a vital role in maintaining track structural stability, transmitting train loads, and ensuring operational safety. Loosening, falling off, or undertightening bolts can cause track structural deformation and abnormal track geometry, leading to serious accidents such as derailment and overturning. Therefore, regular inspection and maintenance of the tightening status of rail bolts is a core task for subway operations and maintenance departments to ensure operational safety.

[0003] Currently, the detection of loose screws in subway inspection operations mainly relies on the following methods:

[0004] 1. Manual inspection: Inspectors visually check bolt status or perform manual torque testing to determine if bolts are loose or misaligned. This method suffers from low efficiency, high labor intensity, reliance on manual experience, a high rate of missed inspections, and limited accuracy in complex subway environments (such as low light, vibration, and dust).

[0005] 2. Image acquisition + manual review: Bolt images are captured using camera systems (such as inspection robots and rail inspection vehicles), and manual back-end review and judgment are performed. Although this improves data acquisition speed, review still requires a lot of manual participation, and subtle deviations in the screw color code are difficult to accurately identify with the naked eye.

[0006] With the development of deep learning technology, convolutional neural networks (CNNs) have performed well in tasks such as image classification and object detection. However, directly using classification networks to judge the status of bolts in subway inspection images still faces the following challenges:

[0007] 1. The characteristic size of the bolt color mark line is small and is easily weakened or lost in the convolution downsampling process;

[0008] 2. Alignment deviations often only involve subtle angular changes, making classification difficult;

[0009] 3. Direct classification lacks explicit modeling of specific color mark line position information, resulting in insufficient detection sensitivity.

[0010] Therefore, there is an urgent need for a new detection method that combines pixel-level segmentation and image-level classification, which can accurately extract bolt color mark line information in complex environments and perform high-sensitivity and high-robustness loosening detection based on this, so as to meet the urgent needs of urban rail transit for automated and intelligent safety inspections. Summary of the Invention

[0011] The present invention provides a bolt color mark line detection method based on image segmentation and classification fusion, which adopts semantic segmentation and feature fusion to realize efficient and intelligent recognition of the bolt tightening status in the subway track system.

[0012] The embodiment of the present invention provides a bolt color code line detection method based on image segmentation and classification fusion, comprising the following steps:

[0013] Collecting a track bolt image and inputting the track bolt image into a pre-trained segmentation model to obtain a mask output and a high-level semantic feature output of the track bolt image, wherein the mask output is a pixel-level prediction mask corresponding to the bolt color line position, and the high-level semantic feature output represents the global context feature of the bolt area;

[0014] The mask output and high-order semantic feature output of the rail bolt image are upsampled and concatenated with the rail bolt image into a five-channel fusion input, which is input into a pre-trained classification model to obtain a classification result of whether the rail bolt image has the correct one.

[0015] Optionally, in one embodiment of the present invention, collecting rail bolt images includes:

[0016] Track bolt images are collected using vehicle-mounted inspection equipment, robotic inspection arms, and handheld inspection terminals. The vehicle-mounted inspection equipment is installed on a track inspection vehicle or a track inspection robot, and has the ability to automatically move and collect batch images, making it suitable for long-distance track inspection tasks; the robotic inspection arm uses a mechanical arm system to take close-range, fixed-point photos of specific track parts, making it suitable for detailed inspections of areas with dense bolts or difficult to access; the handheld inspection terminal is manually carried and operated, and is suitable for local emergency inspections or supplementary inspections in special environments.

[0017] Optionally, in one embodiment of the present invention, building a segmentation model includes:

[0018] Collect multiple rail bolt images, scale them to a standard size, and convert them to the HSV color space. Extract the mask of the bolt color-coded line area by setting an HSV threshold, perform a morphological opening operation on the mask to obtain a preliminary color-coded line mask image, annotate the preliminary color-coded line mask image to obtain mask labels, and establish a training dataset based on the rail bolt images and their corresponding mask labels.

[0019] The training dataset is input into a segmentation network with a U-Net structure, and the corresponding pixel-level prediction mask is input. The difference between the predicted mask and the true annotation mask is optimized using a pixel-level cross entropy loss function to obtain a segmentation model for extracting the bolt color line area.

[0020] Optionally, in one embodiment of the present invention, the structure of the segmentation model includes:

[0021] The encoder consists of convolution + downsampling modules;

[0022] The decoder consists of upsampling + convolution modules;

[0023] Skip connections that transfer high-resolution features between the encoder and decoder.

[0024] Optionally, in one embodiment of the present invention, the input of the segmentation model is a track bolt image, and the output is a prediction mask and a high-order semantic feature output, where the high-order semantic feature output is taken from the last layer feature map of the U-Net encoder part, representing the global contextual features of the bolt area.

[0025] Optionally, in one embodiment of the present invention, the classification model uses the EfficientNet backbone network as the backbone structure, adapts to five-channel input, inherits the pre-trained weights of the RGB channels, and randomly initializes the remaining channels. The network output is a binary Softmax probability vector as the classification result of whether the bolts are aligned, and the optimization target is the cross entropy loss.

[0026] Optionally, in one embodiment of the present invention, when training the segmentation model and the classification model, a joint training mechanism is adopted to simultaneously minimize the mask segmentation error and the classification discrimination error during the training process, and end-to-end collaborative training is performed by optimizing the total loss function.

[0027] The bolt color-coded line detection method, based on image segmentation and classification fusion, captures rail bolt images and combines image preprocessing with mask extraction to accurately segment the color-coded line region on the bolt head. Furthermore, deep semantic feature extraction and fusion are used to integrate the pixel-level mask with global contextual information, which is then fed into a classification network to determine whether the bolt color-coded lines are aligned. Finally, a joint training mechanism is used to optimize the entire network structure, improving the system's accuracy and robustness in complex environments.

[0028] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A flowchart of a bolt color mark line detection method based on image segmentation and classification fusion according to an embodiment of the present invention;

[0031] Figure 2 A network structure diagram for determining whether a bolt color code line is aligned with a network according to an embodiment of the present invention;

[0032] Figure 3 Schematic diagram of the execution process of the bolt color mark line detection method based on image segmentation and classification fusion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and are not to be construed as limiting the present invention.

[0034] Figure 1 The present invention provides a flowchart of a bolt color mark line detection method based on image segmentation and classification fusion according to an embodiment of the present invention.

[0035] like Figure 1 As shown in FIG, the bolt color mark line detection method based on image segmentation and classification fusion includes the following steps:

[0036] In step S101, a track bolt image is collected and input into a pre-trained segmentation model to obtain a mask output and a high-order semantic feature output of the track bolt image. The mask output is a pixel-level prediction mask corresponding to the position of the bolt color line, and the high-order semantic feature output represents the global context feature of the bolt area.

[0037] In actual inspection operations, in order to achieve efficient monitoring of the tightening status of bolts in the subway track system, a variety of intelligent inspection equipment can be used for image acquisition, including but not limited to the use of vehicle-mounted inspection equipment, robotic inspection arms and handheld inspection terminals. Vehicle-mounted inspection equipment is installed on track inspection vehicles or track inspection robots, and has the ability to automatically move and collect images in batches, which is suitable for long-distance track inspection tasks; robotic inspection arms use mechanical arm systems to take close-range, fixed-point photos of specific track parts, which is suitable for fine inspection of areas with dense bolts or difficult to access; handheld inspection terminals are manually carried and operated, and are suitable for local emergency inspections or supplementary inspections in special environments.

[0038] To ensure that the details of the bolt head and color-coded lines can be accurately captured and to avoid inspection failures due to image blur or insufficient resolution, the resolution of the captured image is generally set to 224×224 pixels or higher. The main reasons for selecting this resolution standard include:

[0039] 1. The bolt head diameter is usually between 5mm and 20mm. 224 pixels or above can ensure a spatial resolution of at least 2 to 5 pixels per millimeter, meeting the requirements for detail extraction.

[0040] 2. Color scale lines are usually small (about 1-3 mm), and it is necessary to ensure that at least dozens of pixels in the image cover the line area so that the segmentation network can accurately learn features;

[0041] 3. Higher-resolution images can tolerate a certain degree of shooting errors (such as slight blur and focus), improving the robustness of the overall system;

[0042] 4. Considering both device storage and data transmission efficiency, 224×224 is a reasonable lower limit for both computational overhead and detection accuracy. In practical applications, higher resolutions can be selected as needed to further improve performance.

[0043] In one embodiment of the present invention, constructing a segmentation model includes: collecting multiple rail bolt images, scaling the rail bolt images to a standard size, and converting them to the HSV color space, extracting a mask of the bolt color mark line area by setting an HSV threshold, performing a morphological opening operation on the mask to obtain a preliminary mask image of the color mark line, annotating the preliminary mask image of the color mark line to obtain a mask label, and establishing a training dataset based on the rail bolt images and their corresponding mask labels; inputting the training dataset into a segmentation network with a U-Net structure, and inputting the corresponding pixel-level predicted mask, and using a pixel-level cross entropy loss function to optimize the difference between the predicted mask and the true annotated mask to obtain a segmentation model for extracting the bolt color mark line area.

[0044] Specifically, for the collected rail bolt RGB image First, the following preprocessing steps are performed:

[0045] 1. Size normalization: Scale to the standard size W = 224, H = 224.

[0046] 2. Color space conversion: Convert RGB image to HSV color space I HSV :

[0047] I HSV =Convert_To_HSV(I RGB ).

[0048] 3. Preliminary extraction of color marking lines: Based on prior knowledge of the paint color of the bolt color marking lines (such as red, yellow, or white), set the HSV threshold range:

[0049] H min ≤H(x,y)≤H max ,S min ≤S(x,y)≤Smax

[0050] Among them, H(x,y) is the hue component at the pixel point (x,y), H min and H max Indicates the hue threshold range of the selected color, which is used to filter the target color; S(x,y) is the saturation component of the pixel point, S min and S max Sets the color purity threshold to exclude dark or low-saturation areas (such as background, shadows, etc.).

[0051] Generate a preliminary mask:

[0052]

[0053] 4. Morphological processing: Open the preliminary mask (erode first and then dilate) to remove small noise:

[0054] Mask clean =Morphological_Open(Mask HSV ).

[0055] 5. Based on the preliminary screening of masks, combined with manual review, create a standard mask label M GT .

[0056] Each image sample includes:

[0057]

[0058] Among them, N is usually set to be greater than 2000 to ensure the stability of model training.

[0059] Bolt color markings are typically thin, continuous straight lines or arcs, and are crucial for determining bolt tightening status. If these markings are broken or missing, the segmentation model will learn incorrect spatial structural features, leading to prediction interruptions and recognition errors during the inference phase, severely impacting detection accuracy and reliability. Therefore, color markings must be continuous and complete, avoiding any gaps or omissions.

[0060] In actual subway inspection environments, bolt surfaces may be partially obscured or blurred due to oil, water, and dust accumulation, or due to limitations in camera angle and lighting conditions. Without individually marking these abnormal areas, the model may mistakenly identify these obscured and blurred areas as normal color-coded lines during training, leading to feature learning bias. By clearly marking these obscured and blurred areas during the annotation phase, the model can be guided to distinguish between normal lines and abnormal areas, improving the system's robustness and generalization capabilities in complex environments.

[0061] The metal surface of the bolt head may contain reflective areas that are similar in color to the color-coded lines, which can be mistaken for line features. If the wireless areas are not clearly marked as background during the annotation phase, the segmentation model is prone to false detections during training, especially in scenarios with drastic lighting changes and strong surface reflections. Therefore, clearly distinguishing the wireless portion of the bolt from the color-coded lines is essential for improving model detection accuracy and reducing error rates.

[0062] In one embodiment of the present invention, the structure of the segmentation model includes:

[0063] The encoder consists of convolution + downsampling modules;

[0064] The decoder consists of upsampling + convolution modules;

[0065] Skip connections that transfer high-resolution features between the encoder and decoder.

[0066] In one embodiment of the present invention, the input of the segmentation model is a track bolt image, and the output is a predicted mask and a high-order semantic feature output. The high-order semantic feature output is taken from the last layer feature map of the U-Net encoder part, which represents the global contextual features of the bolt area.

[0067] Specifically, the segmentation model design and training (U-Net): The segmentation model uses the classic U-Net structure for pixel-level extraction of bolt color marking lines. The network structure includes:

[0068] Encoder part: consists of convolution + downsampling modules;

[0069] Decoder part: consists of upsampling + convolution modules;

[0070] Skip connection: transfer high-resolution features between encoder and decoder.

[0071] The input of the segmentation model is a three-channel image I RGB , the output is the prediction mask

[0072] The loss function uses pixel-level binary cross entropy:

[0073]

[0074] in, It is the prediction result of the model for the pixel point (x, y), which indicates the probability value of the pixel being the color-coded line, and its value range is between [0, 1].

[0075] The training goal is to minimize the difference between the mask and the annotation so that the model can accurately segment the highlighted area.

[0076] In step S102, the mask output and high-order semantic feature output of the rail bolt image are upsampled and spliced ​​with the rail bolt image into a five-channel fusion input, which is input into a pre-trained classification model to obtain a classification result of whether the rail bolt image is aligned with the mask output and the high-order semantic feature output.

[0077] Specifically, mask inference and high-order feature fusion are performed. In the inference stage, given a test image I RGB , processed by the trained U-Net network, two parts of output are obtained, such as Figure 2 As shown:

[0078] 1. Mask output Pixel-level prediction mask, corresponding to the color scale line position.

[0079] 2. High-level semantic feature output F high : The feature map of the last layer of the U-Net encoder part represents the global context features of the bolt area. high The dimensions are generally Upsample back to the original image size (W, H, 1) through bilinear interpolation.

[0080] In traditional joint segmentation and classification methods, only mask information is used as auxiliary input. Although it can highlight the spatial position information of the color-coded line, when faced with complex scenes (such as lighting changes, oil reflections on the bolt surface, foreign object occlusion, etc.), the mask information is easily interfered by local noise and lacks the ability to perceive the overall bolt structure and environmental characteristics.

[0081] The high-level semantic features F of the U-Net encoder high It includes abstract descriptions of the overall image structure, contextual relationships, bolt shape features, etc., which can make up for the lack of spatial details of the mask and provide richer judgment basis for subsequent classification tasks.

[0082] Because F high The original size of F is usually much smaller than the input image (the downsampling ratio is 1 / 32). If it is used directly, it will cause the feature and the original image space to be misaligned. Therefore, it is necessary to use bilinear interpolation to convert F high Upsampling is done back to the same resolution (W, H) as the original image. The upsampled high-order feature map corresponds one-to-one with the original image at the pixel level, ensuring that spatial position information is not lost during feature fusion, which helps improve the classifier's sensitivity to local alignment deviations.

[0083] By using mask information and high-order semantic features as additional input channels, the fine local information of the bolt color line is retained, and the contextual information of the bolt's overall shape and environmental background is integrated, thereby effectively improving the accuracy, robustness and anti-interference ability of the classification network in the bolt alignment detection task.

[0084] The high-order feature processing formula is:

[0085] F high_resized =Upsample(F high ,size=(W,H))

[0086] Among them, the Upsample operation is bilinear interpolation.

[0087] If F high For multi-channel, take a part of the feature channels (such as the mean or the maximum activation channel) to form a single-channel feature map.

[0088] Finally, the original RGB image, mask, and high-order semantic features are stacked to obtain a five-channel fusion input:

[0089]

[0090] That is, stacking in the channel dimension:

[0091]

[0092] In one embodiment of the present invention, the classification model uses the EfficientNet backbone network as the backbone structure, adapts to five-channel input, inherits the pre-trained weights of the RGB channels, and randomly initializes the remaining channels. The network output is a binary Softmax probability vector as the classification result of whether the bolts are aligned, and the optimization target is the cross entropy loss.

[0093] The classification model uses the EfficientNet backbone network, and the following changes are made to the input layer to adapt to five-channel input:

[0094] The initial convolution kernel weights are expanded to five channels:

[0095] W 5ch =[W RGB ,W mask ,W feature ]

[0096] Where: W RGB Inherit the original pre-training weights, W mask 、W feature Channels are randomly initialized, or the blue channel weights are copied.

[0097] The final output of the classifier is a binary Softmax probability vector:

[0098] P=[p aligned ,p misaligned ]

[0099] Among them, p aligned The model predicts the probability that the bolt color lines in the image are aligned, p misaligned The model predicts the probability that the bolt color mark line in the image is offset or misaligned, and satisfies the probability constraint: p aligned +p misaligned =1.

[0100] Classification label settings: y = 1 (aligned), y = 0 (unaligned).

[0101] Training using cross entropy loss:

[0102]

[0103] In one embodiment of the present invention, when training the segmentation model and the classification model, a joint training mechanism is adopted to simultaneously minimize the mask segmentation error and the classification discrimination error during the training process, and end-to-end collaborative training is performed by optimizing the total loss function.

[0104] In order to further improve the overall system performance, the segmentation task and the classification task can be jointly optimized, and the total loss function is defined as:

[0105]

[0106] The weight coefficients λ1λ2 are adjusted according to the importance of the task's contribution to the final performance. Generally, the empirical values ​​are λ1=0.4 and λ2=0.6, which slightly favor the classification task and take into account the dual requirements of segmentation mask quality and classification accuracy.

[0107] In traditional separate training processes, the segmentation model and classification model are typically trained independently. This involves first training the segmentation network to extract the mask, then freezing the segmentation output and feeding the mask as static input to the classification network for classification training. While simple to implement, this two-stage approach has the following limitations: the segmentation network's optimization objective focuses solely on the pixel-level accuracy of the mask, potentially overlooking important features relevant to classification, such as the bolt's overall shape and minor local deviations; the mask received by the classification network during training is statically generated, making it impossible to dynamically adjust the mask's features based on classification needs. This limits the mask's auxiliary effect on the classifier; and the two subtask optimization objectives are disconnected, preventing optimal overall system performance.

[0108] To this end, the present invention introduces a joint training mechanism of segmentation and classification, that is, to minimize the mask segmentation error and classification discrimination error simultaneously during the training process, and to use the total loss function Optimize the entire system end-to-end.

[0109] The advantages of joint training include: during the optimization process, the segmentation network not only learns the pixel accuracy of the mask, but also, through classification error feedback, prompts it to extract more discriminative spatial features, such as the position, direction, and completeness of the color mark line, which helps subsequent classifiers better distinguish between aligned and misaligned states; the mask features received by the classification network are dynamically optimized and can be adaptively adjusted according to the needs of the classification task, thereby improving the overall robustness and generalization ability of the system; during the system training process, each module makes collaborative progress, ultimately achieving higher detection accuracy and better reasoning performance.

[0110] The pseudo code of the overall process of the reasoning stage is as follows:

[0111]

[0112] like Figure 3 As shown, the execution process of the bolt color mark line detection method based on image segmentation and classification fusion according to an embodiment of the present invention is demonstrated.

[0113] Step 1: Use an image acquisition device mounted on a track inspection vehicle, inspection robot, or handheld device to collect bolt image data. The image resolution is generally set to 224×224 pixels or higher to ensure that the details of the bolt color marking lines can be effectively captured. The collected RGB images are uniformly scaled to a standard size and converted to the HSV color space. A mask of the bolt color marking line region is initially extracted by setting a threshold. A morphological opening operation is performed on the mask to remove noise, resulting in a clear preliminary mask of the color marking line.

[0114] Step 2: The data obtained in Step 1 is annotated with standard masks in conjunction with manual review to form a training dataset containing images and their corresponding accurate mask labels. During the annotation process, the continuity and integrity of the color scale lines must be ensured to prevent the model from learning incorrect features.

[0115] Step 3: Input the dataset from Step 2 into the U-Net segmentation network along with the corresponding pixel-level predicted mask. A pixel-level cross-entropy loss function is used to optimize the difference between the predicted mask and the ground-truth mask, thereby training a segmentation model that can accurately extract the bolt color-coded line area.

[0116] Step 4: During the training process, the mask and high-order semantic features extracted by the segmentation network are upsampled to the original image size and concatenated with the original RGB image to form a five-channel fusion input, which contains spatial details and global structure information to enhance classification and discrimination capabilities.

[0117] Step 5: Use EfficientNet as the backbone architecture, adapting it to a five-channel input. It inherits the pre-trained weights for the RGB channels and randomly initializes the remaining channels. The network outputs a classification result indicating whether the bolts are aligned or not, optimizing for the cross-entropy loss.

[0118] Step 6: By jointly optimizing the segmentation loss and classification loss, the segmentation network is guided to extract more discriminative structural features, while improving the classifier's adaptability to mask features, achieving end-to-end collaborative training of the system and enhancing the overall detection accuracy and robustness.

[0119] According to the bolt color mark line detection method based on image segmentation and classification fusion proposed in an embodiment of the present invention, rail bolt images are collected through vehicle-mounted inspection equipment, robot inspection arms or handheld terminals, and the color mark line area of ​​the bolt head is accurately segmented by combining image preprocessing and mask extraction. On this basis, the pixel-level mask is integrated with the global context information through deep semantic feature extraction and fusion, and sent to the classification network to determine whether the bolt color mark line is aligned. Finally, the entire network structure is optimized through a joint training mechanism to improve the accuracy and robustness of the system in complex environments. The present invention can accurately extract bolt color mark lines under complex conditions such as weak light, reflection, and occlusion, and can effectively identify subtle offsets with alignment deviations of less than 15°. In actual subway inspection applications, the detection accuracy rate exceeds 95%, greatly reducing the burden of manual review. It can be widely used in subway lines, urban rail vehicle maintenance, automatic inspection robots and other fields.

[0120] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0121] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "N" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0122] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N executable instructions for implementing a custom logical function or step of a process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention pertain.

Claims

1. A bolt color mark line detection method based on image segmentation and classification fusion, characterized in that: The following steps are involved: Collecting a track bolt image and inputting the track bolt image into a pre-trained segmentation model to obtain a mask output and a high-level semantic feature output of the track bolt image, wherein the mask output is a pixel-level prediction mask corresponding to the bolt color line position, and the high-level semantic feature output represents the global context feature of the bolt area; The mask output and high-order semantic feature output of the rail bolt image are upsampled and concatenated with the rail bolt image into a five-channel fusion input, which is input into a pre-trained classification model to obtain a classification result of whether the rail bolt image has the correct one.

2. The method according to claim 1, characterized in that Collecting rail bolt images includes: Track bolt images are collected using vehicle-mounted inspection equipment, robotic inspection arms, and handheld inspection terminals. The vehicle-mounted inspection equipment is installed on a track inspection vehicle or a track inspection robot, and has the ability to automatically move and collect batch images, making it suitable for long-distance track inspection tasks; the robotic inspection arm uses a mechanical arm system to take close-range, fixed-point photos of specific track parts, making it suitable for detailed inspections of areas with dense bolts or difficult to access; the handheld inspection terminal is manually carried and operated, and is suitable for local emergency inspections or supplementary inspections in special environments.

3. The method according to claim 1, characterized in that Building a segmentation model involves: Collect multiple rail bolt images, scale them to a standard size, and convert them to the HSV color space. Extract the mask of the bolt color-coded line area by setting an HSV threshold, perform a morphological opening operation on the mask to obtain a preliminary color-coded line mask image, annotate the preliminary color-coded line mask image to obtain mask labels, and establish a training dataset based on the rail bolt images and their corresponding mask labels. The training dataset is input into a segmentation network with a U-Net structure, and the corresponding pixel-level prediction mask is input. The difference between the predicted mask and the true annotation mask is optimized using a pixel-level cross entropy loss function to obtain a segmentation model for extracting the bolt color line area.

4. The method according to claim 3, characterized in that The structure of the segmentation model includes: The encoder consists of convolution + downsampling modules; The decoder consists of upsampling + convolution modules; Skip connections that transfer high-resolution features between the encoder and decoder.

5. The method according to claim 4, characterized in that The input of the segmentation model is the rail bolt image, and the output is the predicted mask and high-level semantic feature output. The high-level semantic feature output is taken from the last layer feature map of the U-Net encoder part, which represents the global context feature of the bolt area.

6. The method according to claim 1, wherein The classification model uses the EfficientNet backbone network as the main structure, adapts to five-channel input, inherits the pre-trained weights of the RGB channels, and randomly initializes the remaining channels. The network output is a binary Softmax probability vector, which serves as the classification result of whether the bolts are aligned. The optimization target is the cross entropy loss.

7. The method according to claim 1, characterized in that When training the segmentation model and the classification model, a joint training mechanism is adopted to simultaneously minimize the mask segmentation error and the classification discrimination error during the training process, and end-to-end collaborative training is performed by optimizing the total loss function.

Images