A bolt recognition management method and system based on a convolutional neural network

Abstract

The embodiment of the present application relates to the maintenance technical field, disclose a kind of based on convolutional neural network's bolt identification management method, comprising: determining the fastener boundary box and fastener class identifier of fastener in initial detection image;According to fastener class identifier and the fastener-bolt mapping table of pre-constructed to determine fastener-bolt spatial position relationship, and according to fastener boundary box and fastener-bolt spatial position relationship, dynamically generate bolt detection area;Bolt detection area is input to second level detection model to carry out bolt identification to obtain corresponding position bolt's current state information and bolt target position;Output bolt target position and bolt's current state information.The bolt identification management method based on convolutional neural network in the embodiment of the present application adopts cascade detection, and computing resources are concentrated into high confidence local area, and detection accuracy and speed are significantly improved under the same computing power.

Description

Technical Field

[0001] This invention relates to the field of maintenance technology, specifically to a bolt identification and management method and system based on convolutional neural networks. Background Technology

[0002] Currently, railway rails are fixed to sleepers using a fastening system (elastic clips, bolts, washers, etc.), among which the fastening bolts are key components ensuring the stability of the track's geometric dimensions. Bolts are exposed to the outdoor environment for extended periods, suffering from wind, rain, and train vibrations, making them highly susceptible to corrosion. This corrosion leads to reduced fastening force, difficulty in disassembly, and even compromises the safety of the track structure. Therefore, regular lubrication and maintenance of bolts is an important aspect of railway maintenance. Consequently, designing a system for identifying and managing bolts has become a pressing technical problem for those skilled in the art. Summary of the Invention

[0003] To address the aforementioned shortcomings, this invention discloses a bolt recognition and management method based on a convolutional neural network, which can achieve accurate and rapid bolt recognition.

[0004] The first aspect of this invention discloses a bolt recognition and management method based on a convolutional neural network, comprising: The initial detection image captured during the movement of the intelligent fuel injection vehicle is acquired. The initial detection image is then identified using a first-level detection model, and the fastener boundary box and fastener category identifier in the initial detection image are determined. The spatial relationship between fasteners and bolts is determined based on the fastener category identifier and the pre-built fastener-bolt mapping table, and the bolt detection area is dynamically generated based on the fastener boundary box and the spatial relationship between fasteners and bolts. The bolt detection area is input into the second-level detection model for bolt identification to obtain the current status information of the bolt at the corresponding position and the target position of the bolt; Output the target position of the bolt and the current status information of the bolt.

[0005] As an optional implementation, in the first aspect of the present invention, the first-level detection model is a lightweight object detection model based on attention mechanism enhancement, and the first-level detection model includes a convolutional neural network and a weight calculation network. The step of identifying the initial detection image using a first-level detection model and determining the fastener bounding box and fastener category identifier in the initial detection image includes: The initial feature map is obtained by extracting features from the initial detection image using the convolutional neural network. The initial feature map is forward-propagated in the convolutional neural network to obtain a multi-scale feature map generated by a preset convolutional layer based on the initial feature map. The multi-scale feature map is input into a weight calculation network, which calculates the weight value of each spatial location in the initial detection image. The weight value represents the probability confidence that each spatial location belongs to the fastener region. An attention weight map of the initial detection image is constructed based on the obtained weight values. The attention weight map is a weight matrix with the same spatial resolution as the initial detection image. The weight values ​​of the fastener candidate region in the attention weight map are higher than those of the non-fastener region. The initial detection image is fused with the attention weight map pixel by pixel to obtain a weighted initial detection image; The weighted initial detection image is input into the detection head of the convolutional neural network for target detection, and the output fastener bounding box and fastener category identifier are generated.

[0006] As an optional implementation, in the first aspect of the present invention, the convolutional neural network of the first-level detection model is an improved YOLOv8n, whose backbone network adopts the CSPDarknet53 structure, and whose neck network adopts a weighted bidirectional feature pyramid network for cross-scale feature fusion; the multi-scale feature map includes a first-scale feature map, a second-scale feature map and a third-scale feature map, which respectively correspond to fastener targets of different scales in the initial detection image.

[0007] As an optional implementation, in the first aspect of the present invention, the second-level detection model is a key point detection model based on heatmap regression, and the second-level detection model includes a convolutional neural network with an encoder-decoder structure and a heatmap prediction head; The bolt detection area is input into the second-level detection model for bolt identification to obtain the target position of the bolt at the corresponding location, including: The local image after cropping the bolt detection area is input into the convolutional neural network of the encoder-decoder structure for feature extraction and upsampling reconstruction, and the output is a feature reconstruction map with the same spatial resolution as the local image. The feature reconstruction map is input into the heatmap prediction head, and the output is a heatmap of the bolt center point. The value of each pixel in the heatmap of the bolt center point represents the probability that the position is the bolt center point. The position with the largest response value in the heat map of the bolt center point is obtained as the pixel-level coordinate of the bolt center point; Using the pixel-level coordinates as the center, Gaussian surface fitting is performed on the thermal response values ​​within its neighborhood, and the peak position of the Gaussian surface is taken as the target bolt position of the bolt center point.

[0008] As an optional implementation, in the first aspect of the present invention, the second-level detection model further includes a semantic segmentation head, wherein the semantic segmentation head and the heatmap prediction head share the convolutional neural network of the encoder-decoder structure; The method further includes: The feature reconstruction map is input into the semantic segmentation head, and the pixel-level semantic segmentation result of the bolt detection region is output. The semantic segmentation result includes the bolt head mask, the spring strip mask and the background mask. The boundary line between the bolt head edge and the spring clip edge is extracted based on the bolt head mask and the spring clip mask. The boundary line is expanded into an annular contact band of a predetermined width, which serves as the contact area between the bolt and the fastener spring clip. The center point coordinates and local normal vector of the contact area are output. The predetermined width is determined based on the actual contact gap width between the bolt head and the spring clip, and the width of the annular contact band is 2mm to 5mm. The annular contact band is the spray target area of ​​the nozzle, and the intelligent oil spraying vehicle is used to spray grease into the area covered by the annular contact band. The target spray attitude of the corresponding nozzle is determined based on the center point coordinates and the local normal vector. The second-level detection model is a multi-task jointly trained network, and its joint loss function during the training phase is: in, To predict losses using heatmaps, the prediction accuracy of the probabilistic heatmap of bolt center points is constrained. The semantic segmentation loss is used to constrain the segmentation accuracy of the bolt head mask, spring bar mask, and background mask. The keypoint regression loss is used to constrain the regression accuracy of the position coordinates of the multiple semantic keypoints. and These are the weighting coefficients for the corresponding loss terms.

[0009] As an optional implementation, in the first aspect of the present invention, the management method further includes: Within the bolt detection area, the bolt target position of the bolt center point is located by the second-level detection model, and a local spatial vector relationship is constructed based on multiple semantic key points within the bolt detection area. The dynamic spatial offset is calculated based on the local spatial vector relationship and the fastener-bolt spatial position relationship. The bolt target position of the bolt center point is corrected using the dynamic spatial offset to obtain the bolt target position at the current moment. The spatial relationship between the fastener and the bolt is the mean vector and covariance matrix of the offset between the center of the fastener frame and the center point of the bolt, obtained by fitting the labeled data using a Gaussian mixture model. The local spatial vector relationship is obtained by detecting multiple semantic key points within the bolt detection area, including the spring clip fulcrum, bolt center point, and rail bottom edge point, and constructing a 2D vector relationship graph with the geometric center of the fastener boundary frame as the origin; The dynamic spatial offset is the residual offset of the local spatial vector relationship relative to the spatial positional relationship of the fastener-bolt.

[0010] As an optional implementation, in the first aspect of the present invention, the construction of the local spatial vector relationship is determined in the following manner: The cropped local image of the bolt detection area is input into the third detection model, which is a multi-keypoint detection model, and outputs the position coordinates of multiple semantic key points in the bolt detection area. The multiple semantic key points include the left support point of the elastic bar, the right support point of the elastic bar, the center point of the bolt, and the edge point of the rail bottom. Using the geometric center of the fastener boundary frame as the origin, calculate the relative position vector of each semantic key point with respect to the origin, and construct the local spatial vector relationship; The third detection model shares the same backbone network as the second-level detection model. The second-level detection model and the third detection model are multi-task joint detection models, which include a shared encoder, a heatmap prediction head, a semantic segmentation head, and a multi-keypoint regression head. The shared encoder outputs a feature reconstruction map of the bolt detection region. The heatmap prediction head, the semantic segmentation head, and the multi-keypoint regression head take the feature reconstruction map as input and output the bolt center point heatmap, semantic segmentation results, and position coordinates of each semantic keypoint in parallel.

[0011] As an optional implementation, in the first aspect of the present invention, the identification management method further includes: A digital profile is established for each bolt on the line. The digital profile includes the bolt's unique identifier, LBS coordinates, historical maintenance time, maintenance records, and historical bolt status. The historical bolt status is updated based on environmental factors. The planned travel path of the intelligent fuel injection vehicle is obtained, and the LBS trajectory of the planned travel path is spatially superimposed with the list of unmaintained bolts to generate a daily task sequence. The daily task sequence is configured to ensure that the intelligent fuel injection vehicle covers the most unmaintained bolts and has the shortest repeated path in a single travel path. During the bolt identification process, when the intelligent fuel injection vehicle reaches the target bolt position, it queries the digital file of the target bolt. If the target bolt has been marked as having a qualified oil film condition, it automatically skips the target bolt and records the reason for skipping.

[0012] A second aspect of this invention discloses a bolt recognition and management system based on a convolutional neural network, comprising: Level 1 Detection Module: Used to acquire initial detection images captured during the movement of the intelligent fuel injection vehicle, identify the initial detection images through the Level 1 Detection Model, and determine the fastener bounding box and fastener category identifier in the initial detection images; Dynamic generation module: used to determine the spatial positional relationship between fasteners and bolts based on the fastener category identifier and the pre-built fastener-bolt mapping table, and dynamically generate the bolt detection area based on the fastener boundary box and the spatial positional relationship between fasteners and bolts; Secondary detection module: used to input the bolt detection area into the secondary detection model to identify the bolts and obtain the current status information of the bolts at the corresponding positions and the target position of the bolts; Output module: Used to output the target position of the bolt and the current status information of the bolt.

[0013] A third aspect of the present invention discloses an electronic device, comprising: a memory storing executable program code; a processor coupled to the memory; the processor calling the executable program code stored in the memory to execute the bolt recognition and management method based on a convolutional neural network disclosed in the first aspect of the present invention.

[0014] A fourth aspect of the present invention discloses a computer-readable storage medium storing a computer program, wherein the computer program causes a computer to execute the bolt identification and management method based on a convolutional neural network disclosed in the first aspect of the present invention.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The bolt recognition and management method based on convolutional neural networks in this invention identifies fasteners across the entire image using a first-level detection model, while a second-level detection model identifies bolts only within a cropped local area. This coarse-to-fine spatial detection strategy allows for collaborative work among the various models: the first-level model quickly locates the region of interest, while the second-level model handles high-resolution detail analysis. Cascaded detection concentrates computational resources on high-confidence local areas, significantly improving detection accuracy and speed with the same computing power. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic flowchart of the bolt identification and management method based on convolutional neural networks disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the detection process of the first-level detection model disclosed in the embodiments of the present invention; Figure 3 This is a schematic diagram of the detection process of the second-level detection model disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of a bolt identification and management system based on a convolutional neural network provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that the terms "first," "second," "third," "fourth," etc., in the specification and claims of this invention are used to distinguish different objects, not to describe a specific order. The terms "comprising" and "having," and any variations thereof, in the embodiments of this invention are intended to cover non-exclusive inclusion. Exemplarily, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0020] Example 1 Please see Figure 1 , Figure 1This is a flowchart illustrating the bolt identification and management method based on a convolutional neural network disclosed in an embodiment of the present invention. The execution entity of the method described in this embodiment is an execution entity composed of software and / or hardware. This execution entity can receive relevant information via wired or / or wireless means and can send certain instructions. It may also have certain processing and storage functions. This execution entity can control multiple devices, such as remote physical servers or cloud servers and related software, or local hosts or servers and related software that perform related operations on devices located in a certain location. In some scenarios, multiple storage devices can also be controlled; these storage devices may be placed in the same location as the devices or in different locations. Figure 1 As shown, the bolt recognition and management method based on convolutional neural networks includes the following steps: S101: Acquire the initial detection image captured during the movement of the intelligent fuel injection vehicle, identify the initial detection image through the first-level detection model, and determine the fastener boundary box and fastener category identifier in the initial detection image. S102: Determine the spatial relationship between fasteners and bolts based on the fastener category identifier and the pre-built fastener-bolt mapping table, and dynamically generate a bolt detection area based on the fastener boundary frame and the spatial relationship between fasteners and bolts; S103: Input the bolt detection area into the second-level detection model to identify the bolts and obtain the current status information of the bolts at the corresponding positions and the target position of the bolts; S104: Output the target position of the bolt and the current status information of the bolt.

[0021] The solution presented in this invention significantly improves detection efficiency and real-time performance. The first-level detection (global fastener location) has low computational complexity and high speed, while the second-level detection (local bolt location) only needs to process a very small cropped area, avoiding sliding windowing or high-precision convolution on the entire large image. This significantly reduces overall inference latency and is more suitable for the high-speed, real-time processing requirements of intelligent fuel injection vehicles in motion.

[0022] The two-level detection model effectively addresses the issues of small targets and background interference. Bolts are typical small targets in railway scenarios, making them prone to missed detection by direct inspection. This scheme narrows the search range from the entire image to a local high-confidence area by utilizing the spatial relationship between fasteners and bolts, effectively removing complex background noise such as rails, track bed, and oil stains. The high input signal-to-noise ratio of the second-level detection model significantly improves the classification accuracy of bolt states (loose, missing, corroded) and the regression accuracy of bolt target locations.

[0023] Because the fuel injection vehicle vibrates during movement, the relative position of the camera and the track is not fixed. This solution does not rely on fixed pixel coordinates, but instead calculates the fastener bounding box of the current frame in real time and then dynamically generates the bolt detection area. This gives the system adaptive anti-shake capability, ensuring that the bolt detection area can always accurately follow the fastener movement under different lighting conditions and different operating speeds, avoiding detection box offset failure due to vehicle bumps.

[0024] The final output bolt target position can be directly used to drive the robotic arm or fuel injector for fixed-point operation, while the current status information is sent back to the management backend. This direct conversion of visual perception results into execution commands and maintenance records ensures the accuracy of the fuel spraying operation (avoiding mis-spraying and missed spraying) and provides quantitative data support for subsequent bolt lifecycle health management.

[0025] Existing solutions involve acquiring bolt images, training a detection model, and directly outputting the bolt's position and status. While feasible in static scenarios, this approach has drawbacks in dynamic railway operation scenarios. Bolts are small, easily obscured by oil and rust, and their imaging is unstable due to vehicle vibrations, resulting in extremely high false negative and false positive rates. However, rusted areas require more maintenance, which increases the risk to rail safety. The solution in this invention employs a different technical concept: first, identifying structurally stable and characteristic fasteners; then, utilizing the fastener-bolt spatial constraint relationship; dynamically calculating the bolt's location; and finally, precisely locating the bolt within a small area.

[0026] The purpose of this invention is to transform the identification of a small, easily disturbed target into the identification of a stable, large target and geometric deduction, which is a spatial relationship-anchored indirect identification method. Fasteners, as fixing components of railway tracks, are larger in size, have obvious structural features, are less affected by oil contamination, and their detection stability is far higher than that of bolts.

[0027] The problem with existing technologies is that bolts, being small targets, occupy a very small percentage of pixels in an image, making it difficult for existing target detection algorithms to reliably detect them under conditions such as oil stains, changes in lighting, and vehicle vibration. This application addresses the inherent limitations of direct detection of small targets by employing indirect spatial reasoning to achieve stable and accurate bolt positioning in dynamic environments.

[0028] Specifically, the first-level detection model performs spatial anchoring, identifying the fastener bounding box and category identifier; the second-level detection model performs fine positioning within the local area where the fastener is anchored, generating and accurately locating the bolt detection area based on spatial relationships. There is a clear division of functions and information transmission between the two, which is not simply a matter of connecting two existing models, but rather the construction of a new spatial perception method.

[0029] This application reduces the search range of the second-level detection model from the entire image to a local area near the fastener by using fastener bounding boxes and spatial mapping. Assuming the image size is 1920×1080, and the fastener area occupies approximately 5%~10% of the image area, the search range is reduced to 1 / 10~1 / 20 of the original, significantly decreasing the false negative rate and significantly improving the positioning accuracy.

[0030] It is important to emphasize that when identifying fasteners, one can identify the fastener as a whole or directly identify the elastic clip within the fastener. The elastic clip's shape is relatively easier to identify than that of a bolt. In railway fastener systems, the elastic clip has a specific elongated, curved rod-like structure with highly geometrical specificity (specific aspect ratio, radius of curvature, and spatial extension direction), maintaining a recognizable outline even in harsh environments such as oil stains and corrosion. In contrast, bolts have a near-circular, compact structure with a single feature dimension. Once the surface texture is covered by oil or the edges are blurred by corrosion, their visual recognizability drops sharply. This application utilizes the elastic clip, with its higher morphological recognizability and stronger anti-interference ability, as the initial anchoring target, rather than directly detecting bolts whose shape is easily degraded and confused. This constitutes a fundamental difference in technical approach from existing technologies that directly detect bolts.

[0031] More preferably, such as Figure 2 As shown, the first-level detection model is a lightweight object detection model enhanced by an attention mechanism. The first-level detection model includes a convolutional neural network and a weight calculation network. The step of identifying the initial detection image using a first-level detection model and determining the fastener bounding box and fastener category identifier in the initial detection image includes: S1011: The initial detection image is extracted using the convolutional neural network to obtain an initial feature map; S1012: The initial feature map is forward-propagated in the convolutional neural network to obtain a multi-scale feature map generated by a preset convolutional layer based on the initial feature map; S1013: Input the multi-scale feature map into the weight calculation network, and calculate the weight value of each spatial location in the initial detection image through the weight calculation network. The weight value represents the probability confidence that each spatial location belongs to the fastener region. S1014: Construct an attention weight map of the initial detection image based on the obtained weight values. The attention weight map is a weight matrix with the same spatial resolution as the initial detection image. The weight values ​​of the fastener candidate region in the attention weight map are higher than those of the non-fastener region. S1015: The initial detection image and the attention weight map are fused pixel by pixel to obtain a weighted initial detection image; S1016: Input the weighted initial detection image into the detection head of the convolutional neural network for target detection, and output the fastener bounding box and fastener category identifier.

[0032] In railway scenarios, fasteners are often obscured by oil, rust, or shadows. Traditional CNNs treat all pixels equally, easily misclassifying obscured objects as background. This solution generates an attention weight map through a weighted calculation network, actively strengthening candidate fastener regions (with high weight values) while suppressing irrelevant backgrounds (rails, ballast). This ensures that even if the fastener's appearance is blurred, its spatial location features can still be perceived by the network, significantly reducing the false negative rate and preventing the inability to generate subsequent bolt detection regions due to missed fasteners.

[0033] Non-fastener areas (such as metal reflections and gravel on the track) are assigned low weights in the attention weight map. Even if the detection head responds in these areas, its contribution is attenuated after pixel-by-pixel weighted fusion. Through a soft masking mechanism, false targets are effectively filtered out, improving the confidence purity of fastener category identification and reducing resource waste caused by generating invalid bolt detection areas due to false detections.

[0034] Conventional detection models simultaneously perform the dual tasks of location finding and classification, which often interfere with each other. In this proposed solution, the weighted calculation network focuses on spatial location confidence (probability map), while the detection head focuses on bounding box regression and classification. This decoupled design allows different parts of the network to perform their respective functions, reducing optimization difficulty and enabling the model to maintain high accuracy with lightweight parameters, making it suitable for deployment on edge computing devices in fuel injection vehicles.

[0035] The solution in this invention explicitly extracts multi-scale feature maps (from preset convolutional layers) and calculates weights based on these feature maps. This means that the attention weight map integrates global semantic (deep) and local detail (shallow) information. This makes the model more robust to different shooting distances or changes in fastener size (such as sudden changes in target scale caused by vehicles going uphill or downhill), ensuring that fasteners at both near and far distances receive reasonable attention allocation.

[0036] The solution of this invention allows the attention weight calculation network and the detection head to perform joint backpropagation optimization under the same loss function, so that the generation of the attention weight map can be adaptively adjusted based on the final detection accuracy, rather than relying on manually preset rules.

[0037] More preferably, the convolutional neural network of the first-level detection model is an improved YOLOv8n, whose backbone network adopts the CSPDarknet53 structure, and whose neck network adopts a weighted bidirectional feature pyramid network for cross-scale feature fusion; the multi-scale feature map includes a first-scale feature map, a second-scale feature map and a third-scale feature map, which correspond to fastener targets of different scales in the initial detection image, respectively.

[0038] In practical implementation, YOLOv8n is the version with the fewest parameters and lowest computational cost in the YOLOv8 series (approximately 3.2M parameters). Combined with the cross-stage partial connection structure of CSPDarknet53, it ensures the effective flow of gradient information while reducing redundant gradient calculations. This allows the model to be deployed on embedded or edge computing devices in intelligent fuel injection vehicles, maintaining a high frame rate even with limited computing power, meeting the real-time processing requirements while in motion. Simultaneously, the feature reuse capability of the CSP structure prevents a sharp drop in accuracy due to its lightweight design.

[0039] Because the scale of fasteners in an image varies drastically with the distance between the fuel injection truck and the track (larger when closer, smaller when farther away), traditional FPN only performs top-down unidirectional fusion, limiting information flow. BiFPN introduces bidirectional (top-down and bottom-up) weighted fusion and learns trainable weight parameters for each fusion path to distinguish the importance of features at different levels. This allows small-scale fasteners (at a distance) to obtain high-resolution shallow detail information (edges, textures), while large-scale fasteners (at a distance) can obtain deep semantic information (class discrimination), achieving stable detection across all distance scales and avoiding missed detections due to distance variations.

[0040] The solution of this invention achieves accurate positioning of coarse, medium, and fine targets (three-scale feature map division of labor). The feature map at each scale is enhanced with BiFPN cross-scale information. This forms a mechanism of collaborative division of labor among the three-level detection heads. Compared with single-scale or dual-scale detection, it significantly improves the bounding box regression accuracy of fasteners of various sizes, ensuring that the output fastener bounding box closely fits the actual target, and providing a more accurate spatial reference for the subsequent dynamic generation of bolt detection areas.

[0041] In practice, BiFPN's learnable weights not only act between different feature layers but also implicitly suppress the propagation of background noise features. For example, when a feature map in a certain layer is rich in rail texture rather than fastener semantics, its fusion weights will be trained to be lower. This enhances the model's feature purity in complex railway backgrounds (ballast, glare, oil stains), enabling the subsequent attention weight calculation network to generate accurate spatial probability maps based on cleaner multi-scale features.

[0042] More preferably, such as Figure 3 As shown, the second-level detection model is a key point detection model based on heatmap regression. The second-level detection model includes a convolutional neural network with an encoder-decoder structure and a heatmap prediction head. The bolt detection area is input into the second-level detection model for bolt identification to obtain the target position of the bolt at the corresponding location, including: S1031: Input the cropped local image of the bolt detection area into the convolutional neural network of the encoder-decoder structure for feature extraction and upsampling reconstruction, and output a feature reconstruction map with the same spatial resolution as the local image; S1032: Input the feature reconstruction map into the heat map prediction head and output the bolt center point heat map. The value of each pixel position in the bolt center point heat map represents the probability that the position is the bolt center point. S1033: Obtain the position with the largest response value in the thermal map of the bolt center point as the pixel-level coordinate of the bolt center point; S1034: Using the pixel-level coordinates as the center, perform Gaussian surface fitting on the thermal response values ​​within its neighborhood, and use the peak position of the Gaussian surface as the target bolt position of the bolt center point.

[0043] Traditional object detection (such as YOLO) requires pre-setting a large number of anchor boxes on the feature map and regressing the width and height offsets of the bounding boxes, which is computationally intensive and requires manual parameter tuning. Heatmap regression, on the other hand, is essentially pixel-by-pixel binary classification and coordinate regression. The encoder-decoder network only needs to output a single-channel heatmap, without needing to pre-set any prior boxes. This allows the second-level model to be designed to be extremely lightweight, with very fast inference speed on cropped local images, ensuring ultra-low latency response during the movement of the fuel-injecting vehicle.

[0044] Conventional detection heads directly output integer pixel coordinates, resulting in quantization errors. The solution in this invention first extracts the integer positions of the peak values ​​from the heatmap, then performs Gaussian surface fitting on the neighborhood response values, using a continuous function to find the extreme points, thus improving coordinate accuracy to the sub-pixel level. For small targets like bolts, sub-pixel accuracy can reduce positioning errors, directly determining the subsequent fuel injector alignment accuracy and preventing lubricant spray deviation due to pixel-level deviations.

[0045] Bolts in images may have blurred edges or even be partially obscured due to rust, oil, or other contaminants. Traditional bounding box regression relies on sharp edges, making it prone to failure. Heatmaps, however, predict the probability density distribution of the bolt's central region; even with contaminated edges, the Gaussian response of the central region remains stable. The model learns the semantic understanding of where the bolt's center should be, rather than the geometric fit of where the bolt's edges are, significantly improving detection robustness and continuity under harsh conditions.

[0046] The encoder is responsible for downsampling to extract high-level semantic features (compressing the receptive field), while the decoder upsamples to restore the resolution to the input size, outputting a reconstructed feature map. This process decouples semantic understanding from spatial localization, and the reconstructed map remains spatially aligned with the local image. This allows the heatmap prediction head to perform probability estimation simply by performing pixel-wise convolutions on the spatially aligned feature map, greatly simplifying training, resulting in fast convergence, and insensitivity to scale changes in the input image.

[0047] During training, only the coordinates of the bolt center point (single pixel) need to be labeled, rather than the complete bounding box (which requires 4.8 coordinates). Furthermore, the Gaussian kernel radius of the heatmap can be adaptively adjusted. Compared to traditional detection box labeling, the labeling cost is reduced, and the labeling consistency is higher, facilitating rapid expansion of the training dataset to adapt to the recognition needs of different lines and bolt models.

[0048] More preferably, the second-level detection model further includes a semantic segmentation head, which shares the encoder-decoder structure of the convolutional neural network with the heatmap prediction head; The method further includes: The feature reconstruction map is input into the semantic segmentation head, and the pixel-level semantic segmentation result of the bolt detection region is output. The semantic segmentation result includes the bolt head mask, the spring strip mask and the background mask. The boundary line between the bolt head edge and the spring clip edge is extracted based on the bolt head mask and the spring clip mask. The boundary line is expanded into an annular contact band of a predetermined width, which serves as the contact area between the bolt and the fastener spring clip. The center point coordinates and local normal vector of the contact area are output. The predetermined width is determined based on the actual contact gap width between the bolt head and the spring clip, and the width of the annular contact band is 2mm to 5mm. The annular contact band is the spray target area of ​​the nozzle, and the intelligent oil spraying vehicle is used to spray grease into the area covered by the annular contact band. The target spray attitude of the corresponding nozzle is determined based on the center point coordinates and the local normal vector. The second-level detection model is a multi-task jointly trained network, and its joint loss function during the training phase is: in, To predict losses using heatmaps, the prediction accuracy of the probabilistic heatmap of bolt center points is constrained. The semantic segmentation loss is used to constrain the segmentation accuracy of the bolt head mask, spring bar mask, and background mask. The keypoint regression loss is used to constrain the regression accuracy of the position coordinates of the multiple semantic keypoints. and These are the weighting coefficients for the corresponding loss terms.

[0049] Traditional solutions only output the coordinates of the bolt center point (a single point), but the actual target of the lubrication operation should be the contact gap between the bolt and the spring clip (i.e., the area where the grease needs to penetrate). This solution uses a semantic segmentation head to accurately segment the bolt head mask and the spring clip mask, then calculates the boundary line between them and expands it into a ring-shaped contact zone, transforming the abstract point target into a surface target with physical width. This achieves a leap from detecting the bolt to accurately locating the contact surface that needs lubrication. The output spray target area directly corresponds to the actual operational requirements, avoiding ineffective operations where grease is sprayed onto the top surface of the bolt or the surface of the spring clip but cannot penetrate the gap.

[0050] The center point coordinates alone are insufficient to guide the nozzle attitude; the spray angle also needs to be determined. This scheme extracts the local normal vector (a 3-DOF orientation perpendicular to the contact strip direction) and combines it with the center point coordinates to form a multi-DOF target pose, improving control accuracy.

[0051] In this embodiment of the invention, the semantic segmentation head and the heatmap prediction head share the same encoder-decoder structure. That is, only feature extraction and upsampling reconstruction need to be performed once during forward propagation, and the two task heads compute their respective outputs in parallel. Compared with running two independent models, the computational load is almost not increased (only a small number of convolutional layer parameters are added at the end of the segmentation head), and the real-time frame rate can still be maintained on embedded devices, achieving a near-free gain in detection accuracy.

[0052] The joint loss function forces the shared encoder to learn a general feature representation that is beneficial for both center point localization and semantic segmentation. The semantic segmentation task provides rich local texture and edge supervision signals, which is equivalent to an implicit data augmentation and regularization. It effectively suppresses the overfitting that may occur in the single heatmap task, making the features extracted by the shared encoder more discriminative and robust. The accuracy of both tasks can be improved in complex scenarios such as oil stains and changes in lighting.

[0053] Setting the width of the annular contact strip to 2mm to 5mm (based on the actual contact gap width between the bolt head and the spring clip) establishes a clear mapping relationship between visual perception results and oil spraying process parameters (such as injection flow rate, injection time, and nozzle sweeping speed). This upgrades oil spraying operations from experience-based estimation to a standardized, reproducible digital process, facilitating subsequent quality traceability and process optimization, and also providing a basis for adaptive adjustment for different bolt types (different gap widths).

[0054] More preferably, the management method further includes: Within the bolt detection area, the bolt target position of the bolt center point is located by the second-level detection model, and a local spatial vector relationship is constructed based on multiple semantic key points within the bolt detection area. The dynamic spatial offset is calculated based on the local spatial vector relationship and the fastener-bolt spatial position relationship. The bolt target position of the bolt center point is corrected using the dynamic spatial offset to obtain the bolt target position at the current moment. The spatial relationship between the fastener and the bolt is the mean vector and covariance matrix of the offset between the center of the fastener frame and the center point of the bolt, obtained by fitting the labeled data using a Gaussian mixture model. The local spatial vector relationship is obtained by detecting multiple semantic key points within the bolt detection area, including the spring clip fulcrum, bolt center point, and rail bottom edge point, and constructing a 2D vector relationship graph with the geometric center of the fastener boundary frame as the origin; The dynamic spatial offset is the residual offset of the local spatial vector relationship relative to the spatial positional relationship of the fastener-bolt.

[0055] The movement of an intelligent fuel injection vehicle inevitably generates random vibrations and changes in its trajectory curves, causing slight variations in the relative pose between the camera and the fasteners in each frame. Relying solely on the dynamic generation of the bolt detection area using the fastener bounding box still results in residual offset errors (such as pixel-level fluctuations in the fastener detection box itself). The solution in this invention extracts multiple semantic key points (elastic fulcrum points, rail bottom edge points, etc.) within the detection area in real time, constructs the local vector relationship for the current frame, and compares it with the prior Gaussian Mixture Model (GMM) statistical relationship to calculate and correct the residual offset. This upgrades bolt positioning from open-loop calculation to closed-loop feedback correction, effectively filtering out inter-frame drift caused by vehicle body vibrations and camera thermal expansion and contraction, ensuring that the bolt target position output in each frame is continuous and smooth over time, and avoiding severe jitter at the nozzle execution end.

[0056] Specifically, the spatial relationship between fasteners and bolts is not a simple fixed value, but a Gaussian mixture model (GMM) fitted from a large amount of labeled data, including a mean vector (typical offset direction) and a covariance matrix (reasonable fluctuation range of the offset). When a frame experiences anomalies in semantic keypoint detection due to local occlusion, the correction amount is constrained by the confidence interval of the covariance matrix, preventing excessive deviation from statistical regularity. This significantly improves the system's resistance to single-point failures and long-term operational stability.

[0057] Purely visual pixel coordinate positioning is susceptible to the influence of shooting angle (e.g., perspective distortion caused by angled shots leads to center offset). This solution uses a 2D vector relationship diagram constructed from multiple coplanar key points (elastic support point, rail bottom edge, bolt center), and utilizes geometric constraints between these points (e.g., parallelism, perpendicularity, distance ratio) to calibrate the bolt center position. This makes the positioning result closer to the actual physical spatial position rather than the image projection position. Even under different shooting angles, the corrected bolt target position remains highly consistent, improving the system's generalization ability across different track sections and lighting conditions.

[0058] In practical implementation, the calculation of residual offsets supports temporal smoothing of historical frames. For example, the dynamic spatial offset of the current frame can be subjected to an exponentially weighted moving average (EWMA) with the offset of the previous frame to achieve a Kalman filter-style recursive estimation.

[0059] The mean vector and covariance matrix obtained by fitting a Gaussian mixture model in this embodiment of the invention essentially represent a parametric encoding of the geometric patterns implicit in a large amount of manually labeled data. This prior knowledge can be trained offline and used online without recalculating each time it runs. This achieves a knowledge reuse model of one-time labeling and long-term benefit. At the same time, the covariance matrix of the GMM can also be used for online anomaly detection. When the residual offset exceeds the 3σ confidence interval, an alarm can be triggered to indicate abnormalities in the fastener or bolt structure (such as deformation or displacement), adding to the system's ability to perceive abnormal working conditions.

[0060] More preferably, the construction of the local spatial vector relationship is determined in the following manner: The cropped local image of the bolt detection area is input into the third detection model, which is a multi-keypoint detection model, and outputs the position coordinates of multiple semantic key points in the bolt detection area. The multiple semantic key points include the left support point of the elastic bar, the right support point of the elastic bar, the center point of the bolt, and the edge point of the rail bottom. Using the geometric center of the fastener boundary frame as the origin, calculate the relative position vector of each semantic key point with respect to the origin, and construct the local spatial vector relationship; The third detection model shares the same backbone network as the second-level detection model. The second-level detection model and the third detection model are multi-task joint detection models, which include a shared encoder, a heatmap prediction head, a semantic segmentation head, and a multi-keypoint regression head. The shared encoder outputs a feature reconstruction map of the bolt detection region. The heatmap prediction head, the semantic segmentation head, and the multi-keypoint regression head take the feature reconstruction map as input and output the bolt center point heatmap, semantic segmentation results, and position coordinates of each semantic keypoint in parallel.

[0061] In practice, the three task heads share the same feature reconstruction map, meaning that the shared encoder must learn a general feature representation that is beneficial for center point localization, semantic segmentation, and keypoint regression simultaneously. Semantic segmentation provides fine-grained boundary supervision signals, keypoint regression provides sparse but precise spatial constraints, and the heatmap provides dense probability distribution supervision.

[0062] A three-in-one collaborative training mechanism is formed: the boundary information of semantic segmentation helps keypoint regression to more accurately locate the spring bar fulcrum; the sparse constraints of keypoints, in turn, correct the blurred edges of the segmentation mask; and the center prior constraints of the heatmap converge together. This deep collaboration makes the quality of shared features significantly better than any single task training, and greatly enhances the generalization ability.

[0063] Using the geometric center of the fastener bounding box as the origin, the relative position vectors of each semantic key point are calculated to construct a local spatial vector relationship. This operation essentially establishes a locally normalized coordinate system independent of image translation, rotation, and scaling. The left and right pivot points of the spring bar and the bottom edge point of the rail constitute a rigid geometric structure, and their relative positional relationships are almost fixed in physical space. Using this rigid geometric constraint, the system can verify the consistency of the detection results of each key point in real time. If a key point deviates from its reasonable position range relative to other points (such as exceeding a predetermined threshold), it can be determined that the detection of that point is abnormal and trigger a retest or weight reduction, greatly improving the system's self-diagnostic capability and fault tolerance.

[0064] The multiple semantic key points output by the third detection model not only directly serve the construction of local vector relationships, but also provide multi-dimensional geometric verification signals for the aforementioned dynamic spatial offset correction. For example, the direction of the line connecting the left and right support points of the spring clip can verify the longitudinal direction of the track (i.e., the direction of travel of the fuel injection vehicle), and the bottom edge point of the track can verify the lateral levelness of the track. This upgrades the offset correction from one-dimensional point alignment to multi-dimensional geometric registration, which is equivalent to using the rigid structure of the entire fastener area for global optimal alignment, rather than relying solely on the center point of a single bolt, further improving the positioning correction accuracy.

[0065] More preferably, the identification management method further includes: A digital profile is established for each bolt on the line. The digital profile includes the bolt's unique identifier, LBS coordinates, historical maintenance time, maintenance records, and historical bolt status. The historical bolt status is updated based on environmental factors. The planned travel path of the intelligent fuel injection vehicle is obtained, and the LBS trajectory of the planned travel path is spatially superimposed with the list of unmaintained bolts to generate a daily task sequence. The daily task sequence is configured to ensure that the intelligent fuel injection vehicle covers the most unmaintained bolts and has the shortest repeated path in a single travel path. During the bolt identification process, when the intelligent fuel injection vehicle reaches the target bolt position, it queries the digital file of the target bolt. If the target bolt has been marked as having a qualified oil film condition, it automatically skips the target bolt and records the reason for skipping.

[0066] In practice, a complete digital archive is created for each bolt on the line, including a unique identifier, LBS coordinates (Location-Based Service), historical maintenance time, maintenance records, and the bolt's historical status. The bolt's historical status is also dynamically updated based on environmental factors (such as humidity, temperature, and dust concentration). By constructing a one-to-one digital twin of the bolts in the physical world in the digital space, a leap is achieved from passive response maintenance (addressing problems after they are discovered) to proactive predictive maintenance (predicting the optimal maintenance time based on historical data and environmental trends), laying a data foundation for the subsequent introduction of AI predictive maintenance.

[0067] The planned travel path LBS trajectory of the intelligent fuel injection vehicle is spatially overlaid with the list of unmaintained bolts to automatically generate a task sequence for the day. The optimization objective of this sequence is to cover the most unmaintained bolts in a single travel path while minimizing the number of repeated paths. This is essentially an engineering approximate optimal solution to the constrained vehicle routing problem (VRP).

[0068] When the injection truck reaches the target bolt location, it first checks the digital record. If the bolt is already marked as having a satisfactory oil film condition (e.g., it was maintained by another vehicle earlier that day, or the oil film thickness has been visually verified to be within the acceptable range), it is automatically skipped, and the reason for skipping is recorded. This effectively avoids the waste of grease and the risk of over-lubrication caused by repeated maintenance (over-lubrication can actually attract dust and accelerate wear), while also preventing missed maintenance. This check-before-action mechanism upgrades operational decision-making from blindly following a plan to precise execution based on demand.

[0069] The results of each identification and maintenance action (whether it was skipped, the reason for skipping, the actual amount of oil injected, the oil injection posture, etc.) can be written back to the digital file of the corresponding bolt in real time, forming a complete work history.

[0070] Specifically, the historical status of bolts is not statically stored, but dynamically updated based on environmental factors such as rainfall, temperature fluctuations, and salt spray concentration. For example, after continuous rainfall, the corrosion risk level of all outdoor bolts is automatically increased; in high-temperature weather, the evaporation rate of lubricating grease accelerates, and the remaining effective lifespan is automatically shortened. This gives the digital archive a living, perceptive capability, enabling a shift from time-based, fixed-cycle maintenance to dynamic, on-demand maintenance based on actual conditions, which can extend the service life of bolts and reduce the total life-cycle maintenance cost.

[0071] Specifically, the generation of the daily task sequence relies on a globally shared state of the list of unmaintained bolts. When multiple fuel injection trucks work collaboratively on the same line, the bolt status in the database is updated immediately after any vehicle completes maintenance, and the daily task sequences of other vehicles can be dynamically rearranged to avoid conflicts. An architectural interface is reserved for the future introduction of a multi-agent collaborative scheduling system (multiple fuel injection trucks operating in parallel), giving the system good horizontal scalability and adapting to large-scale line maintenance needs.

[0072] Specifically, to improve detection robustness, the sensing unit also includes a line structured light sensor; the steps for identifying fastener targets in the initial detection image through the first-level detection model include: Simultaneously acquire 2D intensity images and 3D depth images of the intelligent fuel injection vehicle during its movement; The 2D intensity image is input into the convolutional neural network of the first-level detection model to detect the fastener target, and the 2D fastener detection result is output, which includes the first fastener bounding box and the first confidence score. The 3D depth image is input into the depth feature extraction network for fastener matching and detection. Regions in the 3D depth image where the depth value changes abruptly are extracted as candidate fastener regions. The 3D fastener detection results are output, which include the second fastener bounding box and the second confidence score. Based on the first and second confidence levels, the 2D fastener detection results and the 3D fastener detection results are fused at the feature pyramid level using attention weighting, and the fused fastener bounding box and fastener category identifier are output.

[0073] Specifically, the steps for attention-weighted fusion of 2D and 3D fastener detection results at the feature pyramid level include: Extracting multi-scale 2D feature maps from 2D intensity images at different stages of the backbone network; Extracting multi-scale 3D feature maps from different stages of a depth feature extraction network for 3D depth images; At the same scale level, channel concatenation features of 2D feature maps and 3D feature maps are calculated, and fusion weights are calculated through the attention-weighted fusion module. The fused feature map is obtained by weighted summation of the 2D feature map and the 3D feature map according to the fusion weight. The fused feature maps at each scale level are input into the neck network and detection head of the first-level detection model for target detection.

[0074] This application incorporates a line structured light sensor into the sensing unit, constructing a heterogeneous multimodal fusion sensing architecture that integrates 2D intensity images and 3D depth images. Unlike existing technologies that simply stitch together 2D and 3D information or fuse only at the decision level, this application adaptively calculates the 2D-3D fusion weights for each spatial location and each feature channel at multiple scale levels of the feature pyramid through an attention-weighted fusion module. This achieves deep complementarity and dynamic synergy between texture and geometric information. The core value of this fusion strategy lies in the fact that when degradation factors such as oil stains, corrosion, and changes in lighting conditions under railway operating conditions cause 2D visual features to fail, the geometric contours represented by depth information (which are inherently insensitive to these degradation factors) can automatically take over, thereby ensuring that the sensing system maintains stable detection performance even under harsh conditions. This technical concept is not a simple superposition of 2D and 3D detection, but a new paradigm of multimodal fusion with an inherent synergistic mechanism.

[0075] The bolt recognition and management method based on convolutional neural networks in this invention identifies fasteners across the entire image using a first-level detection model, while a second-level detection model identifies bolts only within a cropped local area. This coarse-to-fine spatial detection strategy allows for collaborative work among the various models: the first-level model quickly locates the region of interest, while the second-level model handles high-resolution detail analysis. Cascaded detection concentrates computational resources on high-confidence local areas, significantly improving detection accuracy and speed with the same computing power.

[0076] Example 2 Please see Figure 4 , Figure 4 This is a schematic diagram of the bolt recognition and management system based on a convolutional neural network disclosed in an embodiment of the present invention. Figure 4 As shown, the bolt recognition and management system based on convolutional neural networks may include: Level 1 detection module 21: used to acquire the initial detection image captured during the movement of the intelligent fuel injection vehicle, identify the initial detection image through the first-level detection model, and determine the fastener boundary box and fastener category identifier in the initial detection image; Dynamic generation module 22: used to determine the spatial position relationship between fasteners and bolts based on the fastener category identifier and the pre-built fastener-bolt mapping table, and dynamically generate the bolt detection area based on the fastener boundary frame and the spatial position relationship between fasteners and bolts; Secondary detection module 23: used to input the bolt detection area into the secondary detection model for bolt identification to obtain the current status information of the bolt at the corresponding position and the target position of the bolt; Output module 24: Used to output the target position of the bolt and the current status information of the bolt.

[0077] The bolt recognition and management method based on convolutional neural networks in this invention identifies fasteners across the entire image using a first-level detection model, while a second-level detection model identifies bolts only within a cropped local area. This coarse-to-fine spatial detection strategy allows for collaborative work among the various models: the first-level model quickly locates the region of interest, while the second-level model handles high-resolution detail analysis. Cascaded detection concentrates computational resources on high-confidence local areas, significantly improving detection accuracy and speed with the same computing power.

[0078] Example 3 Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of the present invention. The electronic device can be a computer, a server, etc. Of course, in certain cases, it can also be a mobile phone, tablet computer, monitoring terminal, or other smart device, as well as an image acquisition device with processing capabilities. Figure 5 As shown, the electronic device may include: Memory 510 storing executable program code; Processor 520 coupled to memory 510; The processor 520 calls the executable program code stored in the memory 510 to execute some or all of the steps in the bolt recognition and management method based on convolutional neural networks in Embodiment 1.

[0079] This invention discloses a computer-readable storage medium storing a computer program that causes a computer to perform some or all of the steps in the bolt identification and management method based on a convolutional neural network in Embodiment 1.

[0080] This invention also discloses a computer program product, wherein when the computer program product is run on a computer, the computer performs some or all of the steps in the bolt identification and management method based on convolutional neural networks in Embodiment 1.

[0081] This invention also discloses an application publishing platform, which is used to publish computer program products. When the computer program products are run on a computer, the computer executes some or all of the steps in the bolt recognition and management method based on convolutional neural networks in Embodiment 1.

[0082] In various embodiments of the present invention, it should be understood that the sequence number of each process does not necessarily imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0083] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they can be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0084] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0085] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-accessible memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several requests to cause a computer device (which can be a personal computer, server, or network device, specifically a processor in the computer device) to execute some or all of the steps of the methods described in the various embodiments of the present invention.

[0086] In the embodiments provided by this invention, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0087] Those skilled in the art will understand that some or all of the steps in the various methods of the embodiments described can be implemented by a program instructing related hardware. This program can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0088] The bolt identification and management method, system, electronic device, and storage medium based on convolutional neural networks disclosed in the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A bolt recognition and management method based on convolutional neural networks, characterized in that, include: The initial detection image captured during the movement of the intelligent fuel injection vehicle is acquired. The initial detection image is then identified using a first-level detection model, and the fastener boundary box and fastener category identifier in the initial detection image are determined. The spatial relationship between fasteners and bolts is determined based on the fastener category identifier and the pre-built fastener-bolt mapping table, and the bolt detection area is dynamically generated based on the fastener boundary box and the spatial relationship between fasteners and bolts. The bolt detection area is input into the second-level detection model for bolt identification to obtain the current status information of the bolt at the corresponding position and the target position of the bolt; Output the target position of the bolt and the current status information of the bolt.

2. The bolt recognition and management method based on convolutional neural networks as described in claim 1, characterized in that, The first-level detection model is a lightweight object detection model enhanced by an attention mechanism. The first-level detection model includes a convolutional neural network and a weight calculation network. The step of identifying the initial detection image using a first-level detection model and determining the fastener bounding box and fastener category identifier in the initial detection image includes: The initial feature map is obtained by extracting features from the initial detection image using the convolutional neural network. The initial feature map is forward-propagated in the convolutional neural network to obtain a multi-scale feature map generated by a preset convolutional layer based on the initial feature map. The multi-scale feature map is input into a weight calculation network, which calculates the weight value of each spatial location in the initial detection image. The weight value represents the probability confidence that each spatial location belongs to the fastener region. An attention weight map of the initial detection image is constructed based on the obtained weight values. The attention weight map is a weight matrix with the same spatial resolution as the initial detection image. The weight values ​​of the fastener candidate region in the attention weight map are higher than those of the non-fastener region. The initial detection image is fused with the attention weight map pixel by pixel to obtain a weighted initial detection image; The weighted initial detection image is input into the detection head of the convolutional neural network for target detection, and the output fastener bounding box and fastener category identifier are generated.

3. The bolt recognition and management method based on convolutional neural networks as described in claim 2, characterized in that, The convolutional neural network of the first-level detection model is an improved YOLOv8n, whose backbone network adopts the CSPDarknet53 structure, and whose neck network adopts a weighted bidirectional feature pyramid network for cross-scale feature fusion; the multi-scale feature map includes a first-scale feature map, a second-scale feature map and a third-scale feature map, which correspond to fastener targets of different scales in the initial detection image, respectively.

4. The bolt identification and management method based on convolutional neural networks as described in claim 1, characterized in that, The second-level detection model is a keypoint detection model based on heatmap regression. The second-level detection model includes a convolutional neural network with an encoder-decoder structure and a heatmap prediction head. The bolt detection area is input into the second-level detection model for bolt identification to obtain the target position of the bolt at the corresponding location, including: The local image after cropping the bolt detection area is input into the convolutional neural network of the encoder-decoder structure for feature extraction and upsampling reconstruction, and the output is a feature reconstruction map with the same spatial resolution as the local image. The feature reconstruction map is input into the heatmap prediction head, and the output is a heatmap of the bolt center point. The value of each pixel in the heatmap of the bolt center point represents the probability that the position is the bolt center point. The position with the largest response value in the heat map of the bolt center point is obtained as the pixel-level coordinate of the bolt center point; Using the pixel-level coordinates as the center, Gaussian surface fitting is performed on the thermal response values ​​within its neighborhood, and the peak position of the Gaussian surface is taken as the target bolt position of the bolt center point.

5. The bolt identification and management method based on convolutional neural networks as described in claim 4, characterized in that, The second-level detection model also includes a semantic segmentation head, which shares the encoder-decoder structure of the convolutional neural network with the heatmap prediction head; The method further includes: The feature reconstruction map is input into the semantic segmentation head, and the pixel-level semantic segmentation result of the bolt detection region is output. The semantic segmentation result includes the bolt head mask, the spring strip mask and the background mask. The boundary line between the bolt head edge and the spring clip edge is extracted based on the bolt head mask and the spring clip mask. The boundary line is expanded into an annular contact band of a predetermined width, which serves as the contact area between the bolt and the fastener spring clip. The center point coordinates and local normal vector of the contact area are output. The predetermined width is determined based on the actual contact gap width between the bolt head and the spring clip, and the width of the annular contact band is 2mm to 5mm. The annular contact band is the spray target area of ​​the nozzle, and the intelligent oil spraying vehicle is used to spray grease into the area covered by the annular contact band. The target spray attitude of the corresponding nozzle is determined based on the center point coordinates and the local normal vector. The second-level detection model is a multi-task jointly trained network, and its joint loss function during the training phase is: in, To constrain the prediction accuracy of the heatmap for predicting losses, the heatmap for predicting the probabilistic center point of the bolt is used. The semantic segmentation loss is used to constrain the segmentation accuracy of the bolt head mask, spring bar mask, and background mask. The keypoint regression loss is used to constrain the regression accuracy of the position coordinates of the multiple semantic keypoints. and These are the weighting coefficients for the corresponding loss terms.

6. The bolt recognition and management method based on convolutional neural networks as described in claim 5, characterized in that, The management method further includes: Within the bolt detection area, the bolt target position of the bolt center point is located by the second-level detection model, and a local spatial vector relationship is constructed based on multiple semantic key points within the bolt detection area. The dynamic spatial offset is calculated based on the local spatial vector relationship and the fastener-bolt spatial position relationship. The bolt target position of the bolt center point is corrected using the dynamic spatial offset to obtain the bolt target position at the current moment. The spatial relationship between the fastener and the bolt is the mean vector and covariance matrix of the offset between the center of the fastener frame and the center point of the bolt, obtained by fitting the labeled data using a Gaussian mixture model. The local spatial vector relationship is obtained by detecting multiple semantic key points within the bolt detection area, including the spring clip fulcrum, bolt center point, and rail bottom edge point, and constructing a 2D vector relationship graph with the geometric center of the fastener boundary frame as the origin; The dynamic spatial offset is the residual offset of the local spatial vector relationship relative to the spatial positional relationship of the fastener-bolt.

7. The bolt recognition and management method based on convolutional neural networks as described in claim 6, characterized in that, The local spatial vector relationships are determined in the following way: The cropped local image of the bolt detection area is input into the third detection model, which is a multi-keypoint detection model, and outputs the position coordinates of multiple semantic key points in the bolt detection area. The multiple semantic key points include the left support point of the elastic bar, the right support point of the elastic bar, the center point of the bolt, and the edge point of the rail bottom. Using the geometric center of the fastener boundary frame as the origin, calculate the relative position vector of each semantic key point with respect to the origin, and construct the local spatial vector relationship; The third detection model shares the same backbone network as the second-level detection model. The second-level detection model and the third detection model are multi-task joint detection models, which include a shared encoder, a heatmap prediction head, a semantic segmentation head, and a multi-keypoint regression head. The shared encoder outputs a feature reconstruction map of the bolt detection region. The heatmap prediction head, the semantic segmentation head, and the multi-keypoint regression head take the feature reconstruction map as input and output the bolt center point heatmap, semantic segmentation results, and position coordinates of each semantic keypoint in parallel.

8. The bolt recognition and management method based on convolutional neural networks as described in claim 1, characterized in that, The identification management method further includes: A digital profile is established for each bolt on the line. The digital profile includes the bolt's unique identifier, LBS coordinates, historical maintenance time, maintenance records, and historical bolt status. The historical bolt status is updated based on environmental factors. The planned travel path of the intelligent fuel injection vehicle is obtained, and the LBS trajectory of the planned travel path is spatially superimposed with the list of unmaintained bolts to generate a daily task sequence. The daily task sequence is configured to ensure that the intelligent fuel injection vehicle covers the most unmaintained bolts and has the shortest repeated path in a single travel path. During the bolt identification process, when the intelligent fuel injection vehicle reaches the target bolt position, it queries the digital file of the target bolt. If the target bolt has been marked as having a qualified oil film condition, it automatically skips the target bolt and records the reason for skipping.

9. A bolt recognition and management system based on a convolutional neural network, characterized in that, include: Level 1 Detection Module: Used to acquire initial detection images captured during the movement of the intelligent fuel injection vehicle, identify the initial detection images through the Level 1 Detection Model, and determine the fastener bounding box and fastener category identifier in the initial detection images; Dynamic generation module: used to determine the spatial positional relationship between fasteners and bolts based on the fastener category identifier and the pre-built fastener-bolt mapping table, and dynamically generate the bolt detection area based on the fastener boundary box and the spatial positional relationship between fasteners and bolts; Secondary detection module: used to input the bolt detection area into the secondary detection model to identify the bolts and obtain the current status information of the bolts at the corresponding positions and the target position of the bolts; Output module: Used to output the target position of the bolt and the current status information of the bolt.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program causes a computer to execute the bolt identification and management method based on a convolutional neural network as described in any one of claims 1 to 7.

Images