Bridge bolt monitoring image recognition method and system based on deep learning

Abstract

The embodiment of the present invention discloses a bridge bolt monitoring image recognition method and system based on deep learning, belonging to the field of bridge image processing technology. The method includes: obtaining a multi-angle monitoring image set of bridge bolt components through an image acquisition device, preprocessing the monitoring image set to generate a standardized target image sequence; locating the bolt monitoring area for each image frame in the target image sequence based on a pre-trained positioning model, and generating a regional coordinate set containing bolt position information; inputting the regional coordinate set into a state recognition model, extracting the structural state characteristics of each bolt monitoring area, and generating a looseness prediction result of the bolt component based on the structural state characteristics; outputting a bolt state abnormality alarm signal based on the matching result of the looseness prediction result with a preset safety threshold range; and associating the bolt state abnormality alarm signal with the corresponding monitoring image frame to generate a monitoring report containing visual markings.

Description

Technical Field

[0001] The embodiments of the present invention belong to the technical field of bridge image processing, and specifically relate to a bridge bolt monitoring image recognition method and system based on deep learning. Background Art

[0002] In the field of bridge maintenance, bolt condition monitoring is crucial. Traditional manual inspection techniques suffer from low efficiency, poor accuracy, and significant safety hazards. Inspectors must climb bridges or use high-altitude equipment to inspect each point, facing high risks such as falls and traffic accidents. This is especially true in inclement weather or on complex structures such as cable-stayed bridges. Inspections rely on subjective methods such as visual observation and tapping, which can take weeks and struggle to cover hidden areas. Defects such as loose bolts and minor rust can easily be overlooked, and inspection results are affected by personnel experience and fatigue, resulting in a high rate of missed detections. Therefore, improving the efficiency and accuracy of bridge bolt monitoring has become a technical challenge that needs to be addressed. Summary of the Invention

[0003] The embodiments of the present invention provide a method and system for bridge bolt monitoring image recognition based on deep learning, which can solve or partially solve the technical problems involved in the above-mentioned background technology.

[0004] An embodiment of the present invention provides a bridge bolt monitoring image recognition method based on deep learning, which is applied to a bridge bolt monitoring image recognition system. The method includes: obtaining a multi-angle monitoring image set of bridge bolt components through an image acquisition device, and preprocessing the monitoring image set to generate a standardized target image sequence; locating the bolt monitoring area for each image frame in the target image sequence based on a pre-trained positioning model, and generating a regional coordinate set containing bolt position information; inputting the regional coordinate set into a state recognition model, extracting the structural state characteristics of each bolt monitoring area, and generating a looseness prediction result of the bolt component based on the structural state characteristics; outputting a bolt state abnormality alarm signal based on a matching result between the looseness prediction result and a preset safety threshold range; and associating the bolt state abnormality alarm signal with the corresponding monitoring image frame to generate a monitoring report containing visual marks.

[0005] An embodiment of the present invention provides a bridge bolt monitoring image recognition system, comprising at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the above-mentioned method.

[0006] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above method are implemented.

[0007] In an embodiment of the present invention, by acquiring a set of monitoring images from multiple angles and preprocessing them into a standardized target image sequence, the data quality of subsequent analysis is ensured, and analysis errors caused by poor image quality are reduced. By locating the bolt monitoring area using a pre-trained positioning model, the bolt position information can be accurately obtained, which is conducive to accurate analysis. By introducing a state recognition model, the degree of looseness can be predicted based on the structural state characteristics to achieve an effective assessment of the state of the bolt components. In addition, according to the matching of the looseness degree with the safety threshold, the output of the alarm signal can timely detect abnormal bolt status, thereby ensuring the safety of the bridge. Finally, associating the alarm signal with the image frame to generate a monitoring report with visual markings can ensure that relevant personnel can intuitively and quickly understand the abnormal state of the bolt, thereby improving monitoring efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a flowchart of a bridge bolt monitoring image recognition method based on deep learning provided by an embodiment of the present invention.

[0009] Figure 2 This is an example diagram of the change in the angle value of the web bolt provided in an embodiment of the present invention.

[0010] Figure 3 This is a structural diagram of a bridge bolt monitoring image recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the embodiments of the present invention.

[0012] The terms "first," "second," and the like in the embodiments of the present invention are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. Furthermore, in the embodiments of the present invention, "and / or" refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0013] Figure 1A bridge bolt monitoring image recognition method based on deep learning is shown, which is applied to a bridge bolt monitoring image recognition system. The method includes the following steps 101 to 103.

[0014] Step 101: Acquire a multi-angle monitoring image set of a bridge bolt component through an image acquisition device, and pre-process the monitoring image set to generate a standardized target image sequence.

[0015] First, the image acquisition device captures images of the bridge bolt components from multiple angles, generating a set of monitoring images. Multi-angle acquisition, such as vertical, horizontal, and diagonal, allows for a comprehensive and complete reflection of the bolt components' characteristics.

[0016] Furthermore, the collected monitoring image set may be affected by various factors, such as uneven lighting, image noise, etc., so preprocessing is required to generate a standardized target image sequence. In an embodiment of the present invention, the preprocessing process involves multiple aspects. In terms of noise removal, some common filtering methods can be used, such as median filtering. Median filtering can effectively remove isolated noise points such as salt and pepper noise. For uneven lighting, a histogram equalization method can be used to adjust the brightness and contrast of the image so that the image achieves a relative balance in brightness and contrast, thereby making the details of the image more clearly visible. Through the above-mentioned preprocessing operations, a monitoring image set with uneven quality and various interference factors can be converted into a standardized target image sequence with higher quality and more suitable for subsequent analysis.

[0017] Step 102: Based on the pre-trained positioning model, the bolt monitoring area is positioned for each image frame in the target image sequence to generate a regional coordinate set containing bolt position information; the regional coordinate set is input into the state recognition model to extract the structural state characteristics of each bolt monitoring area, and a looseness prediction result of the bolt component is generated based on the structural state characteristics.

[0018] In this step, a pre-trained localization model is trained based on sample data containing images of various types and states of bridge bolt components. For each image frame in the target image sequence, the localization model accurately locates the bolt monitoring area. For example, the localization model can be built based on a convolutional neural network (CNN), which has powerful feature extraction capabilities and can identify the unique features of the bolt monitoring area in the image to determine its location. Processing by the localization model generates a set of regional coordinates containing bolt location information. These regional coordinates precisely indicate the position of each bolt in the image. For example, with the upper left corner of the image as the origin and the coordinates measured in pixels, the upper left corner coordinates of a bolt might be (100, 200), the lower right corner coordinates might be (150, 230), and so on.

[0019] The set of regional coordinates is then input into the state recognition model. The state recognition model is also trained based on a large amount of labeled sample data. The annotations for the sample data include information such as the structural state characteristics of the bolts at different degrees of looseness. The state recognition model extracts structural state characteristics for each bolt monitoring area. This structural state feature extraction process may involve the shape characteristics of the bolts. For example, under normal circumstances, the outline of a bolt should be a regular circle or square, but loosening may cause the outline to deform. It may also involve the relative positional relationship characteristics of the bolt and surrounding components. For example, under normal circumstances, the bolt and nut fit tightly together, but loosening may cause gaps. Based on the extracted structural state features, the state recognition model can generate a prediction result for the looseness level of the bolt component. The looseness prediction result can be a qualitative description, such as being classified into different levels such as no loosening, slightly loosening, moderate loosening, and severe loosening.

[0020] Step 103: Outputting a bolt status abnormality alarm signal based on the matching result between the looseness degree prediction result and the preset safety threshold range; associating the bolt status abnormality alarm signal with the corresponding monitoring image frame, and generating a monitoring report including a visual mark.

[0021] In an embodiment of the present invention, step 103 is the decision-making and output part of the entire process. The operation is performed based on the matching result of the looseness prediction result and the preset safety threshold range. The preset safety threshold range is comprehensively determined based on the design standards of the bridge, engineering experience, and a large amount of experimental data. If the looseness prediction result exceeds the safety threshold range, it means that the state of the bolt may be at risk, and the system will output a bolt state abnormality alarm signal. The bolt state abnormality alarm signal contains key information such as the position information and looseness degree of the bolt. Then, the system will associate the bolt state abnormality alarm signal with the corresponding monitoring image frame, and visually mark the abnormal bolts on the monitoring image frame. For example, the position of the abnormal bolt can be circled with a red circle, and the looseness degree and other information can be marked next to it, thereby generating a monitoring report containing visual marks. Such a monitoring report can intuitively show the status of the bolt to relevant personnel.

[0022] The bridge bolt monitoring image recognition system can effectively monitor and analyze bridge bolt components and provide intuitive monitoring result reports through the process of step 101 to step 103, thereby providing an important basis for the safe maintenance of the bridge.

[0023] In another feasible technical solution, when the image acquisition device collects monitoring images, the more comprehensive and comprehensive the image acquisition angles, the more accurately it can reflect the true state of the bolt component. For example, if only a single angle is used to capture images, the features of certain key parts of the bolt may be missed. However, capturing images from multiple angles, such as every 30 degrees horizontally and every 20 degrees vertically, can minimize this problem.

[0024] For pre-trained positioning models, the quality and quantity of training data crucially impact their positioning accuracy. If the training data includes images of bolts in a variety of complex environments, such as those under varying lighting conditions and weather conditions, the positioning model can more accurately locate the bolt monitoring area in real-world applications. For example, a positioning model trained on a training dataset containing 100,000 images of bolts in various environments will achieve significantly higher positioning accuracy than a model trained on only 10,000 images.

[0025] When the state recognition model extracts structural state features, the type of features it focuses on and the accuracy of the feature extraction directly determine the reliability of the looseness prediction results. In addition to the bolt shape and relative position characteristics mentioned above, other features can also include texture features on the bolt surface. For example, the texture of a normal bolt surface is uniform, but if loosening and wear occur, the texture changes. If the state recognition model can accurately identify these changes in texture features, the looseness prediction will be more accurate.

[0026] The rationality of the preset safety threshold range is crucial for the effectiveness of the entire system. If the safety threshold range is set too wide, some bolts that already pose a risk may be misidentified as normal; if it is set too narrow, some normal bolts may be misidentified as abnormal. Therefore, it is necessary to comprehensively consider multiple factors, including the actual use of the bridge, design requirements, and past engineering experience, to determine an appropriate safety threshold range.

[0027] Therefore, the bridge bolt monitoring image recognition system realizes the effective monitoring and status assessment of bridge bolt components through the above-mentioned collaborative work, and presents the results in the form of intuitive monitoring reports.

[0028] In another feasible technical solution, there are also many factors to consider when selecting an image acquisition device. For example, a high-resolution camera, such as one with a resolution of 1920×1080 pixels, can be selected to capture clearer bolt details. At the same time, the camera's focal length, field of view, and other parameters also need to be adjusted according to actual monitoring needs. If the bolt area to be monitored is small and precise observation of details is required, a camera with a longer focal length and a smaller field of view should be selected; conversely, if a larger range of bolts needs to be monitored as a whole, a camera with a shorter focal length and a larger field of view can be selected.

[0029] For pre-trained positioning and state recognition models, the structural design of the models is also crucial. Taking the positioning model as an example, in addition to the previously mentioned convolutional neural network (CNN)-based model, other improved network structures, such as ResNet (residual network), can also be adopted. By introducing residual connections, ResNet effectively addresses the vanishing gradient problem that occurs with increasing network depth, thereby improving model performance. In the state recognition model, similar advanced network structures can also be adopted, or multi-model fusion can be used to combine the results of models with different structures to improve the ability to extract structural state features and the accuracy of looseness prediction.

[0030] When generating a monitoring report with visual markings, the method and content of the visual markings can be customized according to actual needs. In addition to marking the location of abnormal bolts with red circles and indicating the degree of looseness, the monitoring report can also include more information, such as the time the image was acquired and the bolt number. This makes the monitoring report more detailed and comprehensive, facilitating management and maintenance work for relevant personnel.

[0031] In actual implementation, the installation location of the image acquisition device also needs to be adjusted based on actual conditions. An inappropriate installation location may result in obstruction or distortion of the captured images. For monitoring bridge bolts, the installation location should ensure a clear camera view of the bolt components while minimizing external interference. For example, the device should be installed at a certain distance directly in front of the bolt components and avoid obstruction by other bridge structural components. If multiple bolts need to be monitored, a distributed installation approach can be adopted, with multiple image acquisition devices installed at different locations to ensure that each bolt is effectively monitored. During the training process of the pre-trained localization model and state recognition model, for the localization model of the bolt monitoring area, the annotations should accurately indicate the bolt's position coordinates in the image; for the state recognition model, the annotations should provide detailed descriptions of the bolt's structural state characteristics and corresponding looseness level. When generating monitoring reports with visual annotations, the format and storage method of the reports should also be considered. A universal document format, such as PDF, can be used to facilitate viewing across different devices and software. For storage, monitoring reports can be stored in a local database or backed up to a cloud server. This can not only ensure the security of data, but also facilitate data sharing and query.

[0032] In another possible application scenario of an embodiment of the present invention, in addition to the aforementioned parameters such as resolution, focal length, and field of view, the image acquisition device can also adjust the frame rate of image acquisition. The frame rate determines the number of frames of image captured per second. If the frame rate is too high, the amount of data and the processing burden of the system will increase; if the frame rate is too low, some short-term changes in the status of the bolts may be missed. For example, for general bridge bolt monitoring, the frame rate can be set to 10-15 frames per second, which can ensure that the status changes of the bolts are captured without generating too much data.

[0033] A variety of evaluation metrics can be used to evaluate the performance of pre-trained positioning and state recognition models. For positioning models, metrics such as accuracy and recall can be used. Accuracy represents the ratio of correctly located bolts to the predicted number of bolts, while recall represents the ratio of correctly located bolts to the actual number of bolts. For state recognition models, metrics such as classification accuracy and confusion matrix can be used. These evaluation metrics provide a precise understanding of model performance, allowing for targeted model optimization.

[0034] When generating looseness prediction results for bolt components, the model's input features, in addition to the previously mentioned bolt shape, relative position, and surface texture, can also consider characteristics of the bolt's surrounding environment. For example, whether there is water accumulation or corrosion around the bolt. These environmental characteristics may affect the bolt's condition, and using them as input features can improve the accuracy of looseness prediction results.

[0035] When outputting an abnormal bolt condition alarm signal based on the match between the looseness prediction result and the preset safety threshold range, in addition to directly outputting the alarm signal, a comprehensive analysis of multiple consecutive monitoring results can also be performed. For example, if three consecutive monitoring results show that the bolt is slightly loose, although the individual results do not exceed the safety threshold range, a comprehensive consideration can indicate that the bolt condition has a potential risk, and an alarm signal can be output in this case.

[0036] In one embodiment, preprocessing the monitoring image set to generate a standardized target image sequence includes:

[0037] Step 1011: performing illumination equalization processing on each original image in the monitoring image set, and eliminating brightness deviation in shadow areas by using an adaptive histogram equalization algorithm.

[0038] Step 1012: Extract noise distribution features from the processed image, suppress high-frequency noise based on a multi-scale filtering algorithm, and generate a set of denoised intermediate images.

[0039] Step 1013: geometrically correct the intermediate image set, and adjust the image tilt angle using an affine transformation algorithm so that the bolt component directions of all image frames in the intermediate image set are aligned with a preset standard direction.

[0040] Step 1014: performing resolution unification processing on the corrected image frames, and cropping the non-monitoring areas to generate a target image sequence with a consistent size.

[0041] In the bridge bolt monitoring image recognition system, the monitoring image set is preprocessed to generate a standardized target image sequence. In this link, illumination equalization is first performed on each original image in the monitoring image set. Uneven illumination will bring many difficulties to subsequent analysis. For example, the shadow area may cover certain features of the bolt. The embodiment of the present invention adopts an adaptive histogram equalization algorithm to eliminate the brightness deviation of the shadow area. The adaptive histogram equalization algorithm can automatically adjust the brightness according to the local contrast information of the image. Compared with ordinary histogram equalization, the adaptive histogram equalization algorithm can better adapt to images under different lighting conditions. For example, in some bridge environments, there may be a situation where the lighting is strong on one side and weak on the other side. The adaptive histogram equalization algorithm can effectively adjust the brightness of the image under such uneven illumination, making the brightness of the entire image more uniform, thereby highlighting the details of the bolt components.

[0042] Next, noise distribution features are extracted from the image after illumination equalization. Image noise can interfere with the accurate identification of bolt features. Noise varies, with high-frequency noise having a significant impact on image quality. High-frequency noise is suppressed using a multiscale filtering algorithm, generating a set of de-noised intermediate images. Multiscale filtering algorithms can filter images at different scales, effectively capturing and suppressing noise at different frequencies. For example, for high-frequency noise generated by electronic equipment interference, multiscale filtering algorithms can filter at an appropriate scale based on the frequency characteristics of the noise, resulting in a smoother image while still preserving important features such as the bolt's edges.

[0043] Then, the intermediate image set is geometrically corrected. During the image acquisition process, the image may be tilted due to the angle of the acquisition device or the structural characteristics of the bridge itself, which will affect the accurate judgment of the position and status of the bolt component. The image tilt angle is adjusted by the affine transformation algorithm so that the direction of the bolt component in all image frames in the intermediate image set is aligned with the preset standard direction. The affine transformation algorithm can rotate, translate, scale, and perform other operations on the image while maintaining the parallelism and collinearity of the image. For example, if the bolt shows a certain tilt in the image, it can be rotated to an angle consistent with the preset standard direction through the affine transformation algorithm. In this way, in the subsequent positioning and status recognition process, operations can be performed based on a unified direction standard to improve accuracy.

[0044] Finally, the corrected image frames are processed with uniform resolution, and the non-monitoring areas are cropped to generate a target image sequence with consistent size. Different image acquisition devices or acquisition parameters may result in different image resolutions, but in subsequent analysis, a uniform resolution helps improve processing efficiency and accuracy. On the basis of uniform resolution, cropping the non-monitoring areas can reduce the amount of unnecessary data, making the target image sequence more focused on the bolt components. For example, for some images that contain a large amount of surrounding environment information, the presence of non-monitoring areas will increase the burden of data storage and processing. By cropping, only the areas related to bolt monitoring can be retained, and the sizes of all images can be adjusted to be consistent. In this way, when input into the subsequent positioning model and state recognition model, data processing and feature extraction can be better performed.

[0045] The preprocessing process in this embodiment processes the monitoring image set in multiple aspects, standardizing the images in terms of illumination, noise, geometry, and size. The standardized target image sequence provides a sound data foundation for subsequent processing based on pretrained positioning and state recognition models. Illumination equalization ensures uniform image brightness, facilitating the highlighting of bolt features; noise reduction reduces noise interference with feature recognition; geometric correction ensures consistent orientation of bolt components, facilitating subsequent standardized operations; and resolution unification and cropping of non-monitored areas improve data processing efficiency and accuracy. The synergistic effect of these operations enhances the accuracy and reliability of the entire bridge bolt monitoring image recognition system, providing a crucial guarantee for effective bridge bolt monitoring. Preprocessing the monitoring image set to generate a standardized target image sequence improves image quality in multiple ways. Illumination equalization reduces the impact of uneven illumination on bolt feature display, and multi-scale filtering and noise reduction prevent noise interference with subsequent feature recognition. Geometric correction ensures consistent orientation of bolt components, facilitating subsequent operations; resolution unification and cropping of non-monitored areas improve data processing efficiency and accuracy. In this way, the accuracy and reliability of bridge bolt monitoring image recognition processing can be improved, and the precision and credibility of bridge bolt monitoring can be improved.

[0046] In another embodiment, locating the bolt monitoring area of ​​the target image sequence based on the pre-trained positioning model includes:

[0047] Step 201: Input the target image sequence into the convolution layer of the positioning model, extract the multi-scale spatial features in the target image sequence, and generate a feature map containing bolt contour information; use the region proposal network to screen candidate regions on the feature map, and calculate the matching score between each candidate region and the preset bolt template.

[0048] Step 202: performing non-maximum suppression processing on the candidate regions according to the matching scores, and retaining regions with confidence levels higher than a first threshold as initial positioning frames.

[0049] Step 203: performing edge refinement correction on the initial positioning frame, adjusting the size of the positioning frame using a bilinear interpolation algorithm, and generating a monitoring area coordinate set including the corrected coordinates.

[0050] In this embodiment, the bolt monitoring area is located in a target image sequence based on a pre-trained positioning model. The target image sequence is first input into the convolutional layer of the positioning model, and feature extraction is performed through the convolutional layer. For example, in a target image sequence of size 1000×1000 pixels, the convolutional layer can extract multi-scale spatial features from receptive fields of different sizes. Multi-scale spatial features are critical for accurately depicting bolt contour information. Large-scale features may correspond to the approximate contour range of the bolt in the image, while small-scale features can capture the finer details of the bolt contour, thereby generating a feature map containing the bolt contour information.

[0051] Next, the region proposal network (RPN) uses this feature map to select candidate regions. The pre-set bolt template plays an important role in this process. For example, the pre-set bolt template has five key shape and texture features. The RPN calculates the matching score between each candidate region and the pre-set bolt template based on these features. For example, if a candidate region has an 80% similarity with the template in terms of shape features and a 70% similarity in terms of texture features, a specific algorithm can determine the matching score for this candidate region.

[0052] Then, non-maximum suppression is performed on the candidate regions based on their matching scores. For example, if the first threshold is set to 0.6, only regions with matching scores above 0.6 will be retained as the initial positioning boxes. This step can effectively reduce the number of overlapping candidate regions with low confidence, avoid unnecessary interference in subsequent processing, and improve positioning accuracy and efficiency.

[0053] The initial positioning frame is then refined and corrected. A bilinear interpolation algorithm is used to adjust the size of the positioning frame to generate a monitoring area coordinate set containing the corrected coordinates (which can be understood as precise coordinates). For example, the initial positioning frame might have a side length of 100 pixels. After bilinear interpolation, its side length becomes more precise, such as 98 pixels or 102 pixels. This more closely fits the actual area of ​​the bolt, allowing the monitoring area coordinate set to accurately represent the bolt's position in the image.

[0054] Based on the above content, the process of extracting structural state features by the state recognition model includes: extracting local image blocks of each bolt monitoring area from the monitoring area coordinate set, and inputting the image blocks into the feature extraction module of the state recognition model; performing hierarchical feature fusion on the image blocks through a multi-layer residual network to generate a composite feature vector containing the bolt surface texture, rust degree and connection gap; inputting the composite feature vector into the time series analysis module, and calculating the cumulative displacement of the current bolt in combination with the bolt displacement change trend in the historical monitoring data; based on the comparison result of the cumulative displacement and the preset displacement threshold, generating the looseness degree prediction result, wherein the part of the cumulative displacement that exceeds the displacement threshold is marked as an abnormal displacement interval.

[0055] In detail, in the process of extracting structural state features based on the state recognition model, local image blocks of each bolt monitoring area can be extracted from the monitoring area coordinate set, and these image blocks can be input into the feature extraction module of the state recognition model. The feature extraction module in the embodiment of the present invention performs hierarchical feature fusion on the image blocks through a multi-layer residual network. The multi-layer residual network can effectively fuse features at different levels. For example, for a multi-layer residual network containing 10 layers, simple features of the bolt surface texture may be extracted at the lower layers. As the number of layers increases, more features about the overall structure of the bolt, such as the degree of rust and the connection gap, can be integrated into the higher layers, thereby generating a composite feature vector containing the bolt surface texture, the degree of rust and the connection gap.

[0056] The composite feature vector is input into the time series analysis module, and combined with the bolt displacement trend in the historical monitoring data, the cumulative displacement of the current bolt is calculated. Historical monitoring data contains the displacement information of the bolt at different points in the past, which is crucial for comprehensively assessing the current state of the bolt. For example, historical monitoring data shows that the bolt displacement in the past five monitorings was 1 mm, 1.5 mm, 2 mm, 2.5 mm, and 3 mm, respectively. Combined with the current composite feature vector, the time series analysis module can calculate the cumulative displacement of the current bolt.

[0057] The system generates a looseness prediction based on the comparison of the cumulative displacement with a preset displacement threshold, and marks any portion of the cumulative displacement exceeding the threshold as an abnormal displacement interval. For example, if the preset displacement threshold is 5 mm, when the cumulative displacement reaches 6 mm, the excess 1 mm is marked as an abnormal displacement interval. This allows the system to intuitively reflect the degree of bolt looseness and the presence of abnormal portions.

[0058] Furthermore, the output bolt status abnormal alarm signal includes: determining the potential risk level of the bolt based on the abnormal displacement interval in the looseness prediction result, wherein the risk level includes the first level, the second level and the emergency level; if the risk level is the first level, a primary alarm signal including a recommended inspection time is generated, and the signal is associated with the corresponding monitoring image frame; if the risk level is the second level, an intermediate alarm signal including a mandatory maintenance prompt is generated, and a real-time monitoring frequency increase instruction is triggered; if the risk level is the emergency level, an immediate maintenance alarm signal is generated, and an emergency notification message including the location coordinates is sent to the remote management terminal.

[0059] For example, in terms of outputting an abnormal bolt status alarm signal, the potential risk level of the bolt is determined according to the abnormal displacement interval in the looseness prediction result, and the risk level is divided into the first level, the second level and the emergency level.

[0060] When the risk level reaches Level 1, a primary alarm signal with a recommended inspection time is generated and associated with the corresponding monitoring image frame. This level indicates that while the bolt presents a potential risk, the situation is relatively less urgent. For example, the bolt's displacement may have slightly exceeded the threshold but has not yet seriously impacted the overall structure. In this case, a recommended inspection time is given, allowing personnel to inspect the bolt at the appropriate time. The associated monitoring image frame allows for quick verification of the bolt's specific status.

[0061] When the risk level reaches Level 2, an intermediate alarm signal is generated, including a mandatory maintenance prompt, and an instruction to increase the real-time monitoring frequency is triggered. This indicates that the bolt's condition has become critical, requiring mandatory maintenance and closer monitoring of its status changes. For example, if the bolt displacement is large and exceeds the threshold by a large margin, it may have already affected the connection structure. In this case, increasing the real-time monitoring frequency can promptly detect further changes in the bolt's condition.

[0062] When the risk level reaches the emergency level, the bolt's condition is critical and an immediate maintenance alert is generated. Simultaneously, an emergency notification message containing the location coordinates is sent to the remote management terminal. For example, if a bolt's displacement exceeds a threshold and could cause severe structural damage, requiring immediate maintenance, the emergency notification message containing the location coordinates can be sent to the remote management terminal to ensure that personnel can quickly locate the problem bolt and take emergency measures.

[0063] This design allows the state recognition model to accurately determine the degree of bolt loosening by integrating multiple features and historical data. Furthermore, abnormal bolt state warning signals are generated based on risk level. Based on this, the system can address varying degrees of bolt risk, from primary inspection recommendations to intermediate mandatory maintenance prompts and emergency immediate maintenance notifications. This effectively improves the accuracy and timeliness of bridge bolt state monitoring, thereby ensuring the safety of bridge structures.

[0064] In a preferred embodiment, generating a monitoring report including visual markers in step 103 includes:

[0065] Step 1031: Mark the monitoring area coordinates of abnormal bolts in the target image sequence, and use color coding to distinguish the marking boxes corresponding to different risk levels.

[0066] Step 1032: Convert the looseness degree prediction result into a visual progress bar in percentage form, and display it in a superimposed manner in the side area of ​​the corresponding mark box.

[0067] Step 1033: extract the displacement change curve of the same bolt in the historical monitoring report, generate a comparative analysis graph and embed it into the appendix of the monitoring report.

[0068] Step 1034: Generate a time-ordered abnormal event list based on the timestamp information of the monitoring image frame, and dynamically associate the list with the visual mark.

[0069] In this preferred embodiment, when generating a monitoring report containing visual markings, the monitoring area coordinates of the abnormal bolts must first be marked in the target image sequence. The monitoring area coordinates clearly indicate the location of the abnormal bolt in the image. For example, in a target image sequence of 800×600 pixels, the monitoring area coordinates of the abnormal bolt can be an area with the upper left corner coordinates of (100, 200) and the lower right corner coordinates of (150, 250). Color coding is then used to distinguish the marking boxes corresponding to different risk levels. Different colors can intuitively allow people viewing the report to distinguish the severity of the bolt risk. For example, a red marking box may be used for a high-risk bolt, a yellow marking box for a medium-risk bolt, and a green marking box for a low-risk bolt. This color coding method can quickly convey bolt risk information without the need to review detailed text descriptions.

[0070] Next, the looseness prediction result is converted into a visual progress bar in percentage format and displayed superimposed on the side of the corresponding marked box. For example, if the looseness prediction result shows that a bolt's looseness is within a certain range, after converting it to a percentage, such as 30%, it can be displayed as a visual progress bar on the side of the marked box. This makes the looseness of the bolt clear at a glance, allowing the viewer to intuitively understand the proportion of the bolt's looseness in the overall situation, which helps to quantitatively assess the bolt's condition.

[0071] Afterwards, the displacement change curve for the same bolt from the historical monitoring report is extracted, and a comparative analysis chart is generated and embedded in the appendix of the monitoring report. The historical monitoring report contains displacement information for the same bolt at different points in the past. By plotting this information as a displacement change curve, the displacement trend of the bolt over a period of time can be clearly seen. For example, over the past 10 monitoring sessions, the displacement of the bolt may have shown a gradual increase or fluctuating trend. This comparative analysis chart can provide more comprehensive information on the bolt's status, and the appendix section makes it convenient for viewers to review when they need to conduct in-depth analysis of bolt status changes.

[0072] Finally, based on the timestamp information of the monitoring image frames, a chronological list of abnormal events is generated, and the list is dynamically associated with the visual marker. The timestamp information records the acquisition time of each monitoring image frame. Based on this time information, the order in which abnormal events occurred can be sorted out. For example, a bolt first showed a slight abnormality during monitoring on March 1st, and the degree of abnormality increased during monitoring on March 10th. These events are arranged in chronological order in the abnormal event list. Dynamically associating this list with the visual marker means that when the viewer clicks on a visual marker, they can also view detailed chronological information of the related abnormal events, thereby accurately tracking the changes in the bolt status over time.

[0073] By marking coordinates and using color-coded markers within the target image sequence, the location and risk level of abnormal bolts can be intuitively displayed. A visual progress bar clearly quantifies the degree of loosening. A comparative analysis of historical displacement curves provides comprehensive historical status information, facilitating in-depth analysis of bolt condition trends. The dynamic association of a chronological list of abnormal events with visual markers allows viewers to better understand how bolt conditions evolve over time, improving the readability of monitoring reports and the comprehensive presentation of bolt condition information.

[0074] In an alternative embodiment, the method also includes the step of jointly optimizing the positioning model and the state recognition model: collecting normal state samples and abnormal state samples of bolts in historical monitoring images to construct a training data set containing multi-category labels; initializing the convolutional layer parameters of the positioning model through a transfer learning algorithm, and optimizing the regional positioning accuracy using a focal loss function; introducing an attention mechanism into the state recognition model to dynamically adjust the weight distribution of key areas of the bolts during the feature extraction process; and using a generative adversarial network to enhance the training data to generate synthetic images containing different lighting conditions and angle changes to improve the model generalization ability.

[0075] Among them, the specific process of introducing the attention mechanism includes: adding a channel attention submodule after the last convolution layer of the feature extraction module of the state recognition model to calculate the global average pooling value of each feature channel; performing a nonlinear transformation on the pooling value through the fully connected layer to generate a channel weight vector and perform channel weighting on the original feature map; using the self-attention mechanism in the spatial dimension to calculate the correlation matrix between each position in the feature map and other positions; and spatially recalibrating the feature map according to the correlation matrix to highlight the detailed feature areas of the bolt joints.

[0076] It can be understood that the above embodiment specifically includes the step of jointly optimizing the positioning model and the state recognition model. First, samples of normal and abnormal bolt states in historical monitoring images are collected to construct a training data set containing multi-category labels. Historical monitoring images cover different states of bolts, and multi-category labels can accurately classify and identify samples. For example, 3,000 samples of normal bolt states and 2,000 samples of abnormal bolt states are selected from the past 5,000 historical monitoring images. The abnormal state samples include images of bolts in different structures and different surface conditions. The constructed training data set can comprehensively reflect the various state characteristics of the bolts.

[0077] Next, the convolutional layer parameters of the positioning model are initialized using a transfer learning algorithm, and the focal loss function is used to optimize regional positioning accuracy. The transfer learning algorithm uses the parameters of an existing pre-trained model to initialize the convolutional layer parameters of the positioning model. For example, if a pre-trained model has performed well in similar image recognition tasks, its convolutional layer parameters can provide a good initial state for the current positioning model. The focal loss function focuses on difficult-to-classify samples, improving the accuracy of bolt monitoring area positioning during regional positioning. If some of the 1,000 candidate areas are difficult to accurately classify due to unclear features or interference, the focal loss function will increase attention to these areas, making positioning more accurate.

[0078] Then, an attention mechanism is introduced into the state recognition model. The attention mechanism can dynamically adjust the weight distribution of key areas of the bolt during the feature extraction process. Specifically, a channel attention submodule is added after the last convolutional layer of the feature extraction module. The global average pooling value of each feature channel is first calculated. For example, for a feature map output by a convolutional layer with 16 feature channels, the global average pooling value of each channel is calculated. The global average pooling value reflects the average response of each channel in the overall feature. The pooling value is then nonlinearly transformed through a fully connected layer to generate a channel weight vector and perform channel weighting on the original feature map. As a result, the state recognition model can adjust the original feature map according to the importance of different channel features, highlighting important channel features.

[0079] A self-attention mechanism is employed in the spatial dimension to calculate the correlation matrix between each position in the feature map and every other position. For example, for a 20×20 feature map, the correlation between each position (400 positions in total) and every other position is calculated, forming a 400×400 correlation matrix. Spatial recalibration of the feature map based on this correlation matrix can highlight detailed feature areas such as bolt joints. Bolt joints are critical areas in bolt monitoring, and their condition significantly impacts the safety of the entire bolt. Spatial recalibration allows the state recognition model to focus more on the characteristics of these key areas.

[0080] Finally, a generative adversarial network (GAN) is used to enhance the training data, generating synthetic images that incorporate different lighting conditions and angle variations to improve the model's generalization capabilities. GANs can generate new synthetic images based on existing training data. These synthetic images incorporate different lighting conditions and angle variations. For example, the images in the original training data may primarily be images of bolts taken from the front under normal lighting. Using a GAN, images of bolts taken under different lighting conditions, such as strong light, weak light, and side light, as well as from different angles, such as those taken at angles of 30 or 60 degrees, can be generated. This enables the model to better perform positioning and state recognition in various real-world situations, improving the model's generalization capabilities.

[0081] This design, combined with transfer learning initialization and optimization of the focal loss function, improves the positioning accuracy of the localization model. Introducing the attention mechanism into the state recognition model to highlight key area features improves the accuracy of bolt state feature extraction. Combining it with a generative adversarial network enhances the generalization capabilities of the localization and state recognition models, enabling them to better handle bolt monitoring tasks under varying lighting conditions and angles, thereby enhancing their performance.

[0082] In another alternative embodiment, the method also includes a dynamic adjustment step during real-time monitoring: counting the displacement change of the same bolt during a continuous monitoring period, and triggering a model parameter update instruction when the change exceeds a dynamic threshold; online fine-tuning the state recognition model based on the latest collected monitoring image data, and updating the feature extraction weights using a sliding window algorithm; when it is detected that the ambient light intensity is continuously lower than a preset value, automatically switching to the infrared image acquisition mode and adjusting the input parameters of the positioning model; dynamically correcting the looseness prediction result according to the bridge vibration sensor data to eliminate instantaneous displacement interference caused by external loads.

[0083] Among them, the process of dynamically correcting the looseness degree prediction result includes: obtaining the acceleration spectrum data of the bridge vibration sensor within the monitoring time window, extracting the main vibration frequency components and their amplitudes; calculating the correlation coefficient between the bolt displacement change and the vibration amplitude, and establishing a displacement-vibration coupling relationship model; denoising the original displacement according to the coupling relationship model to generate a corrected displacement that eliminates environmental vibration interference; inputting the corrected displacement into the state recognition model, recalculating the looseness degree prediction result and updating the monitoring report.

[0084] The above embodiment includes a dynamic adjustment step in the real-time monitoring process, and the displacement change of the same bolt is counted in a continuous monitoring cycle. This operation is based on continuous attention to the state of the bolt. For example, in each continuous monitoring cycle set to 1 hour, the displacement of a specific bolt is counted. When this displacement change exceeds the dynamic threshold, the model parameter update instruction will be triggered. This dynamic threshold is an important indicator set according to the displacement change range of the bolt under normal working conditions and the accuracy requirements of the system. Once this threshold is exceeded, it means that the state of the bolt may have changed significantly, and the model parameters need to be updated to adapt to this change.

[0085] The state recognition model is fine-tuned online based on the latest monitored image data, using a sliding window algorithm to update feature extraction weights. The latest monitored image data contains information about the current state of the bolts, and the sliding window algorithm plays a key role in this process. For example, in a sequence of 100 recently acquired monitored images, the sliding window algorithm can gradually update feature extraction weights based on a set window size of, say, 20 images. This allows the model to promptly adapt its bolt feature extraction method using new data, enabling it to more accurately identify the bolt's state.

[0086] When the ambient light intensity is detected to be consistently below a preset value, the system automatically switches to infrared image acquisition mode and adjusts the input parameters of the positioning model. Ambient light intensity is a significant factor affecting image acquisition quality. When light intensity consistently falls below a preset value, for example, below 50 lux, standard image acquisition modes may not be able to provide clear images of bolts. In this case, infrared image acquisition mode is able to capture images of bolts in low-light environments. At the same time, the input parameters of the positioning model need to be adjusted (infrared images differ from standard visible light images in terms of feature representation). For example, features such as the bolt's outline and temperature distribution are more pronounced in infrared images, and the positioning model needs to adapt to these changes to accurately locate the bolt monitoring area.

[0087] In addition, the looseness prediction results are dynamically corrected based on bridge vibration sensor data to eliminate instantaneous displacement interference caused by external loads. Specifically, acceleration spectrum data is first acquired from the bridge vibration sensor within a monitoring time window, which can be set to a period of time, such as 10 minutes. Within this monitoring time window, the primary vibration frequency components and their amplitudes are extracted. For example, the vibration sensor may detect multiple vibration frequency components, among which the component with a frequency of 10 Hz and an amplitude of 2 m / s² may be the primary component related to the bridge structure and external loads. The correlation coefficient between the bolt displacement change and the vibration amplitude is then calculated to establish a displacement-vibration coupling relationship model. This displacement-vibration coupling relationship model reflects the inherent relationship between bolt displacement and bridge vibration. Based on the displacement-vibration coupling relationship model, the original displacement is de-noised to generate a corrected displacement that eliminates environmental vibration interference. Finally, the corrected displacement is input into the state recognition model, which recalculates the looseness prediction result and updates the monitoring report. This operation more accurately reflects the actual looseness of the bolts and avoids misjudgments caused by interference caused by external loads.

[0088] As can be seen, by triggering model parameter updates based on bolt displacement changes during continuous monitoring cycles, the system can promptly adapt to changes in bolt status. Online fine-tuning of the state recognition model improves the accuracy of bolt feature extraction. Switching acquisition modes and adjusting positioning model parameters in low-light conditions ensures effective monitoring in diverse environments. Using bridge vibration sensor data to correct loosening predictions eliminates external load interference and makes loosening predictions more accurate.

[0089] In actual application, the method further includes the steps of bolt key point detection and matching:

[0090] Step 301: extracting key points of the head edge of each bolt from the monitoring area coordinate set, and generating a key point coordinate set using a sub-pixel positioning algorithm.

[0091] Step 302: Match the key point coordinate sets in adjacent monitoring periods and calculate the key point displacement vectors of the same bolt at different time points.

[0092] Step 303: Based on the directional consistency analysis of the displacement vector, determine whether the overall displacement trend of the bolt is progressive loosening or instantaneous offset.

[0093] Step 304: When it is detected that the direction difference of the displacement vectors of the plurality of key points exceeds a tolerance range, a recalibration instruction of the image acquisition device is triggered.

[0094] Among them, the implementation of the sub-pixel positioning algorithm includes: extracting the grayscale gradient distribution information of the area around the key point based on the pixel-level key point detection results; constructing a quadratic surface model to fit the grayscale gradient distribution, and calculating the extreme points of the surface as the sub-pixel key point coordinates; adjusting the quadratic surface parameters of the quadratic surface model through an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold; performing bidirectional optical flow matching on the optimized key point coordinates with adjacent image frames, and eliminating key points with a matching degree lower than a second threshold.

[0095] Specifically, the technical solution described in steps 301 to 304 includes bolt key point detection and matching steps. First, the key points of the head edge of each bolt are extracted from the monitoring area coordinate set, and a sub-pixel positioning algorithm is used to generate a key point coordinate set. The monitoring area coordinate set determines the approximate position of the bolt in the image, while the key points of the bolt head edge are the key parts that more accurately reflect the state of the bolt. For example, for a bolt image area with a size of 500×500 pixels covered by a monitoring area coordinate set, the key points of the bolt head edge may be concentrated in a 100×100 pixel area near the connection between the head and the nut. The sub-pixel positioning algorithm can improve the accuracy of key point positioning.

[0096] The implementation of the sub-pixel localization algorithm involves extracting the grayscale gradient distribution information of the area around the keypoint based on the pixel-level keypoint detection results. For example, the keypoint detected by pixel-level detection is located in a 20×20 pixel area. The grayscale value of this area varies at each pixel, and based on this, the grayscale gradient distribution can be determined.

[0097] Furthermore, a quadratic surface model is constructed to fit this grayscale gradient distribution. This quadratic surface model is established based on the grayscale value distribution law of the area around the key point. For example, for an area containing 20×20=400 pixel grayscale values, a quadratic surface model is constructed through a specific algorithm. The extreme points of the surface are calculated as sub-pixel key point coordinates. This extreme point is more accurate than pixel-level positioning. For example, pixel-level positioning may determine that the key point is within a certain pixel range, while the sub-pixel extreme point coordinates can be accurate to decimal places, such as 0.1 pixel accuracy.

[0098] Furthermore, the quadratic surface parameters of the quadratic surface model are adjusted through an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold. For example, if the preset accuracy threshold is set to 0.05 pixels, the iterative optimization algorithm will continue to adjust the quadratic surface parameters of the quadratic surface model. After 10-20 iterations, the error between the calculated key point coordinates and the actual position will continue to decrease until it is less than 0.05 pixels. The optimized key point coordinates are matched with adjacent image frames through bidirectional optical flow matching. The bidirectional optical flow matching can establish a corresponding relationship between key points between adjacent image frames and eliminate key points with a matching degree lower than a second threshold. For example, the second threshold is set to 0.8. If the matching degree is lower than 0.8, the key point is eliminated to ensure the accuracy and stability of the key points.

[0099] Based on the above steps, the key point coordinate sets within adjacent monitoring cycles can be matched to calculate the key point displacement vectors for the same bolt at different time points. Adjacent monitoring cycles can be set at regular time intervals, such as every 30 minutes. The state of the bolt may change during different cycles. By matching the key point coordinate sets, the position changes of the same bolt at different time points can be determined, and the key point displacement vector can be calculated. For example, in the first monitoring cycle, the coordinates of a key point on the bolt head are (100.5, 200.3). After 30 minutes in the next monitoring cycle, the coordinates of the key point change to (101.2, 200.8). A displacement vector is then calculated with a horizontal displacement of 0.7 pixels and a vertical displacement of 0.5 pixels. This displacement vector contains the displacement information of the bolt key point in the horizontal and vertical directions, or in other spatial directions.

[0100] Based on the directional consistency analysis of the displacement vector, determine whether the overall displacement trend of the bolt is progressive loosening or instantaneous offset. If the direction of the displacement vector shows a relatively consistent change trend over multiple consecutive monitoring cycles, for example, within 5-6 consecutive monitoring cycles, the direction of the displacement vector is roughly in the same direction and the displacement gradually increases, such as the horizontal displacement gradually increases from 0.5 pixels to 1.5 pixels, and the vertical displacement gradually increases from 0.3 pixels to 1.0 pixels, then it may indicate that the bolt is in a state of progressive loosening. If the direction of the displacement vector suddenly changes significantly in a short period of time, for example, the horizontal displacement suddenly changes from 0.2 pixels to 2.0 pixels within a monitoring cycle, and then recovers or there is no trend of continuous change, it may be a transient offset.

[0101] In practical applications, when the displacement vector direction differences of multiple key points exceed the tolerance range, a recalibration command for the image acquisition device is triggered. For example, if the displacement vector direction differences of four key points on a bolt exceed the set tolerance range (e.g., 30°), this may indicate a deviation in the image acquisition device or interference with the bolt's condition assessment. Triggering a recalibration command at this time ensures image acquisition accuracy and, consequently, the reliability of bolt condition monitoring.

[0102] Based on the above embodiments, the sub-pixel positioning algorithm improves the accuracy of bolt key point positioning and ensures the accuracy of subsequent analysis; by matching the coordinates of key points in adjacent monitoring cycles to calculate the displacement vector, the displacement trend of the bolt can be accurately analyzed, and progressive loosening and instantaneous offset can be distinguished; when the direction difference of the key point displacement vector exceeds the range, the image acquisition device recalibration instruction is triggered, ensuring the accuracy of image acquisition, thereby improving the reliability and accuracy of the entire bolt monitoring system and ensuring the accuracy of the bolt status judgment.

[0103] In an exemplary technical solution, the method also includes an interpretable analysis step of abnormal conditions: embedding a feature importance heat map in the monitoring report to display the image area that the state recognition model focuses on during the prediction process; extracting the key feature dimensions that cause abnormal alarm signals, and generating a diagnostic suggestion text based on natural language; matching the current abnormal state with similar cases in the historical case library, and displaying the disposal records and effect evaluation data of the matching cases; calculating the probability distribution of bolt condition improvement under different maintenance strategies based on the Bayesian network model, and generating a recommended list of optimized maintenance plans.

[0104] The process of generating a feature importance heat map includes: extracting a feature activation map in the last convolutional layer of the state recognition model, calculating the gradient relationship between the activation value of each channel and the prediction result; generating an initial heat map through a gradient weighted class activation mapping algorithm, marking image areas with high contribution to abnormality prediction; performing a morphological closing operation on the initial heat map to eliminate discrete noise points and smooth the boundaries of important areas; and overlaying the processed heat map with the original monitoring image with transparency to generate a visually displayed feature importance heat map.

[0105] Specifically, the above-mentioned interpretability analysis step of the abnormal state involves a process of providing in-depth analysis and explanation of the abnormal state of the bolt from multiple perspectives.

[0106] First, a feature importance heatmap is embedded in the monitoring report. The purpose of embedding the feature importance heatmap is to visually display the image regions that the state recognition model focuses on during the prediction process. The feature activation map is extracted from the last convolutional layer of the state recognition model. This feature activation map contains the activation information of each channel when the model identifies the bolt state. For example, for a convolutional layer with 16 channels, each channel corresponds to a different feature activation. The gradient relationship between each channel activation value and the prediction result is calculated. This gradient relationship reflects the influence of each channel's feature on the final prediction result. An initial heatmap is generated using the gradient-weighted class activation mapping algorithm. This initial heatmap can identify image regions that contribute significantly to anomaly prediction. For example, in the feature activation map corresponding to an original monitoring image of 800×600 pixels, after calculation and mapping, the initial heatmap may show areas that contribute significantly to anomaly prediction, such as the connection between the bolt head and nut or specific texture areas on the bolt surface. These areas may be represented by different shades of color to indicate their contribution, with darker colors indicating higher contributions.

[0107] Next, in order to eliminate discrete noise points and smooth the boundaries of important areas, the initial heat map can be processed by morphological closing operations. In the initial heat map, there may be some discrete noise points caused by calculation errors or data fluctuations. These noise points will interfere with the accurate judgment of important areas. For example, there may be some isolated, lighter-colored pixels in the initial heat map, which do not represent areas that truly contribute significantly to anomaly prediction. Through morphological closing operations, these discrete noise points are eliminated, and the boundaries of important areas are made smoother. The processed heat map is transparently superimposed with the original monitoring image to generate a visual display of the feature importance heat map. This transparency superposition method allows people who view the monitoring report to intuitively see the important areas that the model focuses on based on the original image, making the model's prediction basis more transparent.

[0108] Then, the key feature dimensions that cause the abnormal alarm signal are extracted, and a natural language-based diagnostic suggestion text is generated. When the state recognition model detects an anomaly and issues an alarm signal, the model's internal mechanisms and data processing are analyzed to identify the feature dimensions that play a key role in generating the abnormal alarm signal. For example, these key feature dimensions may include the degree of change in the bolt's surface texture, the size of the connection gap between the bolt and the nut, etc. Based on these key feature dimensions, a natural language-based diagnostic suggestion text is generated. This diagnostic suggestion text can include a description of the abnormal state and preliminary treatment suggestions for this abnormal state. For example, if the degree of change in the bolt's surface texture is found to be large, the diagnostic suggestion text may indicate "The bolt's surface texture has changed significantly, and corrosion or wear may be present. It is recommended to further check the material integrity of the bolt."

[0109] Afterwards, the current abnormal state is matched with similar cases in the historical case library, and the disposal records and effect evaluation data of the matching cases are displayed. The historical case library stores various cases of abnormal bolt states encountered previously. Each case contains a detailed description of the abnormal state, the disposal measures taken, and the effect evaluation after the disposal. By matching the current abnormal state with historical cases, similar cases can be found. For example, the current abnormal state of the bolt is manifested as a looseness exceeding a certain range and accompanied by slight surface rust. Similar cases may be found in the historical case library. The disposal record may be that a tightening operation was performed and the rusted part was derusted. The effect evaluation data shows that the bolt state returned to normal within a period of time after the treatment. Displaying the disposal records and effect evaluation data of these matching cases can provide a reference for the handling of the current abnormal state.

[0110] Finally, the Bayesian network model calculates the probability distribution of bolt condition improvement due to different maintenance strategies, generating a recommended list of optimized maintenance solutions. The Bayesian network model can calculate the likelihood of bolt condition improvement due to different maintenance strategies based on existing data and knowledge. For example, for an abnormal bolt condition, different maintenance strategies may include tightening the bolt, replacing the bolt, and performing anti-corrosion treatment on the bolt. The Bayesian network model comprehensively considers factors such as the bolt's current condition, historical data, and the effectiveness of different maintenance strategies to calculate the probability distribution of bolt condition improvement due to each maintenance strategy. Based on this probability distribution, a recommended list of optimized maintenance solutions is generated, with maintenance strategies with higher probabilities prioritized, providing a decision-making reference for relevant personnel.

[0111] This design, by introducing a feature importance heat map, can visualize the prediction basis of the state recognition model and increase the model's transparency. The diagnostic suggestion text generated based on key feature dimensions facilitates understanding of abnormal conditions and preliminary treatment methods. Historical case matching provides a reference for past experience, which helps to formulate a treatment plan for the current abnormal condition. The recommendation list generated by the maintenance strategy probability distribution calculated by the Bayesian network model provides a basis for bolt maintenance decisions, thereby improving the interpretability of bolt abnormal condition analysis and the scientific nature of treatment decisions.

[0112] Based on the above content, in an independent embodiment, the method further includes a multi-sensor data fusion step:

[0113] Step 401: Receive data from strain sensors deployed at bolt joints and calculate a bolt preload force variation curve.

[0114] Step 402: synchronously obtain ambient temperature data collected by the temperature sensor and establish a temperature-preload compensation model.

[0115] In detail, the synchronous acquisition of ambient temperature data collected by the temperature sensor and the establishment of a temperature-preload compensation model include: collecting the ambient temperature data stream continuously generated by the temperature sensor during the monitoring period through a timestamp synchronization mechanism to generate a temperature time series sequence that is time-aligned with the strain sensor data; performing sliding window filtering on the temperature time series sequence to eliminate instantaneous temperature fluctuation noise and retain the trend temperature change component to generate a smoothed temperature trend curve; extracting segmented marking points of the temperature rising stage, stable stage and falling stage from the temperature trend curve, and dividing the temperature time series sequence into multiple temperature time series segments based on the marking points; calculating the mean temperature change rate and the temperature fluctuation frequency in each temperature time series segment to generate a characteristic curve containing temperature change characteristics. a set of eigenvectors; performing a time-aligned correlation analysis on the set of eigenvectors and the change amplitude of the bolt preload change curve in the corresponding time window, and determining the linear relationship coefficient between the mean temperature change rate and the preload attenuation; generating a temperature-preload coupling weight matrix based on the linear relationship coefficient, and the temperature-preload coupling weight matrix is ​​used to describe the compensation ratio of the unit temperature change to the preload change; performing a reverse compensation calculation on the original preload change curve according to the temperature-preload coupling weight matrix, and generating a modified preload curve that eliminates temperature interference; verifying the correlation between the modified preload curve and the displacement change obtained by image recognition within a continuous monitoring period, and confirming that the temperature-preload compensation model meets the validity condition when the correlation coefficient is greater than the verification threshold.

[0116] More specifically, further, the reverse compensation calculation is performed on the original preload change curve according to the temperature-preload coupling weight matrix to generate a modified preload curve that eliminates temperature interference, including: obtaining the preload measurement value corresponding to each sampling time point in the original preload change curve and the synchronously collected ambient temperature value; based on the preload compensation ratio corresponding to the unit temperature change stored in the temperature-preload coupling weight matrix, calculating the preload influence component caused by the deviation value of the temperature change at each sampling time point relative to the reference temperature; arranging the preload influence components in a time series to generate a temperature interference curve consistent with the time dimension of the original preload change curve; performing phase correction on the temperature interference curve to eliminate the interference between the temperature sensor and the strain sensor through a time delay matching algorithm. The method comprises the following steps: a) performing smoothness verification on the initial corrected preload curve, detecting the mutation point introduced by temperature compensation, and performing trend fitting on the data in the neighborhood of the mutation point using a local weighted regression algorithm; replacing the abnormal fluctuation segment in the initial corrected preload curve based on the fitted data, and generating a continuous and smooth corrected preload curve; performing a time window correlation analysis on the corrected preload curve and the bolt displacement change obtained by image recognition; and when the degree of change trend matching between the corrected preload curve and the bolt displacement change obtained by image recognition exceeds a verification threshold, determining that the temperature interference elimination of the corrected preload curve is completed.

[0117] Step 403: Time-align the compensated preload force variation curve with the displacement variation obtained by image recognition.

[0118] Step 404: When the decrease in the preload force and the increase in the displacement satisfy a linear relationship, the reliability of the looseness prediction result is verified.

[0119] Step 405: If there is a significant conflict between the image recognition result and the sensor data, a manual review process is triggered and the automatic alarm function is suspended.

[0120] When applying the above steps 401 to 405, first receive the data from the strain sensors deployed at the bolt joints and calculate the bolt preload change curve. The strain sensor can detect the strain caused by the force at the bolt joints, and calculate the preload change curve based on this data. For example, the strain sensor collects a series of strain data points within a certain period of time, and converts these strain data into corresponding preload data points through a specific mechanical conversion relationship. As time goes by, these data points are connected to form a preload change curve. This curve reflects the trend of the bolt preload over time. The change in preload may indicate a change in the state of the bolt. For example, a decrease in preload may indicate that the bolt has a tendency to loosen.

[0121] Ambient temperature data collected by the temperature sensor is synchronously acquired to establish a temperature-preload compensation model. Ambient temperature data collected by the temperature sensor has a significant impact on the bolt preload, as temperature changes cause the bolt material to expand and contract, which in turn affects the preload measurement. A timestamp synchronization mechanism is used to collect the ambient temperature data stream continuously generated by the temperature sensor during the monitoring period. For example, if the monitoring period is one day, the temperature sensor collects temperature data every 10 minutes. This generates a series of temperature data points throughout the day, generating a temperature time series sequence that is time-aligned with the strain sensor data. This temperature time series sequence ensures the temporal correspondence between the temperature data and the preload data in subsequent analysis.

[0122] Sliding window filtering is performed on the temperature time series. This operation aims to eliminate transient temperature fluctuation noise while retaining the trend-based temperature change component, thereby generating a smoothed temperature trend curve. For example, the sliding window size can be set to 5 data points. Within this window, a smoothed value is calculated using a specific filtering algorithm. The entire temperature time series is processed sequentially in this manner. Segment markers are extracted from the temperature trend curve to indicate the rising, stabilizing, and decreasing temperature phases. Based on these markers, the temperature time series is divided into multiple temperature time series segments. For example, a temperature trend curve may show a gradual rise in the morning, a steady state at noon, and a gradual decline in the afternoon. Markers are set based on the turning points of these phases and used to create segments. Within each temperature time series segment, the mean temperature change rate and temperature fluctuation frequency are calculated to generate a set of feature vectors containing temperature variation characteristics. For example, if a temperature time series segment contains 20 data points, the mean temperature change rate is calculated by dividing the temperature change within the segment by the time. The fluctuation frequency is also calculated by counting the number of temperature fluctuations. These values ​​are then combined into a feature vector.

[0123] A time-aligned correlation analysis is performed on the feature vector set and the amplitude of the bolt preload curve within the corresponding time window to determine the linear relationship coefficient between the mean temperature change rate and the preload attenuation. For example, within a specific time window, the mean temperature change rate is a certain value, and the bolt preload has a corresponding attenuation. By analyzing multiple such corresponding relationships, the linear relationship coefficient between the two is found. Based on this linear relationship coefficient, a temperature-preload coupling weight matrix is ​​generated. The temperature-preload coupling weight matrix is ​​used to describe the compensation ratio of the preload change per unit temperature change. Based on the temperature-preload coupling weight matrix, a reverse compensation calculation is performed on the original preload curve to generate a modified preload curve that eliminates temperature interference.

[0124] When performing reverse compensation calculations on the original preload variation curve based on the temperature-preload coupling weight matrix to generate a corrected preload curve that eliminates temperature interference, the preload measurement value and the simultaneously acquired ambient temperature value corresponding to each sampling time point in the original preload variation curve are first obtained. For example, the original preload variation curve has 100 sampling time points, each of which has a corresponding preload measurement value and the ambient temperature value collected simultaneously with that time point. Based on the preload compensation ratio per unit temperature change stored in the temperature-preload coupling weight matrix, the preload impact component caused by the deviation of the temperature change at each sampling time point from the baseline temperature is calculated. For example, if the baseline temperature is 20°C and the temperature at a certain sampling time point is 25°C, the impact component of this 5°C temperature change on the preload is calculated based on the preload compensation ratio per unit temperature change specified in the coupling weight matrix.

[0125] The preload-influencing components are arranged in a time series to generate a temperature interference curve that matches the time dimension of the original preload curve. The temperature interference curve reflects the interference caused by temperature changes on the preload. Phase correction is performed on the temperature interference curve, and a time delay matching algorithm is used to eliminate the time difference in data acquisition between the temperature sensor and the strain sensor to generate a synchronized temperature interference component. (Data acquisition from different sensors may have a time delay; this operation ensures that the temperature interference component accurately matches the original preload curve in time.) The synchronized temperature interference component is subtracted point by point from the original preload curve to generate an initial corrected preload curve. For example, if the preload value at a point in the original preload curve is 100N and the corresponding synchronized temperature interference component is 5N, then the initial corrected preload value after subtraction is 95N. Similarly, the entire initial corrected preload curve is obtained.

[0126] The smoothness of the initial corrected preload curve is verified to detect mutation points introduced by temperature compensation. For example, in the initial corrected preload curve, if the preload values ​​of two adjacent points change too much, it can be a mutation point. A local weighted regression algorithm is used to perform trend fitting on the data in the neighborhood of the mutation point. For example, the 10 data points around the mutation point are used as the neighborhood, and a reasonable trend curve is fitted based on the distribution of these data points using the local weighted regression algorithm. The abnormal fluctuation segment in the initial corrected preload curve is replaced based on the fitted data to generate a continuous and smooth corrected preload curve. A time window correlation analysis is performed on the corrected preload curve and the bolt displacement change obtained by image recognition. When the matching degree of the change trend between the corrected preload curve and the bolt displacement change obtained by image recognition exceeds the verification threshold, it is determined that the temperature interference of the corrected preload curve has been eliminated.

[0127] The compensated preload change curve is time-aligned with the displacement change obtained by image recognition. This time alignment is to compare preload and displacement, two important indicators reflecting the bolt status, on the same time scale. When the decrease in preload and the increase in displacement satisfy a linear relationship, the reliability of the looseness prediction result is verified. For example, if the decrease in preload and the increase in displacement show a stable linear relationship, such that the displacement increases by a certain amount for every decrease in preload, this indicates that the bolt displacement change obtained by image recognition and the preload change obtained by the strain sensor are mutually confirmed, thereby verifying the reliability of the looseness prediction result.

[0128] If there is a significant conflict between the image recognition results and the sensor data, a manual review process is triggered and the automatic alarm function is suspended. For example, a significant conflict can be caused by an error in image recognition or an anomaly in sensor data. For example, if image recognition shows no significant change in bolt displacement, but sensor data shows a significant decrease in preload, manual intervention and review are required to determine whether the problem lies with the bridge bolt monitoring image recognition system or a sensor failure. At the same time, the automatic alarm function is suspended to avoid issuing erroneous alarm signals.

[0129] By establishing a temperature-preload compensation model, we can eliminate temperature interference with preload measurement, making the preload curve more accurately reflect the bolt's true condition. Time-aligning the compensated preload curve with the displacement change from image recognition and verifying the linear relationship enhances the reliability of bolt loosening predictions. Conflicts between image recognition results and sensor data trigger a manual review process and suspend the automatic alarm function, avoiding false alarms and improving the accuracy and reliability of the entire bolt monitoring system. This helps to more accurately monitor bolt status and promptly identify potential problems.

[0130] The above content is explained below through two application scenario examples.

[0131] Application scenario example 1

[0132] During routine monitoring, the operations and maintenance team of a large cross-sea bridge discovered that some high-strength bolts at the main beam joints frequently triggered loosening warnings during high summer temperatures, yet manual verification revealed no significant movement. To identify the cause of the false alarms, the technical team deployed a deep learning-based bolt monitoring system and initiated a multi-sensor data fusion analysis process. Over a continuous 72-hour monitoring period, the system simultaneously collected visible light images, infrared thermal images, strain sensor data, and temperature sensor data from the bolt area.

[0133] 8:00-12:00 AM (temperature rising phase):

[0134] The ambient temperature rose steadily from 20°C to 35°C, with the temperature sensor recording an increase of 2.5°C per hour. The strain sensor indicated that the bolt preload decreased linearly from an initial value of 450kN to 420kN, a decrease of approximately 6.7%. Simultaneously, the bridge bolt monitoring image recognition system, using a sub-pixel keypoint matching algorithm, measured the displacement of the bolt head center point to be 0.12mm, which did not exceed the preset safety threshold (0.15mm). The temperature-preload compensation model detected a correlation coefficient of 0.92 between the preload drop and the temperature increase, determining that the preload change was primarily caused by thermal expansion. The system automatically applied reverse compensation to the original preload curve. The corrected preload curve showed an actual attenuation of only 5kN (a 1.1% decrease), weakly correlated with the 0.12mm displacement, and therefore no alarm was triggered.

[0135] 2:00 PM - 4:00 PM (extremely high temperature period):

[0136] After reaching a peak of 38°C, the temperature entered a stable period. At 14:23, the bridge bolt monitoring image recognition system detected abnormal displacement of bolt No. B-203: the displacement of the key point suddenly increased to 0.18mm (exceeding the threshold by 20%), and the direction of the displacement vector showed an asymmetric distribution. The strain sensor showed that its preload force dropped sharply from 425kN to 395kN, a decrease of 7.1%. The temperature compensation model calculated that the expected preload attenuation caused by the current temperature fluctuation should be 4.2kN, but the measured attenuation reached 30kN, which exceeded the compensation range by more than 6 times. The system immediately launched multi-dimensional verification:

[0137] 1) Infrared images show localized high-temperature spots on the bolt-flange contact surface (differences of up to 8°C), indicating abnormal contact pressure.

[0138] 2) The vibration sensor detected a 2.5 Hz resonant frequency at this location between 13:50 and 14:20, with an amplitude exceeding the baseline value by 30%;

[0139] 3) Matching the historical case database shows that a similar "high temperature + vibration" combination caused the same type of bolt to break three years ago.

[0140] Based on the above data, the system raised the risk level of the B-203 bolt to the emergency level, corrected the preload-displacement correlation coefficient to 0.87, generated a monitoring report including a temperature compensation curve comparison chart and a thermal map focus area, and automatically dispatched a drone for high-definition image review.

[0141] Nighttime 20:00-next day 6:00 (temperature drop phase):

[0142] As the temperature dropped back to 25°C, the system continued to track the status of bolt B-203. The corrected preload curve showed that its attenuation stabilized at 28kN (compensated value), while the compensated attenuation of adjacent bolts was less than 5kN. The displacement of the key point continued to accumulate to 0.22mm, and the displacement vector directional consistency index deteriorated from 0.85 to 0.63, consistent with progressive loosening. At 3:17 AM, the system detected a step-wise decrease in the preload of this bolt (12kN attenuation in 5 minutes). Despite temperature compensation, it still exceeded the safe range, ultimately triggering a red alarm and generating a maintenance work order.

[0143] Technical verification results:

[0144] By introducing a temperature-preload compensation model, the system reduced the false alarm rate due to temperature disturbances from 32% to 4.7% within 72 hours. For a real-world B-203 bolt failure, the time from initial abnormality detection to confirmed diagnosis was reduced to 47 minutes, an eightfold improvement in efficiency compared to traditional manual inspections. Using the compensated preload curves and displacement-temperature correlation maps provided by the system, the maintenance team accurately located deformation defects on the flange contact surface, avoiding potential structural failures.

[0145] This case study validated the effectiveness of the multi-sensor data fusion solution: when the temperature-compensated preload attenuation exceeded 15kN and the displacement exceeded 0.15mm, the system's diagnostic accuracy reached 98.6%, compared to 82.3% and 76.8%, respectively, for those relying solely on image recognition or sensor data. This fusion approach significantly improved monitoring reliability in complex environments and provided a quantitative basis for decision-making in bridge bolt maintenance.

[0146] Application scenario example 2

[0147] The superstructure of a highway steel-concrete composite box girder bridge utilizes a steel box girder and concrete deck. The intersegmental node connections, including the web, floor, and stiffening ribs, are all connected using 10.9-grade M24 high-strength bolts. The steel box girder web connections contain 208 sets of high-strength bolts, while the floor connections contain 248 sets.

[0148] The intelligent switch is controlled by a preset program, which takes regular photos to obtain image data on the bolt panel. Then, the intelligent algorithm installed on the rack server identifies the current angles of all the bolts on the panel. By comparing the bolt angle data at different time periods, the loosening status and trend of the bolts are obtained, and early warning and alarm information are issued in case of abnormality.

[0149] Depend on Figure 2 It can be seen that the changes in the bolt angles at each position are highly consistent and basically distributed in a straight line. This indicates that when using the deep learning method to install cameras on site for real-time bolt loosening monitoring, the algorithm has strong stability and strong resistance to interference in complex environments. The bolt loosening detection results are very reliable and practical.

[0150] In summary, the embodiments of the present invention ensure the data quality of subsequent analysis and reduce analysis errors caused by poor image quality by acquiring a set of monitoring images from multiple angles and preprocessing them into a standardized target image sequence. By locating the bolt monitoring area using a pre-trained positioning model, the bolt position information can be accurately obtained, which is conducive to accurate analysis. By introducing a state recognition model, the degree of looseness can be predicted based on the structural state characteristics to achieve an effective assessment of the state of the bolt components. In addition, the alarm signal output based on the matching of the looseness degree with the safety threshold can timely detect abnormal bolt status, thereby ensuring the safety of the bridge. Finally, associating the alarm signal with the image frame to generate a monitoring report with visual markings can ensure that relevant personnel can intuitively and quickly understand the abnormal state of the bolt, thereby improving monitoring efficiency and accuracy.

[0151] Furthermore, Figure 3 Schematic diagram of a bridge bolt monitoring image recognition system 200 provided by an embodiment of the present invention. Figure 3 The bridge bolt monitoring image recognition system 200 shown includes a processor 210 , which can call and run a computer program from a memory to implement the method in the embodiment of the present invention.

[0152] Alternatively, as Figure 3 As shown, the bridge bolt monitoring image recognition system 200 may further include a memory 230. The processor 210 may call and run a computer program from the memory 230 to implement the method in the embodiment of the present invention.

[0153] The memory 230 may be a separate device independent of the processor 210 , or may be integrated into the processor 210 .

[0154] Alternatively, as Figure 3 As shown, the bridge bolt monitoring image recognition system 200 may further include a transceiver 220 , and the processor 210 may control the transceiver 220 to interact with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices.

[0155] Optionally, the bridge bolt monitoring image recognition system 200 can implement the corresponding processes corresponding to the storage engine or components in the storage engine (such as a processing module) or the device deployed with the storage engine in each method of the embodiments of the present invention. For the sake of brevity, they will not be repeated here.

[0156] It should be understood that the processor in the embodiment of the present invention may be an integrated circuit chip with signal processing capabilities.

[0157] It is understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. It should be noted that the memory of the systems and methods described herein is intended to include but is not limited to suitable types of memory.

[0158] Based on the above, a readable storage medium is provided, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above method are implemented.

[0159] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0160] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solutions of the embodiments of the present invention, or the part that contributes to the prior art, can be embodied in the form of a computer software product. The computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0161] The above describes an embodiment of the present invention in conjunction with the accompanying drawings, but the embodiment of the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the embodiment of the present invention, ordinary technicians in this field can also make many forms without departing from the purpose of the embodiment of the present invention and the scope of protection of the embodiment of the present invention, all of which are protected by the embodiment of the present invention.

Claims

1. A bridge bolt monitoring image recognition method based on deep learning, characterized in that: The method comprises: Acquire a multi-angle monitoring image set of a bridge bolt component through an image acquisition device, and pre-process the monitoring image set to generate a standardized target image sequence; Positioning the bolt monitoring area for each image frame in the target image sequence based on a pre-trained positioning model to generate a regional coordinate set containing bolt position information; Inputting the regional coordinate set into a state recognition model, extracting the structural state characteristics of each bolt monitoring area, and generating a looseness prediction result of the bolt component based on the structural state characteristics; Outputting a bolt state abnormality alarm signal according to a matching result between the looseness degree prediction result and a preset safety threshold range; Associating the abnormal bolt status alarm signal with the corresponding monitoring image frame to generate a monitoring report including a visual mark; The method also includes an interpretable analysis step for abnormal conditions: embedding a feature importance heat map in the monitoring report to display the image areas that the condition recognition model focuses on during the prediction process; extracting key feature dimensions that lead to abnormal alarm signals and generating diagnostic suggestion text based on natural language; matching the current abnormal condition with similar cases in a historical case library and displaying the disposal records and effect evaluation data of the matching cases; and calculating the probability distribution of bolt condition improvement under different maintenance strategies based on a Bayesian network model to generate a recommended list of optimized maintenance plans. The process of generating a feature importance heat map includes: extracting a feature activation map in the last convolutional layer of the state recognition model, calculating the gradient relationship between the activation value of each channel and the prediction result; generating an initial heat map through a gradient-weighted class activation mapping algorithm, and marking image areas with high contribution to abnormality prediction; performing a morphological closing operation on the initial heat map to eliminate discrete noise points and smooth the boundaries of important areas; and overlaying the processed heat map with the original monitoring image with transparency to generate a visually displayed feature importance heat map.

2. The method according to claim 1, characterized in that The preprocessing of the monitoring image set to generate a standardized target image sequence includes: Performing illumination equalization processing on each original image in the monitoring image set, and eliminating brightness deviation in shadow areas by using an adaptive histogram equalization algorithm; Extract the noise distribution characteristics in the processed image, suppress the high-frequency noise based on the multi-scale filtering algorithm, and generate a set of denoised intermediate images; Performing geometric correction on the intermediate image set, adjusting the image tilt angle by using an affine transformation algorithm, so that the bolt component directions of all image frames in the intermediate image set are aligned with a preset standard direction; The rectified image frames are processed to uniform resolution and the non-monitored areas are cropped to generate a target image sequence with consistent size.

3. The method according to claim 1, characterized in that Positioning the bolt monitoring area on the target image sequence based on the pre-trained positioning model includes: Inputting the target image sequence into the convolution layer of the positioning model, extracting multi-scale spatial features in the target image sequence, and generating a feature map containing bolt profile information; The feature map is screened for candidate regions using a region proposal network, and a matching score between each candidate region and a preset bolt template is calculated; Performing non-maximum suppression processing on the candidate regions according to the matching scores, and retaining regions with confidence scores higher than a first threshold as initial positioning frames; The initial positioning frame is subjected to edge refinement correction, and the positioning frame size is adjusted using a bilinear interpolation algorithm to generate a monitoring area coordinate set including the corrected coordinates.

4. The method according to claim 3, characterized in that The process of extracting structural state features by the state recognition model includes: Extracting a local image block of each bolt monitoring area from the monitoring area coordinate set, and inputting the image block into a feature extraction module of the state recognition model; Performing hierarchical feature fusion on the image blocks through a multi-layer residual network to generate a composite feature vector including the bolt surface texture, rust degree, and connection gap; The composite feature vector is input into the time series analysis module, and the cumulative displacement of the current bolt is calculated based on the bolt displacement change trend in the historical monitoring data; generating the looseness degree prediction result based on a comparison result of the cumulative displacement amount and a preset displacement threshold, wherein the portion of the cumulative displacement amount exceeding the displacement threshold is marked as an abnormal displacement interval; The output bolt status abnormal alarm signal includes: Determining a potential risk level of the bolt according to the abnormal displacement interval in the looseness prediction result, wherein the risk level includes a first level, a second level, and an emergency level; If the risk level is the first level, generating a primary alarm signal including a recommended inspection time, and associating the signal with a corresponding monitoring image frame; If the risk level is the second level, a medium-level alarm signal including a mandatory maintenance prompt is generated, and a real-time monitoring frequency increase instruction is triggered; If the risk level is an emergency level, an immediate maintenance alarm signal is generated, and an emergency notification message including location coordinates is sent to the remote management terminal.

5. The method according to claim 1, wherein The generating of the monitoring report containing visual markings includes: Marking the monitoring area coordinates of abnormal bolts in the target image sequence, and using color coding to distinguish the marking boxes corresponding to different risk levels; Convert the looseness prediction result into a visual progress bar in percentage form, and display it in a superimposed manner on the side area of ​​the corresponding mark box; Extract the displacement change curve of the same bolt in the historical monitoring report, generate a comparative analysis chart and embed it into the appendix of the monitoring report; A time-ordered list of abnormal events is generated according to the timestamp information of the monitoring image frames, and the list is dynamically associated with the visual mark.

6. The method according to claim 1, characterized in that The method further comprises the step of jointly optimizing the positioning model and the state recognition model: Collect normal and abnormal bolt state samples from historical monitoring images and construct a training dataset containing multi-category labels; Initializing the convolutional layer parameters of the positioning model through a transfer learning algorithm and optimizing the regional positioning accuracy using a focal loss function; Introducing an attention mechanism into the state recognition model to dynamically adjust the weight distribution of key areas of the bolt during feature extraction; Use a generative adversarial network to enhance training data and generate synthetic images with different lighting conditions and angle changes to improve model generalization capabilities; The specific process of introducing the attention mechanism includes: Add a channel attention submodule after the last convolutional layer of the feature extraction module of the state recognition model to calculate the global average pooling value of each feature channel; Performing a nonlinear transformation on the pooled value through a fully connected layer to generate a channel weight vector and perform channel weighting on the original feature map; A self-attention mechanism is used in the spatial dimension to calculate the correlation matrix between each position in the feature map and other positions; The feature map is spatially recalibrated according to the correlation matrix to highlight the detailed feature areas of the bolted joint.

7. The method according to claim 1, characterized in that The method further includes a dynamic adjustment step during real-time monitoring: The displacement change of the same bolt is counted during the continuous monitoring period, and the model parameter update instruction is triggered when the change exceeds the dynamic threshold; The state recognition model is fine-tuned online based on the latest collected monitoring image data, and the feature extraction weights are updated using a sliding window algorithm; When it is detected that the ambient light intensity is continuously lower than the preset value, it automatically switches to the infrared image acquisition mode and adjusts the input parameters of the positioning model; Dynamically correcting the looseness prediction result based on bridge vibration sensor data to eliminate instantaneous displacement interference caused by external loads; The process of dynamically correcting the looseness degree prediction result includes: Obtain acceleration spectrum data of the bridge vibration sensor within the monitoring time window and extract the main vibration frequency components and their amplitudes; Calculate the correlation coefficient between the bolt displacement change and the vibration amplitude, and establish a displacement-vibration coupling relationship model; De-noising the original displacement according to the coupling relationship model to generate a corrected displacement that eliminates environmental vibration interference; The corrected displacement is input into the state recognition model, the looseness degree prediction result is recalculated and the monitoring report is updated.

8. The method according to claim 1, characterized in that The method further comprises the steps of detecting and matching bolt key points: Extracting key points of the head edge of each bolt from the monitoring area coordinate set, and generating a key point coordinate set using a sub-pixel positioning algorithm; Match the key point coordinate sets in adjacent monitoring cycles and calculate the key point displacement vectors of the same bolt at different time points; Based on the directional consistency analysis of the displacement vector, determining whether the overall displacement trend of the bolt is progressive loosening or instantaneous displacement; When it is detected that the difference in the displacement vector directions of multiple key points exceeds the tolerance range, a recalibration instruction of the image acquisition device is triggered; The implementation of the sub-pixel positioning algorithm includes: Based on the pixel-level key point detection results, the grayscale gradient distribution information of the area around the key point is extracted; Constructing a quadratic surface model to fit the grayscale gradient distribution, and calculating the extreme points of the surface as sub-pixel key point coordinates; Adjusting the quadratic surface parameters of the quadratic surface model by an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold; The optimized key point coordinates are matched with the adjacent image frames in a bidirectional optical flow, and key points with a matching degree below the second threshold are eliminated.

9. A bridge bolt monitoring image recognition system, characterized in that: The method comprises at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method according to any one of claims 1 to 8.

Images