Bridge bolt monitoring image recognition method and system based on deep learning

Abstract

The embodiment of the invention discloses a bridge bolt monitoring image recognition method and system based on deep learning, and belongs to the technical field of bridge image processing. The method comprises the following steps: acquiring a multi-angle monitoring image set of a bridge bolt part through an image acquisition device, and preprocessing the monitoring image set to generate a standardized target image sequence; positioning a bolt monitoring area for each image frame in the target image sequence based on a pre-trained positioning model, and generating an area coordinate set containing bolt position information; inputting the region coordinate set into a state recognition model, extracting structural state features of each bolt monitoring region, and generating a looseness degree prediction result of the bolt part according to the structural state features; according to a matching result of the looseness degree prediction result and a preset safety threshold range, a bolt state abnormity alarm signal is output; and associating the bolt state abnormity alarm signal to the corresponding monitoring image frame, and generating a monitoring report containing a visual mark.

Description

Technical Field

[0001] The embodiments of the present invention belong to the technical field of bridge image processing, and specifically relate to a bridge bolt monitoring image recognition method and system based on deep learning. Background Art

[0002] In the field of bridge maintenance, the condition monitoring of bolts is crucial. Traditional manual inspection technology generally has defects such as low efficiency, poor accuracy, and prominent safety hazards. Inspectors need to climb bridges or use high-altitude equipment to check point by point, facing high risks such as falling and traffic accidents, especially in bad weather or complex structures (such as cable-stayed bridges). The hidden dangers of operation are exacerbated. Detection relies on subjective means such as visual observation, tapping and listening, which takes up to several weeks and is difficult to cover hidden parts. Defects such as loose bolts and minor rust are easily overlooked. The detection results are affected by the experience and fatigue of the personnel, and the missed detection rate is high. Therefore, how to improve the monitoring efficiency and accuracy of bridge bolts has become a technical problem that needs to be solved at present. Summary of the invention

[0003] The embodiments of the present invention provide a bridge bolt monitoring image recognition method and system based on deep learning, which can solve or partially solve the technical problems involved in the above-mentioned background technology.

[0004] An embodiment of the present invention provides a bridge bolt monitoring image recognition method based on deep learning, which is applied to a bridge bolt monitoring image recognition system. The method includes: acquiring a multi-angle monitoring image set of a bridge bolt component through an image acquisition device, and preprocessing the monitoring image set to generate a standardized target image sequence; locating the bolt monitoring area for each image frame in the target image sequence based on a pre-trained positioning model, and generating a regional coordinate set containing bolt position information; inputting the regional coordinate set into a state recognition model, extracting the structural state characteristics of each bolt monitoring area, and generating a looseness prediction result of the bolt component according to the structural state characteristics; outputting a bolt state abnormality alarm signal according to a matching result between the looseness prediction result and a preset safety threshold range; associating the bolt state abnormality alarm signal with the corresponding monitoring image frame, and generating a monitoring report containing visual marks.

[0005] An embodiment of the present invention provides a bridge bolt monitoring image recognition system, comprising at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the above method.

[0006] An embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above method are implemented.

[0007] In an embodiment of the present invention, by acquiring a set of monitoring images from multiple angles and preprocessing them into a standardized target image sequence, the data quality of subsequent analysis is ensured, and the analysis error caused by poor image quality is reduced. By locating the bolt monitoring area using a pre-trained positioning model, the bolt position information can be accurately obtained, which is helpful for accurate analysis. By introducing a state recognition model, the degree of looseness can be predicted based on the structural state characteristics to achieve an effective assessment of the state of the bolt components. In addition, according to the matching of the looseness degree with the safety threshold, the output alarm signal can detect the abnormal state of the bolt in time, thereby ensuring the safety of the bridge. Finally, associating the alarm signal with the image frame to generate a monitoring report with visual markings can ensure that relevant personnel can intuitively and quickly understand the abnormal state of the bolt, thereby improving the monitoring efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flowchart of a bridge bolt monitoring image recognition method based on deep learning provided in an embodiment of the present invention.

[0009] Figure 2 This is an example diagram of the change in the angle value of the web bolt provided in an embodiment of the present invention.

[0010] Figure 3 A schematic structural diagram of a bridge bolt monitoring image recognition system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0011] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments in the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present invention.

[0012] The terms "first", "second", etc. in the embodiments of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the embodiments of the present invention can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, in the embodiments of the present invention, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0013] Figure 1A bridge bolt monitoring image recognition method based on deep learning is shown, which is applied to a bridge bolt monitoring image recognition system. The method includes the following steps 101-103.

[0014] Step 101: Acquire a multi-angle monitoring image set of a bridge bolt component through an image acquisition device, and pre-process the monitoring image set to generate a standardized target image sequence.

[0015] First, the image acquisition device will collect images of the bridge bolt components from multiple angles to obtain a monitoring image set. Among them, multi-angle acquisition can fully and completely reflect the various characteristics of the bolt components, for example, it can be collected from different angles such as vertical, horizontal and oblique.

[0016] Furthermore, the collected monitoring image set may be affected by various factors, such as uneven lighting, image noise, etc., so preprocessing is required to generate a standardized target image sequence. In an embodiment of the present invention, the preprocessing process involves multiple aspects. In terms of noise removal, some common filtering methods can be used, such as median filtering. Median filtering can effectively remove isolated noise points such as salt and pepper noise. For the case of uneven lighting, a histogram equalization method can be used to adjust the brightness and contrast of the image so that the image achieves a relative balance in brightness and contrast, thereby making the details of the image clearer and more visible. Through the above-mentioned preprocessing operation, a monitoring image set with uneven quality and various interference factors can be converted into a standardized target image sequence with higher quality and more suitable for subsequent analysis.

[0017] Step 102: Based on the pre-trained positioning model, each image frame in the target image sequence is positioned in the bolt monitoring area to generate a regional coordinate set containing bolt position information; the regional coordinate set is input into the state recognition model to extract the structural state characteristics of each bolt monitoring area, and a looseness prediction result of the bolt component is generated according to the structural state characteristics.

[0018] Among them, in this step, the pre-trained positioning model is trained based on sample data, and the sample data contains images of bridge bolt components of various types and in different states. For each image frame in the target image sequence, the positioning model can accurately locate the bolt monitoring area. For example, the positioning model can be built based on a convolutional neural network (CNN), which has a powerful feature extraction capability and can identify the unique features of the bolt monitoring area in the image to determine its position. Through the processing of the positioning model, a set of regional coordinates containing bolt position information is generated. These regional coordinates accurately indicate the position of each bolt in the image. For example, with the upper left corner of the image as the origin and the coordinates in pixels, the upper left corner coordinate of a bolt can be (100, 200), the lower right corner coordinate can be (150, 230), etc.

[0019] Subsequently, the regional coordinate set is input into the state recognition model. The state recognition model is also trained based on a large amount of labeled sample data. The labeling of the sample data includes information such as the structural state characteristics of the bolts under different degrees of looseness. The state recognition model extracts the structural state characteristics of each bolt monitoring area. In the process of extracting the structural state characteristics, the shape characteristics of the bolts may be involved. For example, under normal circumstances, the outline of the bolt should be a regular circle or square. If looseness occurs, the outline may be deformed. The relative position relationship characteristics between the bolts and the surrounding components may also be involved. For example, the bolts and nuts fit tightly under normal circumstances, and gaps may appear when loosening. Based on the extracted structural state characteristics, the state recognition model can generate the prediction results of the looseness of the bolt components. The prediction results of the looseness degree can be qualitative descriptions, such as different levels such as no loosening, slight loosening, moderate loosening and severe loosening.

[0020] Step 103: outputting a bolt status abnormality alarm signal according to the matching result between the looseness prediction result and the preset safety threshold range; associating the bolt status abnormality alarm signal with the corresponding monitoring image frame, and generating a monitoring report including visual marks.

[0021] In an embodiment of the present invention, step 103 is the decision-making and output part of the entire process. The operation is performed according to the matching result of the looseness prediction result and the preset safety threshold range. The preset safety threshold range is comprehensively determined based on the design standards of the bridge, engineering experience, and a large amount of experimental data. If the looseness prediction result exceeds the safety threshold range, it means that the state of the bolt may be at risk, and the system will output an abnormal bolt state alarm signal at this time. The bolt state abnormal alarm signal contains key information such as the position information and looseness degree of the bolt. Then, the system will associate the bolt state abnormal alarm signal with the corresponding monitoring image frame, and visually mark the abnormal bolts on the monitoring image frame, for example, the position of the abnormal bolt can be circled with a red circle, and the information such as the looseness degree can be marked next to it, thereby generating a monitoring report containing visual marks. Such a monitoring report can intuitively show the status of the bolt to relevant personnel.

[0022] The bridge bolt monitoring image recognition system can effectively monitor and analyze bridge bolt components and provide intuitive monitoring result reports through the process of step 101 to step 103, thereby providing an important basis for the safe maintenance of the bridge.

[0023] In another feasible technical solution, when the image acquisition device acquires the monitoring image set, the more and more comprehensive the acquired angles are, the more accurately the real state of the bolt component can be reflected. For example, if the image is acquired from only a single angle, the features of some key parts of the bolt may be missed, while acquiring from multiple angles, such as acquiring every 30 degrees in the horizontal direction and every 20 degrees in the vertical direction, can avoid this situation to the greatest extent.

[0024] For pre-trained positioning models, the quality and quantity of training data have a crucial impact on the accuracy of their positioning. If the training data contains images of bolts in various complex environments, such as images under different lighting conditions and weather conditions, then the positioning model can more accurately locate the bolt monitoring area in actual applications. Taking a training dataset containing 100,000 images of bolts in different environments as an example, the positioning accuracy of the positioning model trained with such a dataset will be significantly improved compared to the model trained with only 10,000 images.

[0025] When the state recognition model extracts structural state features, the type of features it focuses on and the accuracy of feature extraction directly determine the reliability of the looseness prediction results. In addition to the bolt shape features and relative position relationship features mentioned above, it can also include texture features on the bolt surface. For example, the texture on a normal bolt surface is uniform, but if looseness occurs and wear occurs, the texture will change. If the state recognition model can accurately identify the changes in these texture features, then the prediction of the looseness will be more accurate.

[0026] The rationality of the preset safety threshold range is an important guarantee for the effectiveness of the entire system. If the safety threshold range is set too wide, some bolts that already have risks may be misjudged as normal; if it is set too narrow, some bolts in normal state may be misjudged as abnormal. This requires comprehensive consideration of multiple factors such as the actual use of the bridge, design requirements, and past engineering experience to determine an appropriate safety threshold range.

[0027] Therefore, the bridge bolt monitoring image recognition system realizes the effective monitoring and status assessment of bridge bolt components through the above-mentioned collaborative work, and presents the results in the form of intuitive monitoring reports.

[0028] In another feasible technical solution, there are also many considerations in the selection of image acquisition devices. For example, you can choose a high-resolution camera, such as a camera with a resolution of 1920×1080 pixels, which can capture clearer bolt details. At the same time, the focal length, field of view and other parameters of the camera also need to be adjusted according to actual monitoring needs. If the monitored bolt area is small and the details need to be observed accurately, you need to choose a camera with a longer focal length and a smaller field of view; conversely, if you need to monitor a larger range of bolts as a whole, you can choose a camera with a shorter focal length and a larger field of view.

[0029] For the pre-trained positioning model and state recognition model, the structural design of the model is also very critical. Taking the positioning model as an example, in addition to the previously mentioned convolutional neural network (CNN)-based construction, some improved network structures can also be used, such as ResNet (residual network). By introducing residual connections, ResNet can effectively solve the gradient vanishing problem that occurs when the network depth increases, thereby improving the performance of the model. In the state recognition model, similar advanced network structures can also be used, or multi-model fusion can be used to fuse the model results of different structures to improve the ability to extract structural state features and the accuracy of looseness prediction.

[0030] When generating a monitoring report containing visual markings, the method and content of the visual markings can be customized according to actual needs. In addition to marking the abnormal bolt position with a red circle and marking the degree of looseness mentioned above, more information can be added to the monitoring report, such as the time of image acquisition, bolt number, etc. This can make the monitoring report more detailed and comprehensive, and facilitate the management and maintenance work of relevant personnel.

[0031] In the actual implementation process, the installation position of the image acquisition device also needs to be adjusted according to the actual situation. If the installation position is unreasonable, the collected images may be blocked or deformed. For the monitoring of bridge bolts, the installation position should ensure that the camera can clearly capture the bolt components and minimize the interference of external factors. For example, it should be installed at a certain distance in front of the bolt components, and avoid being blocked by other bridge structure components. If there are multiple bolts to be monitored, a distributed installation method can be adopted to install multiple image acquisition devices at different locations to ensure that each bolt can be effectively monitored. Pre-trained positioning model and state recognition model During the training process, for the positioning model of the bolt monitoring area, the annotation needs to accurately indicate the position coordinates of the bolt in the image; for the state recognition model, the annotation needs to describe the structural state characteristics of the bolt and the corresponding degree of looseness in detail. When generating a monitoring report containing visual marks, the format and storage method of the report also need to be considered. A common document format, such as PDF format, can be used so that different devices and software can be easily viewed. In terms of storage, the monitoring report can be stored in a local database and can also be backed up to a cloud server. This can not only ensure the security of data, but also facilitate data sharing and query.

[0032] In another possible application scenario of an embodiment of the present invention, in addition to the parameters such as resolution, focal length, and field of view mentioned above, the parameter adjustment of the image acquisition device may also consider adjusting the frame rate of image acquisition. The frame rate determines the number of frames of images collected per second. If the frame rate is too high, the amount of data and the processing burden of the system will increase; if the frame rate is too low, some short-term changes in the state of the bolts may be missed. For example, for general bridge bolt monitoring, the frame rate can be set to 10-15 frames / second, which can ensure that the state changes of the bolts are captured without generating too much data.

[0033] When evaluating the performance of the pre-trained positioning model and state recognition model, a variety of evaluation indicators can be used. For the positioning model, indicators such as accuracy and recall can be used. Accuracy indicates the ratio of the number of correctly positioned bolts to the number of predicted bolts, and recall indicates the ratio of the number of correctly positioned bolts to the actual number of bolts. For the state recognition model, indicators such as classification accuracy and confusion matrix can be used. Through these evaluation indicators, the performance of the model can be accurately understood, so that the model can be optimized in a targeted manner.

[0034] When generating the looseness prediction results of bolt components, in addition to the previously mentioned features such as bolt shape, relative position relationship, surface texture, etc., the input features of the model can also consider the characteristics of the environment around the bolt. For example, whether there is water accumulation around the bolt, whether there is corrosion, etc. These environmental characteristics may affect the state of the bolt, and using them as input features of the model can improve the accuracy of the looseness prediction results.

[0035] When outputting an abnormal bolt status alarm signal based on the matching result of the looseness prediction result and the preset safety threshold range, in addition to directly outputting an alarm signal, a comprehensive analysis of multiple consecutive monitoring results can also be performed. For example, if three consecutive monitoring results show that the bolt is slightly loose, although the single result does not exceed the safety threshold range, it can be considered that the bolt status has potential risks based on comprehensive considerations, and an alarm signal can also be output at this time.

[0036] In one embodiment, preprocessing the monitoring image set to generate a standardized target image sequence includes: Step 1011: Perform illumination equalization processing on each original image in the monitoring image set, and eliminate brightness deviation in shadow areas by using an adaptive histogram equalization algorithm.

[0037] Step 1012: extract the noise distribution features in the processed image, suppress the high-frequency noise based on the multi-scale filtering algorithm, and generate a set of intermediate images after noise reduction.

[0038] Step 1013: geometrically correct the intermediate image set, and adjust the image tilt angle by using an affine transformation algorithm so that the bolt component directions of all image frames in the intermediate image set are aligned with a preset standard direction.

[0039] Step 1014: The corrected image frames are processed to have uniform resolution, and non-monitored areas are cropped to generate a target image sequence with a consistent size.

[0040] In the bridge bolt monitoring image recognition system, the monitoring image set is preprocessed to generate a standardized target image sequence. In this link, each original image in the monitoring image set is first subjected to illumination equalization. Uneven illumination will bring many difficulties to subsequent analysis. For example, the shadow area may cover certain features of the bolt. The embodiment of the present invention adopts an adaptive histogram equalization algorithm to eliminate the brightness deviation of the shadow area. The adaptive histogram equalization algorithm can automatically adjust the brightness according to the local contrast information of the image. Compared with the ordinary histogram equalization, the adaptive histogram equalization algorithm can better adapt to images under different lighting conditions. For example, in some bridge environments, there may be a situation where the illumination is strong on one side and weak on the other side. The adaptive histogram equalization algorithm can effectively adjust the brightness of the image under such uneven illumination, so that the brightness of the entire image is more uniform, thereby highlighting the details of the bolt components.

[0041] Next, the noise distribution features are extracted from the image after illumination equalization. The noise in the image will interfere with the accurate recognition of the bolt features. There are various types of noise, among which high-frequency noise has a more obvious impact on image quality. The high-frequency noise is suppressed based on the multi-scale filtering algorithm to generate a set of intermediate images after noise reduction. The multi-scale filtering algorithm can filter the image at different scales and can effectively capture and suppress noise information of different frequencies. For example, for some high-frequency noise generated by interference from electronic equipment, the multi-scale filtering algorithm can filter at a suitable scale according to the frequency characteristics of the noise, making the image smoother while better retaining important features such as the edges of the bolts.

[0042] Then, the intermediate image set is geometrically corrected. During the image acquisition process, the image may be tilted due to the angle of the acquisition device or the structural characteristics of the bridge itself, which will affect the accurate judgment of the position and status of the bolt component. The image tilt angle is adjusted through the affine transformation algorithm so that the direction of the bolt component in all image frames in the intermediate image set is aligned with the preset standard direction. The affine transformation algorithm can rotate, translate, scale and other operations on the image while maintaining the parallelism and colinearity of the image. For example, if the bolt shows a certain tilt in the image, it can be rotated to an angle consistent with the preset standard direction through the affine transformation algorithm, so that in the subsequent positioning and status recognition process, operations can be performed based on a unified direction standard to improve accuracy.

[0043] Finally, the corrected image frames are processed with uniform resolution, and the non-monitoring areas are cropped to generate a target image sequence with consistent size. Different image acquisition devices or acquisition parameters may result in different image resolutions, but in subsequent analysis, uniform resolution helps improve processing efficiency and accuracy. On the basis of uniform resolution, cropping the non-monitoring areas can reduce the amount of unnecessary data, making the target image sequence more focused on the bolt components. For example, for some images that contain a lot of surrounding environment information, the presence of non-monitoring areas will increase the burden of data storage and processing. By cropping, only the areas related to bolt monitoring can be retained, and the sizes of all images can be adjusted to be consistent, so that when they are input into the subsequent positioning model and state recognition model, data processing and feature extraction can be better performed.

[0044] The preprocessing process in this embodiment processes the monitoring image set from multiple aspects, so that the image is standardized in terms of illumination, noise, geometric shape and size. The completion of the standardized target image sequence provides a good data basis for the subsequent processing based on the pre-trained positioning model and state recognition model. The illumination equalization processing makes the brightness of the image uniform, which is conducive to highlighting the bolt features; the noise reduction processing reduces the interference of noise on feature recognition; the geometric correction ensures the consistency of the direction of the bolt components, which is convenient for the subsequent unified standard operation; the resolution unification and the non-monitoring area cropping improve the efficiency and accuracy of data processing. Through the synergy of these operations, the accuracy and reliability of the entire bridge bolt monitoring image recognition system are improved, which provides an important guarantee for the effective monitoring of bridge bolts. By preprocessing the monitoring image set to generate a standardized target image sequence, the image quality can be improved in many aspects. Illumination equalization reduces the impact of uneven illumination on the display of bolt features, and multi-scale filtering noise reduction avoids noise interference with subsequent feature recognition. Geometric correction makes the direction of the bolt components consistent, which is conducive to subsequent operations, and resolution unification and cropping of non-monitoring areas improve data processing efficiency and accuracy. Thereby, the accuracy and reliability of bridge bolt monitoring image recognition processing can be improved, and the precision and credibility of bridge bolt monitoring can be improved.

[0045] In another embodiment, the positioning of the bolt monitoring area of ​​the target image sequence based on the pre-trained positioning model includes: Step 201: Input the target image sequence into the convolution layer of the positioning model, extract the multi-scale spatial features in the target image sequence, and generate a feature map containing bolt profile information; screen the candidate regions of the feature map through a region proposal network, and calculate the matching score between each candidate region and a preset bolt template.

[0046] Step 202: Perform non-maximum suppression processing on the candidate regions according to the matching scores, and retain regions with confidence levels higher than a first threshold as initial positioning frames.

[0047] Step 203: Perform edge refinement correction on the initial positioning frame, use a bilinear interpolation algorithm to adjust the size of the positioning frame, and generate a monitoring area coordinate set containing corrected coordinates.

[0048] In this embodiment, the bolt monitoring area is located for the target image sequence based on the pre-trained positioning model. First, the target image sequence is input into the convolution layer of the positioning model, and feature extraction is performed through the convolution layer. For example, in a target image sequence with a size of 1000×1000 pixels, the convolution layer can extract multi-scale spatial features from receptive fields of different sizes. Multi-scale spatial features are critical for accurately depicting the bolt contour information. Large-scale features may correspond to the approximate contour range of the bolt in the image, while small-scale features can capture the finer parts of the bolt contour, thereby generating a feature map containing the bolt contour information.

[0049] Next, the region proposal network screens candidate regions for the feature map. In the process of candidate region screening, the preset bolt template plays an important reference role. For example, the preset bolt template has five main shape and texture features, and the region proposal network calculates the matching score of each candidate region with the preset bolt template in terms of these features. For example, if a candidate region has 80% similarity with the template in terms of shape features and 70% similarity in terms of texture features, the matching score of this candidate region can be obtained through a specific algorithm.

[0050] Then, non-maximum suppression is performed on the candidate regions according to the matching scores. For example, if the first threshold is set to 0.6, only those regions with matching scores higher than 0.6 will be retained as the initial positioning boxes. This step can effectively reduce a large number of overlapping candidate regions with low confidence, avoid unnecessary interference in subsequent processing, and improve positioning accuracy and efficiency.

[0051] After that, the initial positioning frame is refined and corrected. The bilinear interpolation algorithm is used to adjust the size of the positioning frame to generate a monitoring area coordinate set containing the corrected coordinates (which can be understood as precise coordinates). For example, the side length of the initial positioning frame can be 100 pixels. After adjustment by the bilinear interpolation algorithm, its side length becomes more accurate 98 pixels or 102 pixels, etc., so that it fits the actual area of ​​the bolt more closely, so that the monitoring area coordinate set can accurately represent the position of the bolt in the image.

[0052] On the basis of the above content, the process of extracting structural state features by the state recognition model includes: extracting local image blocks of each bolt monitoring area from the monitoring area coordinate set, and inputting the image blocks into the feature extraction module of the state recognition model; performing hierarchical feature fusion on the image blocks through a multi-layer residual network to generate a composite feature vector containing the bolt surface texture, rust degree and connection gap; inputting the composite feature vector into a time series analysis module, and calculating the cumulative displacement of the current bolt in combination with the bolt displacement change trend in the historical monitoring data; based on the comparison result of the cumulative displacement and the preset displacement threshold, generating the looseness degree prediction result, wherein the part of the cumulative displacement exceeding the displacement threshold is marked as an abnormal displacement interval.

[0053] In detail, in the process of extracting structural state features based on the state recognition model, local image blocks of each bolt monitoring area can be extracted from the monitoring area coordinate set, and these image blocks can be input into the feature extraction module of the state recognition model. The feature extraction module in the embodiment of the present invention performs hierarchical feature fusion on the image blocks through a multi-layer residual network. The multi-layer residual network can effectively fuse features at different levels. For example, for a multi-layer residual network containing 10 layers, simple features of the bolt surface texture may be extracted at the lower layers. As the number of layers increases, more features about the overall structure of the bolt, such as the degree of rust and the connection gap, can be integrated into the higher layers, thereby generating a composite feature vector containing the bolt surface texture, the degree of rust and the connection gap.

[0054] The composite feature vector is input into the time series analysis module, and the cumulative displacement of the current bolt is calculated by combining the displacement trend of the bolt in the historical monitoring data. The historical monitoring data contains the displacement information of the bolt at different time points in the past, which is crucial for a comprehensive assessment of the current state of the bolt. For example, the historical monitoring data shows that the displacement of the bolt in the past five monitorings was 1 mm, 1.5 mm, 2 mm, 2.5 mm, and 3 mm, respectively. Combined with the current composite feature vector, the cumulative displacement of the current bolt can be calculated through the time series analysis module.

[0055] The looseness prediction result is generated based on the comparison result of the cumulative displacement and the preset displacement threshold, and the part where the cumulative displacement exceeds the displacement threshold is marked as an abnormal displacement interval. For example, if the preset displacement threshold is 5 mm, when the cumulative displacement reaches 6 mm, the excess 1 mm is marked as an abnormal displacement interval, so that the looseness of the bolt and the abnormal part can be intuitively reflected.

[0056] Furthermore, the output of the abnormal bolt status alarm signal includes: determining the potential risk level of the bolt according to the abnormal displacement interval in the looseness prediction result, wherein the risk level includes the first level, the second level and the emergency level; if the risk level is the first level, a primary alarm signal including a recommended inspection time is generated, and the signal is associated with the corresponding monitoring image frame; if the risk level is the second level, an intermediate alarm signal including a mandatory maintenance prompt is generated, and a real-time monitoring frequency increase instruction is triggered; if the risk level is the emergency level, an immediate maintenance alarm signal is generated, and an emergency notification message including the location coordinates is sent to the remote management terminal.

[0057] Exemplarily, in terms of outputting an abnormal alarm signal of the bolt status, the potential risk level of the bolt is determined according to the abnormal displacement interval in the looseness prediction result, and the risk levels are divided into the first level, the second level and the emergency level.

[0058] When the risk level is the first level, a primary alarm signal including a recommended inspection time is generated and associated with the corresponding monitoring image frame. This level indicates that although there is a certain potential risk in the bolt, the situation is relatively less urgent. For example, although the displacement of the bolt slightly exceeds the threshold, it has not yet had a serious impact on the overall structure. At this time, a recommended inspection time is given to facilitate relevant personnel to inspect the bolt at the appropriate time. At the same time, the specific status of the bolt can be quickly viewed through the associated monitoring image frame.

[0059] When the risk level is the second level, an intermediate alarm signal including a mandatory maintenance prompt is generated, and a real-time monitoring frequency increase instruction is triggered. This means that the bolt's condition is already serious and mandatory maintenance is required, and its condition changes must be monitored more closely. For example, if the bolt's displacement is large and exceeds the threshold by a large amount, it may have had a certain impact on the connection structure. At this time, increasing the real-time monitoring frequency can timely grasp further changes in the bolt's condition.

[0060] When the risk level is the emergency level, the bolt condition is already very critical, and an immediate maintenance alarm signal is generated, and an emergency notification message containing the location coordinates is sent to the remote management terminal. For example, if the displacement of the bolt far exceeds the threshold and may cause serious damage to the structure, maintenance must be carried out immediately. By sending an emergency notification message containing the location coordinates to the remote management terminal, it can ensure that relevant personnel can quickly locate the problem bolt and take emergency measures.

[0061] With this design, the state recognition model can accurately determine the degree of bolt loosening by integrating multiple features and historical data. In addition, the alarm signal of abnormal bolt status is generated according to the risk level classification. Based on the above content, from the primary recommended inspection to the intermediate mandatory maintenance prompt to the emergency level immediate maintenance notification, it can respond to different degrees of bolt risks in a targeted manner, effectively improving the accuracy and timeliness of bridge bolt status monitoring, thereby realizing the safety of monitoring bridge structures.

[0062] In a preferred embodiment, generating a monitoring report including visual markings in step 103 includes: Step 1031: marking the monitoring area coordinates of abnormal bolts in the target image sequence, and using color coding to distinguish the marking boxes corresponding to different risk levels.

[0063] Step 1032: Convert the looseness prediction result into a visual progress bar in percentage form, and display it in a superimposed manner on the side area of ​​the corresponding mark box.

[0064] Step 1033: extract the displacement change curve of the same bolt in the historical monitoring report, generate a comparative analysis chart and embed it into the appendix of the monitoring report.

[0065] Step 1034: Generate a time-ordered list of abnormal events based on the timestamp information of the monitoring image frame, and dynamically associate the list with the visual mark.

[0066] In this preferred embodiment, in the process of generating a monitoring report containing visual marks, it is first necessary to mark the monitoring area coordinates of the abnormal bolts in the target image sequence. The monitoring area coordinates clearly indicate the position of the abnormal bolts in the image. For example, in a target image sequence with a size of 800×600 pixels, the monitoring area coordinates of the abnormal bolts can be an area with the coordinates of the upper left corner (100, 200) and the coordinates of the lower right corner (150, 250). Then color coding is used to distinguish the marking boxes corresponding to different risk levels. Different colors can intuitively allow people who view the report to distinguish the severity of the bolt risk. For example, a red marking box may be used for bolts with a high risk level, a yellow marking box for a medium risk level, and a green marking box for a low risk level. This color coding method can quickly convey bolt risk information without having to view detailed text descriptions.

[0067] Next, the looseness prediction result is converted into a visual progress bar in percentage form and displayed superimposed on the side area of ​​the corresponding mark box. For example, if the looseness prediction result shows that the looseness of a bolt is in a certain range, after converting it into a percentage form, such as 30%, it can be displayed in the form of a visual progress bar on the side of the mark box. This makes the looseness of the bolt clear at a glance, and the viewer can intuitively understand the proportion of the bolt looseness in the overall situation, which helps to quantitatively evaluate the bolt status.

[0068] After that, the displacement change curve of the same bolt in the historical monitoring report is extracted, and a comparative analysis chart is generated and embedded in the appendix of the monitoring report. The historical monitoring report contains the displacement information of the same bolt at different time points in the past. By plotting this information into a displacement change curve, the displacement trend of the bolt over a period of time can be clearly seen. For example, in the past 10 monitorings, the displacement of the bolt may show a trend of gradual increase or fluctuation. The above comparative analysis chart can provide more comprehensive bolt status information, which is convenient for viewers to view in the appendix when they need to conduct in-depth analysis of bolt status changes.

[0069] Finally, based on the timestamp information of the monitoring image frames, a time-ordered list of abnormal events is generated, and the list is dynamically associated with the visual mark. The timestamp information records the acquisition time of each monitoring image frame. Based on this time information, the order in which the abnormal events occurred can be sorted out. For example, a bolt first showed a slight abnormality in the monitoring on March 1, and the degree of abnormality worsened in the monitoring on March 10. These events are arranged in the abnormal event list in chronological order. Dynamically associating this list with the visual mark means that when the viewer clicks on a visual mark, they can simultaneously view the detailed time sequence information of the abnormal events related to it, thereby accurately tracking the changes in the bolt status over time.

[0070] In this way, by marking coordinates in the target image sequence and using color-coded marker boxes, the abnormal bolt location and risk level can be intuitively displayed. The visual progress bar clearly presents the quantitative situation of the looseness. The comparative analysis chart of the historical displacement change curve provides comprehensive historical status information, which helps to deeply analyze the development trend of the bolt status. The dynamic association of the time-ordered abnormal event list and the visual marker enables the viewer to better understand the change process of the bolt status over time, thereby improving the readability of the monitoring report and the comprehensive display ability of the bolt status information.

[0071] In an alternative embodiment, the method also includes the step of jointly optimizing the positioning model and the state recognition model: collecting normal state samples and abnormal state samples of bolts in historical monitoring images to construct a training data set containing multi-category labels; initializing the convolution layer parameters of the positioning model through a transfer learning algorithm, and using a focal loss function to optimize the regional positioning accuracy; introducing an attention mechanism into the state recognition model to dynamically adjust the weight allocation of key areas of the bolts during feature extraction; using a generative adversarial network to enhance the training data to generate synthetic images containing different lighting conditions and angle changes to improve the model generalization ability.

[0072] Among them, the specific process of introducing the attention mechanism includes: adding a channel attention submodule after the last convolution layer of the feature extraction module of the state recognition model to calculate the global average pooling value of each feature channel; performing a nonlinear transformation on the pooling value through a fully connected layer to generate a channel weight vector and perform channel weighting on the original feature map; using a self-attention mechanism in the spatial dimension to calculate the correlation matrix between each position in the feature map and other positions; and spatially recalibrating the feature map according to the correlation matrix to highlight the detailed feature areas of the bolt joints.

[0073] It can be understood that the above embodiment specifically includes the step of jointly optimizing the positioning model and the state recognition model. First, the normal state samples and abnormal state samples of the bolts in the historical monitoring images are collected to construct a training data set containing multi-category labels. The historical monitoring images cover the different states of the bolts, and the multi-category labels can accurately classify and identify the samples. For example, 3,000 normal state samples and 2,000 abnormal state samples of the bolts are selected from the past 5,000 historical monitoring images. The abnormal state samples include images of the bolts in different structures and different surface conditions. The constructed training data set can comprehensively reflect the various state characteristics of the bolts.

[0074] Next, the convolutional layer parameters of the positioning model are initialized through the transfer learning algorithm, and the focal loss function is used to optimize the regional positioning accuracy. The transfer learning algorithm uses the existing pre-trained model parameters to initialize the convolutional layer parameters of the positioning model. For example, the pre-trained model has good performance in similar image recognition tasks, and its convolutional layer parameters can provide a good initial state for the current positioning model. The focal loss function focuses on samples that are difficult to classify, and can improve the accuracy of bolt monitoring area positioning during regional positioning. If some of the 1,000 candidate areas are difficult to accurately classify because of unclear features or interference, the focal loss function will increase attention to these areas to make positioning more accurate.

[0075] Then, the attention mechanism is introduced into the state recognition model. The attention mechanism can dynamically adjust the weight distribution of the key areas of the bolts during the feature extraction process. Specifically, a channel attention submodule is added after the last convolution layer of the feature extraction module, and the global average pooling value of each feature channel is first calculated. For example, for a feature map output by a convolution layer with 16 feature channels, the global average pooling value of each channel is calculated. The global average pooling value reflects the average response degree of each channel in the overall feature. Then, the pooling value is nonlinearly transformed through the fully connected layer to generate a channel weight vector and perform channel weighting on the original feature map. As a result, the state recognition model can adjust the original feature map according to the importance of different channel features and highlight important channel features.

[0076] The self-attention mechanism is used in the spatial dimension to calculate the correlation matrix between each position in the feature map and other positions. For example, for a feature map of size 20×20, the correlation between each position (400 positions in total) and other positions is calculated to form a 400×400 correlation matrix. According to this correlation matrix, the feature map is spatially recalibrated to highlight the detailed feature areas such as bolt joints. In bolt monitoring, bolt joints are key parts, and their status has an important impact on the safety of the entire bolt. Through spatial recalibration, the state recognition model can pay more attention to the characteristics of these key areas.

[0077] Finally, the training data is enhanced using a generative adversarial network to generate synthetic images with different lighting conditions and angle changes to improve the generalization ability of the model. The generative adversarial network can generate new synthetic images based on the existing training data. These synthetic images contain different lighting conditions and angle changes. For example, the images in the original training data may mainly be images of bolts taken from the front under normal lighting. Through the generative adversarial network, images of bolts taken under different lighting conditions such as strong light, weak light, and side light, and from different angles, such as images taken at an angle of 30 degrees, 60 degrees, etc. can be generated. This enables the model to better perform positioning and state recognition when facing various actual situations, improving the generalization ability of the model.

[0078] With this design, the optimization of transfer learning initialization and focal loss function can improve the positioning accuracy of the positioning model. Introducing the attention mechanism into the state recognition model to highlight the key area features can improve the accuracy of bolt state feature extraction. Combined with the adversarial generative network, the generalization ability of the positioning model and the state recognition model can be improved, so that the positioning model and the state recognition model can better cope with the bolt monitoring task under different lighting and angles, thereby enhancing the performance of the positioning model and the state recognition model.

[0079] In another alternative embodiment, the method also includes a dynamic adjustment step during real-time monitoring: counting the displacement change of the same bolt within a continuous monitoring period, and triggering a model parameter update instruction when the change exceeds a dynamic threshold; online fine-tuning the state recognition model based on the latest collected monitoring image data, and updating the feature extraction weights using a sliding window algorithm; when it is detected that the ambient light intensity is continuously lower than a preset value, automatically switching to the infrared image acquisition mode and adjusting the input parameters of the positioning model; dynamically correcting the looseness prediction result according to the bridge vibration sensor data to eliminate instantaneous displacement interference caused by external loads.

[0080] Among them, the process of dynamically correcting the looseness degree prediction result includes: obtaining the acceleration spectrum data of the bridge vibration sensor within the monitoring time window, extracting the main vibration frequency components and their amplitudes; calculating the correlation coefficient between the bolt displacement change and the vibration amplitude, and establishing a displacement-vibration coupling relationship model; denoising the original displacement according to the coupling relationship model to generate a corrected displacement that eliminates environmental vibration interference; inputting the corrected displacement into the state recognition model, recalculating the looseness degree prediction result and updating the monitoring report.

[0081] The above-mentioned embodiment includes a dynamic adjustment step in the real-time monitoring process, and the displacement change of the same bolt is counted in a continuous monitoring cycle. This operation is based on continuous attention to the state of the bolt. For example, in each continuous monitoring cycle set to 1 hour, the displacement of a specific bolt is counted. When this displacement change exceeds the dynamic threshold, the model parameter update instruction will be triggered. This dynamic threshold is an important indicator set according to the displacement change range of the bolt under normal working conditions and the accuracy requirements of the system. Once this threshold is exceeded, it means that the state of the bolt may have changed significantly, and the model parameters need to be updated to adapt to this change.

[0082] The state recognition model is fine-tuned online based on the latest acquired monitoring image data, and the sliding window algorithm is used to update the feature extraction weights. The latest acquired monitoring image data contains the current state information of the bolt, and the sliding window algorithm plays a key role in this process. For example, in a sequence containing 100 latest monitoring images, the sliding window algorithm can gradually update the feature extraction weights according to the set window size, such as 20 images. In this way, new data can be used in a timely manner to adjust the model's extraction method of bolt features, so that the model can more accurately identify the state of the bolt.

[0083] When it is detected that the ambient light intensity is continuously lower than the preset value, it automatically switches to the infrared image acquisition mode and adjusts the input parameters of the positioning model. Ambient light intensity is an important factor affecting the quality of image acquisition. When the light intensity is continuously lower than the preset value, such as lower than 50 lux, the ordinary image acquisition mode may not be able to provide a clear image of the bolt. At this time, the infrared image acquisition mode can obtain the image of the bolt in a low-light environment. At the same time, the input parameters of the positioning model need to be adjusted (infrared images differ from ordinary visible light images in feature expression). For example, the features such as the outline and temperature distribution of the bolt in the infrared image will be more obvious, and the positioning model needs to adapt to this change to accurately locate the bolt monitoring area.

[0084] In addition, the looseness prediction result is dynamically corrected according to the bridge vibration sensor data. This process is to eliminate the instantaneous displacement interference caused by external loads. In detail, the acceleration spectrum data of the bridge vibration sensor in the monitoring time window is first obtained. The monitoring time window can be set to a period of time, such as 10 minutes. In the monitoring time window, the main vibration frequency components and their amplitudes are extracted. For example, the vibration sensor may detect multiple vibration frequency components, among which the vibration component with a frequency of 10Hz and an amplitude of 2m / s² can be the main component related to the bridge structure and external loads. Then the correlation coefficient between the bolt displacement change and the vibration amplitude is calculated, and a displacement-vibration coupling relationship model is established. The displacement-vibration coupling relationship model reflects the intrinsic connection between the bolt displacement and the bridge vibration. According to the displacement-vibration coupling relationship model, the original displacement is denoised to generate a corrected displacement that eliminates the environmental vibration interference. Finally, the corrected displacement is input into the state recognition model, the looseness prediction result is recalculated, and the monitoring report is updated. In this way, the operation can more accurately reflect the actual looseness of the bolt and avoid misjudgment caused by interference caused by external loads.

[0085] It can be seen that by triggering the update of model parameters according to the change in bolt displacement during the continuous monitoring cycle, the change in bolt state can be adapted in time. Online fine-tuning of the state recognition model can improve the accuracy of bolt feature extraction. Switching the acquisition mode and adjusting the positioning model parameters when the light is insufficient ensures effective monitoring in different environments. Using the bridge vibration sensor data to correct the looseness prediction results can eliminate external load interference and make the looseness prediction more accurate.

[0086] In actual application, the method further includes the steps of bolt key point detection and matching: Step 301: extracting key points of the head edge of each bolt from the monitoring area coordinate set, and generating a key point coordinate set using a sub-pixel positioning algorithm.

[0087] Step 302: Match the key point coordinate sets in adjacent monitoring periods and calculate the key point displacement vectors of the same bolt at different time points.

[0088] Step 303: Based on the directional consistency analysis of the displacement vector, determine whether the overall displacement trend of the bolt is progressive loosening or instantaneous offset.

[0089] Step 304: When it is detected that the direction difference of the displacement vectors of the plurality of key points exceeds the tolerance range, a recalibration instruction of the image acquisition device is triggered.

[0090] Among them, the implementation of the sub-pixel positioning algorithm includes: based on the pixel-level key point detection results, extracting the grayscale gradient distribution information of the area around the key point; constructing a quadratic surface model to fit the grayscale gradient distribution, and calculating the extreme points of the surface as sub-pixel key point coordinates; adjusting the quadratic surface parameters of the quadratic surface model through an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold; performing bidirectional optical flow matching on the optimized key point coordinates with adjacent image frames, and eliminating key points with a matching degree lower than a second threshold.

[0091] Specifically, the technical solution recorded in steps 301 to 304 includes bolt key point detection and matching steps. First, the key points of the head edge of each bolt are extracted from the monitoring area coordinate set, and a sub-pixel positioning algorithm is used to generate a key point coordinate set. The monitoring area coordinate set determines the approximate position of the bolt in the image, and the key points of the bolt head edge are the key parts that more accurately reflect the state of the bolt. For example, for a bolt image area with a size of 500×500 pixels covered by a monitoring area coordinate set, the key points of the bolt head edge may be concentrated in a 100×100 pixel area near the connection between the head and the nut. The sub-pixel positioning algorithm can improve the accuracy of key point positioning.

[0092] The implementation of the sub-pixel positioning algorithm includes extracting the grayscale gradient distribution information of the area around the key point based on the pixel-level key point detection results. For example, the key point detected at the pixel level is located in a 20×20 pixel area. The grayscale value of this area is different at each pixel point, based on which the grayscale gradient distribution can be determined.

[0093] Furthermore, a quadratic surface model is constructed to fit the grayscale gradient distribution. This quadratic surface model is established based on the grayscale value distribution law of the area around the key point. For example, for an area containing 20×20=400 pixel grayscale values, a quadratic surface model is constructed through a specific algorithm. The extreme points of the surface are calculated as sub-pixel key point coordinates, which are more accurate than pixel-level positioning. For example, pixel-level positioning may determine that the key point is within a certain pixel range, while the sub-pixel extreme point coordinates can be accurate to decimal places, such as 0.1 pixel accuracy.

[0094] Furthermore, the quadratic surface parameters of the quadratic surface model are adjusted through an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold. For example, the preset accuracy threshold is set to 0.05 pixels, and the iterative optimization algorithm will continue to adjust the quadratic surface parameters of the quadratic surface model. After 10-20 iterations, the error between the calculated key point coordinates and the actual position continues to decrease until it is less than 0.05 pixels. The optimized key point coordinates are matched with adjacent image frames through bidirectional optical flow matching. The bidirectional optical flow matching can establish a corresponding relationship between key points between adjacent image frames and eliminate key points with a matching degree lower than a second threshold. For example, the second threshold is set to 0.8. If the matching degree is lower than 0.8, the key point is eliminated to ensure the accuracy and stability of the key points.

[0095] Based on the above steps, the key point coordinate sets in adjacent monitoring cycles can be matched to calculate the key point displacement vectors of the same bolt at different time points. Adjacent monitoring cycles can be set at certain time intervals, for example, every 30 minutes is a monitoring cycle. In different cycles, the state of the bolt may change. By matching the key point coordinate sets, the position change of the same bolt at different time points can be obtained, and then the key point displacement vector can be calculated. For example, in the first monitoring cycle, the coordinates of a key point on the bolt head are (100.5, 200.3). After 30 minutes in the next monitoring cycle, the key point coordinates become (101.2, 200.8), and then the displacement vector with a horizontal displacement of 0.7 pixels and a vertical displacement of 0.5 pixels is calculated. The displacement vector contains the displacement information of the bolt key point in the horizontal and vertical directions or other spatial directions.

[0096] Based on the directional consistency analysis of the displacement vector, determine whether the overall displacement trend of the bolt is progressive loosening or instantaneous offset. If the direction of the displacement vector shows a relatively consistent change trend in multiple continuous monitoring cycles, for example, in 5-6 consecutive monitoring cycles, the direction of the displacement vector is roughly in the same direction and the displacement gradually increases, such as the horizontal displacement gradually increases from 0.5 pixels to 1.5 pixels, and the vertical displacement gradually increases from 0.3 pixels to 1.0 pixels, then it may indicate that the bolt is in a progressive loose state; if the direction of the displacement vector suddenly changes significantly in a short period of time, for example, the horizontal displacement suddenly changes from 0.2 pixels to 2.0 pixels in a monitoring cycle, and then recovers or has no trend of continuous change, it may be an instantaneous offset.

[0097] In actual application, when the displacement vector direction difference of multiple key points is detected to exceed the tolerance range, the recalibration instruction of the image acquisition device is triggered. For example, for a bolt, if the displacement vector direction difference of 4 key points exceeds the set tolerance range, such as the tolerance range is set to 30°, this may mean that there is a deviation in the image acquisition device or the state judgment of the bolt is disturbed. At this time, triggering the recalibration instruction can ensure the accuracy of image acquisition, thereby ensuring the reliability of bolt state monitoring.

[0098] Based on the above embodiments, the sub-pixel positioning algorithm improves the accuracy of bolt key point positioning and ensures the accuracy of subsequent analysis; by matching the coordinates of key points in adjacent monitoring cycles to calculate the displacement vector, the displacement trend of the bolt can be accurately analyzed to distinguish between progressive loosening and instantaneous offset; when the direction difference of the key point displacement vector exceeds the range, the image acquisition device recalibration instruction is triggered to ensure the accuracy of image acquisition, thereby improving the reliability and accuracy of the entire bolt monitoring system and ensuring the accuracy of the bolt status judgment.

[0099] In an exemplary technical solution, the method also includes an interpretable analysis step of abnormal status: embedding a feature importance heat map in the monitoring report to display the image area that the status recognition model focuses on during the prediction process; extracting key feature dimensions that cause abnormal alarm signals, and generating diagnostic suggestion text based on natural language; matching the current abnormal status with similar cases in the historical case library, and displaying the disposal records and effect evaluation data of the matching cases; calculating the probability distribution of bolt status improvement by different maintenance strategies based on a Bayesian network model, and generating a recommended list of optimized maintenance plans.

[0100] Among them, the process of generating the feature importance heat map includes: extracting the feature activation map in the last convolutional layer of the state recognition model, calculating the gradient relationship between the activation value of each channel and the prediction result; generating an initial heat map through a gradient weighted class activation mapping algorithm, marking the image areas with high contribution to abnormal prediction; performing morphological closing operations on the initial heat map to eliminate discrete noise points and smooth the boundaries of important areas; and overlaying the processed heat map with the original monitoring image for transparency to generate a visually displayed feature importance heat map.

[0101] Specifically, the above-mentioned interpretability analysis step of the abnormal state involves a process of providing in-depth analysis and explanation for the abnormal state of the bolt from multiple perspectives.

[0102] First, embed the feature importance heat map in the monitoring report. The purpose of embedding the feature importance heat map is to intuitively display the image area that the state recognition model focuses on during the prediction process. The feature activation map is extracted in the last convolution layer of the state recognition model. This feature activation map contains the activation information of each channel when the model recognizes the bolt state. For example, for a convolution layer with 16 channels, each channel corresponds to a different feature activation. The gradient relationship between the activation value of each channel and the prediction result is calculated, and the gradient relationship reflects the degree of influence of the features of each channel on the final prediction result. The initial heat map is generated by the gradient weighted class activation mapping algorithm. In this initial heat map, the image areas with high contribution to abnormal prediction can be marked. For example, in the feature activation map corresponding to an original monitoring image with a size of 800×600 pixels, after calculation and mapping, the initial heat map may show areas that make important contributions to abnormal prediction, such as the connection area between the bolt head and the nut or the specific texture area on the bolt surface. These areas may be represented by different shades of color to indicate the degree of contribution, and the darker the color, the higher the contribution.

[0103] Next, in order to eliminate discrete noise points and smooth the boundaries of important areas, the initial heat map can be processed by morphological closing operations. In the initial heat map, there may be some discrete noise points caused by calculation errors or data fluctuations, which will interfere with the accurate judgment of important areas. For example, there may be some isolated, lighter-colored pixels in the initial heat map, which do not represent areas that really contribute to anomaly prediction. Through morphological closing operations, these discrete noise points are eliminated, and the boundaries of important areas are made smoother. The processed heat map is overlaid with the original monitoring image for transparency to generate a visually displayed feature importance heat map. This transparency overlay method allows people who view the monitoring report to intuitively see the important areas that the model focuses on based on the original image, making the model's prediction basis more transparent.

[0104] Then, the key feature dimensions that cause abnormal alarm signals are extracted, and diagnostic suggestion text based on natural language is generated. When the state recognition model detects an abnormality and issues an alarm signal, the feature dimensions that play a key role in generating abnormal alarm signals are found by analyzing the internal mechanism and data processing process of the model. For example, these key feature dimensions may include the degree of change in the surface texture of the bolt, the size of the connection gap between the bolt and the nut, etc. Based on these key feature dimensions, a diagnostic suggestion text based on natural language is generated. The diagnostic suggestion text can include a description of the abnormal state and preliminary treatment suggestions for this abnormal state. For example, if it is found that the surface texture of the bolt has changed to a large extent, the diagnostic suggestion text may point out that "the surface texture of the bolt has changed significantly, and there may be corrosion or wear. It is recommended to further check the material integrity of the bolt."

[0105] After that, the current abnormal state is matched with similar cases in the historical case library, and the disposal records and effect evaluation data of the matching cases are displayed. The historical case library stores various bolt abnormal state cases encountered before. Each case contains a detailed description of the abnormal state, the disposal measures taken, and the effect evaluation after the disposal. By matching the current abnormal state with historical cases, similar cases can be found. For example, the abnormal state of the current bolt is manifested as looseness exceeding a certain range and accompanied by slight surface rust. Similar cases may be found in the historical case library, and their disposal records may be tightening operations and rust removal of the rusted parts. The effect evaluation data shows that the bolt state returns to normal within a period of time after the treatment. Displaying the disposal records and effect evaluation data of these matching cases can provide a reference for the handling of the current abnormal state.

[0106] Finally, the probability distribution of different maintenance strategies for improving the bolt status is calculated based on the Bayesian network model, and a recommended list of optimized maintenance plans is generated. The Bayesian network model can calculate the possibility of different maintenance strategies for improving the bolt status based on existing data and knowledge. For example, for the abnormal status of bolts, there may be different maintenance strategies such as tightening the bolts, replacing the bolts, and anti-corrosion treatment of the bolts. The Bayesian network model will comprehensively consider factors such as the current status of the bolts, historical data, and the effects of different maintenance strategies, and calculate the probability distribution of each maintenance strategy for improving the bolt status. Based on this probability distribution, a recommended list of optimized maintenance plans is generated, and maintenance strategies with higher probabilities are ranked first, providing decision-making references for relevant personnel.

[0107] Such a design, by introducing the feature importance heat map, can visualize the prediction basis of the state recognition model and increase the transparency of the model; the diagnostic suggestion text generated based on the key feature dimensions facilitates the understanding of abnormal status and preliminary treatment methods; historical case matching provides a reference for past experience and helps to formulate a treatment plan for the current abnormal status; the recommendation list generated by the maintenance strategy probability distribution calculated by the Bayesian network model provides a basis for bolt maintenance decisions, thereby improving the interpretability of bolt abnormal status analysis and the scientific nature of treatment decisions.

[0108] Based on the above content, in an independent embodiment, the method further includes a multi-sensor data fusion step: Step 401: receiving data from strain sensors deployed at bolt joints, and calculating a bolt preload force variation curve.

[0109] Step 402: synchronously obtain ambient temperature data collected by a temperature sensor and establish a temperature-preload force compensation model.

[0110] In detail, the synchronous acquisition of ambient temperature data collected by the temperature sensor and the establishment of the temperature-preload compensation model include: collecting the ambient temperature data stream continuously generated by the temperature sensor during the monitoring period through a timestamp synchronization mechanism to generate a temperature time series sequence that is time-aligned with the strain sensor data; performing sliding window filtering on the temperature time series sequence to eliminate instantaneous temperature fluctuation noise and retain the trend temperature change component to generate a smoothed temperature trend curve; extracting segmented marking points of the temperature rising stage, stable stage and falling stage from the temperature trend curve, and dividing the temperature time series sequence into multiple temperature time series segments based on the marking points; calculating the mean temperature change rate and the temperature fluctuation frequency in each temperature time series segment to generate a characteristic curve containing temperature change characteristics. A set of eigenvectors; performing time-aligned correlation analysis on the set of eigenvectors and the change amplitude of the bolt preload change curve in the corresponding time window, and determining the linear relationship coefficient between the mean temperature change rate and the preload attenuation; generating a temperature-preload coupling weight matrix based on the linear relationship coefficient, and the temperature-preload coupling weight matrix is ​​used to describe the compensation ratio of the unit temperature change to the preload change; performing reverse compensation calculation on the original preload change curve according to the temperature-preload coupling weight matrix, and generating a modified preload curve that eliminates temperature interference; verifying the correlation between the modified preload curve and the displacement change obtained by image recognition within a continuous monitoring period, and confirming that the temperature-preload compensation model meets the validity condition when the correlation coefficient is greater than the verification threshold.

[0111] More specifically, further, the reverse compensation calculation is performed on the original preload change curve according to the temperature-preload coupling weight matrix to generate a modified preload curve that eliminates temperature interference, including: obtaining the preload measurement value corresponding to each sampling time point in the original preload change curve and the synchronously collected ambient temperature value; based on the preload compensation ratio corresponding to the unit temperature change stored in the temperature-preload coupling weight matrix, calculating the preload influence component caused by the deviation value of the temperature change at each sampling time point relative to the reference temperature; arranging the preload influence components in time series to generate a temperature interference curve consistent with the time dimension of the original preload change curve; performing phase correction on the temperature interference curve to eliminate the interference between the temperature sensor and the strain sensor through a time delay matching algorithm. The synchronized temperature interference component is generated by calculating the data acquisition time difference between the two, and the synchronized temperature interference component is deducted point by point from the original preload change curve to generate an initial corrected preload curve; the smoothness of the initial corrected preload curve is verified, the mutation point introduced by temperature compensation is detected, and the data in the neighborhood of the mutation point is trend-fitted using a local weighted regression algorithm; the abnormal fluctuation segment in the initial corrected preload curve is replaced based on the fitted data to generate a continuous and smooth corrected preload curve; the time window correlation analysis is performed between the corrected preload curve and the bolt displacement change obtained by image recognition, and when the matching degree of the change trend between the corrected preload curve and the bolt displacement change obtained by image recognition exceeds a verification threshold, it is determined that the temperature interference elimination of the corrected preload curve is completed.

[0112] Step 403: Time-align the preload force variation curve after compensation with the displacement variation obtained by image recognition.

[0113] Step 404: When the decrease amplitude of the preload force and the increase amount of the displacement satisfy a linear relationship, the reliability of the looseness prediction result is verified.

[0114] Step 405: If there is a significant conflict between the image recognition result and the sensor data, a manual review process is triggered and the automatic alarm function is suspended.

[0115] When applying the above steps 401-405, first receive the strain sensor data deployed at the bolt connection and calculate the bolt preload change curve. The strain sensor can detect the strain caused by the force at the bolt connection, and calculate the preload change curve based on these data. For example, the strain sensor collects a series of strain data points within a certain period of time, and converts these strain data into corresponding preload data points through a specific mechanical conversion relationship. As time goes by, these data points are connected to form a preload change curve. This curve reflects the change trend of the bolt preload over time. The change of preload may indicate a change in the state of the bolt. For example, a decrease in preload may indicate that the bolt has a tendency to loosen.

[0116] The ambient temperature data collected by the temperature sensor is synchronously acquired to establish a temperature-preload compensation model. The ambient temperature data collected by the temperature sensor has an important influence on the preload of the bolt, because the temperature change will cause the thermal expansion and contraction of the bolt material, which in turn affects the measurement of the preload. The ambient temperature data stream continuously generated by the temperature sensor during the monitoring period is collected through the timestamp synchronization mechanism. For example, if the monitoring period is one day, the temperature sensor collects temperature data every 10 minutes, thereby obtaining a series of temperature data points within a day, and generating a temperature time series sequence that is time-aligned with the strain sensor data. The temperature time series sequence ensures the temporal correspondence between the temperature data and the preload data in subsequent analysis.

[0117] The temperature time series is subjected to sliding window filtering, which aims to eliminate instantaneous temperature fluctuation noise and retain the trend temperature change component, thereby generating a smoothed temperature trend curve. For example, the size of the sliding window can be set to 5 data points, and a smoothed value is calculated within this window through a specific filtering algorithm, and the entire temperature time series is processed in sequence. The segmentation marking points of the temperature rising stage, stable stage and falling stage are extracted from the temperature trend curve, and the temperature time series is divided into multiple temperature time series segments based on these marking points. For example, in a temperature trend curve, it may be found that the temperature gradually rises in the morning, tends to be stable at noon, and gradually decreases in the afternoon. Marking points are set and segments are performed according to the turning points of these stages. The mean temperature change rate and the temperature fluctuation frequency are calculated in each temperature time series segment to generate a feature vector set containing temperature change characteristics. For example, there are 20 data points in a temperature time series segment. The mean temperature change rate is obtained by calculating the temperature change amount in this segment and dividing it by time. At the same time, the fluctuation frequency is obtained by counting the number of temperature fluctuations, and these values ​​are combined into a feature vector.

[0118] The characteristic vector set is time-aligned with the change amplitude of the bolt preload change curve in the corresponding time window to perform correlation analysis, and the linear relationship coefficient between the mean temperature change rate and the preload attenuation is determined. For example, in a specific time window, the mean temperature change rate is a certain value, and the bolt preload has a corresponding attenuation. By analyzing multiple such corresponding relationships, the linear relationship coefficient between the two is found. Based on the linear relationship coefficient, the temperature-preload coupling weight matrix is ​​generated. The temperature-preload coupling weight matrix is ​​used to describe the compensation ratio of the unit temperature change to the preload change. According to the temperature-preload coupling weight matrix, the original preload change curve is reversely compensated to generate a modified preload curve that eliminates temperature interference.

[0119] In the process of performing reverse compensation calculation on the original preload change curve according to the temperature-preload coupling weight matrix to generate a modified preload curve that eliminates temperature interference, first obtain the preload measurement value corresponding to each sampling time point in the original preload change curve and the ambient temperature value collected synchronously. For example, there are 100 sampling time points in the original preload change curve, each time point has a corresponding preload measurement value, and there is also an ambient temperature value collected synchronously with this time point. Based on the preload compensation ratio corresponding to the unit temperature change stored in the temperature-preload coupling weight matrix, the preload influence component caused by the deviation value of the temperature change at each sampling time point relative to the reference temperature is calculated. For example, if the reference temperature is 20°C and the temperature at a certain sampling time point is 25°C, the influence component of the 5°C temperature change on the preload is calculated according to the preload compensation ratio corresponding to the unit temperature change specified in the coupling weight matrix.

[0120] Arrange the preload influence components in time series to generate a temperature interference curve that is consistent with the time dimension of the original preload change curve. The temperature interference curve reflects the interference caused by temperature change on the preload. Perform phase correction on the temperature interference curve, eliminate the data acquisition time difference between the temperature sensor and the strain sensor through the delay matching algorithm, and generate a synchronized temperature interference component (the data acquisition of different sensors may have a time delay, so this operation ensures that the temperature interference component and the original preload change curve are accurately matched in time). Deduct the synchronized temperature interference component point by point from the original preload change curve to generate the initial corrected preload curve. For example, if the preload value of a point in the original preload change curve is 100N, and the corresponding synchronized temperature interference component is 5N, then the initial corrected preload value after deduction is 95N, and so on to obtain the entire initial corrected preload curve.

[0121] The smoothness of the initial corrected preload curve is verified to detect the mutation points introduced by temperature compensation. For example, in the initial corrected preload curve, if the preload values ​​of two adjacent points change too much, it can be a mutation point. The local weighted regression algorithm is used to fit the trend of the data in the neighborhood of the mutation point. For example, the 10 data points around the mutation point are the neighborhood, and a reasonable trend curve is fitted according to the distribution of these data points through the local weighted regression algorithm. The abnormal fluctuation segment in the initial corrected preload curve is replaced based on the fitted data to generate a continuous and smooth corrected preload curve. The time window correlation analysis is performed on the corrected preload curve and the bolt displacement change obtained by image recognition. When the matching degree of the change trend between the corrected preload curve and the bolt displacement change obtained by image recognition exceeds the verification threshold, it is determined that the temperature interference elimination of the corrected preload curve is completed.

[0122] The compensated preload change curve is time-aligned with the displacement change obtained by image recognition. This time alignment is to compare the preload and displacement, two important indicators reflecting the bolt status, on the same time scale. When the decrease in preload and the increase in displacement satisfy a linear relationship, the reliability of the looseness prediction result is verified. For example, if the decrease in preload and the increase in displacement show a stable linear relationship, such as the displacement increases by a certain value every time the preload decreases by a certain value, this indicates that the bolt displacement change obtained by image recognition and the preload change obtained by the strain sensor are mutually confirmed, thereby verifying the reliability of the looseness prediction result.

[0123] If there is a significant conflict between the image recognition result and the sensor data, the manual review process is triggered and the automatic alarm function is suspended. For example, a significant conflict may be caused by an error in image recognition or an abnormality in sensor data. For example, image recognition shows that there is no obvious change in bolt displacement, but the sensor data shows that the preload has dropped significantly. In this case, manual intervention is required to review to determine whether it is a problem with the bridge bolt monitoring image recognition system or a sensor failure. At the same time, the automatic alarm function is suspended to avoid issuing erroneous alarm signals.

[0124] It can be seen that by establishing a temperature-preload compensation model, the interference of temperature on preload measurement is eliminated, so that the preload change curve can more accurately reflect the real state of the bolt. The compensated preload change curve is time-aligned with the displacement change of image recognition and the linear relationship is verified, which enhances the reliability of the prediction results of the bolt looseness. When the image recognition result conflicts with the sensor data, the manual review process is triggered and the automatic alarm function is suspended, which avoids false alarms, improves the accuracy and credibility of the entire bolt monitoring system, and helps to monitor the bolt status more accurately and detect potential problems in time.

[0125] The above content is described below through two application scenario examples.

[0126] Application scenario example 1 The operation and maintenance team of a large cross-sea bridge found in daily monitoring that some high-strength bolts at the main beam connection frequently triggered loosening warnings during the high temperature period in summer, but no obvious displacement was found during manual review. To find out the cause of the false alarm, the technical team deployed a deep learning-based bolt monitoring system and started a multi-sensor data fusion analysis process. During the 72-hour continuous monitoring period, the system simultaneously collected visible light images, infrared thermal imaging, strain sensor and temperature sensor data of the bolt area.

[0127] 8:00-12:00 am (temperature rising stage): The ambient temperature rose uniformly from 20°C to 35°C, and the temperature sensor recorded an increase of 2.5°C per hour. The strain sensor showed that the bolt preload dropped linearly from the initial value of 450kN to 420kN, a decrease of about 6.7%. At the same time, the bridge bolt monitoring image recognition system measured the displacement of the center point of the bolt head to be 0.12mm through the sub-pixel key point matching algorithm, which did not exceed the preset safety threshold (0.15mm). At this time, the temperature-preload compensation model detected that the correlation coefficient between the decrease in preload and the increase in temperature reached 0.92, and it was determined that the change in preload was mainly caused by the thermal expansion effect. The system automatically reversely compensated the original preload curve. The corrected preload curve showed that the actual attenuation was only 5kN (a decrease of 1.1%), which was weakly correlated with the displacement of 0.12mm, so no alarm was triggered.

[0128] 14:00-16:00 p.m. (extremely high temperature stage): The temperature reached a peak of 38°C and entered a stable period. The bridge bolt monitoring image recognition system detected abnormal displacement of bolt No. B-203 at 14:23: the displacement of the key point suddenly increased to 0.18mm (exceeding the threshold by 20%), and the direction of the displacement vector showed an asymmetric distribution. The strain sensor showed that its preload force dropped sharply from 425kN to 395kN, a decrease of 7.1%. The temperature compensation model calculated that the expected preload attenuation caused by the current temperature fluctuation should be 4.2kN, but the measured attenuation reached 30kN, which exceeded the compensation range by more than 6 times. The system immediately launched multi-dimensional verification: 1) Infrared images show that there are local high-temperature spots on the bolt flange contact surface (the difference is up to 8°C), indicating abnormal contact pressure; 2) The vibration sensor detected a 2.5 Hz resonance frequency at this location between 13:50 and 14:20, with an amplitude exceeding the baseline value by 30%; 3) Matching of the historical case library shows that a similar combination of "high temperature + vibration" caused similar bolts to break three years ago.

[0129] Based on the above data, the system raised the risk level of the B-203 bolt to the emergency level, corrected the preload-displacement correlation coefficient to 0.87, generated a monitoring report including a temperature compensation curve comparison chart and a thermal map focus area, and automatically dispatched a drone for high-definition image review.

[0130] Nighttime 20:00-next day 6:00 (temperature drop phase): As the temperature dropped back to 25°C, the system continued to track the status of the B-203 bolt. The corrected preload curve showed that its attenuation was stable at 28kN (compensated value), while the compensated attenuation of adjacent bolts was less than 5kN. The displacement of the key point continued to accumulate to 0.22mm, and the consistency index of the displacement vector direction deteriorated from 0.85 to 0.63, which was consistent with the progressive loosening characteristics. At 3:17 am, the system detected a step-like decrease in the preload of the bolt (attenuation of 12kN in 5 minutes). Although it was temperature compensated, it was still beyond the safety range, eventually triggering a red alarm and generating a maintenance work order.

[0131] Technical verification results: By introducing the temperature-preload compensation model, the system reduced the false alarm rate caused by temperature interference from 32% to 4.7% within 72 hours. For the real failure of the B-203 bolt, the time from the first abnormal detection to the diagnosis was shortened to 47 minutes, which is 8 times more efficient than traditional manual inspections. The maintenance team accurately located the deformation defects of the flange contact surface based on the compensated preload curve and displacement-temperature correlation map provided by the system, avoiding potential structural accidents.

[0132] This case verifies the effectiveness of the multi-sensor data fusion solution: when the preload attenuation after temperature compensation is greater than 15kN and the displacement is greater than 0.15mm, the system diagnosis accuracy reaches 98.6%, while the accuracy rates of image recognition or sensor data alone are 82.3% and 76.8%, respectively. This fusion method significantly improves the monitoring reliability in complex environments and provides a quantitative decision-making basis for bridge bolt maintenance.

[0133] Application scenario example 2 The upper structure of a highway steel-concrete composite box girder bridge adopts steel box girder-concrete bridge deck. The web, bottom plate and stiffening ribs of the node connection between segments are connected by 10.9 grade M24 high-strength bolts. The web connection of the steel box girder contains 208 sets of high-strength bolts, and the bottom plate connection contains 248 sets of high-strength bolts.

[0134] The smart switch is controlled by a preset program, so that photos are taken at regular intervals to obtain image data on the bolt panel. The intelligent algorithm installed on the rack server then identifies the current angles of all the bolts on the panel. By comparing the bolt angle data in different time periods, the loosening status and trend of the bolts are obtained, and early warning and alarm information are issued in case of abnormalities.

[0135] Depend on Figure 2 It can be seen that the changes in the bolt angles at each position are highly consistent and basically distributed in a straight line. This shows that when the deep learning method is used to install cameras on site for real-time bolt loosening monitoring, the algorithm has strong stability and strong resistance to interference in complex environments. The bolt loosening detection results are very reliable and have good practicality.

[0136] In summary, the embodiments of the present invention ensure the data quality of subsequent analysis and reduce analysis errors caused by poor image quality by acquiring a set of monitoring images from multiple angles and preprocessing them into a standardized target image sequence. By using a pre-trained positioning model to locate the bolt monitoring area, the bolt position information can be accurately obtained, which is helpful for accurate analysis. By introducing a state recognition model, the degree of looseness can be predicted based on the structural state characteristics to achieve an effective assessment of the state of the bolt component. In addition, according to the matching of the degree of looseness with the safety threshold, outputting an alarm signal can promptly detect abnormal bolt status, thereby ensuring the safety of the bridge. Finally, associating the alarm signal with the image frame to generate a monitoring report with visual markings can ensure that relevant personnel can intuitively and quickly understand the abnormal state of the bolt, thereby improving monitoring efficiency and accuracy.

[0137] Further, Figure 3 Schematic diagram of a bridge bolt monitoring image recognition system 200 provided in an embodiment of the present invention. Figure 3 The bridge bolt monitoring image recognition system 200 shown includes a processor 210, and the processor 210 can call and run a computer program from a memory to implement the method in the embodiment of the present invention.

[0138] Alternatively, if Figure 3 As shown, the bridge bolt monitoring image recognition system 200 may further include a memory 230. The processor 210 may call and run a computer program from the memory 230 to implement the method in the embodiment of the present invention.

[0139] The memory 230 may be a separate device independent of the processor 210 , or may be integrated into the processor 210 .

[0140] Alternatively, if Figure 3As shown, the bridge bolt monitoring image recognition system 200 may also include a transceiver 220, and the processor 210 may control the transceiver 220 to interact with other devices, specifically, may send information or data to other devices, or receive information or data sent by other devices.

[0141] Optionally, the bridge bolt monitoring image recognition system 200 can implement the corresponding processes corresponding to the storage engine or components in the storage engine (such as a processing module) or a device deployed with a storage engine in each method of the embodiments of the present invention. For the sake of brevity, they will not be repeated here.

[0142] It should be understood that the processor of the embodiment of the present invention may be an integrated circuit chip with signal processing capability.

[0143] It is understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. It should be noted that the memory of the systems and methods described herein is intended to include but is not limited to suitable types of memory.

[0144] Based on the above, a readable storage medium is provided, on which a program or instruction is stored, and the program or instruction implements the steps of the above method when executed by a processor.

[0145] It should be noted that, in this article, the term "comprises", "includes" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned implementation methods can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the embodiment of the present invention.

[0147] The embodiments of the embodiments of the present invention are described above in conjunction with the accompanying drawings, but the embodiments of the present invention are not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the embodiments of the present invention, ordinary technicians in this field can also make many forms without departing from the purpose of the embodiments of the present invention and the scope of protection of the embodiments of the present invention, all of which are within the protection of the embodiments of the present invention.

Claims

1. A bridge bolt monitoring image recognition method based on deep learning, characterized in that: The method comprises: Acquire a multi-angle monitoring image set of a bridge bolt component through an image acquisition device, and pre-process the monitoring image set to generate a standardized target image sequence; Based on the pre-trained positioning model, each image frame in the target image sequence is positioned in the bolt monitoring area to generate a regional coordinate set containing bolt position information; Inputting the regional coordinate set into the state recognition model, extracting the structural state characteristics of each bolt monitoring area, and generating a looseness prediction result of the bolt component according to the structural state characteristics; Outputting a bolt state abnormality alarm signal according to a matching result between the looseness degree prediction result and a preset safety threshold range; The abnormal bolt status alarm signal is associated with the corresponding monitoring image frame to generate a monitoring report containing visual marks.

2. The method according to claim 1, characterized in that The preprocessing of the monitoring image set to generate a standardized target image sequence includes: Performing illumination equalization processing on each original image in the monitoring image set, and eliminating brightness deviation in shadow areas by using an adaptive histogram equalization algorithm; Extract the noise distribution characteristics in the processed image, suppress the high-frequency noise based on the multi-scale filtering algorithm, and generate a set of intermediate images after noise reduction; Performing geometric correction on the intermediate image set, and adjusting the image tilt angle by using an affine transformation algorithm, so that the bolt component directions of all image frames in the intermediate image set are aligned with a preset standard direction; The rectified image frames are processed to unify the resolution and the non-monitored areas are cropped to generate a target image sequence with consistent size.

3. The method according to claim 1, characterized in that The positioning of the bolt monitoring area for the target image sequence based on the pre-trained positioning model includes: Inputting the target image sequence into the convolution layer of the positioning model, extracting multi-scale spatial features in the target image sequence, and generating a feature map containing bolt profile information; The feature map is screened for candidate regions through a region proposal network, and a matching score between each candidate region and a preset bolt template is calculated; Performing non-maximum suppression processing on the candidate regions according to the matching scores, and retaining regions with confidence levels higher than a first threshold as initial positioning frames; The initial positioning frame is subjected to edge refinement correction, and the size of the positioning frame is adjusted using a bilinear interpolation algorithm to generate a monitoring area coordinate set containing corrected coordinates.

4. The method according to claim 3, characterized in that The process of extracting structural state features by the state recognition model includes: Extracting a local image block of each bolt monitoring area from the monitoring area coordinate set, and inputting the image block into a feature extraction module of the state recognition model; Performing hierarchical feature fusion on the image blocks through a multi-layer residual network to generate a composite feature vector including the bolt surface texture, rust degree and connection gap; The composite feature vector is input into the time series analysis module, and the cumulative displacement of the current bolt is calculated in combination with the bolt displacement change trend in the historical monitoring data; Based on the comparison result between the cumulative displacement and the preset displacement threshold, the looseness degree prediction result is generated, wherein the portion of the cumulative displacement exceeding the displacement threshold is marked as an abnormal displacement interval; The output bolt state abnormal alarm signal includes: Determine the potential risk level of the bolt according to the abnormal displacement interval in the looseness prediction result, wherein the risk level includes the first level, the second level and the emergency level; If the risk level is the first level, generating a primary warning signal including a recommended inspection time, and associating the signal with a corresponding monitoring image frame; If the risk level is the second level, a medium-level alarm signal including a mandatory maintenance prompt is generated, and a real-time monitoring frequency increase instruction is triggered; If the risk level is an emergency level, an immediate maintenance alarm signal is generated, and an emergency notification message including location coordinates is sent to the remote management terminal.

5. The method according to claim 1, characterized in that The generating of the monitoring report containing the visual mark includes: Marking the monitoring area coordinates of abnormal bolts in the target image sequence, and using color coding to distinguish the marking boxes corresponding to different risk levels; Convert the looseness prediction result into a visual progress bar in percentage form, and display it in a superimposed manner on the side area of ​​the corresponding mark box; Extract the displacement change curve of the same bolt in the historical monitoring report, generate a comparative analysis chart and embed it into the appendix of the monitoring report; A time-ordered list of abnormal events is generated according to the timestamp information of the monitoring image frame, and the list is dynamically associated with the visual mark.

6. The method according to claim 1, characterized in that The method further comprises the step of jointly optimizing the positioning model and the state recognition model: Collect normal state samples and abnormal state samples of bolts in historical monitoring images and construct a training dataset containing multi-category labels; Initializing the convolutional layer parameters of the positioning model through a transfer learning algorithm, and optimizing the regional positioning accuracy using a focal loss function; Introducing an attention mechanism into the state recognition model to dynamically adjust the weight distribution of the key areas of the bolt during the feature extraction process; Use a generative adversarial network to enhance training data and generate synthetic images with different lighting conditions and angle changes to improve the model's generalization ability; The specific process of introducing the attention mechanism includes: Add a channel attention submodule after the last convolutional layer of the feature extraction module of the state recognition model to calculate the global average pooling value of each feature channel; Performing a nonlinear transformation on the pooled value through a fully connected layer, generating a channel weight vector and performing channel weighting on the original feature map; A self-attention mechanism is used in the spatial dimension to calculate the correlation matrix between each position in the feature map and other positions; The feature map is spatially recalibrated according to the correlation matrix to highlight the detailed feature areas of the bolted joint.

7. The method according to claim 1, characterized in that The method also includes a dynamic adjustment step during real-time monitoring: The displacement change of the same bolt is counted during the continuous monitoring period, and when the displacement exceeds the dynamic threshold, the model parameter update instruction is triggered; Based on the latest collected monitoring image data, the state recognition model is fine-tuned online, and the feature extraction weights are updated using a sliding window algorithm; When it is detected that the ambient light intensity is continuously lower than the preset value, it automatically switches to the infrared image acquisition mode and adjusts the input parameters of the positioning model; Dynamically correct the looseness prediction result according to the bridge vibration sensor data to eliminate the instantaneous displacement interference caused by external loads; The process of dynamically correcting the looseness prediction result includes: Obtain the acceleration spectrum data of the bridge vibration sensor within the monitoring time window, and extract the main vibration frequency components and their amplitudes; Calculate the correlation coefficient between the bolt displacement change and the vibration amplitude, and establish the displacement-vibration coupling relationship model; De-noising the original displacement according to the coupling relationship model to generate a corrected displacement that eliminates environmental vibration interference; The corrected displacement is input into the state recognition model, the looseness degree prediction result is recalculated and the monitoring report is updated.

8. The method according to claim 1, characterized in that The method also includes the steps of bolt key point detection and matching: Extracting the key points of the head edge of each bolt from the monitoring area coordinate set, and generating a key point coordinate set using a sub-pixel positioning algorithm; Match the key point coordinate sets in adjacent monitoring cycles and calculate the key point displacement vectors of the same bolt at different time points; Based on the directional consistency analysis of the displacement vector, determining whether the overall displacement trend of the bolt is progressive loosening or instantaneous displacement; When it is detected that the difference in the displacement vector directions of multiple key points exceeds the tolerance range, a recalibration instruction of the image acquisition device is triggered; The implementation of the sub-pixel positioning algorithm includes: Based on the pixel-level key point detection results, the grayscale gradient distribution information of the area around the key point is extracted; Constructing a quadratic surface model to fit the grayscale gradient distribution, and calculating the extreme points of the surface as sub-pixel key point coordinates; Adjusting the quadratic surface parameters of the quadratic surface model by an iterative optimization algorithm until the convergence error of the key point coordinates is less than a preset accuracy threshold; The optimized key point coordinates are matched with adjacent image frames in a bidirectional optical flow manner, and key points with a matching degree below a second threshold are eliminated.

9. The method according to claim 1, characterized in that: The method also includes an interpretability analysis step of the abnormal state: embedding a feature importance heat map in the monitoring report, showing the image regions that the state recognition model focuses on during the prediction process; Extract key feature dimensions that lead to abnormal alarm signals and generate diagnostic suggestion text based on natural language; Match the current abnormal state with similar cases in the historical case library, and display the disposal records and effect evaluation data of the matching cases; Based on the Bayesian network model, the probability distribution of bolt condition improvement due to different maintenance strategies is calculated, and a recommended list of optimized maintenance plans is generated; The process of generating a feature importance heat map includes: Extracting a feature activation map in the last convolutional layer of the state recognition model, and calculating the gradient relationship between the activation value of each channel and the prediction result; Generate an initial heat map through the gradient weighted class activation mapping algorithm to mark the image areas that contribute most to anomaly prediction; Performing morphological closing operation on the initial heat map to eliminate discrete noise points and smooth important area boundaries; The processed heat map is overlaid with the original monitoring image with transparency to generate a heat map of feature importance for visual display.

10. A bridge bolt monitoring image recognition system, characterized in that: The method comprises at least one processor and a memory; the memory stores computer-executable instructions; the at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the method according to any one of claims 1 to 9.

Images