Deflection monitoring method for large-span steel structures based on machine vision and Mean Shift clustering algorithm

Abstract

The present invention proposes a method for monitoring the deflection of large-span steel structures based on machine vision and Mean Shift clustering algorithms. At the recognition method level, a deep learning framework is constructed, which breaks through the limitations of the low resolution of traditional black-and-white industrial cameras, greatly broadens the deployment range of the monitoring system, and realizes efficient automatic recognition and precise target matching of speckle features in digital images. Furthermore, an automated denoising framework based on the Mean Shift clustering algorithm is developed, focusing on in-depth processing and analysis of the collected displacement data to ensure the accuracy of the calculation of the average deflection value. Compared with traditional contact monitoring methods, the present invention can effectively implement non-contact deflection monitoring of multiple key nodes of large-span steel structures, and can safely, quickly and accurately monitor the deflection changes of large-span steel structures during the construction process or operation stage, providing a solid data foundation for structural safety assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of deflection monitoring during the construction or operation phase of a large-span steel structure, and specifically to a large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm. Background Art

[0002] During the construction and operation of large-span steel structures, accurate monitoring of deformation states plays a crucial role. It not only affects the rationality of construction strategies and the assurance of project quality, but also directly impacts the assessment of structural safety and the timeliness of damage identification. However, current measurement technologies face many challenges in practical applications. Contact measurement methods, such as dial indicators, micrometers, wire displacement gauges, and fiber Bragg grating sensors, while offering the advantage of high precision, are limited by their limited measurement range, cumbersome installation and maintenance, and complex and changing engineering environments. In contrast, non-contact measurement technologies, such as GPS and laser Doppler vibrometers, offer the convenience of remote monitoring, but GPS lacks vertical accuracy, while laser Doppler vibrometers impose stringent requirements on on-site vibration isolation conditions. These factors restrict their effectiveness in practical applications.

[0003] Digital image correlation (DIC) is gaining popularity in the field of visual measurement. This technique uses a digital camera to capture images of a structure before and after deformation and uses algorithms to analyze correlation changes within subregions within the image, thereby accurately tracking the deformation trajectory. However, when applying DIC to monitoring large-span outdoor steel structures, the following difficulties need to be addressed:

[0004] (1) Performance vs. Resolution Tradeoff: In complex construction environments, industrial cameras face a difficult trade-off between image resolution, frame rate, and storage capacity. While high resolution can enhance detail capture, it often comes with a decrease in frame rate and an increase in memory load, limiting the efficiency and response speed of real-time monitoring.

[0005] (2) Feature matching challenges: Traditional matching strategies based on local feature descriptors (such as SIFT and SURF) are highly sensitive to environmental factors such as illumination changes, rotation, and scaling, and are easily disturbed by noise and occlusion, resulting in matching failures or frequent mismatches. In addition, the manual selection of matching areas in the DIC method also limits the automation and efficiency of monitoring.

[0006] (3) Dealing with noise interference: Noise introduced by external environmental factors such as atmospheric disturbances, thermal fog, and camera self-heating affects the accuracy of subsequent analysis. Currently, most methods rely solely on simple filtering, which is not effective enough to effectively suppress noise interference. Summary of the Invention

[0007] In response to the above technical problems, the present invention proposes a large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm. This monitoring method utilizes an integrated non-contact monitoring system to effectively implement non-contact deflection monitoring of multiple key nodes of large-span steel structures. It can safely, quickly and accurately monitor the deflection changes of large-span steel structures during the construction process or operation stage, providing a solid data foundation for structural safety assessment.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm comprises the following steps:

[0010] S1. Obtain the motion trajectory of the key nodes of the large-span steel structure;

[0011] S2. Build a deep learning framework for large-span steel structure deflection monitoring based on machine vision;

[0012] S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the scale factor corresponding to each key node in the image;

[0013] S22. Based on the scale factor of each key node obtained in step S21, select an upsampling ratio of the ESRGAN super-resolution reconstruction algorithm to obtain a scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel;

[0014] S23. Apply the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading, pass the target box coordinate position information output by the YOLO v5 detection algorithm to the LoFTR algorithm, use the LoFTR algorithm to calculate the image pixel displacement corresponding to each key node on the large-span steel structure, and combine the scale factor of the reconstructed image obtained in step S22 to accurately convert the image displacement of each key node into a physical displacement, and finally intuitively display the measurement results, i.e., structured data;

[0015] S3: Input the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, which specifically includes the following sub-steps:

[0016] S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region;

[0017] S32 divides the entire stable region into multiple data segments and sorts them by time series, using the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering;

[0018] S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment.

[0019] diff i =d i -d i+1 (1)

[0020]

[0021] Where, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections

[0022] window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain;

[0023] Indicates the average deflection value in the time domain at this point.

[0024] In step S22, the ESRGAN super-resolution reconstruction algorithm is first trained on a large-span steel structure and speckle dataset containing high-resolution and low-resolution black-and-white image pairs to ensure that the ESRGAN super-resolution reconstruction algorithm accurately maps low-resolution to high-resolution images.

[0025] The super-resolution images generated by the test set are evaluated by the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM), and are adjusted to maximize the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to achieve the best performance.

[0026] In step S23, the YOLOv5 detection algorithm first constructs a black and white image dataset with speckle annotations for training to achieve accurate speckle detection and positioning. By optimizing the YOLOv5 detection algorithm parameters and hyperparameters, the detection accuracy and overall performance are improved.

[0027] In step S23, the LoFTR algorithm relies on a large-span steel structure image set containing displacement information for training. The YOLOv5 detection algorithm outputs the target frame coordinates to the LoFTR algorithm, which only performs local matching on the pixels within the frame.

[0028] Step S1 specifically includes:

[0029] Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site leveling site to key locations on the long-span steel structure. Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. The pre-prepared digital image speckle matte paper was then affixed to the key nodes of the long-span steel structure.

[0030] The black and white industrial camera is used to completely record the motion trajectory of the digital image speckle on the digital image speckle matte paper.

[0031] Step S21 specifically includes:

[0032] The ultrasonic rangefinder is used to measure the distance from the center of each digital image speckle matte paper on the large-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, a computer is used to identify the pixel coordinates of these key nodes in the black-and-white industrial camera view, and based on the geometric relationship, the scale factor corresponding to each key node is calculated.

[0033] Step S22 specifically includes:

[0034] A black-and-white industrial camera is used to capture a series of images containing key locations of the long-span steel structure during the construction or operation phase. One image before loading is designated as a reference image. Next, the long-span steel structure is used as the core area of ​​study and the image is cropped to retain only the main part of the long-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel.

[0035] The image cropping operation is specifically for all images captured by black and white industrial cameras, focusing on the square pixel area occupied by large-span steel structure beams in the image, that is, the area is defined by four coordinate points: the top, bottom, left, and right. The union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

[0036] Beneficial effects:

[0037] First, in terms of recognition technology, the present invention proposes a deep learning framework that breaks through the low-pixel limitations of traditional black-and-white industrial cameras, greatly expands the deployment range of the monitoring system, and realizes automatic recognition and target matching of digital image speckles.

[0038] Second, the present invention developed an automatic denoising framework based on the Mean Shift clustering algorithm to process and analyze the collected displacement data. This framework can effectively filter out abnormal data and accurately calculate the average deflection value, providing a solid data foundation for structural safety assessment.

[0039] Third, compared with traditional contact monitoring methods, the present invention can effectively implement non-contact deflection monitoring of multiple key nodes of large-span steel structures, and can safely, quickly and accurately monitor the deflection changes during the construction process or operation stage of large-span steel structures, providing a solid data basis for structural safety assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 Schematic diagram of the structure of the large-span steel structure deflection monitoring system of the present invention;

[0041] Figure 2 This is a flow chart of the method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm of the present invention;

[0042] Figure 3 This is a flowchart of the automatic denoising framework based on the three-stage Mean Shift clustering algorithm of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings. However, the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.

[0045] For example, please refer to the attached Figure 1 A long-span steel structure deflection monitoring system based on machine vision and Mean Shift clustering ， The apparatus mainly comprises a digital image speckle matte paper 1, a heavy-duty tripod 2, a black-and-white industrial camera 3, a fixed-focus lens 4, a computer 5, and an ultrasonic rangefinder 6. The black-and-white industrial camera 3 and the fixed-focus lens 4 are fixed to the heavy-duty tripod 2 to capture the dynamic motion trajectory of the digital image speckle matte paper 1. The computer 5 is connected to the black-and-white industrial camera 3 and controls the camera to capture the deflection changes at key nodes during the construction or operation phase of a large-span steel structure.

[0046] The manufacturing method of the digital image speckle matte paper 1 is to print random digital image speckles on matte paper with a matte film on the front and an adhesive on the back. The size of the matte paper can be A3 or A4.

[0047] The digital image speckle matte paper 1 is firmly attached to the key node position of the large-span steel structure and can represent its deflection.

[0048] The black and white industrial camera 3 and the fixed focus lens 4 are fixed to the ground via a heavy tripod 2 equipped with a pan / tilt head.

[0049] The heavy-duty tripod 2 uses a Manfrotto 475B tripod and a 400 gear head.

[0050] The black and white industrial camera 3 is a FLIRGS3-U3-41C6M-C black and white industrial camera.

[0051] The fixed-focus lens 4 is a KOWALM16HC lens.

[0052] The computer 5 is configured with CPU AMD Ryzen 5 5600X, 32G RAM, 512G memory, and graphics card NVIDIA GeForce RTX 3060.

[0053] The ultrasonic rangefinder 6 is a Xima AR841 handheld ultrasonic rangefinder.

[0054] A method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm, the method comprising the following steps:

[0055] S1. Obtain the motion trajectory of the key nodes of the large-span steel structure;

[0056] Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site leveling to key locations on the long-span steel structure with an accuracy of ±(1cm + 0.5% × distance). Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. The pre-prepared digital image speckle matte paper was then affixed to the key nodes of the long-span steel structure.

[0057] Using the black and white industrial camera to completely record the motion trajectory of the digital image speckle on the digital image speckle matte paper;

[0058] S2. Build a deep learning framework for large-span steel structure deflection monitoring based on machine vision;

[0059] S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the scale factor corresponding to each key node in the image, specifically:

[0060] The ultrasonic rangefinder is used to accurately measure the distance from the center of each digital image speckle matte paper on the long-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, a computer is used to identify the pixel coordinates of these key nodes in the camera view and calculate the corresponding scale factor of each key node based on the geometric relationship.

[0061] S22 selects an upsampling ratio of the ESRGAN super-resolution reconstruction algorithm based on the scale factor of each key node obtained in step S21 to obtain a scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel. Specifically,

[0062] A series of images containing key locations of the long-span steel structure are captured using a black-and-white industrial camera during the construction or operation phase. One image before loading is designated as a reference image. The long-span steel structure is then used as the core area of ​​study, and the image is cropped to retain only the main portion of the long-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel.

[0063] S23 uses the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading. The target box coordinate position information output by the YOLO v5 detection algorithm is passed to the LoFTR algorithm. The LoFTR algorithm is used to calculate the image pixel displacement corresponding to each key node on the long-span steel structure. Combined with the scale factor of the reconstructed image obtained in the previous step, the image displacement of each key node is accurately converted into physical displacement. Finally, the measurement results are intuitively displayed as structured data.

[0064] S3: Input the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, which specifically includes the following sub-steps:

[0065] S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region;

[0066] S32 divides the entire stable region into multiple data segments and sorts them by time series, using the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering;

[0067] S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment.

[0068] diff i =d i -d i+1 (1)

[0069]

[0070] Where, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections

[0071] window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain;

[0072] Indicates the average deflection value in the time domain at this point.

[0073] As a further preference of the present invention, in step S22, the image cropping operation is specifically for all images captured by the black and white industrial camera, focusing on the square pixel area occupied by the large-span steel structure beam in the image, that is, the area is defined by four coordinate points of the top, bottom, left and right, and the union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

[0074] As a further preferred embodiment of the present invention, the implementation of the ESRGAN super-resolution reconstruction algorithm primarily relies on a large-span steel structure and digital image speckle dataset containing high-resolution and low-resolution black-and-white image pairs. Subsequently, this large-span steel structure and digital image speckle dataset is used to train the ESRGAN super-resolution reconstruction algorithm to ensure that it can accurately grasp the complex mapping relationship from low-resolution images to their high-resolution counterparts. In order to comprehensively evaluate the algorithm's performance in super-resolution reconstruction, the ESRGAN super-resolution reconstruction algorithm is fine-tuned and optimized by calculating the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) of the super-resolution images generated by the test dataset. The larger the peak signal-to-noise ratio and structural similarity index, the better, to ensure its optimal performance.

[0075] As a further preferred embodiment of the present invention, the YOLOv5 detection algorithm requires pre-collection of a black-and-white image dataset containing annotated information on digital image speckle. Subsequently, the YOLOv5 detection algorithm is trained based on the annotated dataset to achieve accurate detection and location of digital image speckle in the image. By fine-tuning parameters and hyperparameters, the detection accuracy and overall performance are further optimized.

[0076] As a further preferred feature of the present invention, the LoFTR algorithm relies on a pre-collected dataset of black-and-white images of large-span steel structures containing known displacement information. Subsequently, the LoFTR algorithm is trained using image sequences containing these displacement changes. In practice, the target bounding box coordinate position information output by the YOLOv5 detection algorithm is passed to the LoFTR algorithm, which then performs local matching calculations only on pixels within these target bounding boxes.

[0077] The foregoing is merely an embodiment of the present invention. Although the present invention is described with reference to several embodiments, it is not intended to be limited to these embodiments. It will be apparent to those skilled in the art that modifications may be made to the present invention without departing from the principles of the present invention, and all such modifications are intended to fall within the scope of the claims.

Claims

1. A method for monitoring the deflection of a large-span steel structure based on machine vision and MeanShift clustering algorithm, characterized in that: The following steps are involved: S1. Obtain the motion trajectory of the key nodes of the large-span steel structure; S2. Build a deep learning framework for large-span steel structure deflection monitoring based on machine vision; S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the scale factor corresponding to each key node in the image; S22. Based on the scale factor of each key node obtained in step S21, select an upsampling ratio of the ESRGAN super-resolution reconstruction algorithm to obtain a scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel; S23. Apply the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading, pass the target box coordinate position information output by the YOLO v5 detection algorithm to the LoFTR algorithm, use the LoFTR algorithm to calculate the image pixel displacement corresponding to each key node on the large-span steel structure, and combine the scale factor of the reconstructed image obtained in step S22 to accurately convert the image displacement of each key node into a physical displacement, and finally intuitively display the measurement results, i.e., structured data; S3: Input the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, which specifically includes the following sub-steps: S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region; S32 divides the entire stable region into multiple data segments and sorts them by time series, using the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering; S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment. diff i =d i -d i+1 (1) Where, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain; Indicates the average deflection value in the time domain at this point.

2. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: In step S22, the ESRGAN super-resolution reconstruction algorithm is first trained on a large-span steel structure and speckle dataset containing high-resolution and low-resolution black-and-white image pairs to ensure that the ESRGAN super-resolution reconstruction algorithm accurately maps low-resolution to high-resolution images. The super-resolution images generated by the test set are evaluated by the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM), and are adjusted to maximize the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to achieve the best performance.

3. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: In step S23, the YOLO v5 detection algorithm first constructs a black and white image dataset with speckle annotations for training to achieve accurate speckle detection and positioning. By optimizing the YOLO v5 detection algorithm parameters and hyperparameters, the detection accuracy and overall performance are improved.

4. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1, characterized in that: In step S23, the LoFTR algorithm relies on a large-span steel structure image set containing displacement information for training. The YOLOv5 detection algorithm outputs the target frame coordinates to the LoFTR algorithm, which only performs local matching on the pixels within the frame.

5. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: Step S1 specifically includes: Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site leveling site to key locations on the long-span steel structure. Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. The pre-prepared digital image speckle matte paper was then affixed to the key nodes of the long-span steel structure. The black and white industrial camera is used to completely record the motion trajectory of the digital image speckle on the digital image speckle matte paper.

6. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 5 is characterized in that: Step S21 specifically includes: The ultrasonic rangefinder is used to measure the distance from the center of each digital image speckle matte paper on the large-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, a computer is used to identify the pixel coordinates of these key nodes in the black-and-white industrial camera view, and based on the geometric relationship, the scale factor corresponding to each key node is calculated.

7. The method for monitoring deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 5, characterized in that: Step S22 specifically includes: A black-and-white industrial camera is used to capture a series of images containing key locations of the long-span steel structure during the construction or operation phase. One image before loading is designated as a reference image. Next, the long-span steel structure is used as the core area of ​​study and the image is cropped to retain only the main part of the long-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image maintains an accuracy level of 0.1 to 10 mm / pixel.

8. The method for monitoring deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 7 is characterized in that: The image cropping operation is specifically for all images captured by black and white industrial cameras, focusing on the square pixel area occupied by large-span steel structure beams in the image, that is, the area is defined by four coordinate points: the top, bottom, left, and right. The union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

Images