Large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm

Abstract

The invention provides a large-span steel structure deflection monitoring method based on machine vision and a Mean Shift clustering algorithm. On the identification method level, a deep learning framework is constructed, the limitation of low resolution of a traditional black and white industrial camera is broken through, the deployment range of a monitoring system is greatly widened, and efficient automatic identification and accurate target matching of speckle features in digital images are achieved. Furthermore, an automatic denoising framework based on a Mean Shift clustering algorithm is developed, deep processing and analysis are focused on the collected displacement data, and the accuracy of calculation of the deflection average value is ensured. Compared with a traditional contact type monitoring means, non-contact type deflection monitoring of a plurality of key nodes of the large-span steel structure can be effectively implemented, deflection changes of the large-span steel structure in the construction process or the operation stage can be safely, rapidly and accurately monitored, and a solid data foundation is provided for structural safety evaluation.

Description

Technical Field

[0001] The invention relates to the technical field of deflection monitoring during the construction process or operation stage of a large-span steel structure, and specifically to a large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm. Background Art

[0002] In the construction and operation of large-span steel structures, accurate monitoring of deformation states plays a vital role. It is not only related to the rationality of construction strategies and the guarantee of engineering quality, but also directly affects the assessment of structural safety and the timeliness of damage identification. However, current measurement technologies face many challenges in practical applications. Contact measurement methods, such as dial indicators, micrometers, wire displacement meters, and fiber grating sensors, have the advantage of high precision, but are limited by limited measurement range, cumbersome installation and maintenance, and complex and changeable engineering environments. In contrast, non-contact measurement technologies such as GPS and laser Doppler vibrometers, although they realize the convenience of remote monitoring, GPS has insufficient vertical accuracy, while laser Doppler vibrometers have strict requirements on on-site vibration isolation conditions, all of which restrict their effectiveness in practical applications.

[0003] Currently, the digital image correlation method (DIC) in the field of visual measurement is gradually gaining popularity. This technology uses a digital camera to capture images of the structure before and after deformation, and uses an algorithm to analyze the correlation changes of sub-areas in the image to accurately track the deformation trajectory. However, when it is applied to the monitoring of outdoor large-span steel structures, the following difficulties need to be solved:

[0004] (1) Performance and resolution trade-off: In complex construction environments, industrial cameras need to make a difficult choice between image resolution, frame rate, and storage capacity. Although high resolution can enhance the ability to capture details, it is often accompanied by a decrease in frame rate and an increase in memory load, which in turn limits the efficiency and response speed of real-time monitoring.

[0005] (2) Feature matching challenges: Traditional matching strategies based on local feature descriptors (such as SIFT and SURF) are highly sensitive to environmental factors such as illumination changes, rotation, and scaling, and are easily disturbed by noise and occlusion, resulting in matching failures or frequent mismatches. In addition, the practice of manually selecting matching areas in the DIC method also limits the automation and efficiency of monitoring.

[0006] (3) Dealing with noise interference: Noise introduced by external environmental factors such as atmospheric disturbances, thermal fog, and camera self-heating affects the accuracy of subsequent analysis. Currently, most methods rely only on simple filtering, which is not effective enough to effectively suppress noise interference. Summary of the invention

[0007] In view of the above technical problems, the present invention proposes a large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm. The monitoring method utilizes an integrated non-contact monitoring system, which can effectively implement non-contact deflection monitoring of multiple key nodes of large-span steel structures. It can safely, quickly and accurately monitor the deflection changes of large-span steel structures during the construction process or operation stage, and provide a solid data foundation for structural safety assessment.

[0008] The technical solution adopted by the present invention to solve the above technical problems is:

[0009] A method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm comprises the following steps:

[0010] S1. Obtain the motion trajectory of the key nodes of the large-span steel structure;

[0011] S2. Construct a deep learning framework for large-span steel structure deflection monitoring based on machine vision;

[0012] S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the proportional factor corresponding to each key node in the image;

[0013] S22, based on the scale factor of each key node obtained in step S21, selecting the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm, obtaining the scale factor of the reconstructed image, and ensuring that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel;

[0014] S23, using the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading, passing the target frame coordinate position information output by the YOLO v5 detection algorithm to the LoFTR algorithm, using the LoFTR algorithm to calculate the image pixel displacement corresponding to each key node on the large-span steel structure, combining the scale factor of the reconstructed image obtained in step S22 to accurately convert the image displacement of each key node into a physical displacement, and finally intuitively displaying the measurement results, i.e., structured data;

[0015] S3, inputting the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, specifically including the following sub-steps:

[0016] S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region;

[0017] S32 divides the entire stable region into multiple data segments and sorts them by time series, and uses the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering;

[0018] S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment.

[0019] diff i =d i -d i+1 (1)

[0020]

[0021] In the formula, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections of

[0022] window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain;

[0023] Indicates the average deflection value in the time domain at which the point is located.

[0024] In step S22, the ESRGAN super-resolution reconstruction algorithm is first trained on a large-span steel structure and speckle dataset containing high / low resolution black-and-white image pairs, to ensure that the ESRGAN super-resolution reconstruction algorithm accurately maps low to high resolution images;

[0025] The super-resolution images generated by the test set are evaluated by the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM), and are adjusted to maximize the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to achieve the best performance.

[0026] In step S23, the YOLOv5 detection algorithm first constructs a black-and-white image dataset containing speckle annotations for training to achieve accurate speckle detection and positioning, and improves detection accuracy and overall performance by optimizing the YOLOv5 detection algorithm parameters and hyperparameters.

[0027] In step S23, the LoFTR algorithm relies on the training of a large-span steel structure image set containing displacement information. The YOLOv5 detection algorithm outputs the target frame coordinates to the LoFTR algorithm, which only performs local matching on the pixels within the frame.

[0028] Step S1 specifically includes:

[0029] Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site to the key positions of the large-span steel structure. Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. Then, the pre-prepared digital image speckle matte paper was attached to the key nodes of the large-span steel structure.

[0030] The black-and-white industrial camera is used to completely record the movement trajectory of the digital image speckle on the digital image speckle matte paper.

[0031] Step S21 specifically includes:

[0032] The ultrasonic rangefinder is used to measure the distance from the center of each digital image speckle matte paper on the large-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, a computer is used to identify the pixel coordinates of these key nodes in the black-and-white industrial camera view, and based on the geometric relationship, the proportional factor corresponding to each key node is calculated.

[0033] Step S22 specifically includes:

[0034] A black-and-white industrial camera is used to capture a series of images containing key positions of the large-span steel structure during the construction process or operation stage, and a pre-loading image is designated as a reference image. Then, the large-span steel structure is taken as the core area of ​​the study, and the image is cropped to retain only the main part of the large-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel.

[0035] The image cropping operation is specifically for all images captured by black and white industrial cameras, focusing on the square pixel area occupied by the large-span steel structure beams in the image, that is, the area is defined by four coordinate points of the top, bottom, left and right. The union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

[0036] Beneficial effects:

[0037] First, in terms of recognition technology, the present invention proposes a deep learning framework that breaks through the low-pixel limitation of traditional black-and-white industrial cameras, greatly expands the deployment range of the monitoring system, and realizes automatic recognition and target matching of digital image speckles.

[0038] Second, the present invention develops an automatic denoising framework based on the Mean Shift clustering algorithm for processing and analyzing the collected displacement data. This framework can efficiently filter out abnormal data and accurately calculate the average deflection value, providing a solid data foundation for structural safety assessment.

[0039] Third, compared with traditional contact monitoring methods, the present invention can effectively implement non-contact deflection monitoring of multiple key nodes of large-span steel structures, and can safely, quickly and accurately monitor the deflection changes of large-span steel structures during the construction process or operation stage, providing a solid data foundation for structural safety assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural schematic diagram of the large-span steel structure deflection monitoring system of the present invention;

[0041] Figure 2 This is a flow chart of a large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm of the present invention;

[0042] Figure 3 This is a flowchart of the automatic denoising framework based on the three-stage Mean Shift clustering algorithm of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] To facilitate understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings. Several embodiments of the present invention are given in the drawings, but the present invention can be implemented in different forms and is not limited to the embodiments described in the text. On the contrary, these embodiments are provided to make the content disclosed in the present invention more thorough and comprehensive.

[0045] For examples, please refer to the attached Figure 1 , a large-span steel structure deflection monitoring system based on machine vision and Mean Shift clustering ， It is mainly composed of digital image speckle matte paper 1, heavy tripod 2, black and white industrial camera 3, fixed focus lens 4, computer 5, and ultrasonic rangefinder 6. The black and white industrial camera 3 and the fixed focus lens 4 are fixed on the heavy tripod 2 to shoot the dynamic motion trajectory of the digital image speckle matte paper 1. The computer 5 is connected with the black and white industrial camera 3 by data, and the computer 5 controls the camera to shoot the deflection change process at the key nodes during the construction process or operation stage of the large-span steel structure.

[0046] The method for making the digital image speckle matte paper 1 is to print random digital image speckles on matte paper with a matte film on the front side and an adhesive on the back side. The size of the matte paper can be A3 or A4.

[0047] The digital image speckle matte paper 1 is firmly attached to the key node position of the large-span steel structure and can represent its deflection.

[0048] The black-and-white industrial camera 3 and the fixed-focus lens 4 are fixed to the ground via a heavy-duty tripod 2 equipped with a pan / tilt head.

[0049] The heavy-duty tripod 2 is a Manfrotto 475B tripod and a 400 gear head.

[0050] The black and white industrial camera 3 is a FLIR GS3-U3-41C6M-C black and white industrial camera.

[0051] The fixed-focus lens 4 is a KOWALM16HC lens.

[0052] The computer 5 is configured with CPU AMD Ryzen 5 5600X, 32G RAM, 512G internal memory, and graphics card NVIDIA GeForce RTX 3060.

[0053] The ultrasonic rangefinder 6 is a Xima AR841 handheld ultrasonic rangefinder.

[0054] A method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm, the method comprising the following steps:

[0055] S1. Obtain the motion trajectory of the key nodes of the large-span steel structure;

[0056] Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site leveling site to the key positions of the large-span steel structure with an accuracy of ±(1cm+0.5%×distance). Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. Then, the pre-prepared digital image speckle matte paper was attached to the key nodes of the large-span steel structure.

[0057] Using the black and white industrial camera to completely record the motion trajectory of the digital image speckle on the digital image speckle matte paper;

[0058] S2. Construct a deep learning framework for large-span steel structure deflection monitoring based on machine vision;

[0059] S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the proportional factor corresponding to each key node in the image, which is specifically:

[0060] The ultrasonic rangefinder is used to accurately measure the distance from the center of each digital image speckle matte paper on the large-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, the pixel coordinates of these key nodes in the camera view are identified by a computer, and the proportional factor corresponding to each key node is calculated based on the geometric relationship.

[0061] S22 selects the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm based on the scale factor of each key node obtained in step S21, obtains the scale factor of the reconstructed image, and ensures that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel, specifically:

[0062] A series of images containing key positions of the large-span steel structure are captured by a black-and-white industrial camera during the construction process or operation stage, and an image before loading is designated as a reference image. Then, the large-span steel structure is taken as the core area of ​​the study, and the image is cropped to retain only the main part of the large-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel.

[0063] S23 uses the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading, passes the target frame coordinate position information output by the YOLO v5 detection algorithm to the LoFTR algorithm, and uses the LoFTR algorithm to calculate the image pixel displacement corresponding to each key node on the large-span steel structure. Combined with the scale factor of the reconstructed image obtained in the previous step, the image displacement of each key node is accurately converted into physical displacement, and finally the measurement results, i.e., structured data, are intuitively displayed;

[0064] S3, inputting the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, specifically including the following sub-steps:

[0065] S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region;

[0066] S32 divides the entire stable region into multiple data segments and sorts them by time series, and uses the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering;

[0067] S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment.

[0068] diff i =d i -d i+1 (1)

[0069]

[0070] In the formula, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections of

[0071] window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain;

[0072] Indicates the average deflection value in the time domain at which the point is located.

[0073] As a further preference of the present invention, in step S22, the image cropping operation is specifically for all images captured by the black and white industrial camera, focusing on the square pixel area occupied by the large-span steel structure beams in the image, that is, the area is defined by four coordinate points of the top, bottom, left and right, and the union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

[0074] As a further preferred embodiment of the present invention, the implementation of the ESRGAN super-resolution reconstruction algorithm primarily relies on a large-span steel structure and digital image speckle dataset containing high-resolution and low-resolution black-and-white image pairs. Subsequently, this large-span steel structure and digital image speckle dataset is used to train the ESRGAN super-resolution reconstruction algorithm to ensure that it can accurately grasp the complex mapping relationship from low-resolution images to their high-resolution counterparts. In order to comprehensively evaluate the performance of the algorithm in super-resolution reconstruction, the ESRGAN super-resolution reconstruction algorithm is fine-tuned and optimized by calculating the peak signal-to-noise ratio PSNR and the structural similarity index SSIM of the super-resolution images generated by the test dataset. The larger the peak signal-to-noise ratio and the structural similarity index, the better, to ensure that its performance is optimal.

[0075] As a further preferred embodiment of the present invention, the YOLOv5 detection algorithm needs to collect a black-and-white image dataset containing digital image speckle annotation information in advance, and then, based on the annotated dataset, the YOLOv5 detection algorithm is trained to achieve accurate detection and positioning of digital image speckle in the image, and the detection accuracy and overall performance are further optimized by fine-tuning parameters and hyperparameters.

[0076] As a further optimization of the present invention, the operation of the LoFTR algorithm depends on a pre-collected black-and-white image dataset of large-span steel structures, which contains known displacement information. Next, the LoFTR algorithm is trained using an image sequence containing displacement changes. In actual applications, the target frame coordinate position information output by the YOLOv5 detection algorithm is passed to the LoFTR algorithm, and the LoFTR algorithm only performs local matching calculations on the pixels within the target frame range.

[0077] The above is only an embodiment of the present invention. Although the present invention describes several embodiments, it is not intended to be limited to these embodiments. For those skilled in the art, the present invention can be improved without departing from the principles of the present invention, and all improvements are included in the scope of the claims of the present invention.

Claims

1. A large-span steel structure deflection monitoring method based on machine vision and MeanShift clustering algorithm, characterized in that: The following steps are involved: S1. Obtain the motion trajectory of the key nodes of the large-span steel structure; S2. Construct a deep learning framework for deflection monitoring of large-span steel structures based on machine vision; S21 uses the image of the key nodes of the large-span steel structure obtained in step S1 to obtain the pixel coordinates of these key nodes, and calculates the physical distance corresponding to each pixel of each key node in the image, that is, the proportional factor corresponding to each key node in the image; S22, based on the scale factor of each key node obtained in step S21, selecting the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm, obtaining the scale factor of the reconstructed image, and ensuring that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel; S23, using the trained YOLO v5 detection algorithm to track the position of each key node in the image sequence taken after loading, passing the target frame coordinate position information output by the YOLO v5 detection algorithm to the LoFTR algorithm, using the LoFTR algorithm to calculate the image pixel displacement corresponding to each key node on the large-span steel structure, combining the scale factor of the reconstructed image obtained in step S22 to accurately convert the image displacement of each key node into a physical displacement, and finally intuitively displaying the measurement results, i.e., structured data; S3, inputting the structured data obtained in step S23 into an automatic denoising framework based on the Mean Shift clustering algorithm to filter abnormal data and accurately calculate the average deflection value, specifically including the following sub-steps: S31 locates the starting point of the stable region and applies the HDBSCAN clustering algorithm to generate the approximate average centroid of the stable region; S32 divides the entire stable region into multiple data segments and sorts them by time series, and uses the average value obtained by HDBSCAN clustering in step S31 as a parameter to guide the Mean Shift clustering algorithm to perform more detailed clustering; S33 generates two new data features: the instantaneous characteristic formula of deflection fluctuation (1) and the characteristic formula of deflection fluctuation area (2). Based on these features, the average deflection value is calculated to obtain the final result required for safety assessment. diff i =d i -d i+1 (1) In the formula, diff i represents the deflection d at the i-th frame i With the subsequent frame d i+1 The difference between the deflections of window_diff i It represents the difference between the deflection at a specific point and the average deflection value in the subsequent time domain; Indicates the average deflection value in the time domain at which the point is located.

2. The large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: In step S22, the ESRGAN super-resolution reconstruction algorithm is first trained on a large-span steel structure and speckle dataset containing high / low resolution black-and-white image pairs, to ensure that the ESRGAN super-resolution reconstruction algorithm accurately maps low to high resolution images; The super-resolution images generated by the test set are evaluated by the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM), and are adjusted to maximize the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) to achieve the best performance.

3. The large-span steel structure deflection monitoring method based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: In step S23, the YOLO v5 detection algorithm first constructs a black-and-white image dataset containing speckle annotations for training to achieve accurate speckle detection and positioning, and improves the detection accuracy and overall performance by optimizing the YOLO v5 detection algorithm parameters and hyperparameters.

4. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: In step S23, the LoFTR algorithm relies on the training of a large-span steel structure image set containing displacement information. The YOLOv5 detection algorithm outputs the target frame coordinates to the LoFTR algorithm, which only performs local matching on the pixels within the frame.

5. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 1 is characterized in that: Step S1 specifically includes: Before construction, an ultrasonic rangefinder was used to measure the horizontal and vertical distances from the construction site to the key positions of the large-span steel structure. Based on the principle of pinhole imaging, the field of view of the black-and-white industrial camera was estimated and the diameter of the speckle dots in the required digital image speckle matte paper was calculated. Then, the pre-prepared digital image speckle matte paper was attached to the key nodes of the large-span steel structure. The black-and-white industrial camera is used to completely record the movement trajectory of the digital image speckle on the digital image speckle matte paper.

6. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 5 is characterized in that: Step S21 specifically includes: The ultrasonic rangefinder is used to measure the distance from the center of each digital image speckle matte paper on the large-span steel structure to the center of the black-and-white industrial camera target surface. Subsequently, a computer is used to identify the pixel coordinates of these key nodes in the black-and-white industrial camera view, and based on the geometric relationship, the proportional factor corresponding to each key node is calculated.

7. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 5 is characterized in that: Step S22 specifically includes: A black-and-white industrial camera is used to capture a series of images containing key positions of the large-span steel structure during the construction process or operation stage, and a pre-loading image is designated as a reference image. Then, the large-span steel structure is taken as the core area of ​​the study, and the image is cropped to retain only the main part of the large-span steel structure beam. Based on the scale factor corresponding to each key node obtained in step S3, the upsampling ratio of the ESRGAN super-resolution reconstruction algorithm is selected to obtain the scale factor of the reconstructed image, ensuring that the scale factor of the reconstructed image is maintained at an accuracy level of 0.1 to 10 mm / pixel.

8. The method for monitoring the deflection of a large-span steel structure based on machine vision and Mean Shift clustering algorithm according to claim 7 is characterized in that: The image cropping operation is specifically for all images captured by black and white industrial cameras, focusing on the square pixel area occupied by the large-span steel structure beams in the image, that is, the area is defined by four coordinate points of the top, bottom, left and right. The union of these square pixel areas in all pictures is accurately cropped and replaced with the original image.

Images