Method and system for monitoring three-dimensional deformation of shock insulation structure support based on binocular vision

Abstract

The invention discloses a seismic isolation structure support three-dimensional deformation monitoring method and system based on binocular vision. The method comprises the steps that a binocular camera is adopted to collect monitoring scenes under the view angle of a left camera and the view angle of a right camera; performing image distortion removal and three-dimensional correction on the monitored image pair by adopting camera internal and external parameters obtained by camera calibration; carrying out target detection on the corrected image by adopting a target detection algorithm; according to the coordinates of the target detection frame, calculating a cutting frame and cutting target areas in the left image and the right image; carrying out stereo matching on the cut image pair; according to the disparity value of each pixel in the disparity map, depth information of the target is calculated through a triangulation method, and finally a three-dimensional dense point cloud of the target is generated; performing optimization processing on the target three-dimensional dense point cloud obtained in the previous step to obtain an optimized point cloud; solving a rigid body transformation matrix of the two-point clouds by using an iterative nearest point algorithm; according to the invention, the cost of monitoring the three-dimensional pose deformation of the rubber support can be reduced, and the robustness and precision are effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional deformation monitoring, and more specifically to a method and system for monitoring the deformation of a support of a seismic isolation structure based on binocular vision. Background Art

[0002] Seismic isolation technology is widely used in large infrastructure projects, especially in public buildings such as bridges, airport terminals and teaching buildings. Since the key functions of these buildings are closely related to social safety, any potential structural failure or accident may bring huge economic losses and casualties. Therefore, it is crucial to ensure the safety of seismic isolation structures and conduct long-term and effective health monitoring.

[0003] The health monitoring of seismic isolation structures is achieved through on-site non-destructive testing technology and structural characteristics analysis, which can timely identify the location and degree of damage to the engineering structure, thereby providing early warning of structural changes. In seismic isolation technology, seismic isolation bearings are core components, and their performance directly affects the stability and seismic resistance of the structure in various environments. Due to the long-term influence of environmental factors such as temperature changes, foundation settlement, and concrete shrinkage, seismic isolation bearings usually experience varying degrees of displacement and deformation. Therefore, using efficient, real-time, and remote monitoring methods to monitor the status of seismic isolation bearings has become the key to ensuring the safe operation of buildings.

[0004] At present, three-dimensional deformation monitoring technology is gradually being applied to the health monitoring of seismic isolation structures. Common methods include structured light and TOF (time of flight) cameras. These methods can capture the deformation of seismic isolation bearings more accurately by collecting three-dimensional point cloud data of objects. However, although these 3D cameras have the ability to collect data in real time, their equipment costs are high, their robustness is poor, and their monitoring accuracy is limited in complex environments, which cannot meet the actual needs of displacement monitoring of seismic isolation bearings. In addition, due to the large horizontal displacement of rubber bearings during use, existing sensors cannot take into account both wide range and high sensitivity, resulting in a technical bottleneck in the design of sensors between high precision and wide range.

[0005] Therefore, at this stage, there is an urgent need for a low-cost, high-precision, and robust monitoring solution that can accurately measure the three-dimensional deformation of the isolation bearing and can be used stably in complex engineering environments for a long time. Summary of the invention

[0006] In order to overcome the shortcomings of high cost, poor robustness and low precision of existing three-dimensional deformation monitoring of seismic isolation structure supports, the present invention provides a three-dimensional deformation monitoring method and device for seismic isolation structure supports based on binocular vision, which effectively reduces the cost of three-dimensional deformation monitoring of seismic isolation structure supports, enhances robustness and improves precision.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A three-dimensional deformation monitoring method for a seismic isolation structure support based on a dual-purpose method comprises the following steps:

[0009] S1. Use a binocular camera to collect monitoring scenes from the perspectives of left and right cameras respectively, and obtain a monitoring image pair containing a target object for structural deformation monitoring;

[0010] S2, using the camera internal and external parameters obtained by camera calibration to perform image dedistortion and stereo correction on the monitoring image pair to obtain a corrected image pair;

[0011] S3, using the target detection algorithm and the pre-trained target detection model to perform target detection on the rectified image, automatically determine the target position, and obtain the target detection frame coordinates;

[0012] S4, according to the target detection frame coordinates obtained in the previous step, calculate the cropping frame and accurately crop the target area in the left and right images to obtain a cropped image pair and remove irrelevant background information;

[0013] S5, performing stereo matching on the cropped image pair, including calculating an initial cost, cost aggregation, obtaining an initial disparity map, and disparity post-processing to obtain a final disparity map;

[0014] S6, according to the disparity value of each pixel in the disparity map, combined with the geometric relationship of the binocular camera, the depth information of the target is calculated by triangulation, and the depth information is converted into coordinate points in three-dimensional space, and finally a three-dimensional dense point cloud of the target is generated;

[0015] S7, optimizing the three-dimensional dense point cloud of the target obtained in the previous step, including point cloud denoising and downsampling processing, to obtain an optimized point cloud;

[0016] S8. Use the iterative closest point algorithm to solve the rigid body transformation matrix of the two point clouds to realize the three-dimensional deformation monitoring of the isolation structure support.

[0017] Furthermore, in step S1, the monitoring image pair includes a first monitoring image and a second monitoring image, wherein the first frame monitoring image pair is used as a template image pair, and the collected images are all grayscale images;

[0018] Further, in step S2, the camera calibration method includes: shooting a calibration plate image, determining a camera calibration tool, and determining camera calibration parameters; shooting the calibration plate image, using a binocular camera to shoot fifteen or more clear calibration images in different spatial states in the same scene, with the calibration plate occupying two-thirds of the image; the binocular camera includes a left camera and a right camera, and the calibration plate is a checkerboard calibration plate; the camera calibration tool uses a binocular camera calibration program provided by the MATLAB platform; the camera calibration parameters include the camera's intrinsic parameters, the camera's extrinsic parameters, and the camera's distortion parameters, and the camera's intrinsic parameters include the camera's focal length (f x ,f y ), imaging origin coordinates (c x ,c y ), f x ,f y are the pixel focal lengths in the x and y directions, respectively, in pixels. The external parameters of the camera include the rotation matrix R and the translation vector T, and the distortion parameters of the camera include radial distortion (K1, K2, K3) and tangential distortion (P1, P2);

[0019] Furthermore, in step S3, the target detection algorithm is a YOLOv8 target detection algorithm, and the YOLOv8 model is trained based on an existing target data set to obtain a weight file to achieve the purpose of target detection;

[0020] Furthermore, in step S3, the target detection frame coordinates are obtained by inputting the corrected first image into the trained YOLOv8 model for target detection to obtain the upper left corner coordinates (u top ,v top ) and the lower right corner coordinate (u btm ,u btm ), where u top , v top The x value, y value, and u value of the upper left corner coordinates respectively btm ,u btm They are the x and y values ​​of the lower right corner coordinates respectively.

[0021] Furthermore, in step S4, the cropping frame is the column coordinate u of the upper left corner coordinate of the first image target monitoring frame to be corrected. top Subtract the clipping offset value d u , get the coordinates of the upper left corner of the cropping box (u top -d u ,v top ), the coordinates of the lower right corner of the cropping frame are (u btm ,v btm ), further, the clipping offset value d u The calculation formula is Where f is the focal length of the camera, B is the length of the line connecting the optical centers of the two cameras, and Z s is the minimum vertical distance between the upper and lower isolation plates of the rubber bearing; the corrected image pair is cropped using the upper left corner coordinates and the lower right corner coordinates of the cropping frame to obtain a cropped image pair, wherein the cropped image pair includes a cropped first image and a cropped second image, wherein the cropped first image and the cropped second image must contain a complete target image.

[0022] Furthermore, in step S5, the stereo matching step includes: calculating the initial cost, cost aggregation, obtaining the initial disparity map, and disparity post-processing; further, the calculation of the initial cost refers to calculating the census transformation value and the AD value for each pixel of the above-mentioned cropped image for fusion; the cost aggregation refers to the use of a one-way dynamic programming idea and an eight-path cost aggregation strategy to aggregate the initial cost; the obtaining of the initial disparity map refers to the use of a winner-takes-all strategy to select the disparity value corresponding to the minimum cost value of each pixel of the cropped first image; the disparity post-processing refers to the optimization processing of the disparity map by sub-pixel fitting, consistency check, disparity hole filling and median filtering.

[0023] Furthermore, in step S6, the triangulation method refers to the use of binocular ranging principle to calculate the depth value, that is, Among them, X R -X L is the disparity of the pixels of the first image and the second image, f is the focal length of the camera, and B is the center distance between the two cameras, and the depth map can be calculated;

[0024] In the step S6, the method for obtaining the target three-dimensional dense point cloud refers to performing coordinate transformation on the cropped depth map to obtain a three-dimensional dense point cloud in the left camera coordinate system; further, the cropped depth map refers to cropping the above depth map, specifically, with (d u , 0) is the boundary, and the pixel column from (0, 0) to (d u -1,0) is cropped and deleted; further, the X, Y coordinate conversion method in the three-dimensional coordinates is: Where Z is the depth value, (u, v) is the pixel coordinate, and f x , f y is the focal length of the camera in the x and y directions, c x , c y is the principal point position, (u, v) is the pixel coordinate;

[0025] Further, in step S7, the optimized point cloud is to obtain a denoised point cloud by using statistical filtering to process the three-dimensional dense point cloud, and then obtain a final point cloud after uniform downsampling, wherein the final point cloud of the template image pair is the template point cloud, and the final point cloud of the subsequent monitoring image pair is the point cloud to be measured;

[0026] Furthermore, in step S8, the method for solving the rigid body transformation matrix calculation specifically includes: setting an initial transformation matrix, and using an iterative nearest point algorithm to iteratively optimize the transformation matrix to obtain an optimal transformation matrix; the method for setting the initial transformation matrix calculates the three-dimensional coordinates of the center of mass of the template point cloud and the point cloud to be measured, and uses the displacement transformation of the two centers of mass as the translation matrix in the initial transformation matrix; the rotation matrix is ​​set to a rotation matrix without rotation, and the above constitutes the initial transformation matrix; the iterative nearest point algorithm includes: for each point in the template point cloud, find the corresponding point in the point cloud to be measured, use the nearest neighbor search method to find the nearest neighbor point of the point in the template point cloud, correspond them, and form corresponding point pairs; optimize the target error function Among them, E(R, t) is the target error function, R, t are the rotation matrix and translation matrix corresponding to the minimum error, p, q are the coordinates of the points in the template point cloud and the point cloud to be measured in the corresponding point pair, respectively. The optimal rigid body transformation is calculated by minimizing the error, and the optimal transformation matrix is ​​obtained by solving it through the optimization algorithm; iterative calculation is continuously performed to update the optimal transformation matrix until the error E is less than the set threshold Y or greater than the preset maximum number of iterations, and the optimal transformation matrix that meets the conditions is returned;

[0027] The beneficial effects of the present invention are mainly manifested in: the present invention uses YOLOv8 to detect the target object on the image pairs acquired by the binocular camera, performs stereo matching, depth calculation, and coordinate system conversion on the target image pairs to obtain the target point cloud, and then obtains the three-dimensional posture transformation of the seismic isolation structure support by optimizing the target point cloud and calculating the rigid body transformation matrix through the iterative closest point algorithm. Compared with the existing visual three-dimensional deformation monitoring method of the seismic isolation structure support, the method adopts a binocular camera and related algorithms to effectively reduce the cost of three-dimensional deformation monitoring of the seismic isolation structure support and improve the robustness and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of the steps of the monitoring method of the present invention;

[0029] Figure 2 A schematic diagram of a target pattern provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of the installation of a camera and a target provided by an embodiment of the present invention;

[0031] Figure 4 A schematic diagram of a calibration target provided by an embodiment of the present invention;

[0032] Figure 5 A block diagram of sub-steps of a stereo matching method provided by an embodiment of the present invention;

[0033] Figure 6A schematic diagram of a method for calculating posture transformation provided by an embodiment of the present invention;

[0034] Figure 7 A three-dimensional dense point cloud image of a target provided by an embodiment of the present invention;

[0035] Figure 8 A schematic diagram of the structure of a monitoring system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to make the technical solution of the present invention clearer, the technical solution provided by the present invention will be described in detail in conjunction with specific embodiments, and the present invention will be further described in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only partial embodiments of the present invention and are used to explain the present invention, rather than to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0037] Attached Figure 1 1 is a flowchart of the steps of the binocular vision-based three-dimensional deformation monitoring method for seismic isolation structure supports provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps S1 to S8.

[0038] S1. Collecting a pair of monitoring images containing a target through a binocular camera, wherein the monitoring image pair includes a first monitoring image and a second monitoring image, wherein the first frame of the monitoring image pair is used as a template image pair;

[0039] The target image obtained by the binocular camera is as follows: Figure 2 As shown, the camera and target are installed as Figure 3 As shown. A binocular camera is installed on the upper isolation plate of the isolation structure support, and a target is installed on the lower isolation plate of the isolation structure support. Specifically, the resolution of the two cameras of the binocular camera is 1920*1080, and the baseline center position of the binocular camera and the target center are guaranteed to be on the same horizontal line during installation.

[0040] S2, obtaining the camera's internal and external parameter monitoring image pair by calibrating the camera, and correcting the image pair so that the two images in the monitoring image pair are aligned in the horizontal direction to obtain a corrected image pair, wherein the corrected image pair includes a corrected first image and a corrected second image;

[0041] Use a binocular camera to take 15 or more clear calibration images in different spatial states in the same scene, with the calibration plate occupying two-thirds of the image. The calibration plate style is as shown in the attached figure. Figure 4 As shown, further, the binocular camera includes a left camera and a right camera;

[0042] Determine a camera calibration tool, wherein the camera calibration tool adopts a binocular camera calibration program provided by the MATLAB platform;

[0043] Performing camera calibration on the checkerboard image pair according to the camera calibration tool, filtering out checkerboard images with a reprojection error greater than 0.1, and obtaining the camera calibration parameters;

[0044] The camera calibration parameters include the camera's intrinsic parameters, camera's extrinsic parameters and camera's distortion parameters. The camera's intrinsic parameters include the camera's focal length (f x , f y ), imaging origin coordinates (c x , c y ),f x ,f y are the pixel focal lengths in the x and y directions, respectively, in pixels. The external parameters of the camera include the rotation matrix R and the translation vector T, and the distortion parameters of the camera include radial distortion (K1, K2, K3) and tangential distortion (P1, P2);

[0045] Based on the camera calibration parameters, the rectifyStereoImages() function in Opencv is used to eliminate the rotation and projection distortion between the real-scene stereo image pair, so that the two images in the monitoring image pair are aligned in the horizontal direction to obtain a corrected image pair, which includes a corrected first image and a corrected second image. The two images in the monitoring image pair are aligned in the horizontal direction, providing necessary conditions for subsequent stereo matching.

[0046] S3. To reduce the redundancy of stereo matching calculation, the target background is removed to speed up the calculation. First, the YOLOv8 target detection algorithm is used to detect the target on the rectified image to obtain the target detection frame coordinates.

[0047] Specifically, a target data set is obtained and divided into a training set, a validation set, and a test set; a YOLOv8 model is trained based on the training set, the validation set, and the test set to obtain a weight file to achieve the purpose of target detection; the corrected first image is input into the target detection network for target detection, and the upper left corner coordinates and the lower right corner coordinates (u top ,v top ), (u btm ,v btm ),u top , v top The x value, y value, and u value of the upper left corner coordinates respectively btm ,u btm They are the x and y values ​​of the lower right corner coordinates respectively.

[0048] S4, cropping the monitoring image pair according to the target detection frame coordinates to obtain a cropped image pair, wherein the cropped image pair includes a cropped first image and a cropped second image;

[0049] Specifically, in order to ensure that the size of the cropped first image is consistent with that of the cropped second image, and to ensure that the cropped first image and the cropped second image both completely contain the target image, the column coordinates u of the upper left corner coordinates of the target monitoring frame of the corrected first image are top Subtract the clipping offset value d u Get the coordinates of the upper left corner of the cropping box (u top -d u ,v top ), the coordinates of the lower right corner of the cropping frame are (u btm ,v btm );

[0050] Among them, d u The calculation is as follows:

[0051]

[0052] Where f is the focal length of the camera, B is the length of the line connecting the optical centers of the two cameras, and Z s It is the minimum vertical distance between the upper and lower isolation plates of the rubber bearing.

[0053] The rectified image pair is cropped using the upper left corner coordinates and the lower right corner coordinates of the cropping frame to obtain a cropped image pair, wherein the cropped image pair includes a cropped first image and a cropped second image, wherein the cropped first image and the cropped second image must contain a complete target image.

[0054] S5, performing stereo matching on the cropped image pair to obtain a disparity map;

[0055] As attached Figure 5 As shown in the block diagram of the sub-steps of the stereo matching method in the deformation monitoring method of the seismic isolation structure support based on binocular vision, the steps of stereo matching may include initial cost calculation, cost aggregation, disparity calculation, and disparity post-processing.

[0056] First, the matching cost is calculated. In the present application example, the Census transformation value is fused with the AD value to calculate the matching cost, so as to obtain the initial cost value of the first image with different parallaxes in the second image.

[0057] The AD algorithm takes the average of the absolute values ​​of the differences between the three color components of the pixels in the left and right views. If it is a grayscale image, it is equal to the grayscale brightness difference. The formula for calculating the cost is shown in formula (2):

[0058]

[0059] Where CAD (p,d) is the absolute difference cost of pixel p under disparity d, Represents the intensity value of the i-th color channel of the left image at pixel p, where i can be R, G or B, represents the pixel intensity value in the right image at disparity d corresponding to p in the left image.

[0060] The Census transformation compares the center value of the sliding window (the window size is X*Y, and the window size used in this algorithm is 7*9) with the grayscale values ​​of each pixel in the neighborhood of the window, and generates a binary bit string as the Census transformation value of the window center value, as shown in formula (3):

[0061]

[0062] Where x and y are the maximum integer values ​​less than the window size X / 2 and Y / 2 respectively. is a bitwise concatenation operator, and the ξ function is shown in formula (4):

[0063]

[0064] In the formula, the parameters x and y in the ξ function are the grayscale values ​​of the pixels to be compared in the 7*9 sliding window, and the ξ function is the result of the Census transformation. The matching cost calculation method based on the Census transformation is to calculate the Hamming distance between the corresponding two pixels of the left image and the right image after the Census transformation, as shown in formula (5):

[0065] C(u,v,d)=Hamming(C sl (u,v),C sr (ud,v)) (5)

[0066] In the formula, C sl The (u, v) function is the Census transformation value of the pixel point (u, v) in the left image as input parameter, C sr The (u,v) function is the Census transformation value of the pixel point (ud,v) in the right figure with input parameters. The parameter d is the disparity, and the function C(u,v,d) is the Hamming distance of the bit string of the pixel point (u,v) after the Census transformation at different disparities d.

[0067] In the cost aggregation step of the example of this application, the cost obtained by the AD algorithm and the Census transform are fused, which makes good use of the advantages of the Census transform in the smooth area of ​​the image and the advantages of the AD algorithm in the texture area of ​​the image. However, due to the inconsistent scale of the results after the AD algorithm and the Census transform, AD is a brightness difference calculation with a value range of 0 to 255, and the Census transform obtains the XOR value of the corresponding bit of the bit string with a value range of 0 to N (N is the number of bits in the bit string, generally 32 or 64). Therefore, it is necessary to normalize the results after the two transformations to the same range. This patent uses a natural function normalized to the value range of [0, 1], as shown in formula (6):

[0068]

[0069] In the formula, c is the cost calculation value of the two algorithms, and λ is a given parameter; when c and λ are positive numbers, the value range of this function is 0 to 1. Therefore, this function can be used to normalize any cost function value to between 0 and 1. The cost calculation after normalization fusion is shown in formula (7):

[0070] C(p,d)=ρ(C census (P,d),λ census )+ρ(C AD (P,d),λ AD ) (7)

[0071] In the formula, C census (P,d) is the cost calculation value of the Census algorithm, λ census is the given parameter of the Census algorithm, C AD (P,d) is the cost calculation value of the AD algorithm, λ AD is the given parameter of AD algorithm. The purpose of aggregation is to find the optimal cost value of each pixel in the first image and the second image, so that the global energy function of the first image and the second image is minimized.

[0072] The global energy function in the embodiment of the present application can be shown as formula (8):

[0073]

[0074] Where E(D) represents the energy value calculated by the global energy function, C(p,D p ) is the matching cost, T[] returns 0 or 1, returns 1 if it is true, and returns 0 if it is not true. The first term of the formula is the data term, which represents the accumulation of the matching costs of all pixels when the disparity map is D, p represents the pixel of the current disparity map, q represents the neighboring pixel of pixel p, D p represents the disparity of pixel p, D qrepresents the disparity of pixel q, the second and third terms are smoothing terms, representing the N p All pixels q in the neighborhood are penalized. P1 and P2 are penalty coefficients, where the second penalty is smaller and the third penalty is larger (P2>P1). Smaller penalty items allow the algorithm to adapt to situations with small parallax changes, such as tilted planes or continuous surfaces, while larger penalty items allow the algorithm to correctly handle discontinuous parallax situations.

[0075] In order to obtain the optimal solution of the energy function, a more efficient energy optimization strategy can be adopted, including algorithms such as image segmentation method, confidence propagation method or cooperative optimization method to obtain the optimal solution of the energy function.

[0076] In a possible implementation, the example of this application introduces a one-way dynamic programming idea in the cost aggregation part, and adopts a four-path cost aggregation strategy to aggregate the above initial cost. The four-path cost aggregation directions are the four directions of the pixel up, down, left, and right. For a single path, the cost value of the pixel point p along all the disparity ranges on the path is aggregated in one dimension to obtain the cost value under the path. The calculation expression is shown in formula (9):

[0077]

[0078] Where, L r represents the currently optimized path; Lr(pr,d) represents the cost aggregation value when the disparity of the adjacent points of the pixel p along the path r is d; the first term on the right side of the formula is the original cost of the pixel; the second term on the right side of the formula is the smoothing term, and the third term on the right side of the formula is to ensure the new path cost value L r Does not exceed a certain upper limit. C(p,d) represents the matching cost of the current pixel p, i is a positive integer, P1 and P2 represent the second and third penalty items respectively, min i L r (pr,i) represents the minimum cost value of all neighboring viewpoints i on the path of pixel p along path r.

[0079] After cost aggregation, the optimized cost matrix can be obtained.

[0080] According to the principle of cost calculation, the smaller the cost value between the pixels of the first image and the second image, the higher the correlation between the two pixels. Therefore, the present application can adopt a winner-takes-all algorithm, that is, among the cost values ​​corresponding to the pixel disparity of a certain pixel in the first image within the matching range of the second image, the disparity corresponding to the minimum cost value is selected as the optimal disparity. In order to make the disparity calculation more accurate, the present application can use the disparity value obtained after sub-pixel fitting, consistency check, disparity hole filling and median filtering as the final disparity value.

[0081] Specifically, sub-pixel fitting uses a quadratic curve interpolation method to perform quadratic curve fitting on the cost value of the optimal disparity and the cost values ​​of the two adjacent disparities before and after so that the disparity accuracy reaches sub-pixel accuracy. The consistency check is to recalculate the disparity value of the second image, compare it with the disparity value of the first image, and eliminate incorrect matches. After eliminating incorrect matches, the eliminated pixels cause holes in the disparity map. With the hole pixel as the center, 8 rays are emitted outward at equal angles, and the first valid pixel touched by each ray is stored and filled with the median value. Finally, the filled disparity map is subjected to median filtering to reduce the impact of noise and obtain the final disparity map.

[0082] S6, calculating the depth information according to the disparity value of the disparity map, and obtaining a three-dimensional dense point cloud of the target;

[0083] After calculating the disparity, it is necessary to calculate the depth information corresponding to each pixel. The calculation formula for calculating the depth through disparity can be shown as formula (10):

[0084]

[0085] Where Z is the depth value, X R -X L The disparity of pixels between the first image and the second image, f is the focal length of the camera, and B is the center distance between the two cameras.

[0086] In order to accurately obtain the displacement transformation of the target in three-dimensional space, the three-dimensional dense point cloud of the target is calculated.

[0087] Since the cropping frame of the first image after cropping is obtained through offset, it contains some redundant background information, so the depth map needs to be cropped again to remove the redundant background information. u , 0) is the boundary, and the pixel column from (0, 0) to (d u All pixels in the pixel column where the pixel is located (-1, 0) are cropped and deleted to obtain a cropped depth map.

[0088] The cropped depth map is back-projected onto the 3D point cloud using the intrinsic parameters of the left camera. The X and Y calculations of the point cloud are shown in formulas (11, 12):

[0089] Where Z is the depth value calculated by formula (10), (u, v) is the pixel coordinate, and f x , f y is the focal length of the camera in the x and y directions, c x , c yis the main point position, (u, v) is the pixel coordinate, and it is worth noting that in order to obtain the actual space coordinates and perform subsequent displacement calculations, the coordinate values ​​here are to be added with the (u top ,v top ). Finally, X, Y, and Z are unified into millimeter units to obtain the three-dimensional dense point cloud of the target.

[0090] S7. Perform point cloud optimization processing on the three-dimensional dense point cloud to obtain a final point cloud.

[0091] Specifically, in order to remove the noise of the target point cloud, statistical filtering is used for the obvious outlier point cloud. Statistical filtering performs denoising operations on the target point cloud: any point in the target point cloud is used as a query point, and the corresponding domain point set is found through nearest neighbor search. The distance between the query point and the neighborhood point set is counted, and the first threshold is set. Points with an average domain distance lower or higher than the first threshold are divided into internal value points or outlier points respectively. After traversing the entire point cloud, all outlier points are filtered out to denoise the target point cloud. In order to speed up the calculation speed while retaining the planar features of the target point cloud, uniform downsampling is used, and one point is downsampled for every five points to obtain the final point cloud, as shown in the attached figure. Figure 7 shown.

[0092] S8. The iterative closest point algorithm is used to calculate the rigid body transformation matrix between each point cloud to be tested and the template point cloud to realize the deformation monitoring of the isolation structure support.

[0093] As attached Figure 6 As shown in the flowchart of the method for calculating the posture transformation in the deformation monitoring method of the seismic isolation structure support based on binocular vision, the point cloud obtained by calculating the initial frame is stored as a template point cloud, and the rigid body transformation matrix of the subsequent point cloud to be tested is calculated relative to the template point cloud to obtain the final displacement and rotation matrix as the basis for the target posture transformation.

[0094] The example of calculating the rigid body transformation matrix in this application mainly includes two steps: using the center of mass to calculate the initial transformation matrix, and using the iterative closest point algorithm to iteratively optimize the initial transformation matrix to obtain the final transformation matrix.

[0095] First, in order to accelerate the iteration of the iterative closest point algorithm, the initial transformation matrix should be set in advance. Specifically, the centroid coordinates P of the template point cloud and the point cloud to be measured are calculated. o With P c , calculated as shown in formula (13):

[0096]

[0097] Then the initial translation matrix t ori =[t X ,t Y ,t Z ] TThe displacement value in is P o -P c .

[0098] Initial rotation matrix R ori Assuming it is a unit diagonal matrix, the initial transformation matrix is ​​shown as (14):

[0099]

[0100] After obtaining the initial transformation matrix, the iterative closest point algorithm is used to iteratively solve the optimal rigid body transformation matrix. The overall goal is to iteratively optimize the following objective error function, as shown in formula (15):

[0101]

[0102] Where E(R, t) is the target error function, R, t is the rotation matrix and translation matrix corresponding to the minimum error, and p, q are the coordinates of the midpoints of the template point cloud and the point cloud to be measured in the corresponding point alignment, respectively.

[0103] Specifically, for each point in the template point cloud, it is transformed into a temporary point cloud through the initial transformation matrix, and the corresponding point is found in the point cloud to be tested. The nearest neighbor search method is used to find the nearest neighbor points of the points in the template point cloud, and they are matched to form corresponding point pairs; the Euclidean distance between each point pair is calculated, and the sum of these distances is used as the error; the optimal rigid body transformation is calculated by minimizing the error, and the optimal transformation matrix is ​​obtained by solving it through the optimization algorithm; iterative calculations are continuously performed to update the optimal transformation matrix until the error is less than the set threshold or greater than the preset maximum number of iterations, and the optimal transformation matrix that meets the conditions is returned. This matrix is ​​also the final rigid body transformation matrix T final , thereby obtaining the target's posture transformation data;

[0104] This application example uses the rotation matrix and translation matrix of each point cloud to be tested and the template point cloud as the basis for posture transformation. Therefore, the final rigid body transformation matrix T final Decompose it as follows: R final =T final [:3, :3], t final =T final [:3,3],R final is the rigid body rotation transformation matrix, t final is the rigid body displacement transformation matrix.

[0105] The above-mentioned three-dimensional deformation monitoring method of seismic isolation structure supports based on binocular vision uses YOLOv8 to monitor the image pairs acquired by the binocular camera, performs stereo matching, depth calculation, and coordinate system conversion on the target image pairs to obtain the target point cloud, and then obtains the rubber bearing posture transformation by optimizing the target point cloud and calculating the rigid body transformation matrix through the iterative nearest point algorithm, thereby reducing the cost of three-dimensional displacement deformation detection of rubber bearings, realizing the monitoring of rubber bearing posture, and improving robustness and accuracy.

[0106] Attached Figure 8 A structural schematic diagram of a seismic isolation structure support deformation monitoring system based on binocular vision provided in an embodiment of the present invention, the system includes: an image acquisition unit, a target detection unit, a stereo matching unit, a point cloud processing unit, and a posture transformation calculation unit.

[0107] The image acquisition unit is used to obtain the binocular image stream acquired by the binocular camera;

[0108] The target detection unit uses the YOLOv8 target detection algorithm to detect and crop the binocular image;

[0109] The stereo matching unit performs stereo matching calculation on the cropped image to obtain a depth map. The specific steps include initial cost calculation, cost aggregation, disparity calculation, and disparity post-processing.

[0110] The point cloud processing unit reversely projects the depth map into three-dimensional space to obtain the point cloud and performs subsequent processing. The specific steps include statistical filtering denoising and uniform downsampling.

[0111] The pose transformation calculation unit uses an iterative closest point algorithm to calculate the rigid body transformation matrix to obtain the target pose transformation.

[0112] It should be noted that those skilled in the art can understand the specific implementation process of the isolation structure support deformation monitoring device based on binocular vision and its various functional units in detail by referring to the relevant description of the above embodiments. For the sake of brevity, it will not be repeated here.

[0113] In addition, it should be noted that the functional units of the above-mentioned isolation structure support deformation monitoring device are divided based on functional logic, and the division method is not limited to this, as long as the corresponding functions can be achieved. The names of the functional units are only set to distinguish different modules and do not limit the protection scope of the present invention.

[0114] The above-mentioned specific embodiments of the present invention are only illustrative and not limiting to the protection scope thereof. Any person familiar with the technical field can easily make equivalent modifications or substitutions within the technical framework of the present invention, and these modifications or substitutions shall be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope of the claims.

Claims

1. A three-dimensional deformation monitoring method for seismic isolation structure supports based on binocular vision, characterized in that: include: S1. Use a binocular camera to collect monitoring scenes from the perspectives of left and right cameras respectively, and obtain a monitoring image pair containing a target object for structural deformation monitoring; S2, using the camera internal and external parameters obtained by camera calibration to perform image dedistortion and stereo correction on the monitoring image pair to obtain a corrected image pair; S3, using the target detection algorithm and the pre-trained target detection model to perform target detection on the rectified image, automatically determine the target position, and obtain the target detection frame coordinates; S4, according to the acquired target detection frame coordinates, calculate the cropping frame and crop the target area in the left and right images to obtain a cropped image pair, and remove irrelevant background information; S5, performing stereo matching on the cropped image pair, including calculating an initial cost, cost aggregation, obtaining an initial disparity map, and disparity post-processing to obtain a final disparity map; S6. According to the disparity value of each pixel in the disparity map, combined with the geometric relationship of the binocular camera, the depth information of the target is calculated by triangulation, and the depth information is converted into coordinate points in three-dimensional space, and finally a three-dimensional dense point cloud of the target is generated; S7, optimizing the obtained three-dimensional dense point cloud of the target, including point cloud denoising and downsampling processing, to obtain an optimized point cloud; S8. Use the iterative closest point algorithm to solve the rigid body transformation matrix of the two point clouds to realize the three-dimensional deformation monitoring of the isolation structure support.

2. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 1, characterized in that: In step S1, the binocular camera installation method is to install a binocular camera on the upper isolation plate of the rubber support and install a target on the lower isolation plate of the rubber support. During installation, the baseline center position of the binocular camera and the target center are on the same horizontal line. The binocular camera is used to collect monitoring image pairs containing the target. The monitoring image pairs include a first monitoring image and a second monitoring image, wherein the first frame of the monitoring image pair is used as a template image pair.

3. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 1, characterized in that: In step S2, the camera calibration method includes: photographing a calibration plate image, determining a camera calibration tool, and determining camera calibration parameters; The photographing of the calibration plate image comprises photographing fifteen or more clear images containing the calibration plate in different spatial states in the same scene with a binocular camera, wherein the calibration plate occupies two-thirds of the image; The binocular camera includes a left camera and a right camera, and the calibration plate is a checkerboard calibration plate; The camera calibration tool uses the binocular camera calibration program provided by the MATLAB platform; The camera calibration parameters include the camera's intrinsic parameters, camera's extrinsic parameters and camera's distortion parameters. The camera's intrinsic parameters include the camera's focal length (f x , f y ), imaging origin coordinates (c x , c y ), are the pixel focal lengths in the x and y directions, respectively, in pixels; The external parameters of the camera include a rotation matrix R and a translation vector T, and the distortion parameters of the camera include radial distortion (K1, K2, K3) and tangential distortion (P1, P2); The stereo correction method is based on the camera calibration parameters and uses the rectifyStereoImages() function in Opencv to eliminate the rotation and projection distortion between the real-scene stereo image pair, so that the two images in the monitoring image pair are aligned in the horizontal direction to obtain a corrected image pair, which includes a corrected first image and a corrected second image.

4. The method for monitoring deformation of a support of a seismic isolation structure based on binocular vision according to claim 1, characterized in that: In step S3, the target detection algorithm is the YOLOv8 target detection algorithm, and the YOLOv8 model is trained based on the existing target data set to obtain a weight file to achieve the target detection purpose; the target detection frame coordinates are: the corrected first image is input into the trained YOLOv8 model for target detection, and the upper left corner coordinates and the lower right corner coordinates (u top ,v top ), (u btm ,v btm ), where u top , v top The x value, y value, and u value of the upper left corner coordinates respectively btm ,u btm They are the x and y values ​​of the lower right corner coordinates respectively.

5. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 1, characterized in that: In step S4, the cropping frame is the column coordinate u of the upper left corner coordinate of the target monitoring frame of the first image. top Subtract the clipping offset value d u , get the coordinates of the upper left corner of the cropping box (u top -d u ,v top ), the coordinates of the lower right corner of the cropping frame are (u btm ,v btm ),in, f is the focal length of the camera, B is the length of the line connecting the optical centers of the two cameras, Z s is the minimum vertical distance between the upper and lower isolation plates of the rubber bearing; the corrected image pair is cropped using the upper left corner coordinates and the lower right corner coordinates of the cropping frame to obtain a cropped image pair, wherein the cropped image pair includes a cropped first image and a cropped second image, wherein the cropped first image and the cropped second image must contain a complete target image.

6. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 1, characterized in that: In the step S5, stereo matching includes: calculating initial cost, cost aggregation, obtaining initial disparity map, and disparity post-processing; The calculation of the initial cost refers to calculating the census transformation value and the AD value for each pixel of the cropped image and fusing them; The cost aggregation refers to the one-way dynamic programming idea, and adopts an eight-path cost aggregation strategy to aggregate the initial cost; Obtaining the initial disparity map refers to selecting the disparity value corresponding to the minimum cost value of each pixel of the cropped first image by using a winner-takes-all strategy; The disparity post-processing refers to the optimization processing of sub-pixel fitting, consistency check, disparity hole filling and median filtering on the disparity map.

7. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 1, characterized in that: In step S6, the triangulation method refers to the binocular distance measurement principle, that is, Among them, X R -X L is the disparity of the pixels of the first image and the second image, f is the focal length of the camera, and B is the center distance between the two cameras, and the depth map can be calculated; The method for obtaining a three-dimensional dense point cloud of a target refers to performing coordinate transformation on the cropped depth map to obtain a three-dimensional dense point cloud in the left camera coordinate system; The cropping depth map refers to cropping the above depth map to (d u , 0) is the boundary, and the pixel column from (0, 0) to (d u -1, 0) is cropped and deleted; The coordinate conversion method: Where Z is the depth value, (u, v) is the pixel coordinate, and f x , f y is the focal length of the camera in the x and y directions, c x , c y is the principal point position, and (u, v) is the pixel coordinate.

8. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 2, characterized in that: In step S7, the optimized point cloud is to obtain a denoised point cloud by using statistical filtering to process the three-dimensional dense point cloud, and then perform uniform downsampling processing to obtain a final point cloud, wherein the final point cloud of the template image pair is the template point cloud, and the final point cloud of the subsequent monitoring image pair is the point cloud to be measured.

9. The method for monitoring deformation of a seismic isolation structure support based on binocular vision according to claim 8, characterized in that: In the step S8, the method for solving the rigid body transformation matrix calculation includes: setting an initial transformation matrix, and iteratively optimizing the transformation matrix using an iterative closest point algorithm to obtain an optimal transformation matrix; The method for setting the initial transformation matrix calculates the three-dimensional coordinates of the centroids of the template point cloud and the point cloud to be measured, and uses the displacement transformation of the two centroids as the translation matrix in the initial transformation matrix; the rotation matrix is ​​set to a rotation matrix without rotation, and the above constitutes the initial transformation matrix; The iterative closest point algorithm includes: for each point in the template point cloud, finding the corresponding point in the point cloud to be tested, using the nearest neighbor search method to find the nearest neighbor point of the point in the template point cloud, and matching them to form a corresponding point pair; optimizing the target error function Where E(R, t) is the target error function, R, t are the rotation matrix and translation matrix corresponding to the minimum error, p, q are the coordinates of the points in the template point cloud and the point cloud to be measured in the corresponding point pair, respectively. The optimal rigid body transformation is calculated by minimizing the error, and the optimal transformation matrix is ​​obtained by solving it through the optimization algorithm. The optimal transformation matrix is ​​updated by continuously iterating the calculation until the error E is less than the set threshold Y or greater than the preset maximum number of iterations, and the optimal transformation matrix that meets the conditions is returned.

10. The isolation structure support deformation monitoring system based on binocular vision is characterized by: include: An image acquisition unit, used to acquire a binocular image stream acquired by a binocular camera; The target detection unit uses the YOLOv8 target detection algorithm to detect and crop the binocular image; A stereo matching unit is used to perform stereo matching calculation on the cropped image to obtain a depth map, including initial cost calculation, cost aggregation, disparity calculation, and disparity post-processing; Point cloud processing unit, used to reversely project the depth map into three-dimensional space to obtain point cloud and perform subsequent processing, including statistical filtering denoising and uniform downsampling; The pose transformation calculation unit is used to calculate the rigid body transformation matrix using the iterative closest point algorithm to obtain the target pose transformation.

Images