Structural deformation measurement method based on geometric correction of multi-view panoramic camera

Abstract

The invention discloses a structural deformation measurement method based on geometric correction of a multi-view panoramic camera. A method for measuring bridge displacement by using a single multi-view panorama camera is provided, multi-view segmentation is performed on a panorama image according to a panorama camera equivalent model and a cube projection method, and independent calibration and correction are performed to obtain a distorted image capable of covering a 360-degree full-field range; therefore, a single panoramic camera is equivalent to a vertical four-camera system without overlapped view fields, so that the reference target in any direction can be used for displacement correction. The invention further provides a camera disturbance elimination method based on a multi-view circle center array target and pose transformation matrix purification optimization, and a coordinate system transformation matrix after purification optimization is obtained through a high-precision multi-feature point tracking algorithm from camera single-view moving target pose measurement. And obtaining a multi-view geometric correction theoretical model of any view of the measurement target and the reference target in a 360-degree range by utilizing a multi-view equivalent camera fixed connection relationship.

Description

Technical Field

[0001] The invention relates to a structural deformation measurement method based on multi-view panoramic camera geometric correction, which can realize displacement measurement of engineering structures with full field of view and wide viewing angle in complex dynamic environment, and belongs to the technical field of structural health monitoring. Background Art

[0003] From the end of the 20th century to the beginning of the 21st century, image-based structural deformation measurement technology began to develop. In 1992, Stephen and Brownjohn et al. measured the displacement of a bridge in the UK through the visual tracking system of a wafer machine and found that computer vision technology is suitable for low-frequency large displacement measurement. In 1996, Gao Jianxin et al. used monocular vision and image correlation method to study the initial deformation of concrete, demonstrating the application of early domestic image technology in deformation measurement. In the same year, He Xiaoyuan's team at Southeast University achieved long-distance displacement measurement through CCD-based technology. In 2003, Jáuregui et al. used close-range digital photography technology to measure the vertical deflection of bridges, proving its accuracy and practicality. In 2005, Zhang Guodong et al. used computer vision to track structural displacement and successfully applied it to on-site bridge measurement, laying the foundation for further research in this field. These studies not only verified the effectiveness of visual and image measurement technology, but also promoted its progress in engineering applications.

[0004] Since then, image-based deformation measurement methods have become increasingly popular due to their low cost and simple system configuration. These methods include digital image correlation (DIC), edge detection, optical flow, rotational coding matching (OCM), etc. They each have unique advantages and application scenarios, and their measurement accuracy has been verified in multiple engineering projects. The development of these visual measurement technologies has not only broadened the scope of measurement applications from micro to macro, but also improved the accuracy and convenience of measurement, especially in the field of structural health monitoring and engineering inspection, which provides strong technical support for accurately capturing and analyzing structural deformation.

[0005] However, image-based deformation measurement methods also have some limitations. First, the image matching process is sensitive to factors such as occlusion, which makes the collected images susceptible to interference. Second, when measuring structures such as bridges, it is challenging to find the right camera layout and target placement. In addition, atmospheric interference and camera micro-vibration have a significant impact on the measurement results. Effective methods need to be adopted to solve these problems.

[0006] In this field, many studies have evaluated the effects of object distance, image size, camera type, and target motion amplitude on image measurement accuracy. Ferrer et al. pointed out the variation of measurement errors in different camera types and the effects of camera motion and atmospheric distortion on measurement results. At the same time, the effects of environmental disturbances, such as vibrations caused by wind and traffic, on camera systems are also key research points for image measurement. In field measurements, ground vibrations caused by the passage of rail trains will cause certain posture disturbances to the camera, which can lead to significant errors in measurement results in severe cases. This negative impact will be further amplified when using zoom lenses and lightweight tripods.

[0007] Yoneyama and Ueda developed a method to correct the effects of camera movement using the perspective equation and verified its effectiveness through rigid body translation and rotation experiments.

[0008] Kim and Chen studied the relative displacement method based on reference points, which effectively reduces the impact of camera disturbances by subtracting the displacement of fixed reference points.

[0009] Jia et al. proposed a displacement measurement technology based on a non-stationary camera, which does not rely on the prior parameters of the camera, but instead uses geometric constraints within the field of view for projective correction.

[0010] The method developed by Olaszek et al. estimates and corrects the camera's motion by setting a fixed reference point in front of the camera. Although this method helps to improve the accuracy of the measurement, it also limits the number of monitoring points.

[0011] Wagner et al. proposed the use of image-assisted total station (IATS) to detect the absolute displacement of bridges. This method combines photogrammetry with the precise angle and distance measurement of the total station to achieve optimal performance.

[0012] Ehrhart et al. evaluated the use of IATS to measure the vibration and displacement of the structure and corrected the data by taking into account the tilt and rotation of the telescope through a multi-station measurement method to improve the measurement accuracy.

[0013] Lee et al. pointed out that environmental factors such as precipitation and wind may also affect visual system performance.

[0014] Anantrasirichai et al. proposed a method based on complex wavelet fusion to mitigate the impact of atmospheric distortion. Most of these methods face the problems of high cost and complex operation due to the need for additional sensors and data processing.

[0015] Overall, these studies on environmental disturbances not only improve the accuracy and reliability of image measurement technology, but also expand its practicality in structural health monitoring.

[0016] With the rapid development of computer vision and image processing, multi-view camera technology has become an important tool in structural monitoring and analysis. When faced with the challenge of large-scale full-field structural deformation measurement, multi-camera systems composed of multiple cameras were first applied. Multi-camera systems have become an important solution because they can obtain panoramic deformation information of a wide area of ​​the structure. In recent years, the development and application of multi-camera systems have made significant progress in the field of deformation measurement. Malesa et al. proposed two spatial data stitching strategies for multi-camera DIC systems and explored the practicality of these strategies in deformation measurement of large industrial devices.

[0017] Malowany et al. proposed a multi-camera DIC method for large engineering objects and successfully applied it to the three-dimensional deformation measurement of a structure with a span of 18 meters.

[0018] Chen et al. built a four-camera vision system and studied the local and global calibration methods related to multiple cameras, laying the foundation for the visual applications of multi-camera systems.

[0019] Yuan et al. explored the performance of the target- and calibration-based multi-camera stereo DIC method in the deformation measurement of slender components, and measured the deformation of a 900 mm long concrete beam using a system consisting of nine cameras. These studies have shown that multi-camera measurement systems can effectively measure the deformation of large-scale structures while ensuring a large field of view and high accuracy. However, this multi-camera system is complex and costly to build, and requires complex calibration work before measurement, so it is rarely used in engineering.

[0020] In contrast, multi-view cameras, especially panoramic cameras, which rely on one camera to obtain multi-view images, have shown significant efficiency and economic advantages in capturing wide scenes. In structural health detection, some scholars have also proposed the use of panoramic cameras or 360° panoramic images to overcome the limitations of traditional cameras. For example, Lenjani et al. proposed a method to use 360° panoramic images of street view services to assess the damage of buildings after disasters. This method automatically identifies and extracts building information from the panoramic images through a region-based convolutional neural network (R-CNN). Luo et al. designed a structural surface damage detection network based on panoramic images, which can handle high-resolution and highly distorted panoramic image problems that traditional deep learning networks find difficult to deal with. These studies have demonstrated that panoramic cameras can provide wider fields of view and higher-resolution images than traditional cameras, showing the potential of panoramic cameras in structural health monitoring.

[0021] However, existing deformation measurement products have limitations in terms of field of view and anti-interference. When measuring in complex environments, the camera is easily affected by disturbances, resulting in inaccurate measurements. Summary of the invention

[0022] In view of the above technical problems, the present invention proposes a structural deformation measurement method based on multi-view panoramic camera geometric correction. The method corrects the measurement technology and can effectively eliminate disturbances and improve measurement accuracy.

[0023] In order to achieve the above technical objectives, the present invention adopts the following technical means:

[0024] A method for measuring structural deformation based on multi-view panoramic camera geometric correction, comprising the following steps:

[0025] Step 1: Projection segmentation:

[0026] The panoramic camera is equivalent to multiple pinhole camera models, and the multiple pinhole camera models are combined with the cube projection of the panoramic image to obtain the orthographic projection images of the six views of front, back, left, right, top and bottom after projection segmentation;

[0027] Step 2: Independent calibration:

[0028] Independently calibrate and correct the orthographic projection images of the six views after the projection segmentation obtained in step 1, and perform subsequent processing of structural deformation measurement based on the images;

[0029] Step 3: Make the target:

[0030] After completing the projection segmentation and independent calibration of the panoramic camera, a circle center array target is made and installed on the structure as the target target and reference target for panoramic camera measurement;

[0031] Step 4: Single target pose measurement:

[0032] The target of the circle array selected in step 3 is tracked by the ellipse fitting centroid method based on sub-pixel edge extraction, and then the IPPE algorithm based on advantage purification algorithm and nonlinear optimization is used to obtain the precise position and posture of the target;

[0033] Step 5: Multi-view geometry correction:

[0034] By using the fixed connection relationship between the multiple pinhole camera models of the panoramic camera in step 1, the relative geometric relationship between the target target and the reference target in any view is obtained. Combined with the single target pose measurement algorithm in step 4, the pose change of the target target after the reference target correction is obtained.

[0035] Step 1 includes the following sub-steps:

[0036] Step 1-1, mapping the fisheye image presented by the binocular fisheye lens of the panoramic camera onto a spatial unit sphere to obtain a spatial spherical image;

[0037] Step 1-2, unfold the spatial spherical image in step 1-1 into a two-dimensional unfolded panoramic image, and then perform cube projection on the two-dimensional unfolded panoramic image to obtain orthographic projection images of six views of front, back, left, right, top and bottom after projection segmentation.

[0038] In step 2, an independent calibration based on cube projection segmentation is used. Before measurement, the projection image of each surface is assumed to be an image imaged by a pinhole camera. The projection images of the six views are calibrated using Zhang's calibration method respectively, and the calibration results are used to perform distortion correction on the projection images of the six views.

[0039] In step 3, the target is arranged with an array of N×N white circular marking points, N=2, 3, 4..., and the target is made of hard material or LED lights, and all points on the target are in the same plane.

[0040] Step 4 specifically includes the following sub-steps:

[0041] Step 4-1, by performing sub-pixel level edge extraction and ellipse fitting centroid algorithm on the center array target, high-precision target tracking of feature points is achieved;

[0042] Step 4-2: First, use the IPPE algorithm based on coplanar points to roughly calculate the target pose; then use the random sampling consistency algorithm based on subset model pre-purification to screen out outliers; and use the VVS algorithm to minimize the reprojection error. After optimization iteration, the precise target pose is obtained, that is, the precise coordinate system transformation matrix is ​​used for subsequent multi-view correction work.

[0043] In step 5, based on the relative geometric relationship between the target and the reference target in any view obtained by the fixed connection relationship between multiple pinhole camera models of the panoramic camera, the rotation angle ΔR and displacement ΔT before and after the movement of the target are obtained as follows:

[0044]

[0045] Where c1 is the measurement camera for photographing the target, c2 is the calibration camera for photographing the reference target, t1 is the time before the movement, and t2 is the time after the movement. represents the rotation matrix between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target, represents the rotation matrix between the target a and the measurement camera c1 that photographs the target, represents the rotation matrix between the reference target b and the calibration camera c2 that photographs the reference target, is the translation vector between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target, represents the translation vector between the target a and the measurement camera c1 that photographs the target, represents the translation vector between the reference target b and the calibration camera c2 that photographs the reference target, All are obtained by the single target pose measurement algorithm in step 4.

[0046] The rotation matrix between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target When the yaw angles of the views of the two cameras are 0°, 90°, 180°, and 270° respectively:

[0047]

[0048] Beneficial effects:

[0049] The structural deformation measurement method based on a multi-view panoramic camera proposed in the present invention is very different from the existing bridge displacement measurement method. It can more effectively perform safety assessment and maintenance management of bridges, improve the measurement accuracy and efficiency of structural deformation, and is a very potential new method for structural deformation monitoring. The main advantages are:

[0050] (1) In order to solve the problem that ordinary industrial cameras in traditional computer vision deformation measurement have a small field of view and incomplete information capture, making them unsuitable for large-scale structural measurement, the present invention innovatively proposes to use panoramic cameras for structural deformation measurement: after analyzing the panoramic camera imaging model, the panoramic camera is equivalent to multiple pinhole camera models, and the equivalent model is combined with the cube projection of the panoramic image to obtain six orthographic projection images after projection segmentation. The multi-view images are then independently calibrated and distorted, and subsequent processing of structural deformation measurement is performed based on them.

[0051] (2) In view of the problem that most existing measurement methods fail to measure displacement and rotation angle synchronously, the present invention uses a synchronous measurement method of displacement and rotation angle based on a circle center array target: firstly, a circle center array target and an ellipse fitting centroid method based on sub-pixel edge extraction are selected as the target and tracking algorithm, and then the IPPE algorithm based on the advantage purification algorithm and nonlinear optimization is used to obtain the precise position and posture of the target, and then the displacement and rotation angle of the target are calculated based on the measured position and posture. The feasibility and accuracy of this method are verified through experiments in multiple scenarios.

[0052] (3) Finally, in order to solve the problem of inaccurate measurement when the camera is disturbed during measurement in complex environments, a structural deformation measurement method based on multi-view panoramic camera geometric correction is proposed, combining the imaging model and posture measurement method of the panoramic camera. This method starts from the single-view perturbation correction model based on the image method, and generalizes it to the multi-view perturbation correction model after combining the multi-view images of the panoramic camera. At the same time, based on this theory, a number of performance test experiments were designed, and the experimental results verified the effectiveness and reliability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is the overall flow chart of the invention;

[0054] Figure 2 This is the equivalent imaging model of the panoramic camera of the invention;

[0055] Figure 3 The invention is based on the panoramic unfolding image of the binocular fisheye lens;

[0056] Figure 4 A cubic projection segmentation map of the panoramic image of the invention;

[0057] Figure 5 A schematic diagram of the calibration process of the calibration plate and panoramic camera of the present invention;

[0058] Figure 6 The equivalent model diagram of the panoramic camera and the vertical four-camera of the invention;

[0059] Figure 7 This is a schematic diagram of multi-view disturbance correction of the invention;

[0060] Figure 8 This is a schematic diagram of the indoor bridge scale model experiment;

[0061] Fig. 9 Schematic diagram of the arrangement of laser displacement meter and other devices;

[0062] Fig.10 This is a comparison chart of the measurement curves under the camera translation condition;

[0063] Fig.11 This is a comparison chart of the measurement curves under camera rotation conditions. DETAILED DESCRIPTION

[0064] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0065] The present invention provides a structural deformation measurement method based on multi-view panoramic camera geometric correction, which provides a new idea and a specific process framework for structural deformation measurement, such as Figure 1 As shown, it includes two parts: proposing the use of panoramic camera equivalent model and multi-view projection segmentation for independent calibration and using circle center array target for target tracking and disturbance correction.

[0066] (1) In order to solve the problem that the field of view of traditional cameras is limited and the multi-camera system is expensive and complicated in large-scale structural deformation measurement, the present invention innovatively uses multi-view panoramic cameras to measure structural deformation. In order to solve the problem that panoramic images are difficult to calibrate in deformation measurement, the present invention proposes a method for independent calibration of multiple views. First, the panoramic camera is equivalent to multiple pinhole cameras with overlapping optical centers and non-overlapping fields of view. Then, the equivalent model is combined with the cube projection of the panoramic image. After projection segmentation, six orthographic projection images are obtained. The multi-view images are independently calibrated using the Zhang Zhengyou calibration method, and the projection images of the six surfaces are distorted, completing the entire process of panoramic image expansion and dedistortion. The panoramic camera calibration example based on this proves that the calibration method has high accuracy and provides technical support for subsequent deformation measurement and geometric correction.

[0067] (2) In view of the fact that the camera system is easily affected by environmental disturbances and other factors during deformation measurement, which may lead to additional errors, the present invention proposes a structural deformation measurement method based on geometric correction of a multi-view panoramic camera. Starting from the multi-feature point pose measurement, the method combines the advantages of the image method and the multi-view panoramic camera, uses the fixed connection relationship of the panoramic camera lens and the multi-view reference target to inversely calculate the camera pose change, and generalizes the single-view disturbance correction model to the multi-view. After mathematical derivation, a disturbance correction method based on a multi-view panoramic camera is obtained, so that the reference target in any direction can be used to correct the measurement target. Finally, through the disturbance experiment simulating multiple scenes, it is verified that the correction measurement technology can effectively eliminate disturbances and improve measurement accuracy.

[0068] The purpose of the present invention is to combine the imaging model and posture measurement method of a panoramic camera to solve the problem of inaccurate measurement when the camera is disturbed during complex environment measurement. To solve the above technology, the technical solution of the present invention is as follows:

[0069] A structural deformation measurement method based on multi-view panoramic camera geometric correction, the method comprising the following steps:

[0070] Step 1: Projection segmentation:

[0071] The panoramic camera is equivalent to multiple pinhole camera models, and the multiple pinhole camera models are combined with the cube projection of the panoramic image to obtain the orthographic projection images of the six views of front, back, left, right, top and bottom after projection segmentation;

[0072] Step 2: Independent calibration:

[0073] Independently calibrate and correct the orthographic projection images of the six views after the projection segmentation obtained in step 1, and perform subsequent processing of structural deformation measurement based on the images;

[0074] Step 3: Make the target:

[0075] After completing the projection segmentation and independent calibration of the panoramic camera, a circle center array target is made and installed on the structure as the target target and reference target for panoramic camera measurement;

[0076] Step 4: Single target pose measurement:

[0077] The target of the circle array selected in step 3 is tracked by the ellipse fitting centroid method based on sub-pixel edge extraction, and then the IPPE algorithm based on advantage purification algorithm and nonlinear optimization is used to obtain the precise position and posture of the target;

[0078] Step 5: Multi-view geometry correction:

[0079] By using the fixed connection relationship between the multiple pinhole camera models of the panoramic camera in step 1, the relative geometric relationship between the target target and the reference target in any view is obtained. Combined with the single target pose measurement algorithm in step 4, the pose change of the target target after the reference target correction is obtained.

[0080] Preferably, the step 1 is dedicated to engineering applications. First, the fisheye image presented by the binocular fisheye lens of the panoramic camera is mapped onto the spatial unit sphere to obtain a spatial spherical image, and then the spatial spherical image is expanded into a flat unfolded panoramic image. Based on the panoramic image, a cube projection is performed on it, and the spherical content can be projected onto the cube model and then expanded on each surface. The principle is as follows:

[0081] In order to facilitate the understanding and use of panoramic images for structural deformation measurement, the image taken by the panoramic camera can be equivalent to the stitching of six ordinary camera images: in addition to the upper and lower cameras aimed at the top and bottom, the four ring-shaped cameras are responsible for obtaining the horizontal 360° field of view, such as Figure 2 For each camera, its imaging mode can be represented by a pinhole imaging model. Since the panoramic camera itself is very small and the distance between the two cameras integrated into one is very close, it can be assumed that the optical center of each of the two cameras coincides with the optical center of the camera as a whole. The imaging model of the camera is as follows: Figure 2 (b) as shown.

[0082] In this panoramic camera measurement model, for a three-dimensional object within the field of view, its imaging coordinate point in the panoramic image of the panoramic camera is actually the transformation from the three-dimensional rectangular coordinate system to the polar coordinate system. Assume that the radius of the polar coordinate system is r i , angle θ irepresents the angle between the line connecting the coordinate point to the pole and the 0° ray. Since the upper and lower viewing angles of the panoramic camera generally do not contain the measured object, it is simplified to only consider the plane perpendicular to the horizontal plane of the panoramic camera imaging. Then the point coordinate p i (r i ,θ i ) and the three-dimensional point p si (x i ,y i ,k i ) is:

[0083]

[0084] Among them, k i It is the ratio parameter between the plane where the point is located and the image plane.

[0085] There are many ways to display panoramic images, such as cylindrical panorama, spherical panorama, cubic panorama, strip panorama, etc. In the actual processing and application process, in order to facilitate the storage and processing of images, the spherical panorama is generally converted into a 2:1 plane unfolded panoramic image through equidistant cylindrical projection. Furthermore, the present invention divides the unfolded panoramic image into six square projection images of the view through cubic projection on this basis. In order to obtain a 2:1 plane unfolded panoramic image, a panoramic camera based on a binocular fisheye lens first obtains two fisheye images through a binocular fisheye lens, and then maps the two fisheye images into two spherical images according to the lens imaging principle, and then unfolds the spherical image through the method of longitude and latitude unfolding to obtain an unfolded panoramic image, such as Figure 3 shown.

[0086] The imaging of a fisheye lens can be described by a spherical perspective imaging model. The surface of the fisheye lens is regarded as a hemispherical surface, and the light of the scene outside the lens is projected onto the spherical surface. After the light passes through the lens, different incident light rays are projected onto the camera's photosensitive components at different positions, forming a circular fisheye image.

[0087] If you want to expand the panoramic image, you need to remap the fisheye image onto the unit sphere. First, use the variable angle scanning line method to obtain the circumference of the circular effective area in the fisheye image, and use the circle fitting algorithm to solve the center point of the effective area (x c ,y c ) and radius r c The pixels in this area can be used To express, where ρ∈[0,r c ], Fisheye image point (x f ,y f ) and the pixel point (x, y) in the image coordinate system with the origin in the upper left corner is:

[0088]

[0089] The effective area pixel point of the fisheye image and the point (x s ,y s ,z s ) is:

[0090]

[0091] Where: θ is the incident angle of the light. The relationship between θ and ρ can be obtained based on the ρ-θ data of the lens using the bilinear interpolation method to obtain all data within the range of θ∈[0,π].

[0092] Then the panoramic spherical image is expanded by the longitude and latitude of the sphere, and finally the point (x p ,y p ):

[0093]

[0094] In summary, the steps of unfolding a panoramic image based on a binocular fisheye lens are: first, map the fisheye image presented by the binocular fisheye lens onto a spatial unit sphere to obtain a spatial spherical image, and then unfold the spatial spherical image into a flat unfolded panoramic image. After obtaining the 2:1 unfolded panoramic image output by the panoramic camera, the key to processing is to analyze the projection and segmentation relationship between the panoramic image and the normal camera image from the panoramic camera imaging model, so as to convert the 2:1 panoramic image into a dedistorted normal viewing angle image.

[0095] Inside the camera, the panoramic image is transformed from a spherical coordinate system to a 2:1 panoramic image by cylindrical projection. The projection process is to divide the pixels uniformly according to the target resolution in a rectangular area with a plane aspect ratio of 2:1, and obtain a division with a length of w and a width of h. Then, the longitude and latitude are uniformly sampled on the spherical surface according to the length and width of the rectangle, and the longitude w and the latitude h are divided equally to obtain a spherical grid.

[0096] Based on the 2:1 panorama, a cube projection is performed on it. The spherical content can be projected on the cube model and then expanded on each surface, such as Figure 4 As shown, the cube projection realizes the mapping from the spherical surface to the cube face in the form of perspective.

[0097] For the projection surface A1B1BA tangent to the panoramic sphere, set the field of view FOV = 90°, the size of the projected plane is (w, h), and first estimate the normalized focal length:

[0098]

[0099] For a point (u, v) on the projection plane, the conversion to the spherical coordinate system is:

[0100]

[0101] Converted to polar coordinates (θ, r), we have:

[0102]

[0103] Then the coordinates (U, V) of the panorama are:

[0104]

[0105] After the projection is completed, the image on each face of the six-sided cube is segmented to obtain six view orthographic projection images of the same size.

[0106] Preferably, the independent calibration based on cube projection segmentation in step 2 is based on the following principle:

[0107] When calibrating and dedistorting the planar images projected and segmented by the panoramic camera, considering that the panoramic camera lens remains unchanged, the distortion caused by the camera itself also remains unchanged. Therefore, before measurement, the projection image of each surface is assumed to be an image imaged by a pinhole camera, and the projection images in six directions are calibrated using Zhang's calibration method. The calibration results are used to correct the distortion of the images of the six surfaces. In addition, since the upper and lower projection surfaces generally do not contain the objects to be measured, only the images of the four projection surfaces of front, right, back and left are calibrated here.

[0108] Use a checkerboard calibration plate to shoot multiple images in the front, right, back, and left directions of the panoramic camera. After the panoramic images are projected and segmented, use the image processing toolbox in MATLAB to calibrate the camera images in these four directions, and then perform distortion correction. The process is as follows:

[0109] 1) Make the calibration plate

[0110] A 12×9 flat black and white checkerboard calibration plate is made. The size of each square in the calibration plate is 30mm. The camera image of each view is calibrated. The calibrated camera is an Insta360 consumer-grade panoramic camera, model ONE RS 1-inch version, lens aperture is F2.2, 35mm equivalent focal length is 6.52mm. The calibration plate is as follows Figure 5 shown.

[0111] 2) Collect calibration images

[0112] First, the calibration image is collected, such as Figure 5As shown, the panoramic camera is fixed in the center of the venue, the black and white grid calibration plate changes position, direction and angle in the front, right, rear and left views of the panoramic camera, and the panoramic camera is used to capture images.

[0113] 3) Computational processing of calibration images

[0114] First, the checkerboard corner points are extracted, and the coordinates of the corner points in the image coordinate system and the world coordinate system are obtained. Then, they are used as known quantities in solving the equations, and the camera intrinsic parameters and distortion parameters are solved based on the distortion equations. The intrinsic parameters and distortion parameters are inherent properties of the camera. As long as the camera hardware does not change, the intrinsic parameters and distortion parameters will not change. After the parameters are calculated, in order to evaluate the calibration reliability, the world coordinates of the corner points are also converted back to pixel coordinates through the obtained external parameter matrix and internal parameter matrix, and compared with the extracted original corner point coordinates of the image, the reprojection error is calculated. The smaller the reprojection error, the more reliable the result.

[0115] 4) Distortion Correction

[0116] After completing the camera calibration, the distorted position of each pixel in the image can be calculated based on the camera intrinsic parameters and distortion parameters according to the camera distortion model. After calculating the distorted position of each pixel, the pixel values ​​of these positions are mapped to the corresponding positions of their original, undistorted image; at the same time, because the distorted positions often do not fall precisely on integer pixel coordinates, this pixel remapping step usually needs to be completed through interpolation methods. After completing the above steps, an image with each pixel corrected for distortion is obtained, which will be closer to the actual scene captured by the camera, thereby providing more accurate data for further image analysis and measurement tasks.

[0117] Preferably, the center array target in step 3 is generally arranged on a black square target with an array of N×N white circular marking points (N=2, 3, 4...), and these circular points provide characteristic information of color contrast and shape contour, so that they can be quickly identified and accurately located and tracked, and thus can be used to estimate the position and posture of the camera or perform three-dimensional reconstruction of spatial points. At the same time, this type of target is generally made of hard materials or light targets, and all points on the target are in the same plane, which increases the convenience of calculation.

[0118] Preferably, in step 4, firstly, by performing sub-pixel edge extraction and ellipse fitting centroid algorithm on the designed circle center array target, high-precision target tracking of feature points is achieved. Then, the IPPE algorithm based on coplanar points is used to roughly calculate the target pose; then the random sampling consensus (RANSAC) algorithm based on subset model pre-purification is used to screen out abnormal points; and the VVS algorithm is used to minimize the reprojection error, and the accurate pose of the target is obtained after optimization iteration, that is, the accurate coordinate system transformation matrix is ​​used for subsequent multi-view correction work.

[0119] Preferably, the principle and steps of the disturbance correction algorithm based on the multi-view geometric model in step 5 are as follows:

[0120] On the basis of solving the pose change and disturbance correction by single view, the panoramic camera model of the equivalent pinhole camera is added, and combined with the fixed connection relationship between the equivalent cameras, it can be extended to the case where the reference target is at any position in the field of view, that is, the disturbance correction method based on the multi-view model.

[0121] Because the upper and lower views in the six views after projection segmentation Figure 1 Generally, it does not include the measured object, so only the situation in the plane perpendicular to the horizontal plane of the panoramic camera imaging is considered, and in a clockwise direction, they are recorded as the front view, right view, rear view, and left view. Therefore, the front, right, rear, and left views divide the horizontal viewing angle of 360°, that is, the field of view of the equivalent camera of each view is 90°. At the same time, since the panoramic camera itself is very small and the distance between the two integrated lenses is very close, it can be considered that the optical center of the equivalent camera of each view of the panoramic camera coincides with the optical center of the panoramic camera as a whole. Therefore, in this case, the panoramic camera can be regarded as a vertical four-camera model with coincident optical centers, no relative movement, and fixed connection to each other, such as Figure 6 shown.

[0122] The rotation is expressed by Euler angles. The corresponding rotation angles around the x, y, and z axes in the camera system are α, β, and γ, respectively. Assume that the fixed connection relationship between the measurement camera c1 that shoots the target and the correction camera c2 that shoots the reference target is:

[0123]

[0124] Among them, represents the rotation matrix between cameras c1 and c2, is the translation vector between cameras c1 and c2, then:

[0125]

[0126] Among them, [Δμ x Δμ y Δμ z ] is the coordinate of the optical center of camera c1 in the coordinate system of camera c2. Since the optical centers of the equivalent cameras of each view coincide, The three components in are all 0; in addition, since the equivalent four vertical cameras are all in the xz plane, each camera only has a yaw angle β relative to the y axis.

[0127] In the panoramic camera cube model, assuming that the front view is the measurement camera c1, the yaw angles β of the front, right, back, and left views when used as the correction camera c2 are 0°, 90°, 180°, and 270° respectively. At this time, the rotation matrix between cameras c1 and c2 is They are:

[0128]

[0129] Based on the above theoretical foundation, the coordinate relationship between the measurement target and the reference target at different times before and after the movement is derived when the measurement target and the reference target are in any camera view. This target arrangement method is more universal and operable in engineering applications, such as Figure 7 shown.

[0130] At time t1 before the movement, let the coordinates of a point on the target a be P a , the coordinates of a point on the stationary reference target b are P b , and is the imaging point in the coordinate system of cameras c1 and c2, then according to the conversion relationship between the target coordinate system and the camera coordinate system and the fixed connection relationship between the cameras, we get:

[0131]

[0132] Combining the two equations, we can get the measurement camera With reference target P b Another layer of relationship:

[0133]

[0134] Then, combining the formulas, we can get the benchmark target P b With the target P a The relationship between:

[0135]

[0136] At this point, we have obtained the reference target P as the fixed point at time t1. b The target P before exercise a The coordinate relationship between them.

[0137] Similarly, at time t2, according to the conversion relationship between the target coordinate system and the camera coordinate system and the fixed connection relationship between the cameras, we get:

[0138]

[0139] Similarly, repeating the same derivation process as at time t1, we can finally obtain the reference target P as the fixed point at time t2. bThe measurement target P after exercise a The coordinate relationship between them is:

[0140]

[0141] Because the reference target b is stationary, P b There is no change in the target target movement and camera disturbance process. Combining the two equations, we can get the target target P at time t1 and time t2 before and after the movement. a The coordinate relationship between them is:

[0142]

[0143] If written The standard form of , after sorting, we can get:

[0144]

[0145] Finally, the rotation angle ΔR and displacement ΔT of the target before and after movement are obtained as follows:

[0146]

[0147] Among them, all R and T in the ΔR and ΔT expressions can be obtained by the target pose measurement method based on the circle center array target and the PnP algorithm. So far, this method has realized the need to eliminate camera disturbances through arbitrary multi-view benchmark target geometric correction, and finally calculated the real pose change of the measurement target that can accurately reflect the structural deformation.

[0148] As bridge structures age and pose more safety risks, high-precision bridge deformation measurement methods are particularly important. Multi-view camera systems have the potential for bridge monitoring because they can provide a wider field of view and finer spatial details.

[0149] Example

[0150] The following example of actual bridge vertical displacement measurement is used to illustrate the implementation steps of the proposed structural deformation measurement method based on multi-view panoramic camera geometric correction.

[0151] In order to deeply understand and verify the application effect and accuracy of the multi-view camera system in actual bridges, a bridge scale model deflection measurement experiment based on the multi-view camera system was first designed. Considering that bridges are affected by various dynamic and static loads in actual operation, this experiment uses a scaled model to simulate actual load conditions in a controlled environment, and then tests the measurement performance of the multi-view camera system after applying the mentioned disturbance correction method, and verifies the measurement capability and data consistency of the system under multiple views, thereby evaluating the accuracy and reliability of the multi-view camera system in actual bridge deflection measurement.

[0152] The experimental equipment for measuring the scaled model of a bridge using a multi-view camera includes a bridge model, a laser displacement meter, a center array target, a multi-view panoramic camera system, a loading trolley, and a traction rope. The bridge model is a simply supported beam, about 4 meters long; the laser displacement ranging sensor has the advantages of high precision, non-contact, and fast measurement, so the displacement data of the laser displacement meter is used as a benchmark; the camera used is the Insta360 ONE RS 1-inch panoramic camera, with a single-view resolution of 1874×1874.

[0153] like Figure 8 As shown in the figure, at the mid-span position of the bridge, a circular center array target is installed on the side of the beam along the length of the bridge as the target target, and a laser displacement meter is installed at the same position. The panoramic camera is placed 2 meters in front of the target target, and the image frame rate is 10fps. The reference target is placed 3 meters in front of the camera and 2 meters to the right, rear, and left of the camera. The reference target also uses a circular center array target, and the loading trolley is placed at one end of the bridge model, as shown in the figure. Fig. 9 shown.

[0154] After the experiment begins, the loading trolley is pulled from the starting end to the other end at a slow and uniform speed to simulate the actual dynamic situation of a heavy-loaded vehicle passing through the bridge. The target is measured and corrected using a panoramic camera measurement system and a multi-view disturbance correction method. The algorithm used according to the target selection is the sub-pixel ellipse fitting centroid method. The vertical displacement of the target represents the deflection at the bridge measurement point. The deflection measured by this method is analyzed with the results of the laser displacement meter, and compared with the deflection curve that has not been corrected by the benchmark target.

[0155] In order to simulate the mechanical vibration of the camera due to external force during use, taking the most common up and down vibration as an example, the camera is disturbed up and down along the y-axis during measurement to make the camera move horizontally. The vertical displacement of the target is measured under this working condition.

[0156] Taking the left view as an example, the comparison of the deflection curves obtained by uncalibrated single target measurement, laser displacement meter measurement, and reference target calibration is shown in Figure 2. Fig.10 As shown:

[0157] Taking the measurement results of the high-precision laser displacement meter as the benchmark, the measured deflection analysis obtained after calibration of the benchmark targets in different views is shown in Table 1:

[0158] Table 1 Analysis of measurement deflection under camera translation conditions

[0159]

[0160] From the curve comparison and data analysis table, we can see that in the case of small deformation of most structures, small displacement disturbances have a great impact on the measurement results. The maximum deflection measured by the uncalibrated single target is 20.3mm, which is nearly 7mm away from the reference value. At the same time, the deflection curve has obvious excess fluctuations and the trend is chaotic and wrong. If the camera fails to return to its original position after the disturbance and the disturbance continues to accumulate, it will have a greater negative impact on the subsequent measurement results. After the benchmark target correction of each view, the deflection curve returned to normal, the trend of the curve was completely correct and there was no abnormal fluctuation. At the same time, it also maintained high measurement accuracy, with a maximum peak error of 0.27mm and an error range of less than 2%, meeting industrial measurement requirements.

[0161] In practical applications, cameras are generally fixed by a single bracket on a relatively static platform, such as a bridge pier. This type of fixing is often subject to inclination disturbances. Although such inclination changes may be small, they will have a significant impact on measurement accuracy, especially in long-distance or high-precision measurement scenarios. Here, the most common camera deflection angle rotation in engineering practice is taken as an example. When designing the measurement, the camera is slightly rotated in the XZ plane to cause inclination disturbances. Under this condition, the vertical displacement of the target is measured.

[0162] Taking the left view as an example, the comparison of the deflection curves obtained by uncalibrated single target measurement, laser displacement meter measurement, and reference target calibration is shown in Figure 2. Fig.11 As shown:

[0163] Taking the measurement results of the high-precision laser displacement meter as the benchmark, the measured deflection analysis obtained after calibration of the benchmark targets in different views is shown in Table 2:

[0164] Table 2 Analysis of measurement deflection under camera translation conditions

[0165]

[0166] From the curve comparison and data analysis table, we can see that when the camera is disturbed in yaw angle, because the camera is not disturbed in the Y-axis direction, the peak value of the deflection curve is not much different from that of the laser displacement meter, but the shape of the deflection curve fluctuates abnormally, and the curvature is completely inconsistent with that of the laser displacement meter. After multi-view benchmark target correction, the deflection curve is completely consistent with the benchmark curve of the laser displacement rangefinder, and the measurement data is still accurate, with a maximum peak error of about 0.2mm and an error range of less than 2%.

[0167] The panoramic view of the actual bridge tested, the monitoring target, and the comparison of the displacement monitoring results of the present invention are shown above. It can be concluded that the present invention can more effectively perform safety assessment and maintenance management on bridges, improve bridge detection efficiency, and is a very potential new displacement monitoring method.

[0168] The technical means disclosed in the scheme of the present invention are not limited to the technical means disclosed in the above-mentioned implementation mode, but also include technical schemes composed of any combination of the above-mentioned technical features. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims

1. A method for measuring structural deformation based on multi-view panoramic camera geometric correction, characterized in that: The following steps are involved: Step 1: Projection segmentation: The panoramic camera is equivalent to multiple pinhole camera models, and the multiple pinhole camera models are combined with the cube projection of the panoramic image to obtain the orthographic projection images of the six views of front, back, left, right, top and bottom after projection segmentation; Step 2: Independent calibration: Independently calibrate and correct the orthographic projection images of the six views after the projection segmentation obtained in step 1, and perform subsequent processing of structural deformation measurement based on the images; Step 3: Make the target: After completing the projection segmentation and independent calibration of the panoramic camera, a circle center array target is made and installed on the structure as the target target and reference target for panoramic camera measurement; Step 4: Single target pose measurement: The target of the circle array selected in step 3 is tracked by the ellipse fitting centroid method based on sub-pixel edge extraction, and then the IPPE algorithm based on advantage purification algorithm and nonlinear optimization is used to obtain the precise position and posture of the target; Step 5: Multi-view geometry correction: By using the fixed connection relationship between the multiple pinhole camera models of the panoramic camera in step 1, the relative geometric relationship between the target target and the reference target in any view is obtained. Combined with the single target pose measurement algorithm in step 4, the pose change of the target target after the reference target correction is obtained.

2. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 1, characterized in that: Step 1 includes the following sub-steps: Step 1-1, mapping the fisheye image presented by the binocular fisheye lens of the panoramic camera onto a spatial unit sphere to obtain a spatial spherical image; Step 1-2, unfold the spatial spherical image in step 1-1 into a two-dimensional unfolded panoramic image, and then perform cube projection on the two-dimensional unfolded panoramic image to obtain orthographic projection images of six views of front, back, left, right, top and bottom after projection segmentation.

3. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 1, characterized in that: In step 2, an independent calibration based on cube projection segmentation is used. Before measurement, the projection image of each surface is assumed to be an image imaged by a pinhole camera. The projection images of the six views are calibrated using Zhang's calibration method respectively, and the calibration results are used to perform distortion correction on the projection images of the six views.

4. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 1, characterized in that: In step 3, the target is arranged with an array of N×N white circular marking points, N=2, 3, 4..., and the target is made of hard material or LED lights, and all points on the target are in the same plane.

5. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 1, characterized in that: Step 4 specifically includes the following sub-steps: Step 4-1, by performing sub-pixel level edge extraction and ellipse fitting centroid algorithm on the center array target, high-precision target tracking of feature points is achieved; Step 4-2: First, use the IPPE algorithm based on coplanar points to roughly calculate the target pose; then use the random sampling consistency algorithm based on subset model pre-purification to screen out outliers; and use the VVS algorithm to minimize the reprojection error. After optimization iteration, the precise target pose is obtained, that is, the precise coordinate system transformation matrix is ​​used for subsequent multi-view correction work.

6. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 1, characterized in that: In step 5, based on the relative geometric relationship between the target and the reference target in any view obtained by the fixed connection relationship between multiple pinhole camera models of the panoramic camera, the rotation angle ΔR and displacement ΔT before and after the movement of the target are obtained as follows: Where c1 is the measurement camera for photographing the target, c2 is the calibration camera for photographing the reference target, t1 is the time before the movement, and t2 is the time after the movement. represents the rotation matrix between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target, represents the rotation matrix between the target a and the measurement camera c1 that photographs the target, represents the rotation matrix between the reference target b and the calibration camera c2 that photographs the reference target, is the translation vector between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target, represents the translation vector between the target a and the measurement camera c1 that photographs the target, represents the translation vector between the reference target b and the calibration camera c2 that photographs the reference target, All are obtained by the single target pose measurement algorithm in step 4.

7. The method for measuring structural deformation based on multi-view panoramic camera geometric correction according to claim 6, characterized in that: The rotation matrix between the measurement camera c1 that photographs the target and the calibration camera c2 that photographs the reference target When the yaw angles of the views of the two cameras are 0°, 90°, 180°, and 270° respectively:

Images