Structural deformation monitoring method and system based on secondary stereoscopic vision calibration

Abstract

The invention belongs to the field of civil engineering structure deformation monitoring, and discloses a structure deformation monitoring method and system based on secondary stereoscopic vision calibration. Internal and external parameter calibration of the stereoscopic vision monitoring system based on secondary stereoscopic vision calibration is completed; acquiring the pixel coordinates of the initial left and right image measuring points and the size of a search area; pixel coordinates of each measuring point on the left and right images at a certain moment except the initial moment in the monitoring process are obtained, and three-dimensional coordinates of each measuring point at the moment under the structural coordinate system are obtained through calculation; according to the displacement difference value of each measuring point in the structure coordinate system at adjacent moments and the displacement difference value of the left and right camera pixel coordinate systems, abnormal point judgment is completed; if an abnormal point is found, re-positioning the abnormal point template; and repeating the steps until the monitoring is finished. The invention aims to solve the problem that a traditional camera calibration method is limited in application under the condition of a large view field on a structural health monitoring site and the problem that a tracking target is lost in the actual structure monitoring process of a digital image correlation method.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of civil engineering structure deformation monitoring, and particularly relates to a structure deformation monitoring method and system based on secondary stereo vision calibration. BACKGROUND

[0002] At present, measurement methods can be divided into contact measurement and non-contact measurement. The measurement methods such as tensioned wire displacement meter, dial gauge and linear variable differential transformer sensor are common contact measurement methods, which have high precision and high reliability, but the sensor arrangement of these methods is troublesome, low in efficiency, and greatly affected by the site and environment. When the measurement conditions are poor, such as severe deformation of components, in high-temperature or severe temperature change environment, the precision of the contact sensor is difficult to guarantee.

[0003] Non-contact measurement does not directly contact the structure, has small influence on the structure, is high in measurement efficiency, can stably work under complex conditions and provide high precision, and is becoming a hot spot concerned by scholars. The computer vision measurement method is an important non-contact measurement method. The vision measurement method can provide rich measurement information, and as a non-contact technology, has multiple advantages such as non-destructive testing, long-distance high-precision measurement, anti-electromagnetic interference, and can simultaneously monitor multiple targets in a large range. In addition, it is also extremely competitive in installation convenience, measurement range and resolution.

[0004] In recent years, the vision monitoring algorithm has made significant progress, but still faces challenges in practical application. First, the on-site calibration problem under a large field of view environment is prominent. It is difficult to arrange a calibration object under complex working conditions and under light change conditions, and the calibration precision in the laboratory is obtained through the calibration object. Finally, the stability and real-time performance of the target tracking algorithm under complex environment need to be improved, and there are problems such as template mismatching, target loss and target occlusion in the image matching process. SUMMARY

[0005] The application provides a structure deformation monitoring method based on secondary stereo vision calibration, in order to solve the problem that the traditional camera calibration method is limited in application under the large field of view condition of the structure health monitoring site when the vision monitoring method is used for structure three-dimensional deformation monitoring, and the problem of lost tracking target in the actual structure monitoring project of the digital image correlation method.

[0006] The application provides a structure deformation monitoring system based on secondary stereo vision calibration, to realize a structure deformation monitoring method based on secondary stereo vision calibration.

[0007] The application is implemented through the following technical solutions: A structure deformation monitoring method based on secondary stereo vision calibration, the method comprises the following steps: Step 1: complete the calibration of the internal and external parameters of the stereo vision monitoring system based on the secondary stereo vision calibration; Step 2: layout the measuring points and search areas on the initial left image structure monitoring area collected by the calibrated stereo vision monitoring system in step 1, and perform stereo matching on the initial right image based on the layout of the measuring points and search areas on the initial left image to obtain the pixel coordinates of the measuring points and the size of the search areas on the initial left image and the initial right image; Step 3: calculate the three-dimensional coordinates of each structural measuring point in the structural coordinate system at the initial time from the stereo vision system calibration results, and calculate the three-dimensional displacement and three-dimensional rotation angle of each measuring point in the structural coordinate system at the initial time; Step 4: obtain the pixel coordinates of each measuring point on the left and right images at a certain time during the monitoring process other than the initial time, and calculate the three-dimensional coordinates of each measuring point in the structural coordinate system at that time; According to the displacement difference of each measuring point at adjacent times in the structural coordinate system and the displacement difference of the pixel coordinate system of the left and right cameras, it is determined whether there is an abnormal point at this time; if an abnormal point is found, the pixel coordinates of each measuring point in the pixel coordinate system of the abnormal camera are repositioned at the next time; Step 5: repeat step 4 to obtain the pixel coordinates of each measuring point on the left and right images at other times during the monitoring process, and calculate the three-dimensional deformation and three-dimensional rotation angle of each measuring point in the structural coordinate system, until the monitoring is completed.

[0008] Further, step 1 specifically includes the following steps: Step 1.1: using a checkerboard calibration board, the internal parameter calibration of the stereo vision monitoring system is performed in an indoor environment using the Zhang Zhengyou calibration method; Step 1.2: using the stereo vision monitoring system with calibrated internal parameters in step 1.1, the external parameter calibration of the stereo vision system based on the essential matrix is completed.

[0009] Further, step 1.2 specifically includes the following steps: Step 1.2.1: move the tripod and stereo vision system to the structural monitoring project site, maintain the camera object distance and focal length unchanged during this process, which should be basically consistent with the object distance during indoor calibration, and simultaneously collect a pair of images of the structure to be monitored using the left and right cameras of the stereo vision system in the monitoring site; Step 1.2.2: extract the RootSIFT feature points in the structure monitoring area images collected by the left and right cameras, and complete feature matching based on the RootSIFT feature descriptors; Step 1.2.3: set the internal point threshold and iteration number of the RANSAC model and the MSAC model, and use the RAN-MSAC combined algorithm to remove the mismatched feature point pairs in the structure monitoring area images collected by the left and right cameras; Step 1.2.4: Use the stereo matching image point pairs retained after eliminating the image mismatching point pairs to calculate the basic matrix between the left and right cameras, and use the basic matrix The left and right camera intrinsic parameter matrix K obtained in step 1 l , K r , calculate the essential matrix E; Step 1.2.5: Decompose the essential matrix E to obtain the rotation matrix R and the translation vector T0 with a modulus of 1 in the extrinsic parameters of the stereo vision system; Step 1.2.6: Mark a reference object of known scale in the structural images captured by the left and right cameras at the monitoring site l , calculate the known scale length when the translation vector modulus is 1 l 0, calculate the translation vector scale factor based on the reference object scale s , using the scale factor s The translation vector T is calculated by summing the translation vector T0 with a modulus of 1 to complete the calibration of the external parameters of the stereo vision system camera.

[0010] Furthermore, the step 3 specifically includes the following steps: Step 3.1: Use the checkerboard calibration plate in step 1 to place it horizontally and vertically close to the surface of the monitored structure. Establish a structural coordinate system based on the position of the checkerboard calibration plate at that moment. The plane where the checkerboard calibration plate is located is the structural reference surface. Step 3.2: Measure the coordinates of the points in the image coordinate system based on the left image ( X l , Y l ) and the coordinates of the right image point in the image coordinate system ( X r , Y r ), using the stereo vision mathematical model formula, obtain the three-dimensional coordinates of all structural measurement points in the left camera coordinate system; Step 3.3: Convert the three-dimensional coordinates of the structural measurement points in the left camera coordinate system to the structural coordinate system established in step 3.1, and finally obtain the three-dimensional coordinates P of each structural measurement point in the structural coordinate system; Step 3.4: After obtaining the three-dimensional coordinates of each measuring point in the structural coordinate system, calculate the three-dimensional coordinates and three-dimensional rotation angle of each measuring point.

[0011] Furthermore, the stereoscopic vision mathematical model formula of step 3.2 is specifically as follows:

[0012] Where, ( x l 、 y l、z l ) is the three-dimensional coordinate of the measuring point in the left camera coordinate system, f l 、 f r is the normalized focal length of the left and right cameras, t x 、 t y 、 t z and r 1, r 2, …, r 9 are elements corresponding to the translation vector and rotation matrix of the stereo vision system external parameters; Step 3.3 is specifically that the three-dimensional coordinates of each structural measuring point in the structural coordinate system P are specifically formulaed as:

[0013] In the formula, X x l , y l , z l is the three-dimensional coordinate of the structural measuring point in the left camera coordinate system, R ls , T ls is the rotation matrix and translation vector from the left camera coordinate system to the structural coordinate system.

[0014] Further, step 3.4 is specifically that three measuring points, i.e. measuring point 1, measuring point 2 and measuring point 3, which are not collinear on the structural surface, are selected, the initial three-dimensional coordinates of the three measuring points are , the three-dimensional coordinates of the three measuring points at a certain time during the structural deformation are ; taking measuring point 2 as the rotation center, the three-dimensional displacement translation vector T s and the rotation matrix R s of the three-dimensional rotation angle of the structure are calculated from the three measuring points.

[0015] Further, the following is obtained:

[0016] From the above formula, the rotation matrix R s of the structure as a whole can be solved, and the rotation matrix R s is decomposed into the rotation angle around the x-axis, the rotation angle around the y-axis and the rotation angle around the z-axis, thereby obtaining the three-dimensional deformation monitoring result of the structure.

[0017] Further, the step 4 specifically comprises the following steps: Step 4.1: calculate the displacement difference of each measuring point at adjacent time in the structure coordinate system, and complete the abnormal point judgment based on the standard deviation according to the displacement difference of each measuring point; and according to the pixel displacement difference of each measuring point at adjacent time in the left and right camera pixel coordinate system, it is judged that the abnormal point is caused by the image mismatch of the left camera or the right camera; Step 4.2: if an abnormal point is found, then use the multi-scale template positioning method to reposition the pixel coordinates of each measuring point in the pixel coordinate system of the abnormal camera at the next time.

[0018] Further, the step 4.1 specifically comprises the following steps: Step 4.1.1: calculate the three-dimensional coordinates of each measuring point at different times in the structure coordinate system; Step 4.1.2: calculate the three-dimensional coordinate displacement difference of each measuring point at adjacent time, and the adjacent time refers to each time and its previous time; Step 4.1.3: calculate the average and standard deviation of the displacement difference of each measuring point, and use the average plus or minus 3 times the standard deviation of the displacement difference of each measuring point as the judgment interval to judge whether the displacement difference of each measuring point at the current time is within the interval, if the displacement difference of any measuring point at the latest time is not within the interval, then judge that the time t n is an abnormal time point; Step 4.1.4: if an abnormal point is found, then calculate the pixel coordinate displacement difference of each measuring point at adjacent time in the left and right pixel coordinate system from the initial time to t n ; Step 4.1.5: calculate the average and standard deviation of the pixel displacement difference of each measuring point in the left and right pixel coordinate system, and use the method of step 4.1.3 to judge whether the abnormal time t n is caused by the left camera or the right camera, and the camera with the abnormal point is called the disturbed camera.

[0019] Further, the step 4.2 specifically comprises the following steps: Step 4.2.1: use the digital image correlation method to use t n the image collected by the undisturbed camera at the time to perform time sequence matching to locate t n+1 the pixel coordinates of each measuring point in the image of the undisturbed camera at the time; Step 4.2.2: set the template size expansion times k ; Step 4.2.3: expand t n+1The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area k The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area t n+1 The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area k The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area Step 4.2.4: Reduction t n+1 The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area k The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area t n+1 The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area k The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area k The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area Step 4.2.5: Gradual reduction t n+1 The template size of each measuring point in the camera not disturbed at the moment is 1 times the size of the search area

[0020] A structure deformation monitoring system based on secondary stereo vision calibration, which uses the structure deformation monitoring method based on secondary stereo vision calibration as described above, and the system comprises: A camera calibration module of the stereo vision monitoring system: calibration of the internal and external parameters of the stereo vision monitoring system; An initial data acquisition and parameter setting module: measuring points and search areas are arranged in the initial left image structure monitoring area, and the measuring points and search areas arranged based on the initial left image are used to perform stereo matching on the initial right image to obtain the pixel coordinates of the measuring points on the initial left image and the initial right image and the size of the search area; Three-dimensional coordinates of each structure measuring point in the structure coordinate system at the initial moment are calculated from the stereo vision system calibration results, and three-dimensional displacement and three-dimensional rotation angle of each measuring point in the structure coordinate system at the initial moment are calculated; A monitoring module: pixel coordinates of each measuring point on the left and right images at a moment other than the initial moment in the monitoring process are obtained, and three-dimensional coordinates of each measuring point in the structure coordinate system at the moment are calculated; It is judged whether there is an abnormal point at this moment; if an abnormal point is found, the pixel coordinates of each measuring point in the pixel coordinate system of the abnormal camera at the next moment are repositioned; Pixel coordinates of each measuring point on the left and right images at other moments in the monitoring process are obtained, and three-dimensional deformation and three-dimensional rotation angle of each measuring point in the structure coordinate system are calculated until the monitoring ends.

[0021] The beneficial effects of the present application are: The present application uses a stereovision camera secondary calibration method to complete the secondary calibration of the stereovision camera, and solves the problem of lack of a fast and convenient calibration method for structure monitoring under a large field of view.

[0022] The present application solves the problem of target loss and target occlusion in structure static monitoring, and effectively completes the visual monitoring of six degrees of freedom motion of the structure.

[0023] The present application realizes the identification and tracking of the target measuring point of the visual monitoring system, obtains the sub-pixel coordinates of the measuring point center, and meets the demand of high-precision measurement. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 It is the software interface diagram of the stereovision camera secondary calibration of the present application.

[0025] Figure 2 It is the software flowchart of the stereovision camera secondary calibration of the present application.

[0026] Figure 3 It is the flowchart of the RAN-MSAC combined algorithm used by the present application.

[0027] Figure 4 It is the software interface diagram of the structure three-dimensional displacement and angle monitoring based on targetless vision of the present application.

[0028] Figure 5 It is the software flowchart of the structure three-dimensional displacement and angle monitoring based on targetless vision of the present application.

[0029] Figure 6 It is the flowchart of the target loss repositioning method used by the present application.

[0030] Figure 7 It is the visual system layout in the example of the structure deformation monitoring method based on secondary stereovision calibration of the present application.

[0031] Figure 8 It is the model size diagram in the example of the structure deformation monitoring method based on secondary stereovision calibration of the present application. Figure 9 It is the physical diagram in the example of the structure deformation monitoring method based on secondary stereovision calibration of the present application.

[0032] Figure 10 It is the measuring point displacement curve diagram calculated in the example of the structure deformation monitoring method based on secondary stereovision calibration of the present application, wherein (a) is the measuring point 1 displacement curve diagram, (b) is the measuring point 2 displacement curve diagram, and (c) is the measuring point 3 displacement curve diagram. DETAILED DESCRIPTION

[0033] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0034] It is to be understood that the terminology "including", when used in the present specification and in the following claims, indicates the presence of the described features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0035] It is also to be understood that the terminology used in the present specification and the appended claims is for the purpose of describing the particular embodiments only and is not intended to be limiting.

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0037] In the following description, a large number of specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced in other manners different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0038] Embodiment one The embodiment discloses a structure deformation monitoring method based on secondary stereo vision calibration, as shown in the figure, the structure deformation monitoring method comprises the following steps: Figure 1 Step 1: complete the stereo vision monitoring system internal parameter calibration based on secondary stereo vision calibration, specifically, use the checkerboard calibration board, and use Zhang Zhengyou calibration method for stereo vision monitoring system internal parameter calibration in indoor environment; Step 2: using the same set of stereo vision monitoring system, complete the stereo vision system external parameter calibration based on essential matrix in the structure monitoring project site; Step 2: using the same set of stereo vision monitoring system, complete the stereo vision system external parameter calibration based on essential matrix in the structure monitoring project site; Step 3: Set up the measurement points and search area for the initial left image structure monitoring area, and perform stereo matching on the initial right image to obtain the pixel coordinates of the measurement points and the size of the search area of the initial left image and the initial right image; Step 4: Calculate the three-dimensional coordinates of each structure measurement point in the structure coordinate system at the initial time from the stereo vision system calibration results, and calculate the three-dimensional displacement and three-dimensional rotation angle of each measurement point in the structure coordinate system at the initial time; Step 5: Obtain the pixel coordinates of each measurement point on the left and right images at a certain time during the monitoring process (other than the initial time), and calculate the three-dimensional coordinates of each measurement point in the structure coordinate system at that time; Step 6: Calculate the displacement difference of each measurement point at adjacent times in the structure coordinate system, and complete the abnormal point judgment based on the standard deviation according to the displacement difference of each measurement point. According to the pixel displacement difference of each measurement point at adjacent times in the left and right camera pixel coordinate systems, it is determined whether the abnormal point is caused by image mismatch of the left camera or the right camera; Step 7: If an abnormal point is found, use the multi-scale template positioning method to reposition the pixel coordinates of each measurement point in the pixel coordinate system of the abnormal camera at the next time; Step 8: Repeat steps 5-7 to obtain the pixel coordinates of each measurement point on the left and right images at other times during the monitoring process, and calculate the three-dimensional deformation and three-dimensional rotation angle of each measurement point in the structure coordinate system, until the monitoring ends.

[0039] Step 2.1: Move the tripod and stereo vision system to the structure monitoring site, and keep the camera object distance and focal length unchanged during this process. The object distance should be basically the same as that during indoor calibration. Use the stereo vision system to simultaneously collect a pair of images of the structure monitoring area using the left and right cameras; Step 2.2: Extract the RootSIFT feature points in the structure monitoring area images collected by the left and right cameras, and complete feature matching based on the RootSIFT feature descriptors; Step 2.3: Set the inlier threshold and iteration number of the RANSAC model and the MSAC model, and use the RAN-MSAC combined algorithm to remove the mismatched feature point pairs in the structure monitoring area images collected by the left and right cameras; Step 2.4: Calculate the fundamental matrix between the left and right cameras using the stereo matching image point pairs retained after removing the image mismatch points , and calculate the essential matrix E using the fundamental matrix and the left and right camera intrinsic parameter matrices K l , K r obtained in step 1; Step 2.5: Decompose the essential matrix E to obtain the rotation matrix R and the translation vector T0 with a length of 1 in the stereo vision system external parameters; Step 2.6: Label a known scale reference in the structure image collected by the left and right cameras on site l , calculate the length of the known scale when the translation vector module is 1 l 0, calculate the scale factor of the translation vector according to the scale of the reference s , calculate the translation vector T using the scale factor s and the translation vector T0 with a module of 1, complete the camera extrinsic parameter calibration of the stereo vision system; The software interface for secondary calibration of stereo vision cameras is shown in Figure 1 . Using the camera secondary calibration software to control the camera can realize the functions of calibration image acquisition, Zhang Zhengyou calibration method intrinsic parameter calibration, and on-site extrinsic parameter calculation.

[0040] The software flow of the secondary calibration of the stereo vision camera is shown in Figure 2 . According to the calibration needs, the secondary calibration software of the stereo vision camera is mainly divided into four modules, which are image setting module, camera calibration module, feature matching module and calibration calculation module. Each module has corresponding operation buttons and input boxes, and the user can input the corresponding parameters according to the actual conditions to obtain the secondary calibration results of the stereo vision camera.

[0041] The flow of the RAN-MSAC combined algorithm is shown in Figure 3 . According to the actual need of eliminating mis-matched feature points, the functions of RANSAC algorithm to eliminate outliers and MSAC algorithm to calculate fundamental matrix are completed.

[0042] Further, the specific formula of step 2.4 is:

[0043] wherein, F 11 to F 33 is the corresponding position element in the fundamental matrix ; u and v are the pixel coordinates of any homonymous point pair p l ( u l , v l ) and p r ( u r , v r ) in the left and right images.

[0044] Further, step 2.5 specifically includes the following steps: Step 2.5.1: Decompose the essential matrix E=USV through singular value decomposition (SVD) T , U, S, and V are the three corresponding matrices after SVD decomposition; Step 2.5.2: Use the U, S, and V matrices after the essential matrix SVD decomposition to calculate the left and right camera extrinsic parameter combinations. There are four possible solutions, which are: ( T1, R1), ( T1, R2), ( T2, R1) and ( T2, R2), the specific formula of step 2.5.2 is shown in formula 4; Step 2.5.3: Calculate the three-dimensional coordinates of any pair of matching points in the left camera coordinate system and the right camera coordinate system, and select the point that meets the condition "the left camera is to the left of the right camera, that is, the horizontal component t of the translation vector T x is a negative value”, “the left camera A is on the left side of the right camera B”, the external parameter combination is the rotation matrix R and the translation vector T0 with a modulus of 1 in the external parameters of the stereo vision system;

[0045] Furthermore, the specific process of step 2.6 is as follows: Step 2.6.1: From the intrinsic parameter matrix of the left camera , the intrinsic parameter matrix of the right camera , the rotation matrix R and the translation vector T with a modulus of 1 0， Given the pixel coordinates of the start and end points of the scale reference object, calculate the three-dimensional coordinates of the two points in the left camera coordinate system 、 ; Step 2.6.2: From the known scale reference object's start and end coordinates 、 , find the distance between the two points when the translation vector T0 modulus is 1 l 0:

[0046] Step 2.6.3: The distance between the two points when the translation vector T0 has a modulus of 1 l 0 and a known scale reference l , calculate the translation vector T scale factor s for:

[0047] Step 2.6.4: Scale factor of translation vector T s And the translation vector T0 with a modulus of 1, calculate the translation vector T:

[0048] Further, the specific process of step 4 is: Step 4.1: The checkerboard calibration plate is placed on the surface of the monitored structure in a horizontal and vertical manner, and a structure coordinate system is established based on the pose of the checkerboard calibration plate at this moment. The plane where the checkerboard calibration plate is located is the structure reference surface. Step 4.2: According to step 3, the coordinates of the left image measurement points in the image coordinate system are obtained ( X l , Y l ) and the coordinates of the right image measurement points in the image coordinate system are obtained ( X r , Y r ). Using the stereo vision mathematical model formula (8), the three-dimensional coordinates of all structure measurement points in the left camera coordinate system are obtained.

[0049] In the formula, ( x l 、 y l 、z l) is the three-dimensional coordinate of the measurement point in the left camera coordinate system, f l 、 f r is the normalized focal length of the left and right cameras, t x 、 t y 、 t z and r 1、 r 2, …, r 9 are the elements corresponding to the position in the translation vector and rotation matrix of the extrinsic parameters of the stereo vision system; Step 4.3: Convert the three-dimensional coordinates of the structure measurement points in the left camera coordinate system to the structure coordinate system established in step 4.1, and finally obtain the three-dimensional coordinates P of each structure measurement point in the structure coordinate system. The specific formula used is:

[0050] In the formula, X( x l , y l , z l ) is the three-dimensional coordinate of the structure measurement point in the left camera coordinate system, and Rls 、T ls The rotation matrix and translation vector from the left camera coordinate system to the structure coordinate system; Step 4.4: After obtaining the three-dimensional coordinates of each measuring point in the structural coordinate system, calculate the three-dimensional coordinates of each measuring point. Select three measuring points that are not collinear on the surface of the structure, namely measuring point 1, measuring point 2, and measuring point 3. The initial three-dimensional coordinates of the three measuring points are , the three-dimensional coordinates at a certain moment in the structural deformation process are ; Taking measuring point 2 as the rotation center, the three-dimensional rotation angle of the structure is calculated from these three measuring points, and the three-dimensional displacement translation vector T of each measuring point is calculated s The rotation matrix R of the overall three-dimensional rotation angle of the structure s The specific formula is:

[0051] Then we get:

[0052] The above formula can be used to solve the rotation matrix R of the entire structure s , the rotation matrix R s Decomposed into rotation angle around the x-axis , rotation angle around the y-axis and the angle around the z axis , thus obtaining the three-dimensional deformation monitoring results of the structure.

[0053] The software interface for structural three-dimensional displacement and rotation angle monitoring based on non-target vision is as follows: Figure 4 As shown in the figure, the monitoring soft-control camera is used to complete the stereo vision system camera calibration, template matching, structural surface construction and structural six-degree-of-freedom calculation, and obtain the absolute displacement, relative displacement and overall rotation angle information of multiple non-target measurement points on the surface of the structure.

[0054] The software process of structural three-dimensional displacement and rotation angle monitoring based on non-target vision is as follows Figure 5 As shown in the figure, the targetless vision-based 3D displacement and rotation monitoring software is divided into six modules based on monitoring needs: image display, displacement display, parameter setting, camera calibration, graphics processing, and display settings. Each module has corresponding operation buttons and input boxes. Users enter the corresponding parameters according to actual conditions to obtain the six-motion-of-freedom measurement results.

[0055] Furthermore, the specific process of step 6 is as follows: Step 6.1: Use the method in step 4 to calculate the three-dimensional coordinates of each measuring point in the structural coordinate system at different times; Step 6.2: Calculate the three-dimensional coordinate displacement difference of each measuring point at adjacent moments. Adjacent moments refer to each moment and its previous moment. Step 6.3: Calculate the mean and standard deviation of the displacement difference of each measuring point. Take the mean of the displacement difference of each measuring point plus or minus 3 times the standard deviation as the judgment interval to determine whether the displacement difference of each measuring point at the current moment is within the interval. If the displacement difference of any measuring point at the latest moment is not within the interval, then determine whether the displacement difference of the measuring point at the current moment is within the interval. t n is the abnormal time point; Step 6.4: If an abnormal point is found, calculate the time from the initial time to the t n The pixel coordinate displacement difference between adjacent moments in the left and right pixel coordinate systems; Step 6.5: Calculate the mean and standard deviation of the pixel displacement difference of each measuring point in the left and right pixel coordinate systems respectively, and use the method in step 6.3 to determine the abnormal moment t n Is it caused by the abnormality in the left camera or the right camera? The camera with the abnormal point is called the disturbed camera; Furthermore, the specific process of step 7 is as follows: Step 7.1: Using digital image correlation, use t n The images captured by the camera without being disturbed are matched in time sequence to locate the t n+1 The pixel coordinates of each measuring point in the camera image that is not disturbed at that moment; Step 7.2: Set the number of template size expansions k ; Step 7.3: Expand t n+1 The template size of each measurement point in the camera that is not disturbed at any moment is the size of the search area k times, use digital image correlation method to select the expanded template for stereo matching, and obtain t n+1 The measurement points in the camera image are disturbed at all times k Times search area template position; Step 7.4: Zoom out t n+1 The template size of each measurement point in the camera that is not disturbed at any moment is the original search area k -1 times, use digital image correlation method to select the reduced template for stereo matching, and obtain t n+1 The measurement points in the camera image are disturbed at all times k times in the search area k -1 times the search area template position; Step 7.5: Zoom out step by step t n+1The template size of each measuring point in the camera which is not disturbed at the moment is not disturbed, step 7.4 is repeated until the template size is consistent with the original template size, and the multi-scale template repositioning is completed.

[0056] The target loss repositioning method flow chart is shown in Figure 6 The target loss repositioning method flow chart is shown in

[0057] The embodiment is a slope deformation monitoring simulation experiment, which is prepared according to the following steps: As shown in Figure 7 A stereovision system composed of two BASLER acA2040-120um cameras (resolution of 2064pixel×1544pixel) and a ThinkPad P53 mobile graphics workstation is used to monitor the vibration stereovision of the model of the rusted concrete frame under the action of the earthquake. The vibration table surface is 5m×5m in size, and the model plane size is 2.4m×2.4m. The model is a double-layer reinforced concrete frame structure, each layer is 1.65m high, and the total height is 3.3m. The width of the structure beam and column is 0.2m. The size of the model is shown in Figure 8 As shown in Figure 9 The base distance between the left and right cameras of the stereovision system is 300mm, the left and right cameras are selected with a 5mm lens, the camera height is 1410mm, the closest end of the stereovision system to the specimen is 3900mm, and the sampling frequency of the stereovision system is 60Hz.

[0058] In the experiment, the waveforms input by the vibration table are controlled, the displacement of each degree of freedom of the model is measured by using the visual monitoring method of the tensioned displacement meter and Zhang Zhengyou calibration method, and the structure deformation monitoring method based on the secondary stereovision calibration. The upper circular target in each degree of freedom of the model is taken as a measuring point, and the measuring points are sequentially arranged from the top layer to the base as measuring point 1, measuring point 2 and measuring point 3. The displacement curves obtained by different methods are shown in Figure 10 As a whole, under the test conditions, the method has high effectiveness in the field of civil engineering structure deformation monitoring.

[0059] Embodiment two The embodiment provides a structure deformation monitoring system based on secondary stereovision calibration, which uses the structure deformation monitoring method based on secondary stereovision calibration as described in embodiment one, and the system comprises: A parameter calibration module of the stereovision monitoring system: calibration of the internal and external parameters of the stereovision monitoring system; An initial data acquisition module: the initial left image structure monitoring area is laid out with measuring points and search areas, and the initial right image is stereomatched based on the measuring points and search areas laid out by the left image, to acquire pixel coordinates of the initial left image and the initial right image measuring points and search area size; The three-dimensional coordinates of each structure measuring point in the structure coordinate system at the initial moment are calculated from the stereovision system calibration results, and the three-dimensional displacement and three-dimensional rotation angle of each measuring point in the structure coordinate system at the initial moment are calculated; A monitoring module: the pixel coordinates of each measuring point on the left and right images at a certain moment in the monitoring process except the initial moment are obtained, and the three-dimensional coordinates of each measuring point in the structure coordinate system at the moment are calculated; According to the displacement difference of each measuring point at adjacent moments in the structure coordinate system and the displacement difference of the left and right camera pixel coordinate systems, it is determined whether there is an abnormal point at the moment; if an abnormal point is found, a multi-scale template positioning method is used to reposition the pixel coordinates of each measuring point in the pixel coordinate system of the abnormal camera at the next moment; The pixel coordinates of each measuring point on the left and right images at other moments in the monitoring process are obtained, and the three-dimensional deformation and three-dimensional rotation angle of each measuring point in the structure coordinate system are calculated, until the monitoring ends.

Claims

1. A structural deformation monitoring method based on secondary stereo vision calibration, characterized in that: The method comprises the following steps: Step 1: Complete the calibration of the intrinsic and extrinsic parameters of the stereo vision monitoring system based on the secondary stereo vision calibration; Step 2: Layout measurement points and search areas for the structural monitoring area of ​​the initial left image collected by the stereo vision monitoring system after calibration in step 1, and perform stereo matching on the initial right image based on the measurement points and search area layout of the initial left image to obtain the pixel coordinates of the measurement points and the search area size on the initial left image and the initial right image; Step 3: Calculate the three-dimensional coordinates of each structural measurement point in the structural coordinate system at the initial moment based on the calibration results of the stereo vision system, and calculate the three-dimensional displacement and three-dimensional rotation angle of each measurement point in the structural coordinate system at the initial moment; Step 4: Obtain the pixel coordinates of each measuring point on the left and right images at a certain moment other than the initial moment during the monitoring process, and calculate the three-dimensional coordinates of each measuring point in the structural coordinate system at that moment; Based on the displacement difference of each measuring point in the structural coordinate system at adjacent moments and the displacement difference of the left and right camera pixel coordinate systems, it is determined whether there is an abnormal point at this moment; if an abnormal point is found, the pixel coordinates of each measuring point in the pixel coordinate system of the abnormal camera are relocated at the next moment; Step 5: Repeat step 4 to obtain the pixel coordinates of each measuring point on the left and right images at other times during the monitoring process, and calculate the 3D deformation and 3D rotation angle of each measuring point in the structural coordinate system until the monitoring is completed.

2. The method according to claim 1, characterized in that The step 1 specifically includes the following steps: Step 1.1: Use the checkerboard calibration plate and Zhang Zhengyou calibration method to calibrate the internal parameters of the stereo vision monitoring system in an indoor environment; Step 1.2: On site, use the stereo vision monitoring system after the intrinsic parameters are calibrated in step 1.1 to complete the calibration of the extrinsic parameters of the stereo vision system based on the essential matrix.

3. The method according to claim 2, characterized in that The step 1.2 specifically includes the following steps: Step 1.2.1: Move the tripod and stereo vision system to the structural monitoring project site. During this process, keep the camera's object distance and focal length unchanged, which should be basically the same as the object distance during indoor calibration. At the monitoring site, use the left and right cameras of the stereo vision system to simultaneously capture a pair of images of the structure area to be monitored. Step 1.2.2: Extract RootSIFT feature points from the images of the structure area to be monitored captured by the left and right cameras at the structure monitoring site, and perform feature matching based on the RootSIFT feature descriptor; Step 1.2.3: Set the inlier threshold and number of iterations for the RANSAC model and the MSAC model, and use the RAN-MSAC combined algorithm to remove mismatched feature point pairs in the structure monitoring area images captured by the left and right cameras; Step 1.2.4: Use the stereo matching image point pairs retained after eliminating the image mismatching point pairs to calculate the fundamental matrix between the left and right cameras , using the fundamental matrix The left and right camera intrinsic parameter matrix K obtained in step 1 l , K r , calculate the essential matrix E; Step 1.2.5: Decompose the essential matrix E to obtain the rotation matrix R and the translation vector T0 with a modulus of 1 in the extrinsic parameters of the stereo vision system; Step 1.2.6: Mark a reference object of known scale in the structural images captured by the left and right cameras at the monitoring site l , calculate the known scale length when the translation vector modulus is 1 l 0, calculate the translation vector scale factor based on the reference object scale s , using the scale factor s The translation vector T is calculated by summing the translation vector T0 with a modulus of 1 to complete the calibration of the external parameters of the stereo vision system camera.

4. The method according to claim 1, characterized in that The step 3 specifically includes the following steps: Step 3.1: Use the checkerboard calibration plate in step 1 to place it horizontally and vertically close to the surface of the monitored structure. Establish a structural coordinate system based on the position of the checkerboard calibration plate at that moment. The plane where the checkerboard calibration plate is located is the structural reference surface. Step 3.2: Measure the coordinates of the points in the image coordinate system based on the left image ( X l , Y l ) and the coordinates of the right image point in the image coordinate system ( X r , Y r ), using the stereo vision mathematical model formula, obtain the three-dimensional coordinates of all structural measurement points in the left camera coordinate system; Step 3.3: Convert the three-dimensional coordinates of the structural measurement points in the left camera coordinate system to the structural coordinate system established in step 3.1, and finally obtain the three-dimensional coordinates P of each structural measurement point in the structural coordinate system; Step 3.4: After obtaining the three-dimensional coordinates of each measuring point in the structural coordinate system, calculate the three-dimensional coordinates and three-dimensional rotation angle of each measuring point.

5. The method according to claim 4, characterized in that: The stereoscopic vision mathematical model formula of step 3.2 is specifically: Where, ( x l 、 y l 、z l ) is the three-dimensional coordinate of the measuring point in the left camera coordinate system, f l 、 f r is the normalized focal length of the left and right cameras, t x 、 t y 、 t z and r 1. r 2, …, r 9 is the translation vector of the external parameters of the stereo vision system and the element at the corresponding position in the rotation matrix; Specifically, step 3.3 is as follows: the specific formula for the three-dimensional coordinate P of each structural measurement point in the structural coordinate system is: Where, X( x l , y l , z l ) is the three-dimensional coordinate of the structure measurement point in the left camera coordinate system, R ls 、T ls The rotation matrix and translation vector from the left camera coordinate system to the structure coordinate system.

6. The method according to claim 5, characterized in that Specifically, step 3.4 is to select three non-collinear measuring points on the surface of the structure, namely measuring point 1, measuring point 2, and measuring point 3. The initial three-dimensional coordinates of the three measuring points are , the three-dimensional coordinates at a certain moment in the structural deformation process are ; Taking measuring point 2 as the rotation center, the three-dimensional rotation angle of the structure is calculated from these three measuring points, and the three-dimensional displacement translation vector T of each measuring point is calculated s The rotation matrix R of the overall three-dimensional rotation angle of the structure s The specific formula is: Then we get: The above formula can be used to solve the rotation matrix R of the entire structure s , the rotation matrix R s Decomposed into rotation angle around the x-axis , rotation angle around the y-axis and the angle around the z axis , thus obtaining the three-dimensional deformation monitoring results of the structure.

7. The method according to claim 1, characterized in that The step 4 specifically includes the following steps: Step 4.1: Calculate the displacement difference of each measuring point in the structural coordinate system at adjacent moments. Use this displacement difference to determine the outliers based on the standard deviation. Also, use the pixel displacement difference of each measuring point in the pixel coordinate systems of the left and right cameras at adjacent moments to determine whether the outliers are caused by image mismatching with the left or right camera. Step 4.2: If an abnormal point is found, the multi-scale template positioning method is used to relocate the pixel coordinates of each measuring point in the pixel coordinate system where the abnormal camera is located at the next moment.

8. The method according to claim 7, characterized in that: The step 4.1 specifically includes the following steps: Step 4.1.1: Calculate the three-dimensional coordinates of each measuring point in the structural coordinate system at different times; Step 4.1.2: Calculate the 3D coordinate displacement difference of each measuring point at adjacent moments. Adjacent moments refer to each moment and its previous moment. Step 4.1.3: Calculate the mean and standard deviation of the displacement difference of each measuring point. Take the mean of the displacement difference of each measuring point plus or minus 3 times the standard deviation as the judgment interval. Judge whether the displacement difference of each measuring point at the current moment is within the interval. If the displacement difference of any measuring point at the latest moment is not within the interval, judge whether the displacement difference of the measuring point at the current moment is within the interval. t n is the abnormal time point; Step 4.1.4: If an abnormal point is found, calculate the time from the initial time to the t n The pixel coordinate displacement difference between adjacent moments in the left and right pixel coordinate systems; Step 4.1.5: Calculate the mean and standard deviation of the pixel displacement difference of each measuring point in the left and right pixel coordinate systems respectively, and use the method in step 4.1.3 to determine the abnormal moment. t n Is it caused by the abnormality in the left camera or the right camera? The camera with the abnormal point is called the disturbed camera.

9. The method according to claim 7, characterized in that: The step 4.2 specifically includes the following steps: Step 4.2.1: Using digital image correlation, use t n The images captured by the camera without being disturbed are matched in time sequence to locate the t n+1 The pixel coordinates of each measuring point in the camera image that is not disturbed at that moment; Step 4.2.2: Set the number of template size expansions k ; Step 4.2.3: Expand t n+1 The template size of each measurement point in the camera that is not disturbed at any moment is the size of the search area k times, use digital image correlation method to select the expanded template for stereo matching, and obtain t n+1 The measurement points in the camera image are disturbed at all times k Double search area template position; Step 4.2.4: Zoom out t n+1 The template size of each measurement point in the camera that is not disturbed at any moment is the original search area k -1 times, use digital image correlation method to select the reduced template for stereo matching, and obtain t n+1 The measurement points in the camera image are disturbed at all times k times in the search area k -1 times the search area template position; Step 4.2.5: Reduce step by step t n+1 Repeat step 4.2.4 until the template size is consistent with the original template size, completing the multi-scale template repositioning.

10. A structural deformation monitoring system based on secondary stereo vision calibration, characterized in that: The system uses the structural deformation monitoring method based on secondary stereo vision calibration as described in any one of claims 1 to 9, and the system includes: Camera calibration module of stereo vision monitoring system: calibrate the intrinsic and extrinsic parameters of the stereo vision monitoring system; Initial data acquisition and parameter setting module: The initial left image structure monitoring area is used to arrange measurement points and search areas, and based on the measurement points and search area arrangements made on the initial left image, the initial right image is stereo matched to obtain the pixel coordinates of the measurement points and the search area size on the initial left and right images; The three-dimensional coordinates of each structural measuring point in the structural coordinate system at the initial moment are calculated based on the calibration results of the stereo vision system, and the three-dimensional displacement and three-dimensional rotation angle of each measuring point in the structural coordinate system at the initial moment are calculated; Monitoring module: obtains the pixel coordinates of each measuring point on the left and right images at a certain moment other than the initial moment during the monitoring process, and calculates the three-dimensional coordinates of each measuring point in the structural coordinate system at that moment; And determine whether there is an abnormal point at this moment; if an abnormal point is found, the pixel coordinates of each measuring point in the pixel coordinate system where the abnormal camera is located are relocated at the next moment; The pixel coordinates of each measuring point on the left and right images at other times during the monitoring process are obtained, and the three-dimensional deformation and three-dimensional rotation angle of each measuring point in the structural coordinate system are calculated until the monitoring is completed.

Images