A method for dynamic displacement measurement of civil engineering structures based on computer vision

Abstract

The invention discloses a method for measuring the dynamic displacement of civil engineering structures based on computer vision. First, the camera is calibrated. Then, a feature fusion method based on unscented Kalman combined with spatiotemporal context and improved optical flow algorithm is used. The spatiotemporal context algorithm focuses on the correlation calculation between image intensity and its local context, which is used to calculate the whole pixel displacement of the structure. It also takes into account the sub-pixel measurement of the optical flow algorithm. Then, the displacement is calculated by combining the two through unscented Kalman filtering, and the calculated covariance is used for the prediction of the next frame, thereby improving the accuracy and stability of displacement prediction. The method can realize accurate displacement measurement of low-texture surfaces under lighting changes, and has higher robustness.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision and civil engineering measurement, and in particular relates to a civil engineering structure dynamic displacement measurement method based on computer vision. Background Art

[0002] Non-contact displacement measurement technology for the field of structural health monitoring in civil engineering. Traditional contact measurement methods have many limitations in practical applications, such as difficulty in eliminating interference caused by structural response data during acquisition. In contrast, non-contact sensors are more flexible and can complete the acquisition of structural response data without contacting the structure, and can avoid further damage to the structure.

[0003] At present, computer vision technology has made great progress in the field of detection and target tracking. However, in practical applications, displacement measurement technology based on computer vision faces limitations such as illumination and structural surface texture, which leads to the relatively insufficient ability of existing computer vision technology to simultaneously handle these error factors. Summary of the invention

[0004] The present invention provides a method for measuring the dynamic displacement of civil engineering structures based on computer vision, aiming to solve the problems of insufficient stability and accuracy in the displacement measurement process of structures with complex illumination and low texture on the surface of structures in the prior art.

[0005] In order to solve the above technical problems, the present invention provides a method for measuring the dynamic displacement of a civil engineering structure, comprising the following steps:

[0006] S1: Camera calibration, obtaining calibration parameters, establishing the projection relationship between the image and the actual physical world, and calculating the scale factor;

[0007] S2: Use a camera to shoot continuous video frames, use a spatiotemporal context algorithm and an optical flow algorithm to track feature points in the region of interest, and calculate the whole pixel displacement and sub-pixel displacement of the feature points;

[0008] S3: using an unscented Kalman filter to combine integer pixel displacement and sub-pixel displacement to obtain a predicted displacement curve and covariance, solving the predicted pixel displacement according to the displacement curve, and updating the covariance frame by frame for prediction of the next frame;

[0009] S4: Convert the second pixel displacement into actual displacement using the scale factor.

[0010] Preferably, the step S2 is specifically:

[0011] S21: generating a priori model according to the predefined target position and image intensity in the first frame, using the spatial relationship between the target and its context information to model and generate a conditional probability function, and generating a likelihood function that maximizes the target position according to the priori model and the conditional probability function;

[0012] S22: establishing a confidence function, where the confidence function is used to solve the optimal position of the target;

[0013] S23: using a fast Fourier algorithm to convert the confidence function and the conditional probability function from the time domain to the frequency domain, and performing an inverse Fourier transform on the ratio of the two frequency domains to obtain a spatial context model function;

[0014] S24: Establishing a spatiotemporal context model according to the spatial context model;

[0015] S25: predicting the best position of the feature point in the next frame according to the spatiotemporal context model to obtain the integer pixel displacement;

[0016] S26: Calculate the displacement of the displacement feature point by using the optical flow algorithm combined with the whole pixel displacement and update the tracking area in the next frame to calculate the sub-pixel displacement.

[0017] Preferably, the method for calculating the sub-pixel displacement in step S26 is specifically as follows:

[0018]

[0019] Where (δx, δy) is the sub-pixel displacement of the pixel point (x, y) between the template image and the target image, g(x, y) is the coordinate value of the pixel point (x, y) in the next frame image, and f w (x, y) is the coordinate value of the pixel point (x, y) after being shifted by an integer pixel.

[0020] Preferably, the sub-pixel displacement is solved using the least squares method, by determining an error function, and then differentiating the error function to find an analytical solution or a numerical solution with a partial derivative of 0, specifically:

[0021]

[0022] Where (δx, δy) is the sub-pixel displacement of the pixel point (x, y) between the template image and the target image, g(x, y) is the coordinate value of the pixel point (x, y) in the next frame image, and f w (x, y) is the coordinate value of the pixel point (x, y) after the integer pixel displacement; the value of the sub-pixel displacement is iteratively adjusted so that the error function reaches the minimum value, that is, satisfies δx<<1, δy<<1.

[0023] Preferably, the step S3 is specifically:

[0024] S31: using an untraceable transformation method to change the integer pixel displacement and the sub-pixel displacement, and generating prediction sigma point sets of the spatiotemporal context algorithm and the optical flow algorithm respectively;

[0025] S32: Use the spatiotemporal context model to update the predicted sigma point set to obtain a first sigma point set, and use the first sigma point set to calculate the mean and covariance of the prior estimate based on the spatiotemporal context by weighted summation;

[0026] S33: performing untraceable transformation again according to the mean and covariance of the prior estimation of the system state quantity based on the optical flow algorithm to obtain the second sigma point set;

[0027] S34: calculating the mean of the second sigma point set, respectively calculating the covariance of the displacement prediction model based on the spatiotemporal context and the displacement prediction model based on the optical flow, and then calculating the Kalman gain matrix;

[0028] S35: Obtain predicted displacement update and covariance update through the Kalman gain matrix, the predicted displacement update is used as the second pixel displacement, and the covariance update is used for prediction of the next frame.

[0029] Preferably, the expression of the predicted sigma point set is:

[0030]

[0031] Where n is the number of input states, i is the generated sigma point, is the point set of generated sigma sampling points, λ is the scale factor, is the estimation of the system state, where the sub-pixel displacement measured by the optical flow method is input, P K-1 is the covariance measured by the unscented Kalman filter at frame K-1.

[0032] Preferably, the Kalman gain matrix calculation method is:

[0033]

[0034] In the formula, K k is the Kalman gain matrix, Represents the covariance matrix of system state and observation, Represents the inverse matrix of the covariance matrix of the system state and observations.

[0035] Preferably, the prediction displacement update and covariance update are specifically:

[0036]

[0037]

[0038] In the formula, is the predicted displacement, is the displacement measured by the optical flow method, K k is the Kalman gain matrix, is the displacement measured by the spatiotemporal context algorithm; P k+1 is the covariance predicted by the unscented Kalman filter, P k is the covariance based on spatiotemporal context, K k is the Kalman gain matrix, is the covariance matrix of the system state and observation.

[0039] Preferably, the camera calibration adopts Zhang Zhengyou calibration method.

[0040] Preferably, the proportional factor is calculated as follows:

[0041]

[0042] Where SF is the scale factor, D is the actual distance between the two feature points, d is the pixel distance between the two feature points, Z is the distance between the camera and the target object, and f is the focal length of the camera.

[0043] Compared with the prior art, the present invention has the following technical effects:

[0044] 1. The present invention utilizes the spatiotemporal context algorithm and the optical flow algorithm to track the target object in the image sequence, and adopts a series of samples to approximate the posterior probability density of the state through the unscented Kalman filter to realize the prediction of the motion trajectory. It is suitable for the dynamic displacement measurement of civil engineering structures under the conditions of complex lighting and low texture of the structure surface, which can effectively improve the measurement accuracy and stability and realize more reliable non-contact displacement measurement.

[0045] 2. The present invention utilizes a spatiotemporal context algorithm to obtain the integer pixel displacement of feature points, combines the changes of feature points in the time series and the spatial context information, and can provide a more accurate estimation of the integer pixel displacement. By considering the consistency information in multiple frames of images, it can filter out noise and improve the accuracy of the estimation.

[0046] 3. The present invention uses an optical flow algorithm to obtain sub-pixel displacement of feature points, provides a more accurate displacement estimation, can detect and track the tiny movements of objects, and improve the accuracy of displacement detection.

[0047] 4. The present invention uses an unscented Kalman filter to fuse the integer pixel displacement obtained by the spatiotemporal context algorithm and the sub-pixel displacement obtained by the optical flow algorithm, which is suitable for nonlinear probability density distribution; by fusing the information of integer pixel displacement and sub-pixel displacement, the error can be reduced and the stability of the estimation can be improved; in the presence of illumination changes, the sub-pixel optical flow can improve the stability of the displacement estimation by fusing its integer pixel displacement.

[0048] 5. The unscented Kalman filter proposed in the present invention combines the spatiotemporal context and the improved optical flow data fusion algorithm to achieve the prediction of the motion trajectory by adopting a series of samples to approximate the posterior probability density of the state. While giving full play to the advantages of the spatiotemporal context algorithm and the optical flow algorithm, it can also achieve regional measurement and can cope with the displacement measurement needs in complex environments such as lighting changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the measurement method proposed by the present invention;

[0050] Figure 2 It is a schematic diagram of tracking feature points at specific intervals based on optical flow according to an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in combination with specific embodiments of the present application and with reference to the accompanying drawings.

[0052] A method for dynamic displacement measurement of civil engineering structures based on computer vision, such as Figure 1 As shown, the following steps are included:

[0053] S1: Camera calibration, obtain calibration parameters, establish the projection relationship between the image and the actual physical world, and calculate the scale factor.

[0054] Obtain the fixed parameters of the camera. The camera calibration adopts the Zhang Zhengyou calibration method to establish the conversion relationship between the three-dimensional coordinate points in the real world and the two-dimensional coordinate points on the image plane. This embodiment adopts the pinhole model, and its mathematical expression is:

[0055]

[0056] In the formula, Represents the coordinate point in the image coordinate system, Represents the coordinate point in the world coordinate system, s is the scale factor used to adjust the scale relationship between the coordinate points, R and t are the rotation matrix and translation matrix respectively, which describe the position and posture of the camera in the world coordinate system. Parameter A is the inherent internal parameter of the camera, which is usually composed of parameters such as the focal length, pixel size and optical center of the camera.

[0057] Therefore, in order to obtain the intrinsic focal length of the camera, we can take photos of the calibration plate at different angles, input these photos into the camera calibration toolbox of Matlab to obtain parameter A, and extract the focal length of the camera from the parameter to calculate the scale factor. The calculation method of the scale factor is:

[0058]

[0059] Where SF is the scale factor, D is the actual distance between the two feature points, d is the pixel distance between the two feature points, Z is the distance between the camera and the target object, and f is the focal length of the camera.

[0060] S2: Region tracking and pixel displacement acquisition.

[0061] A camera is used to shoot continuous video frames, and a spatiotemporal context algorithm and an optical flow algorithm are used to track feature points in an area of ​​interest, and the whole pixel displacement and sub-pixel displacement of the feature points are calculated.

[0062] First, the spatiotemporal context algorithm is used to track the target. A prior model is generated according to the predefined target position and image intensity in the first frame. The spatial relationship between the target and its context information is used to model and generate a conditional probability function. The likelihood function that maximizes the target position is generated according to the prior model and the conditional probability function. The expression is:

[0063]

[0064] The likelihood function can be divided into two parts, which are composed of the conditional probability function P(x|c(z),o) that models the spatial relationship between the target and its context information and the context prior model P(c(z)|o) that models each point in the local context. x is the pixel coordinate in the target area, o is the target in the scene, and the feature set X in the region of interest is defined as:

[0065] X C ={c(z)=(I(z),z)|z∈Ω c (x * )}

[0066] Where x * is the coordinate obtained by maximizing the likelihood function, usually used as the center point of the measurement area, Ω c (x * ) is represented by x * The neighborhood of , I(z) is the image intensity, that is, the grayscale.

[0067] The prior model can be calculated based on the predefined target position and image intensity in the first frame. The prior model can be regarded as a model of a specific target, that is, a pre-learned target model. Its model expression is as follows:

[0068] P(c(z)|o)=I(z)ω σ (zx * )

[0069] In the formula, ω σ (z), the expression is:

[0070]

[0071] Where a is the normalization function, σ is the scaling function, and its value is the average value of the length and width of the target area.

[0072] The conditional probability function is used to describe the probability of a target appearing under given local context conditions and to estimate and predict the location of the target. The conditional probability function is defined as:

[0073] P(x|c(z),o)=h st (xz)

[0074] Where h st (xz) is a spatial context model function, which is used to describe the spatial relationship between the pixels in the target area and the surrounding local background pixels. This function uses the distance weight function to weight the spatial distance between different pixels, thereby taking into account the spatial relationship between the target and the local context, and solving the impact of distance and similar background.

[0075] Secondly, in order to solve the optimal position of the target, a confidence function is introduced. The confidence function is expressed as:

[0076]

[0077] Where b is the normalization constant, α is the scale parameter, and β is the shape parameter.

[0078] In order to track the target area, it is necessary to calculate the optimal estimate of the target position. However, there are many pixels in the current tracking area, which will lead to low efficiency and long time consumption. Therefore, in order to solve this problem, the fast Fourier algorithm can be used to convert the confidence function and the conditional probability function from the time domain to the frequency domain, and the ratio of the two frequency domains can be inverse Fourier transformed to obtain the spatial context model function, which is expressed as follows:

[0079]

[0080] In the formula Stands for Inverse Fourier Transform.

[0081] With the improvement of the spatial context model function, the spatiotemporal context model is established. The spatiotemporal context model function expression is:

[0082]

[0083] Where ρ is a learning parameter used to weigh the relationship between the spatial context model and the target motion model; t represents the tth image in the measured video; is the spatiotemporal context model. In addition, when t is 1, the spatiotemporal context model Equal to spatial context model

[0084] Finally, the optimal position of the target in the next frame is predicted by calculating the maximum likelihood function to obtain the integer pixel displacement.

[0085] However, the accuracy of the whole pixel level is insufficient for non-contact measurement. Therefore, in order to achieve sub-pixel accuracy, the optical flow algorithm is used to calculate the displacement of the feature points and update the tracking area in the next frame, and feature points are selected at fixed intervals in the tracking area of ​​each frame image, such as Figure 2 As shown, the sub-pixel displacement is calculated. The expression of the sub-pixel movement distance of the pixel point between two frames of images is:

[0086]

[0087] Where g(x,y) is the coordinate value of the target point in the next frame image; (δx,δy) represents the sub-pixel displacement of the pixel point between the template image and the target image; f w (x, y) is the coordinate value obtained by the pixel point after being shifted by an integer pixel.

[0088] The sub-pixel displacement can be solved by minimizing the sum of squared errors, which is expressed as:

[0089]

[0090] Then use the least squares method to solve. By determining the error function or cost function, and then taking the derivative, find the analytical solution or numerical solution with a partial derivative of 0, the expression is:

[0091]

[0092] The partial derivatives in the formula are calculated using finite differences, namely:

[0093]

[0094]

[0095] In this process, the value of the sub-pixel displacement can be adjusted through repeated iterations so that the error function reaches a minimum value, that is, δx<<1, δy<<1 is satisfied.

[0096] S3: Accurate pixel displacement prediction. Use the unscented Kalman filter to combine the whole pixel displacement and sub-pixel displacement to obtain the predicted displacement curve and covariance, solve the predicted pixel displacement according to the displacement curve, and update the covariance frame by frame for the prediction of the next frame;

[0097] In order to overcome the problem that algorithms based on computer vision are easily affected by illumination and achieve sub-pixel level displacement measurement accuracy. The present invention proposes a displacement response prediction algorithm that combines unscented Kalman filtering technology with spatiotemporal context and optical flow. First, the whole pixel displacement and sub-pixel displacement are changed using the unscented transformation method to generate the predicted sigma point sets of the spatiotemporal context algorithm and the optical flow algorithm, that is, the preliminary displacement prediction model, which is expressed as follows:

[0098]

[0099] Where n is the number of input states, i is the generated sigma point, is the point set of generated sigma sampling points, λ is the proportional factor, and its value determines the sigma distribution state. is the estimation of the system state, where the sub-pixel displacement measured by the optical flow method is input, P K-1 is the covariance measured by the unscented Kalman filter at frame K-1.

[0100] Then, the spatiotemporal context model is used to update the predicted sigma point set to obtain the first sigma point set, which is expressed as:

[0101]

[0102]

[0103] In the formula is the displacement measured by the optical flow algorithm; The displacement measured by the spatiotemporal context algorithm uses the first sigma point set to calculate the mean and covariance of the prior estimate based on the spatiotemporal context by weighted summation, and its expression is:

[0104]

[0105]

[0106] In the formula and are the weights for calculating the prior estimated mean and covariance, respectively, and Q is a constant used to simulate a specific covariance.

[0107] Secondly, the untraceable transformation is performed again according to the mean and covariance of the prior estimation of the system state quantity based on the optical flow algorithm to obtain the second sigma point set. Then, the mean of the second sigma point set is calculated, and the covariance of the displacement prediction model based on spatiotemporal context and the displacement prediction model based on optical flow is calculated respectively. The expression is:

[0108]

[0109]

[0110]

[0111] In the formula is the sigma point set of the optical flow algorithm, and are the sigma point set and its mean of the predicted motion model, R is a constant, represents the covariance matrix of the predicted observations, Represents the covariance matrix of the system state and observation. Covariance is an indicator used to measure the degree of association between two random variables. By combining the covariance between the two, the Kalman gain matrix can be calculated. The calculation formula is as follows:

[0112]

[0113] Where K k is the Kalman gain matrix.

[0114] Finally, the predicted displacement update and covariance update are obtained through the Kalman gain matrix, which is expressed as:

[0115]

[0116]

[0117] In the formula is the predicted displacement; is the displacement measured by the optical flow method; is the displacement measured by the spatiotemporal context algorithm; P k+1 is the covariance predicted by the unscented Kalman filter in this process, and the covariance is applied to the next frame.

[0118] S4: Acquisition of actual displacement: The second pixel displacement is converted into actual displacement using the proportional factor, thereby achieving high-precision measurement of dynamic displacement.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several modifications and improvements without departing from the inventive concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A method for measuring dynamic displacement of civil engineering structures based on computer vision, characterized in that: The following steps are involved: S1: Camera calibration, obtaining calibration parameters, establishing the projection relationship between the image and the actual physical world, and calculating the scale factor; S2: Use a camera to shoot continuous video frames, use a spatiotemporal context algorithm and an optical flow algorithm to track feature points in the region of interest, and calculate the whole pixel displacement and sub-pixel displacement of the feature points; S3: using an unscented Kalman filter to combine integer pixel displacement and sub-pixel displacement to obtain a predicted displacement curve and covariance, solving the predicted pixel displacement according to the displacement curve, and updating the covariance frame by frame for prediction of the next frame; S4: Convert the second pixel displacement into actual displacement using the scale factor.

2. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 1, characterized in that: The step S2 is specifically as follows: S21: generating a priori model according to the predefined target position and image intensity in the first frame, using the spatial relationship between the target and its context information to model and generate a conditional probability function, and generating a likelihood function that maximizes the target position according to the priori model and the conditional probability function; S22: establishing a confidence function, where the confidence function is used to solve the optimal position of the target; S23: using a fast Fourier algorithm to convert the confidence function and the conditional probability function from the time domain to the frequency domain, and performing an inverse Fourier transform on the ratio of the two frequency domains to obtain a spatial context model function; S24: Establishing a spatiotemporal context model according to the spatial context model; S25: predicting the best position of the feature point in the next frame according to the spatiotemporal context model to obtain the integer pixel displacement; S26: Calculate the displacement of the displacement feature point by using the optical flow algorithm combined with the whole pixel displacement and update the tracking area in the next frame to calculate the sub-pixel displacement.

3. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 2, characterized in that: The method of calculating the sub-pixel displacement in step S26 is specifically as follows: Where (δx, δy) is the sub-pixel displacement of the pixel point (x, y) between the template image and the target image, g(x, y) is the coordinate value of the pixel point (x, y) in the next frame image, and f w (x, y) is the coordinate value of the pixel point (x, y) after being shifted by an integer pixel.

4. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 3, characterized in that: The sub-pixel displacement is solved using the least squares method, by determining the error function and then taking the derivative of the error function to find an analytical solution or a numerical solution with a partial derivative of 0, specifically: Where (δx, δy) is the sub-pixel displacement of the pixel point (x, y) between the template image and the target image, g(x, y) is the coordinate value of the pixel point (x, y) in the next frame image, and f w (x, y) is the coordinate value of the pixel point (x, y) after the integer pixel displacement; the value of the sub-pixel displacement is iteratively adjusted so that the error function reaches the minimum value, that is, satisfies δx<<1, δy<<1.

5. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 1, characterized in that: The step S3 is specifically as follows: S31: using an untraceable transformation method to change the integer pixel displacement and the sub-pixel displacement, and generating prediction sigma point sets of the spatiotemporal context algorithm and the optical flow algorithm respectively; S32: Use the spatiotemporal context model to update the predicted sigma point set to obtain a first sigma point set, and use the first sigma point set to calculate the mean and covariance of the prior estimate based on the spatiotemporal context by weighted summation; S33: performing untraceable transformation again according to the mean and covariance of the prior estimation of the system state quantity based on the optical flow algorithm to obtain the second sigma point set; S34: calculating the mean of the second sigma point set, respectively calculating the covariance of the displacement prediction model based on the spatiotemporal context and the displacement prediction model based on the optical flow, and then calculating the Kalman gain matrix; S35: Obtain predicted displacement update and covariance update through the Kalman gain matrix, the predicted displacement update is used as the second pixel displacement, and the covariance update is used for prediction of the next frame.

6. A method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 5, characterized in that: The expression of the predicted sigma point set is: Where n is the number of input states, i is the generated sigma point, is the point set of generated sigma sampling points, λ is the scale factor, is the estimation of the system state, where the sub-pixel displacement measured by the optical flow method is input, P K-1 is the covariance measured by the unscented Kalman filter at frame K-1.

7. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 5, characterized in that: The Kalman gain matrix calculation method is: In the formula, K k is the Kalman gain matrix, Represents the covariance matrix of system state and observation, Represents the inverse matrix of the covariance matrix of the system state and observations.

8. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 5, characterized in that: The prediction displacement update and covariance update are specifically as follows: In the formula, is the predicted displacement, is the displacement measured by the optical flow method, K k is the Kalman gain matrix, is the displacement measured by the spatiotemporal context algorithm; P k+1 is the covariance predicted by the unscented Kalman filter, P k is the covariance based on spatiotemporal context, K k is the Kalman gain matrix, is the covariance matrix of the system state and observation.

9. The method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 1, characterized in that: The camera calibration adopts Zhang Zhengyou calibration method.

10. A method for measuring dynamic displacement of civil engineering structures based on computer vision according to claim 1 or 9, characterized in that: The calculation method of the proportional factor is: Where SF is the scale factor, D is the actual distance between the two feature points, d is the pixel distance between the two feature points, Z is the distance between the camera and the target object, and f is the focal length of the camera.

Images