Structural dynamic displacement monitoring method based on interaction and fusion of dual-channel visual signals

Abstract

The application discloses a kind of structural dynamic displacement monitoring methods based on double-channel visual signal interaction and fusion, first, the motion video of structure is recorded in real time using industrial camera, and Gaussian filter denoising and gray processing are carried out frame by frame;Second, wavelet amplitude screening and corner response factor screening are comprehensively used, pixel set with measurement robustness is established, and monitoring point is selected;Then, template matching algorithm is used to measure the displacement of integral pixel accuracy, as the displacement signal of visual channel one, and the signal is used as dynamic window size reference value and visual channel two interaction, optical flow algorithm is used to measure the speed of subpixel accuracy, as the speed signal of visual channel two;Finally, based on Kalman filtering algorithm, the displacement signal of visual channel one and the speed signal of visual channel two are data fusion, so as to obtain the structural dynamic displacement signal of subpixel accuracy.The application effectively combines the advantages of double-channel visual signal, can realize high-precision non-contact structural dynamic displacement monitoring, and does not need to spray artificial mark on the surface of structure, and has good practicability.

Description

Technical Field

[0001] This invention belongs to the field of structural health monitoring technology, and specifically relates to a method for monitoring structural dynamic displacement based on dual-channel visual signal interaction and fusion. Background Technology

[0002] Dynamic displacement monitoring is a crucial component of structural health monitoring, providing vital information for understanding the performance and safety of civil engineering structures. In early stages, contact sensors such as accelerometers were primarily used for monitoring. However, these contact sensors face significant limitations in practical applications, including sparse measurement points, the need for physical contact with the measured structure, and high manpower and resource consumption. In contrast, non-contact methods such as scanning laser Doppler vibration meters and GPS have alleviated these problems. However, they still retain significant drawbacks, such as limited measurement distance, high cost, and operational complexity.

[0003] In recent years, visual monitoring methods have received increasing attention due to their advantages such as high flexibility, high efficiency, and high spatial resolution. However, existing methods still struggle to overcome inherent limitations when facing dynamic structural monitoring in real-world scenarios. For example, digital image correlation algorithms heavily rely on high-quality artificial speckle patterns on the surface of the measured structure; template matching algorithms can only provide displacement measurements with integer pixel accuracy, resulting in unsatisfactory measurement results and the presence of jagged edges; while optical flow algorithms can provide relatively accurate sub-pixel velocity measurements, the integration operations during dynamic processes lead to error accumulation and severe motion drift. Summary of the Invention

[0004] To overcome the shortcomings of existing structural dynamic displacement monitoring methods and address the inherent limitations of single-vision algorithms, this invention proposes a structural dynamic displacement monitoring method based on the interaction and fusion of dual-channel visual signals. This method effectively combines dual-channel visual signals, offering advantages such as high measurement accuracy, good noise robustness, and no need for manual surface marking.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for monitoring structural dynamic displacement based on dual-channel visual signal interaction and fusion includes the following steps:

[0007] Step 1: Use an industrial camera to record the movement video of the structure in real time, and perform Gaussian filtering for noise reduction and grayscale processing frame by frame;

[0008] Step 2: By combining wavelet amplitude screening and corner response factor screening, establish a set of pixels with measurement robustness, and select monitoring points;

[0009] Step 3: Use template matching algorithm to measure displacement at integer pixel precision, and use it as displacement signal for visual channel one. Use this signal as a dynamic window size reference value to interact with visual channel two. Use optical flow algorithm to measure velocity at sub-pixel precision, and use it as velocity signal for visual channel two.

[0010] Step 4: Based on the Kalman filter algorithm, the displacement signal of visual channel one and the velocity signal of visual channel two are fused to obtain the sub-pixel precision structural dynamic displacement signal.

[0011] Compared with existing methods for monitoring dynamic displacement of structures, the advantages of this invention are as follows:

[0012] (1) This invention automatically filters the texture information and corner characteristics of all pixels by establishing wavelet amplitude and corner response screening criteria, thereby providing a reliable set of monitoring points without the need for manual marking on the surface of the structure under test to provide monitoring points.

[0013] (2) By performing dual-channel processing on motion video and performing signal interaction and fusion, this invention effectively combines the advantages of template matching algorithm and optical flow algorithm, which can improve the accuracy of dynamic displacement monitoring while ensuring computational stability and noise resistance, and overcome the inherent defects of single vision algorithm.

[0014] (3) The implementation of this invention only requires an industrial camera and a computer. The monitoring system used is simple and efficient and has good applicability in actual industrial application scenarios. Attached Figure Description

[0015] Figure 1 This is a flowchart of a structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion.

[0016] Figure 2 This is a schematic diagram of the dual-channel visual signal fusion strategy used in this invention.

[0017] Figure 3 The results are experimental findings of using this invention for dynamic displacement monitoring of civil structures. Detailed Implementation

[0018] The present invention will now be further described with reference to the accompanying drawings.

[0019] like Figure 1 As shown, a method for monitoring structural dynamic displacement based on dual-channel visual signal interaction and fusion includes the following steps:

[0020] Step 1: Use an industrial camera to record the movement video of the structure in real time, and perform Gaussian filtering for noise reduction and grayscale processing frame by frame;

[0021] Step 2: By combining wavelet amplitude screening and corner response factor screening, establish a set of pixels with measurement robustness, and select monitoring points;

[0022] First, texture information is quantized using wavelet amplitude values. A two-dimensional Gabor wavelet convolution kernel is used to convolve the initial frame image to extract its wavelet amplitude values.

[0023]

[0024] In the formula, x and y are the horizontal and vertical coordinates of the pixel, respectively; A(x,y) is the wavelet amplitude of the pixel at (x,y); φ(x,y) is the wavelet phase of the pixel at (x,y); I(x,y) is the gray level of the pixel at (x,y); and G... 2D H represents the real part of the two-dimensional Gabor wavelet convolution kernel. 2D is the imaginary part of the two-dimensional Gabor wavelet convolution kernel, where i is the imaginary unit;

[0025] After obtaining the wavelet amplitude values ​​of all pixels, sort them and take the median A. th As a threshold, pixels with rich texture information are selected, and a set of pixels with rich texture information E1 is established:

[0026] E1={(x,y)A(x,y)>A th}

[0027] Then, the corner characteristics are quantified and characterized using the corner response factor. For each pixel in the initial frame image, its corner response matrix can be calculated as follows:

[0028]

[0029] In the formula, M(x,y) is the corner response matrix of the pixel at (x,y), and I x (m,n) and I y (m,n) are the gray-level gradients of the pixel at (m,n) in the horizontal and vertical coordinate directions, respectively, and W(x,y) is the neighborhood window of the pixel at (x,y).

[0030] Based on the corner response matrix, its eigenvalues ​​are solved, and the corner response factor is calculated based on the eigenvalues ​​of the corner response matrix.

[0031]

[0032] In the formula, R(x,y) is the corner response factor of the pixel at (x,y), and λ1(x,y) and λ2(x,y) are two eigenvalues ​​of the corner response matrix M(x,y), respectively.

[0033] After obtaining the corner response factors of all pixels, sort them and take the median R. th As a threshold, pixels with rich corner information are selected, and a set of pixels with rich corner information E2 is established:

[0034] E2={(x,y)R(x,y)>R th}

[0035] By performing an intersection operation on the set of points rich in texture information and the set of points rich in corner information, a set of pixels E with measurement robustness is finally obtained. r And select monitoring points from them:

[0036] E r =E1∩E2

[0037] Step 3: Use template matching algorithm to measure displacement at integer pixel precision, and use it as displacement signal for visual channel one. Use this signal as a dynamic window size reference value to interact with visual channel two. Use optical flow algorithm to measure velocity at sub-pixel precision, and use it as velocity signal for visual channel two.

[0038] Let the coordinates of the reliable monitoring point selected in the initial frame be (x0, y0), and establish a matching template centered on this point. For each motion frame, firstly, the template matching algorithm is used to measure the displacement signal with integer pixel precision. Then, the matching template is slid across the motion frame image, and the similarity score when overlapping with each image block is calculated.

[0039]

[0040] In the formula, C is the correlation coefficient, w and h are the width and height of the matching template, respectively, T(i,j) and I(i,j) are the gray levels of the matching template and the motion frame image block at position (i,j), respectively, and T(m,n) and I(m,n) are the gray levels of the matching template and the motion frame image block at position (m,n), respectively.

[0041] After obtaining the correlation coefficients for all positions in the motion frame, the position with the highest correlation coefficient is the location of the reliable monitoring point in the current motion frame. Subtracting the coordinates of the reliable monitoring point in the current motion frame from those in the reference frame yields the displacement signal with integer pixel precision, which is used as the displacement signal for visual channel one. Considering that this displacement signal can accurately represent the motion trend of the structure, it is used as a reference value for the dynamic window size in interaction with visual channel two.

[0042]

[0043] In the formula, and These are the integer pixel-precision displacement signals in the x and y directions at time t, respectively, where Δt is the inter-frame time. and These represent the dynamic window sizes of the optical flow algorithm in the x and y directions at time t, respectively.

[0044] After obtaining the dynamic window size, an optical flow algorithm is used to measure the sub-pixel accuracy velocity signal. Based on the optical flow constraint equation:

[0045]

[0046] In the formula, p1, p2, ..., p k For pixels within a dynamic window, I t (p) represents the gray level of the motion frame image at pixel p at time t. and Let be the grayscale gradients in the x and y directions of the motion frame image at pixel p at time t. and These are the velocity signals in the x and y directions at time t, respectively;

[0047] The least squares method can be used to solve the optical flow constraint equation, thereby obtaining a velocity signal with sub-pixel accuracy, which can be used as the velocity signal of visual channel two.

[0048] Step 4: Based on the Kalman filter algorithm, perform data fusion on the displacement signal of visual channel one and the velocity signal of visual channel two to obtain the sub-pixel precision structural dynamic displacement signal, such as... Figure 2 As shown.

[0049] The evolution of the structural motion state is described by the state equation and the observation equation:

[0050] s t =Fs t-Δt +Bc t +ω

[0051] z t =Hs t +γ

[0052] In the formula, It is the true motion state vector of the monitoring point at time t. To monitor the actual displacement signal of the point at time t, The signal represents the actual velocity of the monitoring point at time t, where Δt is the inter-frame time. Here is the state transition matrix. Let c be the motion control matrix. t =(0,v) t ) T Let v be the motion input vector. t Let z be the sub-pixel precision velocity signal of visual channel two at time t. t =(ut ,0) T Let u be the observation vector. t This represents the integer pixel precision displacement signal of visual channel one at time t. Let ω be the observation matrix, and γ be the process noise and observation noise, respectively.

[0053] First, based on the state equation, a priori estimation of the motion state is provided by inputting the sub-pixel precision velocity signal from visual channel two:

[0054]

[0055] in, It is a priori estimate of the motion state at time t. It is a reliable posterior estimate of the motion state at time t-Δt. It is the covariance of the prior estimate of the motion state. Q is the covariance of the reliable posterior estimate of the motion state, and Q is the variance of the process noise.

[0056] Then, based on the observation equation, a reliable posterior estimate of the motion state is provided by inputting the integer-pixel precision displacement signal of visual channel one:

[0057]

[0058] In the formula, K t R is the filter gain, R is the observation noise variance, and E is the identity matrix. This provides a reliable posterior estimate of the motion state at time t.

[0059] Thus, a reliable posterior estimate of the motion state of the monitoring point with measurement robustness at each moment is obtained, and the displacement signal is extracted to achieve high-precision monitoring of the dynamic displacement of the structure.

[0060] The method of this invention was used to monitor the dynamic displacement of a reinforced concrete structure. The sensor results were used as a benchmark and compared with those of a single template matching algorithm and an optical flow algorithm. The experimental results are as follows: Figure 3 As shown in the figure, the measurement results obtained by the method of the present invention match the sensor reference value better, and the measurement accuracy is higher.

Claims

1. A method for monitoring structural dynamic displacement based on dual-channel visual signal interaction and fusion, characterized in that, Includes the following steps: Step 1: Use an industrial camera to record the movement video of the structure in real time, and perform Gaussian filtering for noise reduction and grayscale processing frame by frame; Step 2: By combining wavelet amplitude screening and corner response factor screening, establish a set of pixels with measurement robustness, and select monitoring points; Step 3: Use template matching algorithm to measure displacement at integer pixel precision, and use it as displacement signal for visual channel one. Use this signal as a dynamic window size reference value to interact with visual channel two. Use optical flow algorithm to measure velocity at sub-pixel precision, and use it as velocity signal for visual channel two. In step 3, considering that the displacement signal can accurately represent the motion trend of the structure, the displacement signal is used as a reference value for the dynamic window size and interacted with visual channel two: In the formula, and They were obtained respectively t Moment x , y Two-way integer pixel precision displacement signal Inter-frame time, and They are respectively t Time-based optical flow algorithm in x , y Dynamic window size in both directions; After obtaining the dynamic window size, an optical flow algorithm is used to measure the sub-pixel accuracy speed: based on the optical flow constraint equation: In the formula, For pixels within a dynamic window, for t Motion frame image at any time pixel p grayscale of the position, and They are respectively t Motion frame image at any time pixel p Location x , y Two-way grayscale gradient, and They are respectively t time x , y Velocity signals in both directions; The least squares method is used to solve the optical flow constraint equation to obtain the sub-pixel accuracy velocity, which is used as the velocity signal of visual channel two. Step 4: Based on the Kalman filter algorithm, the displacement signal of visual channel one and the velocity signal of visual channel two are fused to obtain the sub-pixel precision structural dynamic displacement signal. In step 4, the displacement signal from visual channel one and the velocity signal from visual channel two are fused using the Kalman filter algorithm to obtain a sub-pixel precision structural dynamic displacement signal, as detailed below: The evolution of the structural motion state is described by the state equation and the observation equation: In the formula, The monitoring point is at t The true motion state vector at any given moment. For monitoring points at t The actual displacement signal at any given moment, For monitoring points at t The actual speed signal at any given moment, Inter-frame time, Here is the state transition matrix. For motion control matrix, For the motion input vector, For visual channel two in t Subpixel precision velocity signal at any given moment For the observation vector, For visual channel one in t The displacement signal with integer pixel precision at any given moment. For the observation matrix, ω and γ These are process noise and observation noise, respectively. First, based on the state equation, a priori estimation of the motion state is provided by inputting the sub-pixel precision velocity signal from visual channel two: in, yes t Prior estimation of the motion state at any given time. yes t -Δ t A reliable posterior estimate of the motion state at any given time. It is the covariance of the prior estimate of the motion state. It is the covariance of a reliable posterior estimate of the motion state. Q It is the variance of the process noise; Then, based on the observation equation, a reliable posterior estimate of the motion state is provided by inputting the integer-pixel precision displacement signal of visual channel one: In the formula, K t For filter gain, R To observe the noise variance, E It is the identity matrix. for t A reliable posterior estimate of the motion state at any given time; Thus, a reliable posterior estimate of the motion state of the monitoring point with measurement robustness at each moment is obtained, and the displacement signal is extracted to achieve high-precision monitoring of the dynamic displacement of the structure.

2. The structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion according to claim 1, characterized in that, In step 2, a set of pixels with measurement robustness is established by combining wavelet amplitude screening and corner response factor screening, and the monitoring points are selected as follows: First, texture information is quantized and represented using wavelet amplitude values: a two-dimensional Gabor wavelet convolution kernel is used to convolve the initial frame image to extract its wavelet amplitude values. In the formula, These are the x and y coordinates of the pixel, respectively. for The wavelet amplitude value of the pixel, for Wavelet phase of the pixel, for The grayscale of the pixel. For the real part of the two-dimensional Gabor wavelet convolution kernel, is the imaginary part of the two-dimensional Gabor wavelet convolution kernel, where i is the imaginary unit; After obtaining the wavelet amplitude values ​​of all pixels, sort them and take the median. As a threshold, pixels with rich texture information are selected to establish a set of pixels with rich texture information. E 1: Then, the corner characteristics are quantified and characterized using the corner response factor: for each pixel of the initial frame image, its corner response matrix is ​​calculated using the following formula: In the formula, for The corner response matrix of the pixel at that location. and They are respectively The grayscale gradient of the pixel in the horizontal and vertical coordinate directions. for The neighborhood window of the pixel; Based on the corner response matrix, solve for its eigenvalues, and then calculate the corner response factor based on the eigenvalues ​​of the corner response matrix. In the formula, for The corner response factor of the pixel. and Corner response matrices Two eigenvalues; After obtaining the corner response factors of all pixels, sort them and take the median. As a threshold, pixels with rich corner information are selected to establish a set of pixels with rich corner information. E 2: By performing an intersection operation on the set of pixels rich in texture information and the set of pixels rich in corner information, a set of pixels with measurement robustness is finally obtained. E r And select monitoring points from them: 。 3. The structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion according to claim 1, characterized in that, In step 3, a template matching algorithm is used to measure the displacement at integer pixel precision, which is then used as the displacement signal for visual channel one. This signal is used as a dynamic window size reference value for interaction with visual channel two. An optical flow algorithm is used to measure the velocity at sub-pixel precision, which is then used as the velocity signal for visual channel two, as detailed below: Let the coordinates of the reliable monitoring point selected in the initial frame be... A matching template is established centered on this point: For each motion frame, the template matching algorithm is first used to measure the displacement with integer pixel precision, the matching template is then slid across the motion frame image, and the correlation with each image block when overlapping is calculated. In the formula, C The correlation coefficient is... w and h These represent the width and height of the matching template, respectively. T ( i , j ) and I ( i , j ) are the matching template and the moving frame image patch, respectively. grayscale of the position, T ( m , n ) and I ( m , n ) are the matching template and the moving frame image patch, respectively. The grayscale value of the location; After obtaining the correlation coefficients of all positions, the position with the largest correlation coefficient is the position of the reliable monitoring point in the current motion frame. Subtracting the position coordinates of the reliable monitoring point in the current motion frame from those in the reference frame gives the displacement with integer pixel precision, which is used as the displacement signal of visual channel one.

Images