Structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion

Through the dual-channel visual signal interaction and fusion method, combined with template matching and optical flow algorithm, the Kalman filtering algorithm is used to fuse the signals, and the problems of low structural dynamic displacement monitoring accuracy and noise sensitivity in the existing technology are solved, achieving high-precision and robustness Monitor the effect.

CN118936329AActive Publication Date: 2024-11-12XI AN JIAOTONG UNIV
View PDF 13 Cites 0 Cited by

Patent Information

Application Number
CN202411291828.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-14
Publication Date
2024-11-12
Estimated Expiration
2044-09-14

AI Technical Summary

Technical Problem

The existing dynamic displacement monitoring methods for structural dynamic displacement have defects such as low measurement accuracy, noise sensitivity, and the need for artificial surface marking, making it difficult to effectively monitor the dynamic displacement of civil engineering structures in actual scenarios.

Method used

Using dual-channel visual signal interaction and fusion method, motion video is recorded through industrial cameras, Gaussian filtering and denoising and grayscale processing are carried out, and a robust pixel point set is established by combining wavelet amplitude filtering and corner point response filtering, and the template matching algorithm is used to complete the whole. Pixel accuracy displacement measurement is carried out, and subpixel accuracy velocity measurement is carried out in combination with optical flow algorithm. The signal is fused through the Kalman filtering algorithm to obtain the structural dynamic displacement signal with subpixel accuracy.

Benefits of technology

It realizes high-precision and strong noise-resistant structural dynamic displacement monitoring, overcomes the problem of insufficient accuracy and stability of a single visual algorithm, and does not require manual marking, and is suitable for practical industrial applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118936329A_ABST
    Figure CN118936329A_ABST
Patent Text Reader

Abstract

The invention discloses a structure dynamic displacement monitoring method based on two-channel visual signal interaction and fusion, which comprises the following steps: firstly, recording a motion video of a structure in real time by adopting an industrial camera, and carrying out Gaussian filtering denoising and graying processing frame by frame; secondly, integrating wavelet amplitude screening and angular point response factor screening, establishing a pixel point set with measurement robustness, and selecting monitoring points; then, a template matching algorithm is adopted to measure displacement of integer pixel precision to serve as a displacement signal of the first visual channel, the signal serves as a dynamic window size reference value to interact with the second visual channel, and an optical flow algorithm is adopted to measure speed of sub-pixel precision to serve as a speed signal of the second visual channel; and finally, carrying out data fusion on the displacement signal of the visual channel I and the speed signal of the visual channel II based on a Kalman filtering algorithm so as to obtain a structure dynamic displacement signal with sub-pixel precision. According to the invention, the advantages of double-channel visual signals are effectively combined, high-precision non-contact structure dynamic displacement monitoring can be realized, manual marks do not need to be sprayed on the surface of the structure, and the practicability is good.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of structural health monitoring, and in particular relates to a structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion. Background Art

[0002] Dynamic displacement monitoring is an important part of structural health monitoring and can provide an important basis for understanding the performance and safety of civil engineering structures. In the early stage, contact sensors such as accelerometers were mainly used to complete the monitoring task. However, these contact sensors are subject to great limitations in practical applications, such as sparse measurement points, the need to contact the measured structure, and the consumption of manpower and material resources. In contrast, the emergence of non-contact methods such as scanning laser Doppler vibrometers and global positioning systems has alleviated the above problems. However, there are still obvious disadvantages, such as limited measurement distance, high price and complex operation.

[0003] In recent years, visual monitoring methods have attracted more attention due to their advantages such as strong flexibility, high efficiency and high spatial resolution. However, existing methods are difficult to overcome their inherent defects when facing dynamic structural monitoring in actual scenarios. For example, the digital image correlation algorithm is heavily dependent on high-quality artificial speckles on the surface of the measured structure; the template matching algorithm can only provide displacement measurement with integer pixel accuracy, which makes it difficult to meet the accuracy of the measurement results and there is a sawtooth phenomenon; although the optical flow algorithm can provide relatively accurate sub-pixel precision speed measurement, the integral operation in the dynamic process will lead to error accumulation and serious motion drift. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned existing structural dynamic displacement monitoring methods and solve the problem that the current single visual algorithm is difficult to overcome its inherent defects, the purpose of the present invention is to propose a structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion. This method effectively combines dual-channel visual signals and has the advantages of high measurement accuracy, good noise robustness, and no need for surface artificial marking.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion, comprising the following steps:

[0007] Step 1: Use an industrial camera to record the motion video of the structure in real time, and perform Gaussian filtering, denoising and grayscale processing frame by frame;

[0008] Step 2: Comprehensive wavelet amplitude screening and corner response factor screening are used to establish a pixel point set with measurement robustness and select monitoring points;

[0009] Step 3: Use the template matching algorithm to measure the displacement with integer pixel accuracy as the displacement signal of visual channel one, and use this signal as the dynamic window size reference value to interact with visual channel two. Use the optical flow algorithm to measure the speed with sub-pixel accuracy as the speed signal of visual channel two.

[0010] Step 4: Based on the Kalman filter algorithm, the displacement signal of visual channel one and the velocity signal of visual channel two are fused to obtain the structural dynamic displacement signal with sub-pixel accuracy.

[0011] Compared with the existing structural dynamic displacement monitoring method, the advantages of the present invention are:

[0012] (1) The present invention automatically screens the texture information and corner point characteristics of all pixel points in the entire field by establishing wavelet amplitude and corner point response screening criteria, thereby providing a reliable monitoring point set without the need for manual marking on the surface of the structure being tested to provide monitoring points.

[0013] (2) The present invention effectively combines the advantages of template matching algorithm and optical flow algorithm by implementing dual-channel processing on motion video and performing signal interaction and fusion. It can improve the accuracy of dynamic displacement monitoring while ensuring calculation stability and noise resistance, and overcome the inherent defects of a single visual algorithm.

[0014] (3) The implementation of the present invention only requires an industrial camera and a computer. The adopted monitoring system is simple and efficient and has good applicability in actual industrial application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion.

[0016] Figure 2 Schematic diagram of the dual-channel visual signal fusion strategy adopted by the present invention.

[0017] Figure 3 The present invention is used to carry out the experimental results of dynamic displacement monitoring of civil engineering structures. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings.

[0019] like Figure 1 As shown, a structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion includes the following steps:

[0020] Step 1: Use an industrial camera to record the motion video of the structure in real time, and perform Gaussian filtering, denoising and grayscale processing frame by frame;

[0021] Step 2: Comprehensive wavelet amplitude screening and corner response factor screening are used to establish a pixel point set with measurement robustness and select monitoring points;

[0022] First, the texture information is quantitatively characterized by wavelet amplitude. The initial frame image is convolved with a two-dimensional Gabor wavelet convolution kernel to extract its wavelet amplitude:

[0023]

[0024] In the formula, x and y are the horizontal and vertical coordinates of the pixel, respectively, A(x,y) is the wavelet amplitude of the pixel at (x,y), φ(x,y) is the wavelet phase of the pixel at (x,y), I(x,y) is the grayscale of the pixel at (x,y), G 2D is the real part of the two-dimensional Gabor wavelet convolution kernel, H 2D is the imaginary part of the two-dimensional Gabor wavelet convolution kernel, and i is the imaginary unit;

[0025] After obtaining the wavelet amplitudes of all pixels, sort them and take the median A th As the threshold, the pixels with rich texture information are screened out, and the pixel set E1 with rich texture information is established:

[0026] E1={(x,y)A(x,y)>A th}

[0027] Then the corner point characteristics are quantitatively characterized by the corner point response factor. For each pixel of the initial frame image, its corner point response matrix can be calculated as:

[0028]

[0029] Where M(x,y) is the corner point response matrix of the pixel at (x,y), I x (m,n) and I y (m,n) are the grayscale gradients of the pixel at (m,n) in the horizontal and vertical directions, respectively, and W(x,y) is the neighborhood window of the pixel at (x,y);

[0030] According to the corner point response matrix, its eigenvalue is solved, and the corner point response factor is calculated based on the eigenvalue of the corner point response matrix:

[0031]

[0032] Where R(x,y) is the corner response factor of the pixel at (x,y), λ1(x,y) and λ2(x,y) are the two eigenvalues ​​of the corner response matrix M(x,y);

[0033] After obtaining the corner response factors of all pixels, sort them and take the median R th As the threshold, the pixels with rich corner information are screened out, and the pixel set E2 with rich corner information is established:

[0034] E2={(x,y)R(x,y)>R th}

[0035] By performing an intersection operation on the texture information-rich point set and the corner point information-rich point set, we can finally obtain a pixel point set E with measurement robustness. r , and select monitoring points from them:

[0036] E r =E1∩E2

[0037] Step 3: Use the template matching algorithm to measure the displacement with integer pixel accuracy as the displacement signal of visual channel one, and use this signal as the dynamic window size reference value to interact with visual channel two. Use the optical flow algorithm to measure the speed with sub-pixel accuracy as the speed signal of visual channel two.

[0038] Assume that the coordinates of the reliable monitoring point selected in the initial frame are (x0, y0), and establish a matching template with this point as the center. For each motion frame, first use the template matching algorithm to measure the displacement signal with integer pixel accuracy, slide the matching template on the motion frame image, and calculate the similarity score when overlapping with each image block:

[0039]

[0040] Where C is the correlation coefficient, w and h are the width and height of the matching template, T(i, j) and I(i, j) are the grayscale of the matching template and the motion frame image block at the (i, j) position, T(m, n) and I(m, n) are the grayscale of the matching template and the motion frame image block at the (m, n) position;

[0041] After obtaining the correlation coefficients of all positions in the motion frame, the position with the largest correlation coefficient is the position of the reliable monitoring point in the current motion frame. Subtracting the position coordinates of the reliable monitoring point in the current motion frame and the reference frame is the displacement signal with integer pixel accuracy, which is used as the displacement signal of visual channel 1. Considering that the displacement signal can correctly characterize the motion trend of the structure, the displacement signal is used as a reference value of the dynamic window size to interact with visual channel 2:

[0042]

[0043] In the formula, and are the integer pixel precision displacement signals in the x and y directions at time t, Δt is the time between frames, and They are the dynamic window sizes of the optical flow algorithm in the x and y directions at time t respectively;

[0044] After obtaining the dynamic window size, the optical flow algorithm is used to measure the sub-pixel precision speed signal. According to the optical flow constraint equation:

[0045]

[0046] Where p1, p2, ..., p k is the pixel point in the dynamic window, I t (p) is the grayscale of the motion frame image at pixel p at time t, and They are the grayscale gradients in the x and y directions of the motion frame image at the pixel point p at time t, and They are the speed signals in the x and y directions at time t respectively;

[0047] The optical flow constraint equation can be solved by the least square method to obtain a speed signal with sub-pixel accuracy as the speed signal of the second visual channel.

[0048] Step 4: Based on the Kalman filter algorithm, the displacement signal of visual channel 1 and the speed signal of visual channel 2 are fused to obtain the structural dynamic displacement signal with sub-pixel accuracy, such as Figure 2 shown.

[0049] The evolution of the structural motion state is described by the state equation and the observation equation:

[0050] s t =Fs t-Δt +Bc t +ω

[0051] z t =Hs t +γ

[0052] In the formula, is the real motion state vector of the monitoring point at time t, is the real displacement signal of the monitoring point at time t, is the real speed signal of the monitoring point at time t, Δt is the time between frames, is the state transfer matrix, is the motion control matrix, c t =(0,v t ) T is the motion input vector, v t is the sub-pixel precision velocity signal of the visual channel 2 at time t, z t =(ut ,0) T is the observation vector, u t is the integer pixel precision displacement signal of the visual channel 1 at time t, is the observation matrix, ω and γ are process noise and observation noise respectively;

[0053] First, based on the state equation, the sub-pixel precision velocity signal of the second visual channel is input to provide a priori estimation of the motion state:

[0054]

[0055] in, is the prior estimate of the motion state at time t, is a reliable a posteriori estimate of the motion state at time t-Δt, is the covariance of the prior estimate of the motion state, is the covariance of the reliable a posteriori estimate of the motion state, Q is the variance of the process noise;

[0056] Then, based on the observation equation, the integer pixel precision displacement signal of the input visual channel 1 provides a reliable a posteriori estimate of the motion state:

[0057]

[0058] In the formula, K t is the filter gain, R is the observation noise variance, E is the unit matrix, is a reliable a posteriori estimate of the motion state at time t;

[0059] Thus, a reliable a posteriori estimate of the motion state of the monitoring point at each moment with measurement robustness is obtained, and the displacement signal is extracted therefrom to achieve high-precision monitoring of the dynamic displacement of the structure.

[0060] The proposed method is used to monitor the dynamic displacement of a reinforced concrete structure, and the sensor results are used as a benchmark to compare with the single template matching algorithm and the optical flow algorithm. The experimental results are as follows: Figure 3 It can be seen that the measurement result of the method of the present invention is more consistent with the sensor reference value and has higher measurement accuracy.

Claims

1. A structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion, characterized in that: The following steps are involved: Step 1: Use an industrial camera to record the motion video of the structure in real time, and perform Gaussian filtering, denoising and grayscale processing frame by frame; Step 2: Comprehensive wavelet amplitude screening and corner response factor screening are used to establish a pixel point set with measurement robustness and select monitoring points; Step 3: Use the template matching algorithm to measure the displacement with integer pixel accuracy as the displacement signal of visual channel one, and use this signal as the dynamic window size reference value to interact with visual channel two. Use the optical flow algorithm to measure the speed with sub-pixel accuracy as the speed signal of visual channel two. Step 4: Based on the Kalman filter algorithm, the displacement signal of visual channel one and the velocity signal of visual channel two are fused to obtain the structural dynamic displacement signal with sub-pixel accuracy.

2. The structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion according to claim 1 is characterized in that: In step 2, wavelet amplitude screening and corner response factor screening are combined to establish a pixel point set with measurement robustness, and select monitoring points, as follows: First, the texture information is quantitatively characterized by wavelet amplitude: the initial frame image is convolved with a two-dimensional Gabor wavelet convolution kernel to extract its wavelet amplitude: In the formula, x and y are the horizontal and vertical coordinates of the pixel, respectively, A(x, y) is the wavelet amplitude of the pixel at (x, y), φ(x, y) is the wavelet phase of the pixel at (x, y), I(x, y) is the grayscale of the pixel at (x, y), G 2D is the real part of the two-dimensional Gabor wavelet convolution kernel, H 2D is the imaginary part of the two-dimensional Gabor wavelet convolution kernel, and i is the imaginary unit; After obtaining the wavelet amplitudes of all pixels, sort them and take the median A th As the threshold, the pixels with rich texture information are screened out to establish the pixel set E1 with rich texture information: E1={(x,y)A(x,y)>A th } Then the corner point characteristics are quantitatively characterized by the corner point response factor: for each pixel point of the initial frame image, its corner point response matrix is ​​calculated by the following formula: Where M(x,y) is the corner point response matrix of the pixel at (x,y), I x (m,n) and I y (m,n) are the grayscale gradients of the pixel at (m,n) in the horizontal and vertical directions, respectively, and W(x,y) is the neighborhood window of the pixel at (x,y); According to the corner point response matrix, solve its matrix eigenvalue, and calculate the corner point response factor based on the eigenvalue of the corner point response matrix: Where R(x,y) is the corner response factor of the pixel at (x,y), λ1(x,y) and λ2(x,y) are the two eigenvalues ​​of the corner response matrix M(x,y); After obtaining the corner response factors of all pixels, sort them and take the median R th As the threshold, the pixels with rich corner information are screened out, and the pixel set E2 with rich corner information is established: E2={(x,y)R(x,y)>R th } By performing an intersection operation on a set of pixels with rich texture information and a set of pixels with rich corner information, we can finally obtain a set of pixels with measurement robustness E. r , and select monitoring points from them: AND r =E1∩E2。 3. The structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion according to claim 1 is characterized in that: In step 3, the template matching algorithm is used to measure the displacement with integer pixel accuracy as the displacement signal of the visual channel 1, and the signal is used as the dynamic window size reference value to interact with the visual channel 2, and the optical flow algorithm is used to measure the speed with sub-pixel accuracy as the speed signal of the visual channel 2, as follows: Assume that the coordinates of the reliable monitoring point selected in the initial frame are (x0, y0), and establish a matching template with this point as the center: For each motion frame, first use the template matching algorithm to measure the displacement with integer pixel accuracy, slide the matching template on the motion frame image, and calculate the correlation when overlapping with each image block: Where C is the correlation coefficient, w and h are the width and height of the matching template, T(i, j) and I(i, j) are the grayscale of the matching template and the motion frame image block at the (i, j) position, T(m, n) and I(m, n) are the grayscale of the matching template and the motion frame image block at the (m, n) position; After obtaining the correlation coefficients of all positions, the position with the largest correlation coefficient is the position of the reliable monitoring point in the current motion frame. Subtracting the position coordinates of the reliable monitoring point in the current motion frame and the reference frame is the displacement with integer pixel accuracy, which is used as the displacement signal of visual channel one. Considering that the displacement signal can correctly characterize the motion trend of the structure, the displacement signal is used as a reference value of the dynamic window size to interact with visual channel two: In the formula, and are the integer pixel precision displacement signals in the x and y directions at time t, Δt is the time between frames, and They are the dynamic window sizes of the optical flow algorithm in the x and y directions at time t respectively; After obtaining the dynamic window size, the optical flow algorithm is used to measure the sub-pixel precision speed: According to the optical flow constraint equation: Where p1, p2, ..., p k is the pixel point in the dynamic window, I t (p) is the grayscale of the motion frame image at pixel p at time t, and are the grayscale gradients in the x and y directions of the motion frame image at the pixel point p at time t, and They are the speed signals in the x and y directions at time t respectively; The least square method is used to solve the optical flow constraint equation to obtain the sub-pixel precision speed as the speed signal of the second visual channel.

4. The structural dynamic displacement monitoring method based on dual-channel visual signal interaction and fusion according to claim 1 is characterized in that: In step 4, the displacement signal of the visual channel 1 and the speed signal of the visual channel 2 are fused based on the Kalman filter algorithm to obtain a structural dynamic displacement signal with sub-pixel accuracy, as follows: The evolution of the structural motion state is described by the state equation and the observation equation: s t =Fs t-Δt +Bc t +oh z t =Hs t +γ In the formula, is the real motion state vector of the monitoring point at time t, is the real displacement signal of the monitoring point at time t, is the real speed signal of the monitoring point at time t, Δt is the time between frames, is the state transfer matrix, is the motion control matrix, c t =(0,v t ) T is the motion input vector, v t is the sub-pixel precision velocity signal of the visual channel 2 at time t, z t =(u t ,0) T is the observation vector, u t is the integer pixel precision displacement signal of the visual channel 1 at time t, is the observation matrix, ω and γ are process noise and observation noise respectively; First, based on the state equation, the sub-pixel precision velocity signal of the second visual channel is input to provide a priori estimation of the motion state: in, is the prior estimate of the motion state at time t, is a reliable a posteriori estimate of the motion state at time t-Δt, is the covariance of the prior estimate of the motion state, is the covariance of the reliable a posteriori estimate of the motion state, Q is the variance of the process noise; Then, based on the observation equation, the integer pixel precision displacement signal of the input visual channel 1 provides a reliable a posteriori estimate of the motion state: In the formula, K t is the filter gain, R is the observation noise variance, E is the unit matrix, is a reliable a posteriori estimate of the motion state at time t; Thus, a reliable a posteriori estimate of the motion state of the monitoring point at each moment with measurement robustness is obtained, and the displacement signal is extracted therefrom to achieve high-precision monitoring of the dynamic displacement of the structure.

Citation Information

Patent Citations

  • Method for designing monocular vision odometer with light stream method and feature point matching method integrated

    CN103325108A

  • Method for detecting vehicle motion information based on integration of binocular stereoscopic vision and optical flow

    CN104318561A

  • Positioning device and method based on depth vision and robot

    CN108406731A

  • Tire vertical force estimating system and method based on binocular vision

    CN108749819A

  • Video image stabilization method combining optical flow method and Kalman filtering

    CN110796010A