Structural motion optical flow measurement method based on preorder motion sequence analysis

Abstract

The invention discloses a structural motion optical flow measurement method based on preorder motion sequence analysis, which comprises the following steps: firstly, collecting a motion video of a measured structure, and carrying out graying and Gaussian smooth denoising processing on each frame of the video; secondly, angular point characteristics are screened through angular point response factors, a pixel point set with measurement robustness is established in a structure imaging area, and measurement points are selected from the pixel point set; then, the window size is initialized, the number of preorder reference frames is set, light streams of the measuring points are calculated, and motion of the measuring points in all the preorder reference frames is obtained and serves as a preorder motion sequence; and finally, establishing a sequence analysis model, predicting the size of the optical flow matching window of the next frame based on the preorder motion sequence, calculating the motion of the measurement point in the next frame, adding the motion to the preorder motion sequence, removing the motion of the earliest frame in the original sequence, and repeating the process until the last frame. According to the method, statistical information of the structure motion history in a time domain is effectively utilized, the problem that a static window cannot give consideration to calculation stability and measurement accuracy is solved, and high-precision optical flow measurement of the structure motion based on vision can be realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of mechanical equipment health monitoring, and in particular relates to a structural motion optical flow measurement method based on preceding motion sequence analysis. Background Art

[0002] Structural motion measurement is an important task for the health monitoring of mechanical equipment, which can reflect the dynamic characteristics of the structure under working conditions. Due to its significant advantages such as non-contact, high spatial resolution of measurement, portability and flexibility, the vision-based structural motion measurement method has gradually received more attention from the industry. At present, a variety of visual algorithms have been developed, including template matching, feature tracking, digital image correlation, and optical flow algorithms. Among them, the optical flow algorithm is a visual algorithm that reflects the motion of the structure by analyzing the temporal change information of the grayscale of image pixels. Compared with several other visual methods, the optical flow algorithm does not require explicit tracking of structural surface features, does not rely on special artificial markings on the surface of the structure, and has high computational efficiency, can realize real-time measurement, and is more suitable for actual engineering tasks.

[0003] However, when the optical flow algorithm is measuring structural motion, it needs to be based on prior knowledge of the consistency of neighborhood motion, that is, to select an optical flow matching window with the measurement point as the center, and solve the optical flow of the central measurement point by least squares. The size of the optical flow matching window has a great impact on the measurement results. A larger window size will improve the calculation stability for fast motion, but will reduce the measurement accuracy of slow motion; while a smaller window size will improve the measurement accuracy of slow motion at the expense of the calculation stability of fast motion. At present, the selection of the optical flow matching window size is still a problem that needs to be solved urgently. In particular, when faced with the strong irregularity of structural motion, the performance of the optical flow algorithm cannot be guaranteed. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned existing structural motion measurement methods based on optical flow algorithms and solve the problem that the performance of the current optical flow algorithm is limited by the size of the optical flow matching window, the purpose of the present invention is to propose a structural motion optical flow measurement method based on the analysis of the preceding motion sequence. This method effectively utilizes the historical statistical information of the structural motion sequence, overcomes the problem that the static window cannot take into account both the calculation stability and the measurement accuracy, and can achieve high-precision optical flow measurement of structural motion.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for measuring structure motion optical flow based on preceding motion sequence analysis comprises the following steps:

[0007] Step 1: Use a high-speed camera to capture the motion video of the structure under test, and perform grayscale conversion and Gaussian smoothing denoising on each frame of the video;

[0008] Step 2: Screen the pixel corner characteristics through the corner response factor, establish a pixel set with measurement robustness in the structural imaging area, and select the measurement points from it;

[0009] Step 3: Initialize the window size and set the number of previous reference frames. Calculate the optical flow by minimizing the optical flow matching error function, and obtain the motion of the measurement point in all previous reference frames as the previous motion sequence.

[0010] Step 4: Establish a structural motion sequence analysis model, predict the optical flow matching window size of the next frame based on the previous motion sequence, calculate the motion of the measurement point in the next frame, add it to the previous motion sequence and remove the motion of the earliest frame in the original sequence, and repeat this process until the last frame.

[0011] Compared with the existing structure motion measurement method based on optical flow algorithm, the advantages of the present invention are as follows:

[0012] (1) The present invention pre-calculates the corner point response factors and automatically screens the corner point characteristics of all pixel points in the field, thereby providing highly robust measurement points that can be used for optical flow calculation, avoiding the blind selection of measurement points, and eliminating the need for special manual marking on the surface of the structure.

[0013] (2) The present invention effectively utilizes the historical statistical information of the structural motion sequence and can dynamically predict and update the window size in real time, thus overcoming the problem that the static window cannot take into account both computational stability and measurement accuracy, and can achieve high-precision optical flow measurement of structural motion. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 Flowchart of the structure-motion optical flow measurement algorithm based on analysis of preceding motion sequences.

[0015] Figure 2 Schematic diagram of the motion of the optical flow matching window between frames.

[0016] Figure 3 It is a schematic diagram of the real-time dynamic window update based on the structural motion sequence analysis of the present invention.

[0017] Figure 4 (a) is the time domain information of the motion of the concrete building measured by the method of the present invention.

[0018] Figure 4 (b) is the dynamic window size adjustment history during the measurement process of the method of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below in conjunction with the accompanying drawings.

[0020] like Figure 1 As shown, a method for measuring structure motion optical flow based on preceding motion sequence analysis comprises the following steps:

[0021] Step 1: Use a high-speed camera to capture the motion video of the structure under test, and perform grayscale conversion and Gaussian smoothing denoising on each frame of the video;

[0022] Step 2: Screen the pixel corner characteristics through the corner response factor, establish a pixel set with measurement robustness in the structural imaging area, and select the measurement points from it;

[0023] First, the corner response factor is calculated to quantify the pixel corner characteristics of the initial frame of the video. The corner response matrix of each pixel can be calculated as:

[0024]

[0025] Where H(x,y) is the corner point response matrix of the pixel at (x,y), I x (m,n) and I y (m,n) are the grayscale gradients of the pixel at (m,n) in the horizontal and vertical directions, respectively. W(x,y) is the neighborhood window of the pixel at (x,y). (m,n) are both in the W(x,y) set.

[0026] When the pixel corner characteristics meet the measurement criteria, the two eigenvalues ​​of the corner response matrix should be at the local peak value. Therefore, the corner response factor is defined as:

[0027]

[0028] Where Ind(x,y) is the corner response factor of the pixel at (x,y), λ1(x,y) and λ2(x,y) are the two eigenvalues ​​of the corner response matrix H(x,y);

[0029] After calculating the corner response factors of all pixels in the field, they are sorted and the median of the corner response factors Ind is used as the median. mid As the screening threshold of pixel corner point characteristics, a set of pixel points E with high measurement robustness is obtained, and measurement points are selected from them:

[0030] E={(x,y)Ind(x,y)>Ind mid |(x,y)∈R}

[0031] Where R is the structural imaging area;

[0032] Step 3: Initialize the window size and set the number of previous reference frames. Calculate the optical flow by minimizing the optical flow matching error function, and obtain the motion of the measurement point in all previous reference frames as the previous motion sequence.

[0033] Assume that the initial window horizontal and vertical sizes are The number of previous reference frames is n. First, based on the initially set window size, the optical flow algorithm is used to calculate the motion of the measurement point in the previous 2n+1 frames, such as Figure 2 The figure shows the schematic diagram of the optical flow matching window moving between frames. When the measurement point moves between two adjacent frames, the optical flow matching error function is defined as:

[0034]

[0035] Where ε is the optical flow matching error function, p = (p x ,p y ) is the coordinate of the measuring point in the current frame, d=(d x ,d y ) is the optical flow of the measurement point in the current frame, I(x,y) is the pixel grayscale at the coordinate (x,y) of the current frame, J(x+d x ,y+d y ) is the coordinate of the next frame at (x+d x ,y+d y ), ωx and ωy are the half-window sizes in the x and y directions respectively;

[0036] By minimizing the optical flow matching error function, the optical flow can be solved as:

[0037]

[0038] Calculate the optical flow between all previous reference frames, and obtain the motion of the measurement point in the previous 2n+1 frames as the previous motion sequence:

[0039] s={d 1 ,d 2 ,...,d 2n+1}

[0040] Where s is the previous motion sequence, d i is the optical flow of the measurement point in the i-th frame;

[0041] Step 4: Establish a structural motion sequence analysis model, predict the optical flow matching window size of the next frame based on the previous motion sequence, calculate the motion of the measurement point in the next frame, add it to the previous motion sequence and remove the motion of the earliest frame in the original sequence, and repeat this process until the last frame;

[0042] Based on the temporal dependency hypothesis, that is, there is a strong correlation between the structural motions of different frames, and the structural motion of a given frame is significantly affected by its past structural motion, a structural motion sequence analysis model is established, as shown in the following formula, the structural motion of the next frame can be predicted by its previous motion:

[0043]

[0044] In the formula, is the predicted motion of the tth frame, a0 is a constant term, a i is the influence factor of the previous motion of the past i frames on the motion of the current frame;

[0045] By substituting the previous motion sequence data into the established structural motion sequence analysis model, the equation group can be obtained:

[0046]

[0047] Thus, we can n Solve. Figure 3 The figure shows a schematic diagram of real-time dynamic window update based on structural motion sequence analysis. The motion of the next frame is predicted based on the structural motion sequence analysis model. It is updated as the reference value of the next frame optical flow matching window size, as shown in the following formula:

[0048]

[0049] Then (2w t +1) is the optical flow matching window size of the current t-th frame;

[0050] Based on the optical flow matching window size, the optical flow algorithm is used to calculate the motion of the measurement point in the next frame, add it to the previous motion sequence and remove the motion of the earliest frame in the original sequence. This is repeated until the last frame of the video to obtain the optical flow of the measurement point in all frames, thereby obtaining high-precision structural motion time domain information.

[0051] The method of the present invention is used to measure the motion of concrete building structures. The measurement results of the time domain information of the structure motion are as follows: Figure 4 As shown in (a), the dynamic window adjustment history during the measurement process is as follows Figure 4 As shown in (b), it can be seen that the method of the present invention can predict and adjust the matching window size in real time according to the situation of structural motion, taking into account both calculation stability and measurement accuracy, and is suitable for structural motion measurement under actual complex working conditions.

Claims

1. A method for measuring structured motion optical flow based on analysis of preceding motion sequences, characterized in that: The following steps are involved: Step 1: Use a high-speed camera to capture the motion video of the structure under test, and perform grayscale conversion and Gaussian smoothing denoising on each frame of the video; Step 2: Screen the pixel corner characteristics through the corner response factor, establish a pixel set with measurement robustness in the structural imaging area, and select the measurement points from it; Step 3: Initialize the window size and set the number of previous reference frames. Calculate the optical flow by minimizing the optical flow matching error function, and obtain the motion of the measurement point in all previous reference frames as the previous motion sequence. Step 4: Establish a structural motion sequence analysis model, predict the optical flow matching window size of the next frame based on the previous motion sequence, calculate the motion of the measurement point in the next frame, add it to the previous motion sequence and remove the motion of the earliest frame in the original sequence, and repeat this process until the last frame.

2. The method for measuring structure motion optical flow based on preceding motion sequence analysis according to claim 1, characterized in that: In step 2, the pixel corner point characteristics are screened by the corner point response factor, a pixel point set with measurement robustness is established in the structural imaging area, and measurement points are selected from the pixel point set, as follows: First, the corner response factor is calculated to quantitatively characterize the pixel corner characteristics of the initial frame of the video: the corner response matrix of each pixel is: Where H(x,y) is the corner point response matrix of the pixel at (x,y), I x (m,n) and I y (m,n) are the grayscale gradients of the pixel at (m,n) in the horizontal and vertical directions, respectively. W(x,y) is the neighborhood window of the pixel at (x,y). (m,n) are both in the W(x,y) set. When the pixel corner characteristics meet the measurement criteria, the two eigenvalues ​​of the corner response matrix should be at the local peak value. Therefore, the corner response factor is defined as: Where Ind(x,y) is the corner response factor of the pixel at (x,y), λ1(x,y) and λ2(x,y) are the two eigenvalues ​​of the corner response matrix H(x,y); After calculating the corner response factors of all pixels in the field, they are sorted and the median of the corner response factors Ind is used as the median. mid As the screening threshold of pixel corner point characteristics, a set of pixel points E with high measurement robustness is obtained, and measurement points are selected from them: E={(x,y)Ind(x,y)>Ind mid |(x,y)∈R} Where R is the structural imaging area.

3. The method for measuring structure motion optical flow based on preceding motion sequence analysis according to claim 1, characterized in that: In step 3, the window size is initialized, and the number of previous reference frames is set. The optical flow is calculated by minimizing the optical flow matching error function, and the motion of the measurement point in all previous reference frames is obtained as the previous motion sequence, which is as follows: Assume that the initial window horizontal and vertical sizes are The number of previous reference frames is n. First, based on the initially set window size, the optical flow algorithm is used to calculate the motion of the measurement point in the previous 2n+1 frames. When the measurement point moves in two adjacent frames, the optical flow matching error function is defined as: Where ε is the optical flow matching error function, p = (p x ,p y ) is the coordinate of the measuring point in the current frame, d=(d x ,d y ) is the optical flow of the measurement point in the current frame, I(x,y) is the pixel grayscale at the coordinate (x,y) of the current frame, J(x+d x ,y+d y ) is the coordinate of the next frame at (x+d x ,y+d y ), ωx and ωy are the half-window sizes in the x and y directions respectively; By minimizing the optical flow matching error function, the optical flow is solved as: In the formula, d(p x ,p y ) is the measuring point (p x ,p y ), I(x,y) is the pixel grayscale at the current frame coordinate (x,y), J(x,y) is the pixel grayscale at the next frame coordinate (x,y), I x (x,y) and I y (x, y) are the grayscale gradients of the pixel at the coordinate (x, y) in the horizontal and vertical directions of the current frame respectively; Calculate the optical flow between all previous reference frames, and obtain the motion of the measurement point in the previous 2n+1 frames as the previous motion sequence: s={d 1 ,d 2 ,...,d 2n+1 } Where s is the previous motion sequence, d i is the optical flow of the measurement point in the i-th frame.

4. The method for measuring structure motion optical flow based on preceding motion sequence analysis according to claim 1, characterized in that: In step 4, a structural motion sequence analysis model is established, and the optical flow matching window size of the next frame is predicted based on the previous motion sequence, and the motion of the measurement point in the next frame is calculated and added to the previous motion sequence and the motion of the earliest frame in the original sequence is removed. The process is repeated until the last frame, as follows: Based on the temporal dependency hypothesis, that is, there is a strong correlation between the structural motions of different frames, and the structural motion of a given frame is significantly affected by its past structural motion, a structural motion sequence analysis model is established, as shown in the following formula, where the structural motion of the next frame is predicted by its previous motion: In the formula, is the predicted motion of the tth frame, a0 is a constant term, a i is the influence factor of the previous motion of the past i frames on the motion of the current frame; By substituting the previous motion sequence data into the established structural motion sequence analysis model, the equation group is obtained: Thus, we can n Solve it and predict the motion of the next frame based on the structural motion sequence analysis model. As a reference value for the size of the optical flow matching window for the next frame, it is shown in the following formula: Then (2w t +1) is the optical flow matching window size of the current t-th frame; Based on the optical flow matching window size, the optical flow algorithm is used to calculate the motion of the measurement point in the next frame, add it to the previous motion sequence and remove the motion of the earliest frame in the original sequence. This is repeated until the last frame of the video to obtain the optical flow of the measurement point in all frames, thereby obtaining high-precision structural motion time domain information.

Images