Computer vision measurement method for unmarked structure vibration
Through the computer vision measurement method of markless structure vibration, the inter-frame tracking is performed using ORB feature points and pyramid LK optical flow method, which solves the problems of low efficiency and unavailability of markers in the prior art, and realizes the vibration displacement of markless cantilever beams Efficient detection and analysis.
Patent Information
- Application Number
- CN202411269517.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-08
AI Technical Summary
Existing computer vision vibration measurement techniques need to rely on manual marking and are inefficient to achieve efficient displacement measurement without marking, especially when marking is not allowed on some critical structures.
The computer vision measurement method of markless structure vibration is adopted, and the reprojection error is obtained through the calibration plate calibration method. The video is captured using a monocular vision system, and the inter-frame tracking is performed by combining ORB feature point detection and pyramid LK optical flow method to calculate the optical flow displacement. And the vibration frequency of the cantilever beam is obtained through fast Fourier transform.
The detection and analysis of vibration displacement of labelless cantilever beams is realized, and an efficient vibration displacement measurement method for labelless experimental subjects is provided, solving the problems of low efficiency and unavailable marking of traditional methods.
Smart Images

Figure CN118918099A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of mechanism vibration measurement, and in particular to a computer vision measurement method for non-marking structure vibration. Background Art
[0002] In the practice of structural health monitoring, it is crucial to accurately obtain the vibration characteristics of the structure. Traditional vibration detection methods, such as accelerometers, must be attached to the surface of the structure. This contact method may have a significant impact on the dynamic behavior of the structure due to the additional mass. At the same time, due to the limitations of the arrangement and number of measurement points, these technologies are usually unable to provide high-resolution local area measurement data. In sharp contrast, computer vision measurement technology, due to its non-contact and high-resolution characteristics, has been widely used and recognized in many fields such as aerospace, civil engineering, and intelligent manufacturing.
[0003] At present, the mainstream methods of computer vision technology in the field of vibration measurement include digital image correlation (DIC), geometric matching and optical flow. DIC and geometric matching techniques often require specific feature markers to be set on the measurement object in advance, such as random speckles or regular geometric patterns. However, in actual operation, these artificial markers may fall off due to external forces or other factors, and on the positive sample surfaces of certain key structures, it is usually not allowed to add any artificial markers due to functional requirements. In addition, a large number of similarity calculations are involved in the inter-frame tracking and stereo matching process, which is not only time-consuming, but also particularly inefficient in displacement field measurement scenarios that require multiple measuring points. These problems limit the application of vibration measurement technology based on computer vision in certain occasions. Summary of the invention
[0004] The purpose of the present invention is to provide a computer vision measurement method for marker-free structural vibration, which solves the problem that the prior art method needs to rely on speckle and artificial marking and has low efficiency and cannot achieve displacement measurement without markers.
[0005] To achieve the above object, the present invention provides a computer vision measurement method for markerless structural vibration, comprising the following steps:
[0006] S1. Obtain the reprojection error through the calibration plate calibration method, analyze the reprojection error of the calibration point in the image, and use it to evaluate whether the installation position and direction of the monocular vision system and the cantilever beam meet the expected accuracy requirements;
[0007] S2, using a monocular vision system to shoot a video of the vibration of the unmarked structure, graying the RGB image of the video frame of the unmarked structure, intercepting the ROI image, and obtaining a preprocessed video;
[0008] S3, select the first frame image of the pre-processed unmarked structure vibration video, and perform feature point detection through the ORB feature point detection algorithm to determine the pixel coordinates of each ORB feature point in the image, and at the same time filter out unnecessary feature points through the corner point response value threshold;
[0009] S4. Based on the assumption of constant brightness, an optical flow equation is established, a multi-scale pyramid is constructed, and the optical flow residual is minimized. By constructing the initial optical flow guess, the optical flow transfer between the upper and lower layers is calculated to obtain the final optical flow calculation result. The ORB feature points detected in the first frame image are tracked by the pyramid LK optical flow method for inter-frame tracking.
[0010] S5. Obtain the pixel displacement vibration curve of the ORB corner point in each frame of the markerless structure vibration video, convert the pixel displacement by a proportional factor to obtain the actual displacement vibration curve in three-dimensional space, and use fast Fourier transform to obtain the cantilever beam vibration frequency.
[0011] Preferably, in step S1, a camera calibration technique using a calibration plate marking method is used to establish a mapping relationship between three-dimensional actual coordinates and two-dimensional pixel coordinates, wherein the mapping relationship involves three basic transformations: rigid body transformation, perspective projection, and affine transformation. The specific steps are as follows:
[0012] S11, capturing a calibration plate image, wherein the calibration plate image includes images of front view, upward view, downward view, and left and right oblique view relative to the monocular vision system;
[0013] S12, obtaining calibration, wherein the method for obtaining calibration is to record a calibration video, continuously change the position of the calibration plate relative to the camera during the recording process, and subsequently select frames with positions that meet the requirements and clear images from the calibration video as calibration images;
[0014] S13, when performing detection, locate the inner corner points, and use the inner corner points as a point set for calibration;
[0015] S14, analyzing the reprojection error of the calibration point in the image, wherein the reprojection error is the deviation between the actual imaging position and the imaging position predicted by the calibration parameters. It is expressed as:
[0016]
[0017] Where n is the total number of points in the image, (x i ,y i ) is the coordinate on the two-dimensional plane after reprojection transformation, (a i ,b i ) are the coordinates on the two-dimensional plane actually measured.
[0018] Preferably, step S2 is specifically as follows: in the preprocessing process, firstly extract the first frame of the marker-free structure vibration video, convert it into a grayscale image, intercept the ROI image, and obtain the preprocessed video.
[0019] Preferably, in step S3, an ORB feature point detection algorithm is used to detect corner point features in the first frame image, and a response value is assigned to each detected feature point. A threshold screening method is used to retain feature points whose response values exceed a predetermined threshold. The specific process is as follows:
[0020] The ORB feature detection algorithm uses the FAST operator in the feature point detection part, and adds the directional feature to the FAST operator to generate the o-FAST algorithm. The o-FAST algorithm uses the Harris corner point detection scoring mechanism to sort the candidate points. By selecting the N points with the highest Harris response values, the o-FAST algorithm selects the best corner points from the candidate points.
[0021] The FAST operator takes a pixel point P, considers the 16 pixels around the pixel point P, calculates the gray value difference between the pixel point P and the surrounding 16 pixels, takes the absolute value, and compares it with the threshold:
[0022] If it is greater than the threshold, the two pixels are considered different. If there are N consecutive pixels around the pixel P that meet this condition, the pixel P is defined as a FAST feature point. The feature point determination method is:
[0023] ∣G P -G i ∣>G th i=1,2,…,16 (2)
[0024] In the formula, G P Represents the gray value of pixel P, G i Represents the gray value of the i-th pixel around the pixel P, G th Represents the gray value threshold.
[0025] Preferably, the o-FAST algorithm is generated by adding directional features to the FAST operator, and the specific process is as follows:
[0026] Through the grayscale centroid method, a coordinate system is established with the feature point as the origin, the centroid position is calculated in the neighborhood S, and a vector is constructed with the feature point P as the starting point and the grayscale centroid Q as the end point. Then the moment of the domain S about I(x,y) is:
[0027]
[0028] In the formula, m pqRepresents the moment of domain S about I(x,y), I(x,y) is the gray value of the image, p and q are the orders of the moment, x and y are the coordinates of the pixel point, x, y∈[-r,r], r is the radius of the feature point domain S;
[0029] The centroid C of the image region is expressed as:
[0030]
[0031] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, m 00 The sum of the grayscale values of all pixels in the image block, about the x-coordinate of the grayscale centroid, is:
[0032]
[0033] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image;
[0034] The y coordinate of the grayscale centroid is:
[0035]
[0036] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image;
[0037] Get the direction angle θ of the FAST feature point P:
[0038] θ=arctan2(c y ,c x )=arctan2(m 01 ,m 10 ) (7)
[0039] In the formula, c x is the x coordinate of the grayscale centroid, c y is the grayscale centroid y coordinate, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, and x and y are the pixel coordinates;
[0040] The o-FAST algorithm is used to calculate the direction angle θ of the feature point P, and further obtain the direction angles of all feature points.
[0041] Preferably, the ORB feature detection algorithm adopts an improved BRIEF descriptor r-BRIEF in the feature description part, and improves the BRIEF descriptor by introducing the direction information of the key points. The specific process is as follows:
[0042] Rewrite the point set into matrix form:
[0043] At position (x i ,y i ), for any n binary criterion feature sets, define a 2n matrix S:
[0044]
[0045] According to the direction angle θ of the feature point p obtained by the o-FAST algorithm, find its corresponding rotation matrix R θ :
[0046]
[0047] This results in the corrected version S θ :
[0048] S θ =R θ S (10)
[0049] Where S is the defined 2n matrix;
[0050] Then the SteeredBRIEF descriptor g n It is expressed as:
[0051] g n (p,θ)=f nd (p)|(x i ,y i )∈S θ (11)
[0052] p(x i ,y i ) represents the position of feature point p, θ represents the direction angle of feature point p, and f nd It is the BRIEF descriptor.
[0053] Preferably, step S4 is specifically:
[0054] Based on the assumption that the image brightness is constant, the optical flow is calculated by solving the optical flow equation. For each pixel brightness I(x,y,t) in the image, its displacement in the time interval from time t to t+Δt is (u,v). The optical flow equation is expressed as:
[0055] I(x+Δx,y+Δy,t+Δt)=I(x,y,t) (12)
[0056] Where Δx is the horizontal displacement of the pixel within Δt time, and Δy is the vertical displacement of the pixel within Δt time;
[0057] The pyramid LK optical flow method assumes that the motion vector in a neighborhood Ω in a space is constant, then in a neighborhood Ω of n pixels, each pixel satisfies the following formula:
[0058] I xi u+I yi v+I ti =0 i=1,2,···n (13)
[0059] In the formula, and are the spatial gradients in the x and y directions respectively, u and v represent the moving components of the optical flow in the horizontal and vertical directions, that is, the required optical flow information, is the rate of change of image grayscale relative to time t, i represents the i-th pixel;
[0060] Let I 0 is the grayscale representation of the original image (w×h), located at the 0th layer of the pyramid, and sequentially generates the 1st, 2nd, ..., N-1th layers of the N-layer Gaussian image pyramid, denoted as I 1 , I 2 ,…,I N-1 ;
[0061] For the original image I 0 The corner point X(x,y) in the Lth layer image I L X in L (x L ,y L ) corresponds to it; for a given two consecutive images I a ,I b , is to find I a The coordinates of the previous point U=(u x ,u y ), in I b The corresponding point on the image has a similar image intensity U+d=(u x +d x ,u y +d y ), where d is the distance from point U to the previous frame I a To the next frame I b The amount of translation;
[0062] The optical flow is defined as the displacement that produces the best fit between image regions at different times, centered at point U. a ,I b Establish U+d=(u x +d x,u y +d y ), and assuming that the pixels in the corresponding window have the same motion displacement vector d, the neighboring point N(U) of point U is expressed as follows:
[0063]
[0064] Where u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the size of the window on the x and y axes. The size of the window is (2w x +1,2w y +1);
[0065] Define the optical flow residual E(d) of the pixels in the window:
[0066]
[0067] Where d is the previous frame I of point U a To the next frame I b The translation amount, d x and d y Respectively expressed as the translation in the x and y directions, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes, I a (x,y) and I b (x+d x ,y+d y ) indicates I a and I b Grayscale values of the two images;
[0068] Find the partial derivative of the translation d and set it to 0, where I b (x+d x ,y+d y ) Perform a first-order Taylor expansion approximation at d = 0. The spatial gradient matrix G is used to describe the spatial change rate of the pixel point. The spatial gradient matrix G is defined as:
[0069]
[0070] In the formula, and Respectively represent the square of the brightness change rate of the pixel in the x and y directions, which is used to characterize the texture information of the image in a specific direction. x I yIndicates the cross-change rate of the brightness of the pixel in the x and y directions, which is used to capture the correlation of the brightness in the image in two directions. x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes;
[0071] The error matching vector is:
[0072]
[0073] Wherein, the error matching vector b is used to store the difference between the optical flow estimation value of the pixel point and its true value. is the rate of change of image grayscale relative to time t, i represents the i-th pixel, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes;
[0074] Then the optimal vector solution d oe for:
[0075] d oe =G -1 b (18)
[0076] Where b is the error matching vector, G -1 is the inverse matrix of the spatial gradient matrix;
[0077] The Newton-Raphson optimization method is used to refine the translation. Let e k =G -1 b k is the estimated translation at the kth iteration, b k is the error matching vector of the kth iteration, minimizing the residual E(d), and we have:
[0078]
[0079] Where d0 is the initial translation, subscript 0 represents the initial iteration, and d k is the displacement of the kth estimation iteration, d k-1 is the displacement of the k-1th estimation iteration;
[0080] For two adjacent images on the same level L of the image sequence pyramid Its optical flow residual E L Defined as:
[0081]
[0082] Where, d L is the optical flow result calculated by L layer, g L is the optical flow guess value corresponding to the L layer, which is recursively calculated from the N-1 to L+1 layers. The initial optical flow guess value g at the top layer N-1 =0, is the gray value corresponding to the previous frame of layer L, is the gray value corresponding to the next frame of layer L, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes;
[0083] The optical flow calculation and transmission process between the upper and lower layers is:
[0084] g N-1 =0 (21)
[0085] g L-1 =2(g L +d L ) (twenty two)
[0086] In the formula, g N-1 is the initial optical flow guess value of the N-1th layer. The superscript N-1 represents the N-1th layer of the pyramid. L is the optical flow guess value corresponding to the L layer, g L-1 is the estimated value of the optical flow at the L-1 layer, d L The optical flow result calculated for the L layer;
[0087] Until the 0th layer is calculated and the final optical flow displacement vector d result is obtained:
[0088]
[0089] In the formula, g 0 is the optical flow guess value of the 0th layer. The superscript 0 represents the 0th layer of the pyramid. 0 is the optical flow result calculated at the 0th layer. The superscript 0 represents the 0th layer of the pyramid, N is the number of pyramid layers, L represents the number of layers the optical flow is being calculated, and d L The optical flow result calculated for the L layer;
[0090] Then the previous frame image I a The pixel point U in the next frame image I b The corresponding points in are:
[0091] ν=u+d (24)
[0092] Where ν is I b The pixel position in the middle, u is I aThe pixel position in , d is the final optical flow displacement vector.
[0093] Preferably, step S5 specifically includes: obtaining a pixel displacement time-history curve of a designated target point by target tracking using an optical flow algorithm, and converting the actual displacement of the target in the world coordinate system by a scaling factor:
[0094]
[0095] Where S represents the scale factor, that is, the actual size of the target corresponding to the unit pixel length in the image coordinate system, and D and d p They represent actual size and pixel length respectively, Z represents the distance between the lens and the object being photographed, and f represents the focal length of the lens.
[0096] Therefore, the present invention adopts the above-mentioned computer vision measurement method for markerless structural vibration, and the beneficial effects are as follows:
[0097] The present invention studies the visual vibration displacement measurement of the fusion feature point method and the sparse optical flow method, and proposes a multi-information fusion pyramid LK optical flow algorithm based on ORB corner points, which solves the problem that the existing technology relies on speckle and artificial marking and is inefficient, and the method without marking cannot achieve displacement measurement, and realizes the research on vibration displacement detection of unmarked cantilever beam. The measurement device of the method proposed by the present invention is simple, and provides a new measurement method for vibration displacement analysis of unmarked experimental objects.
[0098] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS
[0099] Figure 1 It is an overall flow chart of an embodiment of a method for computer vision measurement of markerless structural vibration according to the present invention. DETAILED DESCRIPTION
[0100] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.
[0101] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.
[0102] The technical solution adopted by the present invention is to study the visual vibration displacement measurement by integrating the feature point method and the sparse optical flow method, and propose a multi-information fusion pyramid LK optical flow algorithm based on ORB corner points to achieve vibration displacement measurement without marking.
[0103] like Figure 1 As shown, a computer vision measurement method for markerless structural vibration includes the following steps:
[0104] S1. Obtain the reprojection error through the calibration plate calibration method, and analyze the reprojection error of the calibration point in the image to evaluate whether the installation position and direction of the monocular vision system and the cantilever beam meet the expected accuracy requirements;
[0105] This embodiment adopts a camera calibration technology based on the calibration plate marking method, by establishing a mapping relationship between three-dimensional actual coordinates and two-dimensional pixel coordinates, which involves three basic transformations: rigid body transformation, perspective projection and affine transformation. In this process, the camera's internal and external parameter matrices are accurately determined. The specific steps are as follows:
[0106] S11. In order to make the calibration result more accurate and reliable, twenty-one calibration plate images are taken, and these calibration plate images include images of front view, upward view, downward view, and left and right oblique view relative to the monocular vision system;
[0107] S12, obtain calibration:
[0108] The method required for obtaining the calibration is to record a calibration video, continuously change the position of the calibration plate relative to the camera during the recording process, and then select frames with positions that meet the requirements and clear images from the calibration video as calibration images;
[0109] S13. In practice, many unnecessary corner points may be extracted. In order to avoid interference from corner points outside the chessboard in the image, the inner corner points are located by using the characteristic that the inner corner points have four adjacent squares during detection, and the inner corner points are used as the point set for calibration.
[0110] S14. Analyze the reprojection error of the calibration points in the image. The reprojection error is the deviation between the actual imaging position and the imaging position predicted by the calibration parameters. The reprojection error is the core evaluation indicator of the calibration process in the field of computer vision. It quantifies the deviation between the projection position of the three-dimensional world point estimated by the camera intrinsic and extrinsic parameters on the two-dimensional image plane and the actual observed image point position. This metric is crucial for verifying the accuracy of the camera model, correcting system errors, and optimizing camera parameters. The smaller the reprojection error, the more accurate the calibration algorithm. When the reprojection error is less than one pixel, it can meet most application requirements.
[0111] Reprojection Error It can be expressed as:
[0112]
[0113] Where n is the total number of points in the image, (x i ,y i ) is the coordinate on the two-dimensional plane after reprojection transformation, (a i ,b i) are the coordinates on the two-dimensional plane actually measured.
[0114] In this embodiment, after calculation with 21 calibration plates, the highest reprojection error is less than 0.25 pixels and the average is 0.17 pixels. This value indicates that the inconsistency between the calculated 2D projection position of the 3D world point and the actual observed image point position is very small. This ensures the reliability of the calibration results and is suitable for subsequent visual measurement and 3D reconstruction applications.
[0115] S2, using a monocular vision system to shoot a video of the vibration of the unmarked structure, graying the RGB image of the video frame of the unmarked structure, intercepting the ROI image, and obtaining a preprocessed video;
[0116] From the reprojection error obtained in step S1, it can be seen that the inconsistency between the two-dimensional projection position and the actual observed image point position is extremely small, and the multi-information fusion LK optical flow algorithm can be used for tracking. The video clip used is 2 seconds long, and the acquisition frame rate of the monocular vision system is 60Hz, corresponding to a 120-frame image sequence. In the preprocessing process, the first frame of the unmarked structure vibration video is first extracted and converted into a grayscale image to reduce the computational complexity; the ROI image is intercepted to obtain the preprocessed video.
[0117] S3, select the first frame image of the unmarked structure vibration video after preprocessing, and perform feature point detection through the ORB feature point detection algorithm to determine the pixel coordinates of each ORB feature point in the image, and at the same time filter out unnecessary feature points through the corner point response value threshold.
[0118] The ORB feature point detection algorithm is used to detect corner point features in the first frame image. These feature points are evaluated by the algorithm based on their significance and uniqueness. The ORB feature point detection algorithm assigns a response value to each detected feature point, which quantifies the significance of the feature point, where feature points with higher response values are considered to be more significant. In order to further refine the selection of feature points, this embodiment adopts a threshold screening method to only retain feature points whose response values exceed a predetermined threshold. The specific process is as follows:
[0119] The ORB feature detection algorithm uses the FAST (Features from Accelerated Segment Test) operator in the feature point detection part, and adds directional features to the FAST operator to generate the o-FAST algorithm, which is used to improve the lack of directionality of the FAST operator. The FAST operator has been widely used in application scenarios with high real-time requirements such as parallel tracking and video stitching due to its accurate, efficient and rapid corner point extraction capabilities. In terms of feature point detection, the ORB algorithm improves the detection performance of feature points by improving the FAST operator, and also enhances the robustness of the algorithm to image transformation.
[0120] At the same time, the o-FAST algorithm uses the Harris corner point detection scoring mechanism to sort the candidate points. Harris scoring can evaluate the stability and significance of corner points. Therefore, by selecting the N points with the highest Harris response values, the o-FAST algorithm can select the best corner points from the candidate points;
[0121] The FAST operator takes a pixel point P, considers the 16 pixels around the pixel point P, calculates the grayscale value difference between the pixel point P and the surrounding 16 pixels, takes the absolute value, and compares it with the threshold:
[0122] If it is greater than the threshold, the two pixels are considered different. If there are N consecutive pixels around the pixel P that meet this condition, the pixel P is defined as a FAST feature point. The feature point judgment method is:
[0123] ∣G P -G i ∣>G th i=1,2,…,16 (2)
[0124] In the formula, G P Represents the gray value of pixel P, G i Represents the gray value of the i-th pixel around the pixel P, G th Indicates the gray value threshold, which depends on the working conditions.
[0125] At the same time, the FAST operator is added with directional features to generate the o-FAST algorithm. This feature makes the feature description operator rotationally invariant. In order to introduce the directional characteristics of feature points when they rotate, the idea of effectively establishing the directional vector is used with the help of the grayscale centroid method to solve this problem. The specific process is as follows:
[0126] Through the grayscale centroid method, a coordinate system is established with the feature point as the origin, the centroid position is calculated in the neighborhood S, and a vector is constructed with the feature point P as the starting point and the grayscale centroid Q as the end point. Then the moment of the domain S about I(x,y) is:
[0127]
[0128] In the formula, m pq Represents the moment of domain S about I(x,y), I(x,y) is the gray value of the image, p and q are the orders of the moment, x and y are the coordinates of the pixel point, x, y∈[-r,r], r is the radius of the feature point domain S;
[0129] The centroid C of the image area can be expressed as:
[0130]
[0131] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, m 00 The sum of the grayscale values of all pixels in the image block, about the x-coordinate of the grayscale centroid, is:
[0132]
[0133] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image;
[0134] The y coordinate of the grayscale centroid is:
[0135]
[0136] In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image;
[0137] Get the direction angle θ of the FAST feature point P:
[0138] θ=arctan2(c y ,c x )=arctan2(m 01 ,m 10 ) (7)
[0139] In the formula, c x is the x coordinate of the grayscale centroid, c y is the grayscale centroid y coordinate, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, and x and y are the pixel coordinates. The o-FAST algorithm is used to effectively calculate the direction angle θ of the feature point P, and further obtain the direction angles of all feature points.
[0140] Based on the improvement of the original BRIEF descriptor, the ORB feature detection algorithm uses the improved BRIEF descriptor r-BRIEF in the feature description part to enhance the rotation invariance. r-BRIEF improves the BRIEF descriptor by introducing the directional information of the key points, correcting the defect that the RIEF descriptor does not have the plane rotation invariance, thereby improving the robustness of the ORB algorithm in feature matching. The specific process is as follows: The ORB operator uses a 256-bit descriptor, namely BRIEF-32. In order to input the directional information into BRIEF, the point set is rewritten into a matrix form:
[0141] At position (x i ,y i ), for any n binary criterion feature sets, define a 2n matrix S:
[0142]
[0143] According to the direction angle θ of the feature point p obtained by the o-FAST algorithm, find its corresponding rotation matrix R θ :
[0144]
[0145] This results in the corrected version S θ :
[0146] S θ =R θ S (10)
[0147] Where S is the defined 2n matrix;
[0148] Then the SteeredBRIEF descriptor g n It is expressed as:
[0149] g n (p,θ)=f nd (p)|(x i ,y i )∈S θ (11)
[0150] p(x i ,y i ) represents the position of feature point p, θ represents the direction angle of feature point p, and f nd It is the BRIEF descriptor.
[0151] S4. Based on the assumption of constant brightness, establish the optical flow equation, construct a multi-scale pyramid, minimize the optical flow residual, construct the initial optical flow guess, calculate the optical flow transfer between the upper and lower layers, and obtain the final optical flow calculation result. Use the pyramid LK optical flow method to track the ORB feature points detected in the first frame image for inter-frame tracking. Specifically:
[0152] By performing threshold screening on the feature point set output by the ORB feature detector, feature points whose response values exceed the predetermined threshold are defined as key points, and the coordinates of these key points are then used for subsequent tracking analysis.
[0153] Based on the assumption that the image brightness is constant, the optical flow can be calculated by solving the optical flow equation. For each pixel brightness I(x,y,t) in the image, its displacement in the time interval from time t to t+Δt is (u,v). The optical flow equation can be expressed as:
[0154] I(x+Δx,y+Δy,t+Δt)=I(x,y,t) (12)
[0155] Where Δx is the horizontal displacement of the pixel within Δt time, and Δy is the vertical displacement of the pixel within Δt time;
[0156] The LK optical flow method is based on three assumptions:
[0157] (1) The brightness is constant assumption: the pixel brightness of the same object does not change in two adjacent frames;
[0158] (2) Small motion assumption: the pixel positions in two adjacent frames cannot change significantly;
[0159] (3) Local consistency assumption: the motion state between adjacent pixels is consistent.
[0160] The basic principle of the LK optical flow method is to estimate the optical flow using multiple pixels in a small window in the image. It assumes that the optical flow field is approximately constant in this small window, which is the spatial consistency assumption of the LK optical flow method. The LK method updates the estimate of the optical flow field in an iterative manner so that the condition of image brightness conservation is satisfied in the small window. Now there are two unknowns, but the basic equation of the optical flow algorithm cannot solve the two unknowns, resulting in an aperture problem. The aperture problem refers to the problem that when estimating motion only through an image sequence, the direction of motion cannot be determined due to the limited image information.
[0161] The pyramid LK optical flow method assumes that the motion vector in a neighborhood Ω in a space is constant, then in a neighborhood Ω of n pixels, each pixel satisfies the following formula:
[0162] I xi u+I yi v+Iti =0 i=1,2,···n (13)
[0163] In the formula, and are the spatial gradients in the x and y directions respectively, u and v represent the moving components of the optical flow in the horizontal and vertical directions, that is, the required optical flow information, is the rate of change of image grayscale relative to time t, i represents the i-th pixel;
[0164] Let I 0 is the grayscale representation of the original image (w×h), located at the 0th layer of the pyramid, and sequentially generates the 1st, 2nd, ..., N-1th layers of the N-layer Gaussian image pyramid, denoted as I 1 , I 2 ,…,I N-1 ;
[0165] For the original image I 0 The corner point X(x,y) in the Lth layer image I L X in L (x L ,y L ) corresponds to it; for a given two consecutive images I a ,I b , is to find I a The coordinates of the previous point U=(u x ,u y ), in I b The corresponding point on the image has a similar image intensity U+d=(u x +d x ,u y +d y ), where d is the distance from point U to the previous frame I a To the next frame I b The amount of translation;
[0166] At this time, the optical flow can be defined as the displacement that produces the best fit between image regions at different times, with point U as the center, at I a ,I b Establish U+d=(u x +d x ,u y +d y ), and assuming that the pixels in the corresponding window have the same motion displacement vector d, the neighboring points N(U) of point U can be expressed as follows:
[0167]
[0168] Where u x and u yis the horizontal and vertical coordinates of point U, w x and w y Defines half of the size of the window on the x and y axes. The size of the window is (2w x +1,2w y +1).
[0169] Taking into account noise interference and distortion due to rotation and the tracked target, there are some small matching point deviations in the positions calculated in the step. Define the optical flow residual E(d) of the pixels in the window:
[0170]
[0171] Where d is the previous frame I of point U a To the next frame I b The translation amount, d x and d y Respectively expressed as the translation in the x and y directions, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes, I a (x,y) and I b (x+d x ,y+d y ) indicates I a and I b Grayscale values of the two images.
[0172] Find the partial derivative of the translation d and set it to 0, where I b (x+d x ,y+d y ) performs a first-order Taylor expansion approximation at d = 0, and the spatial gradient matrix G is used to describe the spatial change rate of the pixel points. At different levels of the pyramid, the spatial gradient matrix G reflects the change rate of the pixel points in different directions, and the spatial gradient matrix G is defined as:
[0173]
[0174] In the formula, and Represent the square of the brightness change rate of the pixel in the x and y directions, which are used to represent the texture information of the image in a specific direction. x I y Indicates the cross-change rate of the brightness of the pixel in the x and y directions, which is used to capture the correlation of brightness in the image in two directions and helps detect the rotation and translation of the pixel. x and u y is the horizontal and vertical coordinates of point U, w x and w yDefines half of the window size on the x and y axes;
[0175] The error matching vector is:
[0176]
[0177] Wherein, the error matching vector b is used to store the difference between the optical flow estimation value of the pixel point and its true value. is the rate of change of image grayscale relative to time t. i represents the i-th pixel, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes;
[0178] Then the optimal vector solution d oe for:
[0179] d oe =G -1 b (18)
[0180] Where b is the error matching vector, G -1 is the inverse matrix of the spatial gradient matrix. The actual solution process needs to be iterated. This embodiment uses the Newton-Raphson optimization method to refine the translation. Let e k =G -1 b k is the estimated translation at the kth iteration, b k is the error matching vector of the kth iteration, minimizing the residual E(d), and we have:
[0181]
[0182] Where d0 is the initial translation, subscript 0 represents the initial iteration, and d k is the displacement of the kth estimation iteration, d k-1 is the displacement of the k-1th estimation iteration.
[0183] For two adjacent images on the same level L of the image sequence pyramid Its optical flow residual E L Defined as:
[0184]
[0185] Where, d L is the optical flow result calculated by L layer, g L is the optical flow guess value corresponding to the L layer, which is recursively calculated from the N-1 to L+1 layers. The initial optical flow guess value g at the top layer N-1 =0, is the gray value corresponding to the previous frame of layer L, is the gray value corresponding to the next frame of layer L, u x and u y is the horizontal and vertical coordinates of point U, w x and w y The window size on the x and y axes is defined as half, and the optical flow calculation and transmission process between the upper and lower layers is:
[0186] g N-1 =0 (21)
[0187] g L-1 =2(g L +d L ) (twenty two)
[0188] In the formula, g N-1 is the initial optical flow guess value of the N-1th layer. The superscript N-1 represents the N-1th layer of the pyramid. L is the optical flow guess value corresponding to the L layer, g L-1 is the estimated value of the optical flow at the L-1 layer, d L The optical flow result calculated for the L layer.
[0189] The above recursive process is carried out from the top layer to the bottom layer until the calculation of layer 0 is completed and the final optical flow displacement vector d is obtained:
[0190]
[0191] In the formula, g 0 is the optical flow guess value of the 0th layer. The superscript 0 represents the 0th layer of the pyramid. 0 is the optical flow result calculated at the 0th layer. The superscript 0 represents the 0th layer of the pyramid, N is the number of pyramid layers, L represents the number of layers the optical flow is being calculated, and d L The optical flow result calculated for the L layer.
[0192] Then the previous frame image I a The pixel point U in the next frame image I b The corresponding points in are:
[0193] ν=u+d (24)
[0194] In the formula, ν is I b The pixel position in the middle, u is I a The pixel position in , d is the final optical flow displacement vector.
[0195] S5. Obtain the pixel displacement vibration curve of the ORB corner point in each frame of the markerless structure vibration video, convert the pixel displacement by the scale factor, obtain the actual displacement vibration curve in the three-dimensional space, and use the fast Fourier transform to obtain the cantilever beam vibration frequency, which is specifically:
[0196] Through target tracking using the optical flow algorithm, the pixel displacement time curve of the specified target point can be obtained, and the actual displacement of the target in the world coordinate system is converted by the scale factor:
[0197]
[0198] Where S represents the scale factor, that is, the actual size of the target corresponding to the unit pixel length in the image coordinate system, and D and d p They represent actual size and pixel length respectively, Z represents the distance between the lens and the object being photographed, and f represents the focal length of the lens.
[0199] Therefore, the present invention adopts the above-mentioned computer vision measurement method of marker-free structural vibration, which solves the problem that the prior art relies on speckle and artificial marking and is inefficient, and the method without marking cannot achieve displacement measurement. It realizes the research on marker-free cantilever beam vibration displacement detection, and the measuring device is simple, which provides a new measurement method for vibration displacement analysis of marker-free experimental objects.
[0200] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.
Claims
1. A computer vision measurement method for markerless structural vibration, characterized in that: The following steps are involved: S1. Obtain the reprojection error through the calibration plate calibration method, analyze the reprojection error of the calibration point in the image, and use it to evaluate whether the installation position and direction of the monocular vision system and the cantilever beam meet the expected accuracy requirements; S2, using a monocular vision system to shoot a video of the vibration of the unmarked structure, graying the RGB image of the video frame of the unmarked structure, intercepting the ROI image, and obtaining a preprocessed video; S3, select the first frame image of the pre-processed unmarked structure vibration video, and perform feature point detection through the ORB feature point detection algorithm to determine the pixel coordinates of each ORB feature point in the image, and at the same time filter out unnecessary feature points through the corner point response value threshold; S4. Based on the assumption of constant brightness, an optical flow equation is established, a multi-scale pyramid is constructed, and the optical flow residual is minimized. By constructing the initial optical flow guess, the optical flow transfer between the upper and lower layers is calculated to obtain the final optical flow calculation result. The ORB feature points detected in the first frame image are tracked by the pyramid LK optical flow method for inter-frame tracking. S5. Obtain the pixel displacement vibration curve of the ORB corner point in each frame of the markerless structure vibration video, convert the pixel displacement by a proportional factor to obtain the actual displacement vibration curve in three-dimensional space, and use fast Fourier transform to obtain the cantilever beam vibration frequency.
2. The computer vision measurement method for markerless structural vibration according to claim 1, characterized in that: In step S1, the camera calibration technology of the calibration plate marking method is used to establish a mapping relationship between the three-dimensional actual coordinates and the two-dimensional pixel coordinates. The mapping relationship involves three basic transformations: rigid body transformation, perspective projection and affine transformation. The specific steps are as follows: S11, capturing a calibration plate image, wherein the calibration plate image includes images of front view, upward view, downward view, and left and right oblique view relative to the monocular vision system; S12, obtaining calibration, wherein the method for obtaining calibration is to record a calibration video, continuously change the position of the calibration plate relative to the camera during the recording process, and subsequently select frames with positions that meet the requirements and clear images from the calibration video as calibration images; S13, when performing detection, locate the inner corner points, and use the inner corner points as a point set for calibration; S14, analyzing the reprojection error of the calibration point in the image, wherein the reprojection error is the deviation between the actual imaging position and the imaging position predicted by the calibration parameters. It is expressed as: Where n is the total number of points in the image, (x i ,y i ) is the coordinate on the two-dimensional plane after reprojection transformation, (a i ,b i ) are the coordinates on the two-dimensional plane actually measured.
3. The computer vision measurement method for markerless structural vibration according to claim 2, characterized in that: Step S2 is specifically as follows: in the preprocessing process, firstly extract the first frame of the unmarked structure vibration video and convert it into a grayscale image, intercept the ROI image, and obtain the preprocessed video.
4. The computer vision measurement method for markerless structural vibration according to claim 3, characterized in that: In step S3, the ORB feature point detection algorithm is used to detect the corner point features in the first frame image, and a response value is assigned to each detected feature point. A threshold screening method is used to retain feature points whose response values exceed a predetermined threshold. The specific process is as follows: The ORB feature detection algorithm uses the FAST operator in the feature point detection part, and adds the directional feature to the FAST operator to generate the o-FAST algorithm. The o-FAST algorithm uses the Harris corner point detection scoring mechanism to sort the candidate points. By selecting the N points with the highest Harris response values, the o-FAST algorithm selects the best corner points from the candidate points. The FAST operator takes a pixel point P, considers the 16 pixels around the pixel point P, calculates the gray value difference between the pixel point P and the surrounding 16 pixels, takes the absolute value, and compares it with the threshold: If it is greater than the threshold, the two pixels are considered different. If there are N consecutive pixels around the pixel P that meet this condition, the pixel P is defined as a FAST feature point. The feature point determination method is: ∣G P -G i ∣>G th i=1,2,…,16 (2) In the formula, G P Represents the gray value of pixel P, G i Represents the gray value of the i-th pixel around the pixel P, G th Represents the gray value threshold.
5. The computer vision measurement method for markerless structural vibration according to claim 4, characterized in that: The specific process of adding directional features to the FAST operator to generate the o-FAST algorithm is as follows: Through the grayscale centroid method, a coordinate system is established with the feature point as the origin, the centroid position is calculated in the neighborhood S, and a vector is constructed with the feature point P as the starting point and the grayscale centroid Q as the end point. Then the moment of the domain S about I(x,y) is: In the formula, m pq Represents the moment of domain S about I(x,y), I(x,y) is the gray value of the image, p and q are the orders of the moment, x and y are the coordinates of the pixel point, x, y∈[-r,r], r is the radius of the feature point domain S; The centroid C of the image region is expressed as: In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, m 00 The sum of the grayscale values of all pixels in the image block, about the x-coordinate of the grayscale centroid, is: In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image; The y coordinate of the grayscale centroid is: In the formula, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, r is the radius of the feature point area S, x and y are the coordinates of the pixel point, and I(x,y) is the grayscale value of the image; Get the direction angle θ of the FAST feature point P: θ=arctan2(c y ,c x )=arctan2(m 01 ,m 10 ) (7) In the formula, c x is the x coordinate of the grayscale centroid, c y is the grayscale centroid y coordinate, m 10 is the weighted sum of the image block in the x direction, m 01 is the weighted sum of the image block in the y direction, and x and y are the pixel coordinates; The o-FAST algorithm is used to calculate the direction angle θ of the feature point P, and further obtain the direction angles of all feature points.
6. The computer vision measurement method for markerless structural vibration according to claim 5, characterized in that: The ORB feature detection algorithm adopts the improved BRIEF descriptor r-BRIEF in the feature description part, and improves the BRIEF descriptor by introducing the direction information of the key points. The specific process is as follows: Rewrite the point set into matrix form: At position (x i ,y i ), for any n binary criterion feature sets, define a 2n matrix S: According to the direction angle θ of the feature point p obtained by the o-FAST algorithm, find its corresponding rotation matrix R θ : This results in the corrected version S θ : S θ =R θ S (10) Where S is the defined 2n matrix; Then the SteeredBRIEF descriptor g n It is expressed as: g n (p,θ)=f nd (p)|(x i ,y i )∈S θ (11) p(x i ,y i ) represents the position of feature point p, θ represents the direction angle of feature point p, and f nd It is the BRIEF descriptor.
7. The computer vision measurement method for markerless structural vibration according to claim 6, characterized in that: Step S4 is specifically as follows: Based on the assumption that the image brightness is constant, the optical flow is calculated by solving the optical flow equation. For each pixel brightness I(x,y,t) in the image, its displacement in the time interval from time t to t+Δt is (u,v). The optical flow equation is expressed as: I(x+Δx,y+Δy,t+Δt)=I(x,y,t) (12) Where Δx is the horizontal displacement of the pixel within Δt time, and Δy is the vertical displacement of the pixel within Δt time; The pyramid LK optical flow method assumes that the motion vector in a neighborhood Ω in a space is constant, then in a neighborhood Ω of n pixels, each pixel satisfies the following formula: I xi u+I yi v+I ti =0 i=1,2,···n (13) In the formula, and are the spatial gradients in the x and y directions respectively, u and v represent the moving components of the optical flow in the horizontal and vertical directions, that is, the required optical flow information, is the rate of change of image grayscale relative to time t, i represents the i-th pixel; Let I 0 is the grayscale representation of the original image (w×h), located at the 0th layer of the pyramid, and sequentially generates the 1st, 2nd, ..., N-1th layers of the N-layer Gaussian image pyramid, denoted as I 1 , I 2 ,…,I N-1 ; For the original image I 0 The corner point X(x,y) in the Lth layer image I L X in L (x L ,y L ) corresponds to it; for a given two consecutive images I a ,I b , is to find I a The coordinates of the previous point U=(u x ,u y ), in I b The corresponding point on the image has a similar image intensity U+d=(u x +d x ,u y +d y ), where d is the distance from point U to the previous frame I a To the next frame I b The amount of translation; The optical flow is defined as the displacement that produces the best fit between image regions at different times, centered at point U. a ,I b Establish U+d=(u x +d x ,u y +d y ), and assuming that the pixels in the corresponding window have the same motion displacement vector d, the neighboring point N(U) of point U is expressed as follows: Where u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the size of the window on the x and y axes. The size of the window is (2w x +1,2w y +1); Define the optical flow residual E(d) of the pixels in the window: Where d is the previous frame I of point U a To the next frame I b The translation amount, d x and d y Respectively expressed as the translation in the x and y directions, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes, I a (x,y) and I b (x+d x ,y+d y ) indicates I a and I b Grayscale values of the two images; Find the partial derivative of the translation d and set it to 0, where I b (x+d x ,y+d y ) Perform a first-order Taylor expansion approximation at d = 0. The spatial gradient matrix G is used to describe the spatial change rate of the pixel point. The spatial gradient matrix G is defined as: In the formula, and Respectively represent the square of the brightness change rate of the pixel in the x and y directions, which is used to characterize the texture information of the image in a specific direction. x I y Indicates the cross-change rate of the brightness of the pixel in the x and y directions, which is used to capture the correlation of the brightness in the image in two directions. x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes; The error matching vector is: Wherein, the error matching vector b is used to store the difference between the optical flow estimation value of the pixel point and its true value. is the rate of change of image grayscale relative to time t, i represents the i-th pixel, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes; Then the optimal vector solution d oe for: d oe =G -1 b (18) Where b is the error matching vector, G -1 is the inverse matrix of the spatial gradient matrix; The Newton-Raphson optimization method is used to refine the translation. Let e k =G -1 b k is the estimated translation at the kth iteration, b k is the error matching vector of the kth iteration, minimizing the residual E(d), and we have: Where d0 is the initial translation, subscript 0 represents the initial iteration, and d k is the displacement of the kth estimation iteration, d k-1 is the displacement of the k-1th estimation iteration; For two adjacent images on the same level L of the image sequence pyramid Its optical flow residual E L Defined as: Where, d L is the optical flow result calculated by L layer, g L is the optical flow guess value corresponding to the L layer, which is recursively calculated from the N-1 to L+1 layers. The initial optical flow guess value g at the top layer N-1 =0, is the gray value corresponding to the previous frame of layer L, is the gray value corresponding to the next frame of layer L, u x and u y is the horizontal and vertical coordinates of point U, w x and w y Defines half of the window size on the x and y axes; The optical flow calculation and transmission process between the upper and lower layers is: g N-1 =0 (21) g L-1 =2(g L +d L ) (22) In the formula, g N-1 is the initial optical flow guess value of the N-1th layer. The superscript N-1 represents the N-1th layer of the pyramid. L is the optical flow guess value corresponding to the L layer, g L-1 is the estimated value of the optical flow at the L-1 layer, d L The optical flow result calculated for the L layer; Until the 0th layer is calculated and the final optical flow displacement vector d result is obtained: In the formula, g 0 is the optical flow guess value of the 0th layer. The superscript 0 represents the 0th layer of the pyramid. 0 is the optical flow result calculated at the 0th layer. The superscript 0 represents the 0th layer of the pyramid, N is the number of pyramid layers, L represents the number of layers the optical flow is being calculated, and d L The optical flow result calculated for the L layer; Then the previous frame image I a The pixel point U in the next frame image I b The corresponding points in are: ν=u+d (24) Where ν is I b The pixel position in the middle, u is I a The pixel position in , d is the final optical flow displacement vector.
8. The computer vision measurement method for markerless structural vibration according to claim 7, characterized in that: Step S5 is specifically as follows: by tracking the target using the optical flow algorithm, a pixel displacement time history curve of the specified target point is obtained, and the actual displacement of the target in the world coordinate system is converted by a scaling factor: Where S represents the scale factor, that is, the actual size of the target corresponding to the unit pixel length in the image coordinate system, and D and d p They represent actual size and pixel length respectively, Z represents the distance between the lens and the object being photographed, and f represents the focal length of the lens.
Citation Information
Cited By
Universe ultra-precision strain measurement method
CN119832092A
Structure large displacement estimation method based on Canny-Hough transformation and KLT optical flow
CN120147398A
Target-free structure vibration signal acquisition method based on unmanned aerial vehicle platform
CN120213192A
Suturing scalpel packaging quality detection method based on optic nerves
CN120510129A
Radiation source individual tracking method and system based on optical flow method
CN120543596A
