Structural three-dimensional displacement measurement method and system based on centroid bounding box tracking algorithm

Abstract

The present invention discloses a method and system for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm, including the following steps: collecting two original vibration videos of a target object; using a centroid bounding box tracking technique to obtain the centroid of the node of interest corresponding to each key area of ​​interest; obtaining the phase information of the centroid of the node of interest of the image sequence relative to the first frame; calculating the effective displacement vector between the centroid of the node of interest of the image sequence and the first frame; obtaining the two-dimensional displacement of the image sequence of the target object relative to the first frame image; determining the true three-dimensional displacement of the centroid of the physical node of the image sequence, the three-dimensional displacement being the spatial displacement of the target object. The method provided by the present invention creatively proposes to determine the three-dimensional displacement as the spatial displacement of the target object by tracking the centroid of the node of interest of the image sequence of two cameras, without the need to arrange targets, and is not easily disturbed by external conditions when using computer vision for three-dimensional displacement monitoring, and has strong anti-noise capability.

Description

Technical Field

[0001] The present invention relates to the field of structural health technology, and in particular to a method and system for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm. Background Art

[0002] The real-time displacement of bridge structures is an important parameter that reflects the health status of bridge structures. Real-time monitoring of the three-dimensional displacement of bridges is an important part of structural health monitoring (SHM). The dynamic displacement of bridges, as a key indicator, reflects the stiffness and deformation capacity of bridge structures. How to accurately and quickly identify the dynamic displacement of bridge structures has attracted widespread attention.

[0003] At present, the measurement of three-dimensional displacement of bridge structure can be divided into two methods: contact and non-contact. Specifically, the classic traditional contact scheme is to use a total station for measurement, set up measuring points at the observation position and use the Yellow Sea elevation coordinate system to establish leveling points. The total station can accurately measure the elevation and plane coordinates of the measuring points, but the deployment conditions of the total station are relatively harsh. It has the disadvantages of relying on manual observation, inconvenient to deploy the total station in some areas with poor geological environment, repeated observations to average and eliminate errors, and the coordinate data of the measuring points must be obtained in advance. It is a relatively accurate observation method but it is difficult to meet the requirements of observing real-time dynamic displacement. Although the laser vibrometer (LDV) can accurately measure the dynamic displacement of the bridge at a high frequency, it can only achieve single-point displacement measurement, and the instrument cost is high, and large-scale deployment increases the cost. Most of the existing non-contact displacement measurement technologies based on computer vision are aimed at the measurement of two-dimensional displacement of structures. For example, the method of measuring the two-dimensional single-point displacement of structures based on motion magnification (MM) has been widely used, but it is necessary to amplify the displacement within a narrow frequency band according to the prior structure frequency information. Template Matching (TM) algorithm requires artificial targets to be prefabricated on the structure to be measured, and displacement is measured by matching the targets. With the development of three-dimensional digital image correlation method (3D-DIC), it has attracted widespread attention. 3D-DIC can measure the displacement of the entire structure, but it requires that the surface of the structure must have rich texture features or artificial textures as the measurement target. In essence, it is still a measurement method with a target. Several disadvantages of the above methods limit their promotion in the field of structural health monitoring.

[0004] In view of this, it is necessary to propose a structural three-dimensional displacement measurement method and system based on a centroid bounding box tracking algorithm to solve or at least alleviate some of the above defects. Summary of the invention

[0005] The main purpose of the present invention is to provide a method and system for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm, aiming to solve the technical problem in the prior art that when using a non-contact method to measure the three-dimensional displacement of a structure, it is necessary to match and calibrate targets, making it difficult to accurately measure the three-dimensional displacement of the structure.

[0006] To achieve the above-mentioned purpose, the present invention provides a method for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm, comprising the following steps: S10, using two calibrated cameras to collect two original vibration videos of the target object; S20, performing frame processing on each original vibration video to obtain a single image sequence, extracting the key interest region corresponding to a single image in the single image sequence, and using the centroid bounding box tracking technology to obtain the centroid of the node of interest corresponding to each key interest region; S30, performing Gaussian smoothing processing on each frame of a single image, extracting the local phase information of each frame of a single image in different directions and at different scales according to multiple filters, and obtaining the phase information of the centroid of the node of interest in the image sequence relative to the first frame; S40, using a phase-based optical flow method to track the phase of the node centroid. The effective displacement vector between the centroid of the node of interest of each single image of each frame and the first frame is calculated; S50, the effective displacement vector of the centroid of the node of interest of each single image of each frame is processed by the constrained intersection method or the vector average method to obtain the two-dimensional displacement of the image sequence of the target object relative to the first frame of the image; S60, the centroid of the node of interest of each single image of each frame of the two cameras is matched to obtain the corresponding physical node centroid in the real coordinate system of each frame; the depth vector of the centroid of the physical node of adjacent frames is obtained by using the particle swarm optimization algorithm; the horizontal displacement and vertical displacement of the centroid of the physical node of adjacent frames are determined based on the two-dimensional displacement of the image; the real three-dimensional displacement of the centroid of the physical node of the image sequence is determined based on the depth vector, the horizontal displacement and the vertical displacement, and the three-dimensional displacement is the spatial displacement of the target object.

[0007] Furthermore, step S40 also includes determining the number of valid displacement vectors; step S50 specifically includes: if the number of valid displacement vectors is not less than 3, the constrained intersection method is used to process the effective displacement vectors of the centroid of the node of interest of each single image in each frame to obtain the two-dimensional image displacement of the image sequence of the target object relative to the first frame image; if the number of valid displacement vectors is less than 3, the vector averaging method is used to process the effective displacement vectors of the centroid of the node of interest of each single image in each frame to obtain the two-dimensional image displacement of the image sequence of the target object relative to the first frame image.

[0008] Furthermore, using the formula Calculate the effective displacement vector between the center of mass of the node of interest of each single image and the first frame, where, The two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the constrained intersection method, is the two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the vector averaging method, is the phase nonlinearity value described in pixels, is the filter response after the image is processed by the filter.

[0009] Furthermore, the real coordinates of the center of mass of the node of interest in the world coordinate system are determined according to the coordinates of the center of mass of the node of interest of the two frames of single images corresponding to the same moment; the physical node center of mass corresponding to the single image is found according to the calibration parameters and the real coordinates of the center of mass of the node of interest of the two frames of single images corresponding to the same moment.

[0010] Furthermore, using the formula Determine the depth vector of a single image of adjacent frames, where is the depth displacement of the centroid of the physical node of the current frame in the defined global coordinate system, is the depth vector of a single image in the current frame, is the depth vector of a single image in the previous frame.

[0011] Furthermore, step S20 specifically includes: S21, performing frame processing on each original vibration video to obtain a single image sequence, and extracting a single image from the single image sequence; S22, processing the single image using digital image processing technology to obtain a preprocessed image file with a key area of ​​interest; S23, using a centroid bounding box tracking algorithm to generate a bounding box surrounding the key area of ​​interest, and extracting the centroid of the node of interest in the key area of ​​interest.

[0012] Furthermore, step S30 specifically includes: S31, calculating and obtaining the Gaussian smooth gradient of a single image sequence containing the centroid of the node of interest; S32, using 10 Gabor filters of different scales and directions to perform local phase extraction on the single image; S33, calculating the phase information of the centroid of the node of interest in 10 directions through the phase motion tracking technology, and obtaining the phase information of the centroid of the node of interest in the image sequence relative to the first frame.

[0013] Furthermore, step S40 specifically includes: S41, using the phase-based optical flow method to calculate multiple center of mass displacement components of the center of mass from the phase information; S42, accurately estimating the motion based on the spatial phase gradient and phase difference and eliminating erroneous center of mass displacement components to obtain a valid displacement vector.

[0014] The present invention also provides a structural three-dimensional displacement measurement system based on a centroid bounding box tracking algorithm, comprising an image acquisition unit, a centroid extraction unit, a filtering two-dimensional processing unit and a three-dimensional processing unit, wherein the image acquisition device is used to acquire two original vibration videos of a target object; the centroid extraction unit is used to perform frame processing on each original vibration video to obtain a single image sequence, extract the key interest region corresponding to a single image in the single image sequence, and use the centroid bounding box tracking technology to obtain the centroid of the interest node corresponding to each key interest region; the filtering two-dimensional processing unit is used to perform Gaussian smoothing processing on each frame of a single image, extract the local phase information of each frame of a single image in different directions and different scales according to multiple filters, and obtain the image sequence. The method is used to calculate the phase information of the centroid of the node of interest of the column relative to the first frame; to track the phase of the node centroid using the phase-based optical flow method to calculate the effective displacement vector between the centroid of the node of interest of each frame of a single image and the first frame, and to determine the number of effective displacement vectors; to process the effective displacement vector of the centroid of the node of interest of each frame of a single image using the constrained intersection method or the vector average method, and obtain two original vibration videos of the image sequence of the target object relative to the first frame image; the three-dimensional processing unit is used to match the image node centroids of the two cameras using the particle swarm optimization algorithm, obtain the depth information of the physical node centroid, and finally obtain the three-dimensional displacement of the physical node centroid in combination with the two-dimensional displacement obtained by the optical flow method, and the three-dimensional displacement is the spatial displacement of the target object.

[0015] Compared with the prior art, the method for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm provided by the present invention has the following beneficial effects:

[0016] The present invention provides a method for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm. Two calibrated cameras are used to collect two original vibration videos of a target object. The centroid bounding box tracking algorithm is used to extract the centroid of the node of interest in a single image sequence. Gaussian smoothing is performed to obtain the phase information of the centroid of the node of interest in the image sequence relative to the first frame. The phase-based optical flow method is used to track the phase of the node centroid to calculate the effective displacement vector between the centroid of the node of interest in each single image and the first frame. After obtaining the two-dimensional displacement of the current frame image of the target object relative to the previous frame, the centroid of the node of interest corresponding to the two cameras is finally used as the binocular stereo matching. The primitive determines the centroid of the physical node, uses the particle swarm optimization algorithm to match the two-dimensional displacement of the two cameras, and obtains the spatial displacement of the target object. It creatively proposes to determine the three-dimensional displacement as the spatial displacement of the target object by tracking the centroid of the node of interest in the pictures of the two cameras. There is no need to arrange the target, and avoids the technical problem that the feature primitives selected by the existing target calibration are easily affected by the lighting conditions and texture features, resulting in inaccurate measurement. The structural three-dimensional displacement measurement method based on the centroid bounding box tracking algorithm provided by the present invention is not easily affected by external conditions when using computer vision for three-dimensional displacement monitoring, and has strong noise resistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0018] Figure 1 It is a schematic flow chart of a method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm in one embodiment of the present invention;

[0019] Figure 2 Schematic diagram of the use scenario of the method for measuring the three-dimensional displacement of a structure based on the centroid bounding box tracking algorithm in one embodiment of the present invention, wherein 2a is a schematic diagram of obtaining a vibration video, 2b is a schematic diagram of extracting the centroid of an interest node, 2c is a schematic diagram of obtaining a two-dimensional displacement, and 2d is a schematic diagram of extracting a three-dimensional displacement;

[0020] Figure 3 yes Figure 2 Flowchart of extracting node centroids based on centroid bounding box technology in, where 3a is a schematic diagram of selecting key interest areas, 3b is a schematic diagram of separating interest nodes, 3c is a schematic diagram of extracting interest node centroids, and 3d is a schematic diagram of producing bounding boxes;

[0021] Figure 44a is a schematic diagram of the process of obtaining a phase gradient image, wherein 4a is the input original image, 4b is the image after Gaussian smoothing to remove noise, 4c is the phase contour image obtained by Gabor filter processing, and 4d is the final phase gradient image;

[0022] Figure 5 It is a schematic diagram of the phase motion vector direction of the image obtained after being processed by the filter;

[0023] Figure 6 It is a schematic diagram of using the constrained intersection method and the vector average method to find the intersection and average of all vectors respectively.

[0024] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0025] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0028] In addition, the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0029] Please refer to the attached Figure 1 , Figure 2 , Figure 3 Figure 4 , Figure 5 as well as Figure 6The present invention provides a method for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm, comprising the following steps: S10, using two calibrated cameras to collect two original vibration videos of a target object; S20, performing frame processing on each original vibration video to obtain a single image sequence, extracting a key area of ​​interest corresponding to a single image in the single image sequence, and using a centroid bounding box tracking technology to obtain the centroid of an interest node corresponding to each key area of ​​interest; S30, performing Gaussian smoothing processing on each frame of a single image, extracting local phase information of each frame of a single image in different directions and at different scales according to multiple filters, and obtaining the phase information of the centroid of the interest node of the image sequence relative to the first frame; S40, using a phase-based optical flow method to track the phase of the node centroid to calculate the interest of each frame of a single image. The effective displacement vector between the centroid of the node of interest and the first frame; S50, using the constrained intersection method or the vector average method to process the effective displacement vector of the centroid of the node of interest of each single image of each frame, and obtain the image two-dimensional displacement of the image sequence of the target object relative to the first frame image; S60, matching the centroid of the node of interest of a single image of each frame of the two cameras to obtain the corresponding physical node centroid in the real coordinate system of each frame; using the particle swarm optimization algorithm to obtain the depth vector of the centroid of the physical node of adjacent frames; based on the centroid of the node of interest, the centroid of the physical node and the two-dimensional displacement of the image, determine the horizontal displacement and vertical displacement of the centroid of the physical node of adjacent frames; based on the depth vector, the horizontal displacement and the vertical displacement, determine the real three-dimensional displacement of the centroid of the physical node of the image sequence, and the three-dimensional displacement is the spatial displacement of the target object.

[0030] The method for measuring the three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm provided by the present invention uses two calibrated cameras to collect two original vibration videos of a target object, uses a centroid bounding box tracking algorithm to extract the centroid of the node of interest of a single image sequence, performs Gaussian smoothing processing to obtain the phase information of the centroid of the node of interest of the image sequence relative to the first frame, uses a phase-based optical flow method to track the phase of the node centroid to calculate the effective displacement vector between the centroid of the node of interest of each single image and the first frame, and after obtaining the two-dimensional displacement of the current frame image of the target object relative to the previous frame, finally uses the centroid of the node of interest corresponding to the two cameras as a binocular stereo matching primitive The centroid of the physical node is determined, and the particle swarm optimization algorithm is used to match the two-dimensional displacement of the two cameras to obtain the spatial displacement of the target object. It is creatively proposed to determine the three-dimensional displacement as the spatial displacement of the target object by tracking the centroid of the node of interest in the pictures of the two cameras. There is no need to arrange targets, which avoids the technical problem that the feature primitives selected by the existing target calibration are easily affected by the lighting conditions and texture features, resulting in inaccurate measurement. The structural three-dimensional displacement measurement method based on the centroid bounding box tracking algorithm provided by the present invention is not easily affected by external conditions when using computer vision for three-dimensional displacement monitoring, and has strong noise resistance.

[0031] It is understandable that the target object may be a bridge structure, a truss structure, a building structure or other mechanical devices. In the embodiments of the present invention, the target object is described as a truss structure. In specific implementation, the target object may be a whole truss structure or a part of a truss structure.

[0032] Understandably, the centroid-based bounding-box tracking algorithm (CBBT) extracts the centroid of the node of interest corresponding to each frame of a single image from a single image sequence by generating a rectangular bounding box that accurately surrounds the node in the region of interest (ROI). When the target object (truss structure) is affected by illumination changes, the centroid of the node of interest of the truss structure (truss node centroid) may be distorted by external conditions in a single image sequence. The use of a bounding box can prevent the erroneous detection of the centroid of the truss node.

[0033] Understandably, the two calibrated cameras form binocular stereo matching, which is based on the principle of parallax and uses imaging equipment to obtain two images of the object to be measured from different positions, and calculates the position deviation between corresponding points in the images to obtain the three-dimensional geometric information of the object. Pure two-dimensional images cannot reflect the three-dimensional information in the real world (under the world coordinate system) due to the lack of depth information. Through the images taken by the two cameras, the focus areas are found for mutual matching, and the parallax value between the two images of the same object is calculated; the depth information of the point at a certain moment can be obtained by dividing the camera focal length by the parallax value, that is, the vertical distance (depth) from the point to the imaging system.

[0034] It can be understood that during calibration, in the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters; under most conditions, these parameters must be obtained through experiments and calculations. The process of solving the parameters is called camera calibration. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very critical link. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera.

[0035] Optionally, in the present invention, the filter adopts a Gabor filter. In the field of image processing, Gabor wavelet is a very effective texture representation method. In the spatial domain, a two-dimensional Gabor filter is the product of a sinusoidal plane wave and a Gaussian kernel function. The local phase map of each Gabor filter can be used to estimate the local component displacement projected by the local total displacement in the filter direction. In order to measure the displacement, multiple Gabor filters covering different directions and scales are needed to provide sufficient center of mass displacement components.

[0036] Understandably, the particle swarm optimization algorithm (Particle Swarm Optimization, PSO), also known as the bird flock foraging algorithm, is a parallel algorithm and a new evolutionary algorithm. The particle swarm optimization algorithm starts from a random solution, searches for the optimal solution through iteration, and evaluates the solution through fitness. It searches for the global optimum by following the best solution currently searched. This algorithm has attracted attention for its advantages such as easy implementation, high accuracy, and fast convergence, and has demonstrated its superiority in solving practical problems.

[0037] It can be understood that in the specific implementation, the background information of a single image is removed by Gaussian smoothing; the phase of the gradient image is extracted by using a Gabor filter; the phase change is tracked by the phase-based optical flow (POF) method to calculate several centroid displacement vectors (centroid displacement components) at the centroid of each node of interest; the phase nonlinearity and filter response are used to eliminate the erroneous centroid displacement vectors, and the effective displacement vector between the centroid of the node of interest and the first frame of each frame of a single image is obtained; the effective displacement vector in the node of the centroid of each node of interest is processed by the IOC and VA methods to obtain the two-dimensional displacement of the current frame image of the target object relative to the previous frame, and obtain the full-field two-dimensional displacement; finally, the calibration matrix of each camera is matched with the two-dimensional centroid coordinates of each camera. The particle swarm optimization algorithm (PSO) is used to determine the three-dimensional displacement from the two-dimensional pixel displacement of the two cameras, and the three-dimensional displacement is obtained as the spatial displacement of the target object.

[0038] Furthermore, camera calibration can be used to eliminate the radial and tangential distortion of the camera lens, and can also be used to solve the conversion of the two camera image coordinate systems to the world coordinate system and the stereo calibration between the two cameras. Camera calibration matrix is determined by setting some control points with known positions in 3D global coordinates and then mapping their respective projections in 2D image coordinates. It should be noted that due to the matrix Contains 12 elements, so at least 6 control points are required.

[0039] The relationship between the world coordinate system and the image coordinate system is as follows:

[0040] (1)

[0041] in, is the scale factor, is the vector form of feature points in image coordinates, is the vector of the feature point in the world coordinate system, is the camera calibration matrix, is the coordinate of point k in the image coordinate system, is the region of interest in the image coordinate system, are real physical coordinates. The calibration matrix contains the intrinsic and extrinsic parameters of the camera. The intrinsic parameters depend on the characteristics of the specific camera, and the extrinsic parameters depend on the observed scene. They represent the position and orientation of the target point relative to the three-dimensional global coordinate system. In order to obtain the camera calibration matrix , the kth point of formula (1) can be expressed as:

[0042] (2)

[0043] in, is a matrix After each camera is calibrated, the next step is to synchronize the two cameras, which requires obtaining images of the same scene taken by the two cameras at the same time.

[0044] Furthermore, the calibration relationship between the two stereo cameras refers to the formula:

[0045] (3)

[0046] in, represents the position of the kth point displayed on the image taken by camera 1, represents the position of the kth point displayed on the image taken by camera 2, Called the fundamental matrix, the fundamental matrix can be determined using 8 pairs of matching points between the two images from the two cameras.

[0047] It can be understood that step S20 specifically includes: S21, frame-processing each original vibration video to obtain a single image sequence, and extracting a single image in the single image sequence; S22, using digital image processing technology to process the single image to obtain a preprocessed image file with a key area of ​​interest; S23, using a centroid bounding box tracking algorithm to generate a bounding box surrounding the key area of ​​interest, and extracting the centroid of the interest node in the key area of ​​interest.

[0048] Furthermore, a centroid bounding box tracking algorithm is used to generate a bounding box surrounding the key area of ​​interest, and the centroid of the node of interest in the key area of ​​interest is extracted. Specifically, the method includes: using the CBBT algorithm to generate a bounding box surrounding the truss node, and extracting the centroid of the node of interest from the image sequence; creating a rectangular boundary through the bounding box to accurately surround the node. Due to the influence of illumination changes during video acquisition, the node is easily distorted in the image sequence, so the use of a bounding box can prevent the incorrect detection of the centroid of the node of interest.

[0049] Furthermore, in the specific implementation, the phase information is extracted based on Gabor filter and Gaussian smoothing. The Gaussian smoothed gradient of the node video sequence (single image sequence) is calculated and obtained; then, the local phase of the gradient image is extracted using Gabor filters of 10 different scales and directions; finally, the phase information of the node in 10 directions is calculated by tracking the phase motion.

[0050] For a single image sequence, vibration information usually appears in local areas with clear edges. Phase-based techniques are difficult to estimate motion in local areas with generally uniform image intensity. Extracting high-quality local phase is critical for measuring local vibration using POF, which requires the filter to retain image edge information as much as possible while reducing noise. Gaussian smoothed gradients of Canny edge detection are used to extract edge information and reduce the influence of uniform patterns and noise. Each frame is first convolved with a Gaussian kernel as shown below:

[0051] (4)

[0052] in is the standard deviation of the Gaussian filter (Gaussian convolution kernel) in the image pixels, is the number of image sequences, is the image frame convolved with the Gaussian convolution kernel, (x, y) is the image coordinate;

[0053] The smoothed image is further filtered using the Sobel operator in the x direction (horizontally inversely) and the y direction (vertically).

[0054] (5)

[0055] in, It is the processed image gradient; it can effectively extract edge motion information and suppress image noise, while removing uniform areas that are not conducive to POF effect; it can also reduce the phase nonlinearity of POF.

[0056] After preprocessing, the Gabor filter is used to extract the phase because the Gabor filter has the flexibility of adjustable parameters and is mathematically describable in both the spatial domain and the frequency domain. The four main parameters that control the filter are: direction, center frequency, angle, and radial standard deviation. In the spatial domain, the expression of the two-dimensional Gabor filter is,

[0057] (6)

[0058] In the formula is the standard deviation of the two-dimensional Gaussian envelope in the angular and radial directions; is the center frequency of the frequency domain filter, and can also be expressed in terms of direction and modulation frequency It is expressed as follows:

[0059] (7)

[0060] The Gabor filter in the frequency domain is the Fourier transform of formula (6), as follows:

[0061] (8)

[0062] In the formula are the angular standard deviation and radial standard deviation of the two-dimensional Gaussian envelope in the frequency domain, which are related to the corresponding standard deviation in the spatial domain. Considering that the filter size for filtering in the spatial domain is small, the Gabor filter in the frequency domain is given by equation (8) for subsequent use; the filter in the spatial domain has both real and imaginary parts, and the processed image is complex-valued. The gradient obtained after image convolution is The filter response Then the phase of the video sequence is calculated as the response parameter, which is expressed as follows:

[0063] (9)

[0064] Local phase map of each Gabor filter It can be used to estimate the local component displacement projected by the local displacement in the filter direction. In order to measure the complete displacement, multiple two-dimensional Gabor filters covering different directions and scales are required to provide sufficient centroid displacement components. This study uses two sets of filters with the same 1-octave bandwidth and different modulation frequencies; radial and angular standard deviations Respectively expressed as ;

[0065] Consider the overlap of filters and the resulting sub-pixel motion errors at the same time; first, the highest spatial frequency of the image is 1 / 2, which uses only two pixels to describe the sine wave. These high spatial frequency components usually contain small vibration information. Using a Gabor filter with a center frequency close to 1 / 2 will result in large sub-pixel motion errors, especially for phase-based techniques, where the center frequency is less than 1 / 5, which can retain high spatial frequency information as much as possible; second, define the spatial frequency domain boundary of the filter as its 95% confidence interval. Less overlap can reduce computational costs and make lateral motion measurements easier to distinguish. For each modulation frequency There are five filters to avoid significant overlap in the angular direction. To avoid significant overlap in the radial direction, the modulation frequency and The filter directions are 1 / 5 and 1 / 8 respectively; after filtering by 10 filters, the multi-directional and multi-scale phase becomes a four-dimensional variable , where m ranges from 1 to 10. Therefore, the phase Estimate the displacement component of the center of mass , which is expressed as follows:

[0066] (10)

[0067] In the formula, is the spatial phase difference, is the spatial phase gradient vector diagram, are the partial derivatives in the horizontal and vertical directions, is the number of the first frame; for each pixel (x, y) of a frame image, 10 vectors can be obtained, and these vectors are the projections of the complete vector in 10 directions.

[0068] In the specific implementation, the center frequency is less than 1 / 5, the empty frequency domain boundary of the filter is defined as its 95% confidence interval, and the modulation frequency and and the filter directions are 1 / 5 and 1 / 8 respectively.

[0069] Considering that image noise and incorrect filter response will bring difficulties to POF, if you want to accurately measure the displacement of the center of mass, you must first measure a stable phase and an almost constant spatial phase gradient. The slope of the phase in an ideal state is usually a straight line after being processed by a filter; however, the actual spatial phase gradient of the pixels on the vibrating object cannot be constant, and the measurement error is large due to complex image patterns and noise; if you want to accurately estimate the motion through the spatial phase gradient and phase difference, you need to pre-process (Gaussian smooth gradient) to help measure the phase with low nonlinearity. The quantification of phase nonlinearity can be defined as the standard deviation of the spatial phase gradient of each pixel, and the nonlinearity of POF is represented by the measured difference between the original spatial phase gradient and its mean, and its expression is as follows:

[0070] (11)

[0071] In the formula is the total number of video frames, is the spatial phase difference between the current frame and the first frame, is the spatial phase gradient vector; the mask of each filter can be created according to equation (8) and threshold TH1,

[0072] (12)

[0073] In the formula Redefine as the quantized phase nonlinearity index The phase nonlinearity is described in pixels; each pixel of the preprocessed image after passing through the filter has a phase nonlinearity value, which is used to suppress the centroid displacement component with large measurement errors. TH1 can be adjusted as a threshold for different applications; it is known that the diagonal motion resolution of an 8-bit image is about 0.003 pixels.

[0074] The filter is also used to remove image noise. The weak filter response caused by low image intensity and incorrect filter parameters causes large errors in POF. Therefore, the filter response is considered as another rejection condition, and its expression is as follows:

[0075] (13)

[0076] Where TH2 is the threshold of the filter response strength, is the response of the video sequence after being processed by the filter; is the redefined image filter response; assuming that the filter impulse response for the bright pixels (255 in an 8-bit image) of the filter described using a Gaussian smoothed gradient and TH2 is about 50, the value of TH2 can exceed 50 (depending on the application). Therefore, the effective centroid displacement component (effective displacement vector) becomes:

[0077] (14)

[0078] All erroneous center-of-mass displacement components can be removed to retain correct information for subsequent motion.

[0079] It can be understood that step S30 specifically includes: S31, calculating and obtaining the Gaussian smooth gradient of a single image sequence containing the centroid of the node of interest; S32, using 10 Gabor filters of different scales and directions to perform local phase extraction on the single image; S33, calculating the phase information of the centroid of the node of interest in 10 directions through the phase motion tracking technology, and obtaining the phase information of the centroid of the node of interest in the image sequence relative to the first frame.

[0080] It can be understood that step S40 specifically includes: S41, using the phase-based optical flow method to calculate the corresponding multiple center of mass displacement components from the phase information; S42, accurately estimating the motion based on the spatial phase gradient and phase difference to eliminate the erroneous center of mass displacement components and then obtain the effective displacement vector.

[0081] Furthermore, the filtering performed by the Gabor filter only retains features based on its direction. It can only perceive displacements perpendicular to the filter direction. In order to restore the complete displacement from the observed centroid displacement component, two motion processing methods, IOC and VA, are usually used; the vector displacement obtained by the image passing through the filter is the vector Projection in the corresponding filter direction, the true displacement vector It should be located at the intersection of all components, however, due to measurement errors, the constraint lines cannot intersect at the same point; the IOC method provides more accurate estimation results than the VA method when there are more centroid displacement components; when there are only a small number of vectors, VA presents more stable results than IOC, therefore, when the number of reliable centroid displacement components is greater than or equal to 3, the IOC pixel displacement integration method is used, while the VA method is used in other cases.

[0082] Alternatively, finding the optimal intersection point for the IOC and VA problems can be considered as finding a point with the minimum sum of perpendicular distances to all constraint lines. Medium effective amount The constraint line in vector form Can be described as points on a line and the unit direction vector :

[0083] (15)

[0084] To any point The square of the vertical distance:

[0085] (16)

[0086] The goal is to find a point The minimum square of the perpendicular distance to all constraint lines:

[0087] (17)

[0088] By setting the gradient to zero, a least squares solution is used to find a unique solution to Equation (14). Taking the derivative, we get: , where is the total number of filters, which can be rearranged into the following linear system of equations:

[0089] (18)

[0090] Furthermore, equation (18) can be directly solved as:

[0091] (19)

[0092] from The starting point a of the orthogonal unit vector n of its pixel (x, y,) is:

[0093] (20)

[0094] in The final displacement According to formulas (16) and (17), it is as follows:

[0095] (twenty one)

[0096] In another VA integration method, the effective vectors for each pixel are averaged to form the complete displacement vector as follows:

[0097] (twenty two)

[0098] The final displacement is calculated as follows:

[0099] (twenty three)

[0100] It can be understood that step S40 also includes determining the number of valid displacement vectors; step S50 specifically includes: if the number of valid displacement vectors is not less than 3, the constrained intersection method is used to process the effective displacement vectors of the centroid of the node of interest of each single image in each frame to obtain the two-dimensional image displacement of the image sequence of the target object relative to the first frame image; if the number of valid displacement vectors is less than 3, the vector averaging method is used to process the effective displacement vectors of the centroid of the node of interest of each single image in each frame to obtain the two-dimensional image displacement of the image sequence of the target object relative to the first frame image.

[0101] Understandably, the formula Calculate the effective displacement vector between the centroid of the node of interest of each single image and the first frame,

[0102] in, The two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the constrained intersection method, is the two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the vector averaging method, is the phase nonlinearity value described in pixels, is the filter response after the image is processed by the filter.

[0103] In practice, the first step to recover the 3D displacement from the 2D image displacement is to find the corresponding points, that is, stereo matching. The proposed CBBT method is mainly used to extract the centroid from the image sequence. The coordinate point of the centroid in the camera image is expressed as , using the principle of epipolar geometry to determine the point The corresponding point on the camera 2 image is denoted by The plane connecting the 3D global point and the two camera centers is defined as the epipolar plane. The intersection of the epipolar plane and the image plane is called the epipolar line. And the basic matrix Substitute into formula (x), then The epipolar line corresponding to the point on the camera 2 image is,

[0104] (twenty four)

[0105] Epipolar Line The intersection point between and the centroid is .

[0106] Reconstruct the 3D displacement (X, Y, Z) based on the pixel displacement; the projections of the 3D point M (X, Y, Z) on the camera 1 image and the camera 2 image are respectively and ; The relationship between the three-dimensional global coordinates and the two-dimensional image pixel coordinates is shown in formula (25).

[0107] (25)

[0108] in Indicates the time-related video frames, and denote the elements of the calibration matrix of camera 1 and camera 2 respectively. The position of the 3D point at time t relative to video frame j is expressed as The 2D displacement of the node centroid has been obtained before, but the image position information in the Z direction, i.e., depth, is missing. To obtain the correct depth information by successfully matching the node centroids of the two images and then using focal length and parallax is to make the estimated value and truth value The difference between them is minimized, then the objective function can be defined as:

[0109] (26)

[0110] The key to successful matching is to make the objective function Get the minimum value. Select the particle swarm optimization algorithm (PSO) to get the desired result. Get the correct depth vector Z through formula (27),

[0111] (27)

[0112] Where f is the focal length of the camera, and d is the disparity between the centroid coordinates of the two single images.

[0113] After obtaining the three-dimensional position (X, Y, Z) of the center of mass point on the recorded video frame, the displacement of the center of mass at the selected point in the global coordinate system can be expressed as:

[0114] (28)

[0115] in, is the horizontal displacement of the center of mass in the defined global coordinate system (X, Y, Z), and is the horizontal displacement vector value of the previous frame image, It is the horizontal displacement vector value of the current frame image.

[0116] is the vertical displacement of the center of mass in the defined global coordinate system (X, Y, Z), and is the vertical vector value of the previous frame image, It is the depth vector value of the current frame image.

[0117] is the depth displacement of the centroid of the physical node of the current frame in the defined global coordinate system, is the depth vector of a single image in the previous frame, is the depth vector of a single image in the current frame.

[0118] It can be understood that the coordinates of the center of mass of the physical node are determined according to the coordinates of the center of mass of the node of interest of the corresponding two frames of single images; the particle swarm optimization algorithm is used to calculate the three-dimensional displacement of the center of mass of the physical node according to the calibration parameters, the coordinates of the center of mass of the node of interest of the corresponding two frames of single images, and the coordinates of the center of mass of the physical node, and the three-dimensional displacement is determined as the spatial displacement of the target object.

[0119] The beneficial effects of the structural three-dimensional displacement measurement method based on the centroid bounding box tracking algorithm of the present invention are as follows:

[0120] Compared with the traditional total station, this method can measure real-time dynamic displacement more effectively and conveniently. Compared with the general two-dimensional displacement measurement method, it can effectively measure the depth information and achieve the effect of three-dimensional displacement measurement;

[0121] A centroid bounding box tracking technology is proposed to generate a bounding box for extracting the node centroid of the truss bridge node, and the node centroid is used as the matching primitive of binocular vision; as a traditional target-free method, the selected feature primitives are easily affected by lighting conditions and texture features, and it is often difficult to achieve ideal working results under outdoor natural lighting conditions; the centroid bounding box tracking technology proposed by us is not easily affected by external conditions and has strong anti-noise ability;

[0122] A phase-based displacement measurement technology is proposed, which has good noise resistance. The general displacement measurement method based on tracking pixel points has poor noise resistance. First, the node centroid texture feature is proposed by using Gaussian smoothing enhancement, which is more conducive to the filter to extract phase information. Considering that image noise and incorrect filter response will bring difficulties to the phase-based optical flow method (POF), the nonlinear phase is quantified and the filter response strength is required. It is more capable of retaining effective phase information while filtering out high-frequency components, so as to obtain a stable phase and an almost constant spatial phase gradient, and accurately estimate the motion through the spatial phase gradient and phase difference. Finally, the unreliable phase motion component is eliminated, and the retained true displacement vector is used by IOC (constrained intersection) and VA (vector average) methods to calculate the two-dimensional displacement of the node centroid;

[0123] Node matching based on particle swarm optimization algorithm is faster. Traditional primitive matching calculates the SSD (Sum of Absolute Differences), SAD (SSD, Sum of Squared Differences), and SAD (NCC, Normalized Cross Correlation) of two feature primitives. This method is to define a window D with a size of (2m+1)×(2n+1) with the source matching point of the left image as the center, and then gradually calculate the difference between the grayscale sum of the left and right windows in the right image. Finally, the central pixel of the area with the smallest difference is the matching point. SAD is similar to NCC. This method uses the proposed CBBT method to extract the centroid from each image frame, obtains the basic matrix between the two cameras through the principle of epipolar geometry, constructs the node relationship on the two images, and uses the particle swarm optimization algorithm to find the minimum value of the objective function corresponding to the node centroid and calculate the disparity to obtain the depth.

[0124] The present invention also provides a structural three-dimensional displacement measurement system based on a centroid bounding box tracking algorithm, including an image acquisition unit, a centroid extraction unit, a filtering two-dimensional processing unit and a three-dimensional processing unit. The image acquisition device is used to acquire two original vibration videos of a target object; the centroid extraction unit is used to perform frame processing on each original vibration video to obtain a single image sequence, extract the key interest area corresponding to a single image in the single image sequence, and use the centroid bounding box tracking technology to obtain the centroid of the interest node corresponding to each key interest area; the filtering two-dimensional processing unit is used to perform Gaussian smoothing processing on each frame of a single image, extract local phase information of different directions and different scales of each frame of a single image according to multiple filters, and obtain an image sequence The method is used to obtain the phase information of the center of mass of the node of interest relative to the first frame; to track the phase of the center of mass of the node using the phase-based optical flow method to calculate the effective displacement vector between the center of mass of the node of interest of each single frame and the first frame, and to determine the number of effective displacement vectors; to process the effective displacement vector of the center of mass of the node of interest of each single frame using the constrained intersection method or the vector average method, and obtain two original vibration videos of the image sequence of the target object relative to the first frame image; the three-dimensional processing unit is used to match the center of mass of the image nodes of the two cameras using the particle swarm optimization algorithm, obtain the depth information of the center of mass of the physical node, and finally obtain the three-dimensional displacement of the center of mass of the physical node in combination with the two-dimensional displacement obtained by the optical flow method, and the three-dimensional displacement is the spatial displacement of the target object.

[0125] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm, characterized in that: The steps include: S10, using two calibrated cameras to collect two original vibration videos of the target object; S20, performing frame processing on each of the original vibration videos to obtain a single image sequence, extracting a key region of interest corresponding to a single image in the single image sequence, and using a centroid bounding box tracking technology to obtain a centroid of an interest node corresponding to each key region of interest; S30, performing Gaussian smoothing processing on the single image of each frame, extracting local phase information of the single image of each frame in different directions and at different scales according to multiple filters, and obtaining phase information of the centroid of the node of interest of the image sequence relative to the first frame; S40, using a phase-based optical flow method to track the node centroid phase to calculate the effective displacement vector between the centroid of the node of interest of the single image of each frame and the first frame, and determine the number of the effective displacement vectors; S50, using a constrained intersection method or a vector average method to process the effective displacement vector of the centroid of the node of interest of each single image of each frame, to obtain a two-dimensional displacement of the image sequence of the target object relative to the first frame of image; If the number of the valid displacement vectors is not less than 3, the effective displacement vectors of the centroids of the nodes of interest of each single image of each frame are processed by a constrained intersection method to obtain a two-dimensional displacement of the image sequence of the target object relative to the first frame of image; If the number of the effective displacement vectors is less than 3, the effective displacement vectors of the centroid of the node of interest of each frame of the single image are processed by a vector averaging method to obtain a two-dimensional displacement of the image sequence of the target object relative to the first frame of the image; S60, performing matching processing on the centroids of the nodes of interest of the single image of each frame of the two cameras to obtain the corresponding physical node centroids in the real coordinate system of each frame; Using a particle swarm optimization algorithm to obtain the depth vector of the centroid of the physical node of adjacent frames; Determine the horizontal displacement and vertical displacement of the centroid of the physical node of adjacent frames based on the two-dimensional displacement of the image; The real three-dimensional displacement of the centroid of the physical node of the image sequence is determined based on the depth vector, the horizontal displacement and the vertical displacement, and the three-dimensional displacement is the spatial displacement of the target object.

2. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to claim 1, characterized in that: Using formula Calculate the effective displacement vector between the centroid of the node of interest of the single image in each frame and the first frame, in, is the two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the constrained intersection method, is the two-dimensional displacement of the current frame image of the target object relative to the first frame image obtained by the vector averaging method, is the phase nonlinearity value described in pixels, is the filter response after the image is processed by the filter, is the image sequence number, (x, y) is the image coordinate, and m is the filter sequence number.

3. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to any one of claims 1 or 2, characterized in that: Determine the real coordinates of the center of mass of the node of interest in the world coordinate system according to the coordinates of the center of mass of the node of interest of the two frames of the single image corresponding to the same time; According to the calibration parameters and the real coordinates of the centroids of the interest nodes of the corresponding two frames of the single image, the centroids of the physical nodes corresponding to the single image are found.

4. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to claim 3, characterized in that: Using formula Determine the depth vector of the single image of the adjacent frame, wherein, is the depth displacement of the centroid of the physical node of the current frame in the defined global coordinate system, is the depth vector of the single image of the current frame, is the depth vector of the single image in the previous frame, is the number of image sequences.

5. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to any one of claims 1 or 2, characterized in that: Step S20 specifically includes: S21, performing frame processing on each of the original vibration videos to obtain a single image sequence, and extracting a single image from the single image sequence; S22, processing the single image using digital image processing technology to obtain a pre-processed image file having a key area of ​​interest; S23, using a centroid bounding box tracking algorithm to generate a bounding box surrounding the key area of ​​interest, and extracting the centroid of the interest node in the key area of ​​interest.

6. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to any one of claims 1 or 2, characterized in that: Step S30 specifically includes: S31, calculating and obtaining the Gaussian smoothed gradient of a single image sequence containing the centroid of the node of interest; S32, local phase extraction of a single image using 10 Gabor filters of different scales and orientations; S33, by tracking the phase motion technology, calculate the phase information of the centroid of the node of interest in 10 directions, and obtain the phase information of the centroid of the node of interest in the image sequence relative to the first frame.

7. The method for measuring three-dimensional displacement of a structure based on a centroid bounding box tracking algorithm according to any one of claims 1 or 2, characterized in that: Step S40 specifically includes: S41, using a phase-based optical flow method to calculate multiple centroid displacement components of the centroid from the phase information of the centroid of the node of interest in the image sequence relative to the first frame; S42, accurately estimating the motion based on the spatial phase gradient and the phase difference and eliminating the erroneous center-of-mass displacement component to obtain the effective displacement vector.

8. A structural three-dimensional displacement measurement system based on a centroid bounding box tracking algorithm, characterized in that: It includes an image acquisition unit, a centroid extraction unit, a filtering two-dimensional processing unit, and a three-dimensional processing unit. The image acquisition unit is used to acquire two original vibration videos of the target object; The centroid extraction unit is used to perform frame processing on each of the original vibration videos to obtain a single image sequence, extract the key interest region corresponding to a single image in the single image sequence, and use the centroid bounding box tracking technology to obtain the centroid of the interest node corresponding to each of the key interest regions; The filtering two-dimensional processing unit is used to perform Gaussian smoothing on each frame of the single image, extract local phase information of different directions and different scales of each frame of the single image according to multiple filters, and obtain the phase information of the centroid of the node of interest of the image sequence relative to the first frame; it is used to track the phase of the node centroid by using the phase-based optical flow method to calculate the effective displacement vector between the centroid of the node of interest of each frame of the single image and the first frame, and determine the number of the effective displacement vectors; it is used to process the effective displacement vector of the centroid of the node of interest of each frame of the single image by using the constrained intersection method or the vector average method to obtain the image two-dimensional displacement of the image sequence of the target object relative to the first frame of the image; if the number of the effective displacement vectors is not less than 3, the effective displacement vector of the centroid of the node of interest of each frame of the single image is processed by the constrained intersection method to obtain the image two-dimensional displacement of the image sequence of the target object relative to the first frame of the image; if the number of the effective displacement vectors is less than 3, the effective displacement vector of the centroid of the node of interest of each frame of the single image is processed by the vector average method to obtain the image two-dimensional displacement of the image sequence of the target object relative to the first frame of the image; The three-dimensional processing unit is used to use a particle swarm optimization algorithm to match the image node centroids of the two cameras, obtain the depth information of the physical node centroid, and finally obtain the three-dimensional displacement of the physical node centroid in combination with the two-dimensional displacement obtained by the optical flow method. The three-dimensional displacement is the spatial displacement of the target object.

Images