Automatic measurement method and system for interlayer displacement of structure based on visual data

Abstract

The application discloses a structure interlayer displacement automatic measurement method and system based on visual data and belongs to the technical field of structure displacement measurement. The measurement method comprises the following steps: acquiring video data in the vibration process of a building based on a monitoring camera; processing the video data by using an improved kernel correlation filtering algorithm to extract building interlayer displacement information; verifying the extracted displacement information and outputting building interlayer displacement results. On the basis of a classical kernel correlation filtering target tracking method, the application utilizes image pyramid, robust feature extraction and discrete Fourier transform up-sampling technology to expand the tracking range, enhance the edge features and improve the tracking precision; the application fully utilizes the existing monitoring camera equipment to measure the interlayer displacement, does not need to additionally arrange sensors and reduces the cost budget.

Description

Technical Field

[0001] This invention relates to structural displacement measurement technology, and in particular to an automatic method and system for measuring inter-story displacement based on visual data. Background Technology

[0002] Earthquakes, tsunamis, storm surges, and other natural disasters often cause severe damage to buildings. After a disaster, it is necessary to repair and rebuild damaged and collapsed buildings, while simultaneously assessing the performance of those buildings that may still be repairable. Minor, visible structural damage, such as cracks or spalling, can be easily identified through visual inspection. However, for invisible structural damage, while sophisticated seismic monitoring systems are installed in important buildings during construction, these systems primarily rely on acceleration sensors to monitor the dynamic characteristics of the structure. However, the installation and maintenance of such monitoring systems require significant human and material resources, limiting their application to large-scale buildings.

[0003] In structural health monitoring, a variety of sensors are used for structural vibration measurement. Besides traditional accelerometers, other types of sensors, such as laser sensors and global navigation satellite systems, are also widely used. In recent years, vision-based structural displacement measurement methods have attracted widespread attention in mechanical and civil engineering fields. Researchers have demonstrated the potential of using cameras to capture structural vibrations (Ding, Z., Yu, Y., Tan, D., Yuen, K.-V., 2023. Adaptivevision feature extractions and reinforced learning-assisted evolution for structural condition assessment. Structural and MultidisciplinaryOptimization 66, 209). This method has evolved from target-based methods to target-free methods, from laboratory environments to field applications, and from two-dimensional measurement to three-dimensional measurement.

[0004] Compared to other sensor-based structural displacement monitoring methods, camera-based measurement methods utilize visual algorithms, offering advantages such as lower cost, simpler deployment, and the ability to capture multiple measurement points. Furthermore, numerous surveillance cameras are installed in busy urban areas and important buildings, primarily for maintaining public safety and order. However, unlike previous vision-based displacement measurement methods that typically rely on professional cameras producing low-noise, clear video, surveillance cameras are relatively inexpensive but produce lower-quality video with low frame rates, low resolution, and high noise levels. Moreover, during periods of high structural vibration caused by natural disasters, video frames are prone to severe image degradation, such as noise due to voltage instability or blurring caused by rapid movement. Therefore, camera-based measurement algorithms must maintain high robustness even under harsh environmental conditions.

[0005] Existing research has explored vision-based structural displacement measurement under complex environmental conditions such as low resolution, noise, blur, or low illumination. There are two main methods for achieving displacement measurement under complex environmental conditions: (1) reconstructing degraded images using image restoration models; and (2) directly improving the robustness and accuracy of displacement measurement algorithms. For the first method, datasets containing pairs of original and degraded images have been developed to train image restoration techniques based on deep neural networks. These trained networks are used to restore degraded images to their original quality, and then the restored images are used to estimate structural vibrations. Many studies have used this method to achieve visual measurement of structural displacement in complex environments, including low resolution, blur, and light spot problems on targets. However, these neural network-based methods are generally time-consuming.

[0006] Besides improving the clarity of degraded images, some researchers have focused on enhancing the robustness of displacement recognition algorithms by refining target tracking algorithms, achieving higher computational efficiency compared to neural network-based methods. However, in structural displacement measurement tasks, especially during periods of high structural vibration caused by natural disasters, severe noise and motion blur problems remain unresolved. Summary of the Invention

[0007] Purpose of the invention: To address the above problems, the purpose of this invention is to provide an automatic measurement method and system for inter-story displacement based on visual data.

[0008] Technical solution: The present invention provides an automatic method for measuring inter-story displacement based on visual data, comprising the following steps:

[0009] Step 1: Acquire video data of the building vibration process using surveillance cameras;

[0010] Step 2: The video data is processed using an improved kernel correlation filtering algorithm to extract inter-story displacement information of the building. The improved kernel correlation filtering algorithm integrates image pyramid, edge enhancement image operator and discrete Fourier transform upsampling. Multi-resolution image sequence is generated through image pyramid, image edge features are extracted using edge enhancement image operator, and discrete Fourier transform is used to locally upsample the result of kernel correlation filtering target tracking algorithm to obtain sub-pixel level target localization.

[0011] Step 3: Verify the extracted displacement information and output the inter-story displacement results of the building.

[0012] Further, step 2 includes:

[0013] Convert surveillance video to grayscale image, grayscale image First, a Gaussian kernel filter is performed to obtain the pixel value, represented as:

[0014] ,

[0015] in, This indicates that after filtering, in The pixel value of the location, Indicates the position of the original image frame grayscale image at that location, Indicates the radius of the Gaussian kernel; Representing coordinates The Gaussian kernel weight at a given location is calculated using the following formula:

[0016] ,

[0017] in, This represents the standard deviation of the Gaussian kernel;

[0018] Then, a downsampling operation is performed to obtain the first image features. , is represented as:

[0019] ,

[0020] First image features Then, Gaussian kernel filtering and downsampling operations are performed sequentially to obtain the second image features. This leads to the image sequence. The image sequence is then used to extract features using image operators.

[0021] Furthermore, the edge enhancement image operator mentioned in step 2 includes at least one of the Sobel operator, Prewitt operator, Canny operator, and Scharr operator, and the operator processing result is combined with the original grayscale image to form a multi-channel feature tensor.

[0022] Furthermore, step 2 also includes:

[0023] The filter was optimized using linear ridge regression, and the multi-channel feature tensor of the image was processed using kernel techniques. Mapping to nonlinear space The parameters are then input into the optimized filter for filtering, resulting in the filter parameter vector. , is represented as:

[0024] ,

[0025] in, The corresponding true values ​​are obtained by generating a two-dimensional Gaussian distribution. It is a regularization term to prevent overfitting. This indicates the contribution of the l-th training sample to the final filter; It is the number of samples generated;

[0026] pass To solve, where, yes The kernel matrix, whose elements are , Represents the kernel matrix;

[0027] No. The target position of the frame is represented as:

[0028] ,

[0029] in, It is the first The final response result of the kernel correlation filter target tracking algorithm for the frame. Indicates will Frame images are mapped to a high-dimensional space; This represents the filter parameters in kernel space.

[0030] Furthermore, the implementation process of discrete Fourier transform upsampling in step 2 is as follows:

[0031] The frequency components are obtained by performing a discrete Fourier transform on the final response of the kernel correlation filter target tracking algorithm, and are expressed as follows:

[0032] ,

[0033] in, , Indicates the size of the region of interest. Indicates the size is The response results of the region of interest It is an index. It is an index in the frequency domain. Represents the imaginary unit;

[0034] The time signal is recovered by inverse discrete Fourier transform, and can be expressed as:

[0035] ,

[0036] Will Written in matrix form, it can be represented as:

[0037] ,

[0038] in, for transpose, This represents the frequency domain signal obtained by Fourier transforming f. and The expressions are as follows:

[0039] ,

[0040] ,

[0041] in, yes The position of the maximum value, express The number of adjacent elements, Indicates the upsampling factor;

[0042] This is the response of the target tracking algorithm after discrete Fourier transform upsampling, and the location of its maximum value. This is the sub-pixel accuracy result for the selected target.

[0043] Furthermore, step 1 also includes preprocessing the video data, correcting the measurement deviation caused by the camera installation angle using a perspective correction method, calculating the physical-pixel conversion factor using a reference target with known real size, and converting the pixel displacement into real physical displacement.

[0044] Another aspect of the present invention provides an automatic structural inter-story displacement measurement system based on visual data, comprising:

[0045] Data acquisition module: Composed of surveillance cameras, used to collect video data of buildings during natural disasters or vibrations;

[0046] Preprocessing module: used to perform perspective correction, resolution adjustment, noise simulation and motion blur simulation on video data to generate data to be processed that conforms to actual working conditions;

[0047] Displacement extraction module: Integrates an improved kernel correlation filtering algorithm to extract inter-story displacement information of buildings from preprocessed video data;

[0048] Data output module: Used to output verified inter-story displacement data and frequency domain characteristic information, providing a basis for structural damage assessment.

[0049] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:

[0050] 1. Based on the classic kernel correlation filter target tracking method, this invention utilizes image pyramid, robust feature extraction, and discrete Fourier transform upsampling techniques to expand the tracking range, enhance edge features, and improve tracking accuracy;

[0051] 2. This invention obtains the most accurate displacement results by improving the kernel correlation filtering algorithm, and can effectively capture the frequency domain characteristics of inter-story displacement. The algorithm is stable and can be used in building disaster environments. It can resist interference such as fuzziness and noise.

[0052] 3. This invention can make full use of existing monitoring camera equipment for inter-floor displacement measurement, without the need for additional sensors, thus reducing cost budget. Attached Figure Description

[0053] Figure 1 This is a flowchart of a method for automatically measuring inter-story displacement based on visual data.

[0054] Figure 2 Flowchart for improving the kernel correlation filter algorithm;

[0055] Figure 3 A flowchart for generating an image sequence;

[0056] Figure 4 The layout is a three-layer test structure;

[0057] Figure 5 A schematic diagram simulating motion blur and noise caused by vibration;

[0058] Figure 6 Displacement and amplitude curves estimated from low-resolution video;

[0059] Figure 7 Box plots showing the errors of different methods;

[0060] Figure 8 Schematic diagram of monitoring images during an earthquake under different simulation conditions;

[0061] Figure 9 The first video image captured by the surveillance camera;

[0062] Figure 10For the first video segment, here are the estimated displacement curves in the x and y directions using different methods;

[0063] Figure 11 The second video image captured by the surveillance camera;

[0064] Figure 12 The figures show the estimated displacement curves in the x and y directions for the second video segment using different methods. Detailed Implementation

[0065] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.

[0066] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0067] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0068] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0069] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0071] Example 1

[0072] The flowchart of the automatic inter-story displacement measurement method based on visual data described in this embodiment is as follows: Figure 1 As shown, the method includes the following steps:

[0073] Step 1: Acquire video data of the building vibration process using surveillance cameras.

[0074] Furthermore, step 1 also includes preprocessing the video data, correcting the measurement deviation caused by the camera installation angle using a perspective correction method, calculating the physical-pixel conversion factor using a reference target with known real size, and converting the pixel displacement into real physical displacement.

[0075] Step 2: The improved kernel correlation filter algorithm is used to process the video data and extract the inter-story displacement information of the building. The improved kernel correlation filter algorithm integrates image pyramid, edge enhancement image operator and discrete Fourier transform upsampling. Multi-resolution image sequence is generated through image pyramid, image edge features are extracted by edge enhancement image operator, and discrete Fourier transform is used to locally upsample the result of kernel correlation filter target tracking algorithm to obtain sub-pixel level target localization.

[0076] like Figure 2 As shown, step 2 further includes:

[0077] Convert surveillance video to grayscale image, grayscale image First, a Gaussian kernel filter is performed to obtain the pixel value, represented as:

[0078] ,

[0079] in, This indicates that after filtering, in The pixel value of the location, Indicates the position of the original image frame grayscale image at that location, Indicates the radius of the Gaussian kernel; Representing coordinates The Gaussian kernel weight at a given location is calculated using the following formula:

[0080] ,

[0081] in, This represents the standard deviation of the Gaussian kernel;

[0082] Then, a downsampling operation is performed to obtain the first image features. , is represented as:

[0083] ,

[0084] like Figure 3 As shown, the first image features Then, Gaussian kernel filtering and downsampling operations are performed sequentially to obtain the second image features. This leads to the image sequence. The image sequence is then used to extract features using image operators.

[0085] Image sequence The image sequence is relatively large, and direct calculation would be computationally expensive. Therefore, feature extraction is performed on the image sequence using image operators. The sequence is first downsampled to a low-resolution image, and an approximate location is calculated in the low-resolution image. Then, further precise calculations are performed in the local area of ​​the high-resolution image to reduce computational overhead.

[0086] Furthermore, in step 2, the edge enhancement image operator includes at least one of the Sobel operator, Prewitt operator, Canny operator, and Scharr operator, and the operator processing result is combined with the original grayscale image to form a multi-channel feature tensor.

[0087] Furthermore, step 2 also includes:

[0088] The filter was optimized using linear ridge regression, and the multi-channel feature tensor of the image was processed using kernel techniques. Mapping to nonlinear space The parameters are then input into the optimized filter for filtering, resulting in the filter parameter vector. , is represented as:

[0089] ,

[0090] in, The corresponding true values ​​are obtained by generating a two-dimensional Gaussian distribution. It is a regularization term to prevent overfitting. This indicates the contribution of the l-th training sample to the final filter; It is the number of samples generated;

[0091] pass To solve, where, yes The kernel matrix, whose elements are , Represents the kernel matrix;

[0092] No. The target position of the frame is represented as:

[0093] ,

[0094] in, It is the first The final response result of the kernel correlation filter target tracking algorithm for the frame. Indicates will Frame images are mapped to a high-dimensional space; This represents the filter parameters in kernel space.

[0095] Furthermore, the implementation process of discrete Fourier transform upsampling in step 2 is as follows:

[0096] The frequency components are obtained by performing a discrete Fourier transform on the final response of the kernel correlation filter target tracking algorithm, and are expressed as follows:

[0097] ,

[0098] in, , Indicates the size of the region of interest. Indicates the size is The response results of the region of interest It is an index. It is an index in the frequency domain. Represents the imaginary unit;

[0099] The time signal is recovered by inverse discrete Fourier transform, and can be expressed as:

[0100] ,

[0101] Will Written in matrix form, it can be represented as:

[0102] ,

[0103] in, for transpose, This represents the frequency domain signal obtained by Fourier transforming f. and The expressions are as follows:

[0104] ,

[0105] ,

[0106] in, yes The position of the maximum value, express The number of adjacent elements, Indicates the upsampling factor;

[0107] This is the response of the target tracking algorithm after discrete Fourier transform upsampling, and the location of its maximum value. This is the sub-pixel accuracy result of the selected target. After performing simple post-processing on this sub-pixel level positioning (e.g., removing initial values), the final displacement measurement result is obtained.

[0108] Step 3: Verify the extracted displacement information and output the inter-story displacement results of the building.

[0109] To test the performance of the visual data-based automatic inter-story displacement measurement method described in this invention, which utilizes a monitoring camera to capture inter-story displacement during an earthquake, a full-scale shaking table test was conducted on a steel frame structure. The experiment was carried out in a seismic laboratory using a three-story steel frame with complete non-framework components. The layout of the three-story test structure is as follows... Figure 4 As shown, the center-to-center spacing of the longitudinal columns is The center-to-center spacing of the horizontal columns is The floor height is To verify this, along each floor of the structure... shaft and A high-precision displacement sensor is installed on the shaft, providing a basis for comparing sensor measurement results with visual measurement results.

[0110] This example included 21 seismic wave loading tests, including those along the [path / line]. direction, Seismic waves were applied in both directions and bidirectionally. These experiments covered a variety of earthquake intensities, simulating different levels of seismic activity. The diversity of loading conditions allowed for a comprehensive evaluation of the algorithm's performance under different seismic scenarios, ensuring that the results accurately reflect its ability to measure inter-story displacement under various earthquake magnitudes and directions. Furthermore, numerous sensors were installed on the experimental structure to comprehensively monitor changes in its mechanical properties and displacement response during the experiments. These sensors included strain sensors, displacement sensors, and accelerometers, ensuring accurate capture of various mechanical and dynamic responses of the structure under seismic wave loading. Particularly important were the sensors installed on each floor of the structure. direction and Displacement sensors are installed in all directions. The data from these displacement sensors not only provides accurate displacement measurements but also serves as reference data for vision-based displacement measurements.

[0111] For visual measurements, a GoPro Hero 10 Black camera was used as an interior building monitoring camera. The camera was set to 4K resolution, 120 Hz frame rate, and image stabilization was completely disabled to ensure that subtle inter-story vibrations were not suppressed. While professional action cameras like GoPros offer superior performance, actual building monitoring cameras are typically lower-spec and more affordable. During an earthquake, monitoring cameras may suffer severe motion blur due to the rapid movement of the building, posing a significant challenge to accurate visual measurements. Therefore, this example focuses on the motion blur problem caused by vibration, aiming to verify the effectiveness of the visual algorithm in detecting inter-story displacement under such extreme conditions. Figure 5 As shown.

[0112] Surveillance cameras are typically installed with their optical axes angled to the ground to capture a wider field of view. However, this installation method can affect the accuracy of certain measurements, such as inter-story drift analysis. To mitigate this effect, the camera's optical axis must be recalibrated to be parallel to the ground before performing such measurements. This process involves manually selecting four points on the structural surface that form a rectangle, and their corresponding projection points in the calibrated image. Perspective correction methods can then be used to correct the perspective, such as... Figure 5 As shown in Figure (a), the corrected region of interest (ROI) video is generated. This correction relies on accurately identifying the four reference points of the quadrilateral surrounding the ROI and calculating the homography transformation to correct the image to a frontal view, effectively mitigating the impact of the camera's installation angle. Therefore, as long as the reference points can be clearly detected and their positions accurately determined, the installation angle of the surveillance camera itself will not affect the final measurement accuracy. This ensures the robustness of the method described in this invention under various camera configurations commonly found in urban surveillance systems. After obtaining the perspective-corrected image, the physical-pixel factor can be calculated by comparing the known real-world dimensions with their corresponding pixel dimensions in the image. Then, it can be used The pixel displacement data detected by the visual algorithm is converted into real-world displacement. This conversion is crucial for ensuring the accuracy of visual measurements because it transforms relative displacement data in the image into absolute physical displacement, enabling more precise structural displacement analysis. In this example, the dimensions of the artificial target on the floor are known and therefore available.

[0113] To simulate the working conditions of cameras in real earthquake scenarios with low frame rates, low resolution, noise, and motion blur, the performance of the algorithm described in this invention was compared with other algorithms under these challenging conditions. These factors significantly degrade image quality and affect the accuracy of displacement measurements. To address this issue, images captured by a GoPro were adjusted, and noise and motion blur were artificially introduced to evaluate the robustness and adaptability of the proposed method. In the experiments, various visual disturbances were simulated using MATLAB, namely, reducing image resolution, reducing video frame rate, adding data noise, and motion blur. Image resolution can be achieved using the `imresize` function. Image noise was introduced using the `imnoise` function, with Gaussian noise chosen as the noise model. The mean and variance of the Gaussian noise were set to predefined values. and noise level Defined as Gaussian noise is used. Motion blur is applied using the `imfilter` function, and a motion blur filter is generated using the `fspecial` function. The motion blur direction for each frame is random, while the blur length is determined based on the number of horizontal pixels in the image and a motion factor. Scaling is applied. Motion factors. The larger the image size, the more severe the simulated motion blur. This method ensures that the blurring effect scales proportionally with the image size. Using these techniques, the experiment simulated common interference found in surveillance cameras during earthquakes, such as… Figure 5 As shown in Figure (b), interference is particularly caused by low resolution, low frame rate, data noise, and motion blur.

[0114] To test the applicability of the algorithm described in this invention, a comparative study of inter-story displacement measurements under different seismic loads was conducted. In this experiment, two different intensities of El Centro seismic wave conditions were selected: Condition 1 (… Unidirectional seismic wave, peak ground acceleration (PGA) is ) and Working Condition 2 ( Unidirectional seismic wave, PGA is The GoPro videos captured under each condition underwent preprocessing, including perspective correction and simulation of earthquake video scenes. These simulations included low-resolution video, blurred video, noisy video, and a combination of noise and blur. First, the resolution of the original GoPro video was reduced to 1080p (a common resolution for surveillance cameras), and the frame rate was reduced to 8Hz to obtain low-resolution video. Under both conditions, the displacement and amplitude estimated from the low-resolution video using the method described in this invention are as follows: Figure 6As shown, Figures (a) and (c) represent the displacement and amplitude under condition 1, respectively, while Figures (b) and (d) represent the displacement and amplitude under condition 2, respectively. In both conditions, the visual target is... Figure 4 Artificial target 1 in the middle. From Figure 6 It can be seen that, The directional results are highly accurate, and the frequency domain closely matches the original signal. In contrast, due to The amplitude of the directional vibration is small, and the result of the time-domain signal is similar to... The directional accuracy is relatively low. However, in the frequency domain, the first five frequencies are accurately captured.

[0115] To demonstrate the superiority of the method described in this invention, the invention (denoted as the IKCF method) is compared with traditional vision-based methods, including the Kanade-Lucas-Tomasi (KLT) tracker based on sparse optical flow (Pan, X., Yang, TY, Xiao, Y., Yao, H., Adeli, H., 2023. Vision-based real-time structural vibration measurement through deep-learning-based detection and tracking methods. Engineering Structures 281, 115676.) and template matching based on upsampled cross-correlation (UCC) (Feng, D., Feng, MQ, Ozer, E., Fukuda, Y., 2015. A Vision-Based Sensor for Noncontact Structural Displacement Measurement. Sensors 15, 16557–16575.). Tables 1 and 2 list the root mean square error (RMSE) and correlation coefficient (CC) of the displacements identified using IKCF, KLT, and UCC. RMSE quantitatively measures the difference between the estimated and actual values, while CC assesses the linear correlation between the estimated and actual displacements. Figure 7 The error box plots of three methods (IKCF, UCC, and IKCF) are shown. The left figure is the error box plot under condition 1, and the right figure is the error box plot under condition 2. The error value is used to represent the square root of the absolute difference between the identified displacement and the corresponding measured value.

[0116] Table 1. RMSE and CC of estimated y-direction displacement under different noise levels

[0117] Table 2. RMSE and CC of estimated x-direction displacement under different noise levels.

[0118] To investigate the combined effects of noise and motion blur, motion blur and image noise were simultaneously applied to the original low-resolution image to simulate potential noise and motion blur in surveillance videos during earthquakes. Three different simulation conditions were tested, as follows: Figure 8 Figure (a) shows a weak earthquake simulation using S1: motion factor. The noise level is 0.01. =0.01; Figure (b), S2 represents the motion factor in a moderate earthquake simulation. The noise level is 0.03. =0.02; Figure (c), S3 represents the strong earthquake simulation: motion factor The noise level is 0.05. The value is 0.03. Furthermore, two targets were analyzed, such as... Figure 8 Figure (a) shows: including complete target F (complete artificial target 2): simulating a scene with obvious texture in the video; and partial target P (partial artificial target 5): simulating a scene with less texture in the video.

[0119] When noise intensity and motion blur are low, the error differences among the three methods are not significant. However, as noise intensity and motion blur increase, the IKCF method consistently maintains the lowest error. Under the S3-P condition, the overall error of the KLT method is similar to that of IKCF, but this does not indicate an improvement in the performance of the KLT method in this scenario; on the contrary, it completely fails, and the identified displacement curve becomes a straight line. Overall, the IKCF algorithm performs best, achieving higher accuracy in both the time and frequency domains compared to the other two traditional algorithms, even in strong earthquake simulation videos.

[0120] To further verify the performance of the method described in this invention, two video clips captured by indoor surveillance cameras during real earthquake events were selected for comparative analysis. The comparison algorithms included KLT optical flow, UCC template matching, CSR (A. Lukežic, T. Vojír, LC Zajc, J. Matas, M. Kristan, 2017. Discriminative Correlation Filter with Channel and Spatial Reliability, in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Presented at the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4847–4856. https: / / doi.org / 10.1109 / CVPR.2017.515.), and C-COT (Danelljan, M., Robinson, A., Shahbaz Khan, F., Felsberg, M., 2016. Beyond Correlation Filters: Learning). Continuous Convolution Operators for Visual Tracking, in: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.), Computer Vision – ECCV2016. Springer International Publishing, Cham, pp. 472–488.).

[0121] The first video is from an indoor security camera in a restaurant during the 5.5 magnitude earthquake that struck Zibo City, Shandong Province in August 2023. Figure 9 As shown, Figure (a) is a frame from the initial video frame, Figure (b) is the corrected viewpoint diagram, and Figure (c) shows that the images captured by the monitoring camera during the earthquake were severely blurred and noisy, which posed a significant challenge to the visual displacement measurement method.

[0122] A small portion of the ground exit markers was used as the target, and inter-story displacement was calculated using five different algorithms (KLT, UCC, C-COT, CSR, and this invention). The final estimated displacement is as follows: Figure 10 As shown in the figures, (a), (b), (c), (d), and (e) correspond to the displacements in the x and y directions of KLT, UCC, C-COT, CSR, and the method described in this invention, respectively. It can be seen that the CSR and C-COT methods exhibit significant deviations and poor performance; the KLT method shows some drift, while the UCC method and the method described in this invention achieve relatively better results.

[0123] The second video is as follows Figure 11 Figure (a) in Figure 11 shows footage recorded by a home security camera during the Jishishan County earthquake in Gansu Province in December 2023. Target tracking was challenging due to the lack of clear ground markings, and the video did not show any obvious relative motion between the table / chair and the ground. Therefore, a bag on the chair in Figure (b) of Figure 11 was selected as the tracking target, and the relative displacement between the camera and the ground was extracted as inter-floor displacement. Figure 11 Figure (c) shows that during the earthquake, the images captured by surveillance cameras were significantly blurrier than the indoor video footage from Shandong. The final estimated displacement is as follows: Figure 12 As shown in the figures, (a), (b), (c), (d), and (e) correspond to the displacements in the x and y directions of KLT, UCC, C-COT, CSR, and the present invention, respectively. The identified target positions of all methods do not significantly deviate from the actual target positions. However, it can be seen that, except for the method described in this invention, the relative vibration displacement curves identified by other methods differ significantly from the actual situation. When severe motion ambiguity exists, the KLT method fails directly, while the UCC and CSR methods exhibit obvious mismatch phenomena, and the C-COT method has significant noise.

[0124] The automatic inter-story displacement measurement method based on visual data described in this invention can be used to measure the displacement of buildings under high vibration conditions. It estimates inter-story displacement using indoor video and has been validated in shaking table tests of steel frames considering various video qualities. Results show that this invention can accurately estimate the inter-story displacement of structures with an error of less than 1%. Compared with template matching algorithms and optical flow methods, the IKCF method achieves better results. Furthermore, this method has been applied to inter-story displacement estimation in two surveillance videos, and compared with other sub-pixel target tracking algorithms, the IKCF method estimates more accurate inter-story displacement and frequency information.

[0125] Example 2

[0126] This example describes an automatic structural inter-story displacement measurement system based on visual data, which includes:

[0127] Data acquisition module: Composed of surveillance cameras, used to collect video data of buildings during natural disasters or vibrations;

[0128] Preprocessing module: used to perform perspective correction, resolution adjustment, noise simulation and motion blur simulation on video data to generate data to be processed that conforms to actual working conditions;

[0129] Displacement extraction module: Integrates an improved kernel correlation filtering algorithm to extract inter-story displacement information of buildings from preprocessed video data;

[0130] Data output module: Used to output verified inter-story displacement data and frequency domain characteristic information, providing a basis for structural damage assessment.

Claims

1. An automatic method for measuring inter-story displacement based on visual data, characterized in that, Includes the following steps: Step 1: Acquire video data of the building vibration process using surveillance cameras; Step 2: The video data is processed using an improved kernel correlation filtering algorithm to extract inter-story displacement information of the building; The improved kernel correlation filter algorithm integrates image pyramid, edge enhancement image operator and discrete Fourier transform upsampling. It generates multi-resolution image sequences through image pyramid, extracts image edge features using edge enhancement image operator, and uses discrete Fourier transform to locally upsample the result of kernel correlation filter target tracking algorithm to obtain sub-pixel level target localization. Step 3: Verify the extracted displacement information and output the inter-story displacement results of the building.

2. The automatic measurement method for inter-story displacement based on visual data according to claim 1, characterized in that, Step 2 includes: Convert surveillance video to grayscale image, grayscale image First, a Gaussian kernel filter is performed to obtain the pixel value, represented as: ， in, This indicates that after filtering, in The pixel value of the location, Indicates the position of the original image frame grayscale image at that location, Indicates the radius of the Gaussian kernel; Representing coordinates The Gaussian kernel weight at a given location is calculated using the following formula: ， in, This represents the standard deviation of the Gaussian kernel; Then, a downsampling operation is performed to obtain the first image features. , represented as: ， First image features Then, Gaussian kernel filtering and downsampling operations are performed sequentially to obtain the second image features. This leads to the image sequence. The image sequence is then used to extract features using image operators.

3. The automatic measurement method for inter-story displacement based on visual data according to claim 2, characterized in that, The edge enhancement image operator mentioned in step 2 includes at least one of the Sobel operator, Prewitt operator, Canny operator, and Scharr operator. The operator processing result is combined with the original grayscale image to form a multi-channel feature tensor.

4. The automatic measurement method for inter-story displacement based on visual data according to claim 3, characterized in that, Step 2 also includes: The filter was optimized using linear ridge regression, and the multi-channel feature tensor of the image was processed using kernel techniques. Mapping to nonlinear space The parameters are then input into the optimized filter for filtering, resulting in the filter parameter vector. , represented as: ， in, The corresponding true values ​​are obtained by generating a two-dimensional Gaussian distribution. It is a regularization term to prevent overfitting. This indicates the contribution of the l-th training sample to the final filter; It is the number of samples generated; pass To solve, where, yes The kernel matrix, whose elements are , Represents the kernel matrix; No. The target position of the frame is represented as: ， in, It is the first The final response result of the kernel correlation filter target tracking algorithm for the frame. Indicates will Frame images are mapped to a high-dimensional space; This represents the filter parameters in kernel space.

5. The automatic measurement method for inter-story displacement based on visual data according to claim 4, characterized in that, The implementation process of discrete Fourier transform upsampling in step 2 is as follows: The frequency components are obtained by performing a discrete Fourier transform on the final response of the kernel correlation filter target tracking algorithm, and are expressed as follows: ， in, , Indicates the size of the region of interest. Indicates the size is The response results of the region of interest It is an index. It is an index in the frequency domain. Represents the imaginary unit; The time signal is recovered by inverse discrete Fourier transform, and can be expressed as: ， Will Written in matrix form, it can be represented as: ， in, for transpose, This represents the frequency domain signal obtained by Fourier transforming f. and The expressions are as follows: ， ， in, yes The position of the maximum value, express The number of adjacent elements, Indicates the upsampling factor; This is the response of the target tracking algorithm after discrete Fourier transform upsampling, and the location of its maximum value. This is the sub-pixel accuracy result for the selected target.

6. The automatic measurement method for inter-story displacement based on visual data according to any one of claims 1 to 5, characterized in that, Step 1 also includes preprocessing the video data, correcting the measurement deviation caused by the camera installation angle using a perspective correction method, calculating the physical-pixel conversion factor using a reference target with known real size, and converting pixel displacement into real physical displacement.

7. An automatic structural inter-story displacement measurement system based on visual data, characterized in that, include: Data acquisition module: Composed of surveillance cameras, used to collect video data of buildings during natural disasters or vibrations; Preprocessing module: used to perform perspective correction, resolution adjustment, noise simulation and motion blur simulation on video data to generate data to be processed that conforms to actual working conditions; Displacement extraction module: Integrates an improved kernel correlation filtering algorithm to extract inter-story displacement information of buildings from preprocessed video data; Data output module: Used to output verified inter-story displacement data and frequency domain characteristic information, providing a basis for structural damage assessment.

Images