Computer vision-based structural vibration displacement monitoring method and system, and storage medium

Abstract

The application provides a computer vision-based structural vibration displacement monitoring method and system and a storage medium, and the method comprises the following steps: step 1, structural vibration video preprocessing; basic information of video data is automatically acquired by using a video basic information automatic acquisition algorithm, and an image gray scale and local histogram equalization processing method is used to simplify the image calculation complexity and enhance the image features; step 2, structural displacement monitoring based on improved FAST corner detection; an automatic threshold calculation method based on the maximum inter-class variance is introduced to form an improved FAST corner detection algorithm, the image features obtained by the improved FAST corner detection algorithm are combined with the pyramid Lucas-Kanade optical flow method, and a whole-process automatic image feature detection and motion tracking scheme is established. The method has the advantages that the overall monitoring process is optimized, and the method is non-contact, simple to operate and high in automation.

Description

Technical Field

[0001] This invention relates to the field of structural health monitoring, and in particular to a method, system, and storage medium for monitoring structural vibration and displacement based on computer vision. Background Technology

[0002] Buildings and structures have a limited design service life, and over time, potential safety issues may arise. Furthermore, during normal use, structures are constantly subjected to external factors. Inevitably, structures experience accumulated damage and weakened resistance, posing significant safety risks. These phenomena underscore the urgent need for structural health monitoring. This work plays a crucial role in assessing structural safety, preventing engineering accidents, and protecting the lives and property of the public. Vibration monitoring is an important branch of health monitoring and forms the basis for subsequent damage analysis and assessment.

[0003] Currently, structural vibration monitoring methods can be broadly classified into two categories: contact vibration measurement and non-contact vibration measurement. Traditional contact vibration measurement methods primarily involve installing sensors on-site and acquiring structural responses by collecting sensor data. Commonly used sensor types in current practice include accelerometers, velocity sensors, tilt sensors, etc. (as shown in Figure 1), capable of acquiring and transmitting data on different responses of building structures.

[0004] With the rapid development of science and technology in recent years, structural displacement monitoring methods based on computer vision have made continuous breakthroughs and have been widely applied. Research on structural displacement monitoring based on computer vision is mainly divided into two categories: those with artificial targets and those without. Artificial targets are generally specific geometric patterns. Before monitoring, the target must be attached to the point on the structure to be measured, and then a camera is used to track and measure the target. Using targets can significantly reduce the difficulty of motion tracking and generally has high accuracy; therefore, scholars at home and abroad have conducted extensive research on structural displacement monitoring based on computer vision with artificial targets. On the other hand, structural vibration monitoring technology without artificial targets has gradually become a research focus and hot topic in recent years. This technology does not require additional target design; it achieves vibration monitoring by identifying and tracking the inherent features of the building structure, including color changes, texture features, etc. Scholars at home and abroad have conducted extensive research on the extraction and tracking of building structural features, achieving relatively rich results.

[0005] Existing structural vibration monitoring methods can be mainly divided into contact methods and non-contact methods. Contact methods have a longer history and a more mature development system, but their processes and operations are relatively complex, and sometimes the work can even affect the normal use of the building structure. In addition, some sensor systems are quite large, resulting in high equipment and labor costs, and the equipment needs to be replaced regularly, causing great inconvenience to the monitoring work.

[0006] To address the aforementioned issues, non-contact monitoring methods based on computer vision have been widely researched and applied. One computer vision-based method for monitoring structural vibration using artificial targets involves pattern detection and matching to track structural vibration responses. However, using artificial targets for structural displacement monitoring typically requires developing matching methods for specific target patterns, thus limiting its applicability. Furthermore, this method still faces many challenges in field operations, such as target damage and detachment. Summary of the Invention

[0007] To address the problems in the prior art, this invention provides a structural vibration displacement monitoring method based on computer vision.

[0008] This invention provides a structural vibration displacement monitoring method based on computer vision, comprising the following steps:

[0009] Step 1: Structural vibration video preprocessing; The basic information of the video data is automatically obtained by an algorithm for automatic acquisition of basic video information, and the image grayscale conversion and local histogram equalization of video frames are used to simplify the image calculation complexity and enhance image features.

[0010] Step 2: Structural displacement monitoring based on improved FAST corner detection; introduce an automatic threshold calculation method based on maximum inter-class variance to form an improved FAST corner detection algorithm; combine the image features obtained by the improved FAST corner detection algorithm with the pyramid Lucas-Kanade optical flow method to establish a fully automated image feature detection and motion tracking scheme.

[0011] As a further improvement of the present invention, in step 1, the basic information includes resolution, frame rate, and sampling duration. In step 1, a video basic information automatic acquisition algorithm is used to automatically acquire the basic information of the video data, and the specific steps are as follows:

[0012] Step 10: Open the computer file management system using the visual interactive commands;

[0013] Step 11: Extract any frame of the obtained image, and use the video frame acquisition integration command in OpenCV (Open Source Computer Vision Library) to obtain the width and height of the corresponding frame, thereby obtaining the image resolution;

[0014] Step 12: As above, use the integrated commands for video frame rate acquisition and video frame count acquisition to obtain the video capture frame rate and the total number of frames in the video stream, and then obtain the total video duration.

[0015] Step 13: Save the obtained information to a document in the same directory as the video file.

[0016] The present invention also discloses a structural vibration displacement monitoring system based on computer vision, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the structural vibration displacement monitoring method of the present invention when called by the processor.

[0017] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the structural vibration displacement monitoring method of the present invention when invoked by a processor.

[0018] The beneficial effects of this invention are: 1. The structural vibration displacement monitoring method of this invention uses a camera and imaging system as a data acquisition system, extracts structural vibration information using digital images as the processing object, and then outputs the actual vibration information of the structure in three-dimensional space according to the imaging principle; 2. The structural vibration displacement monitoring method of this invention has developed functions such as visualization operation and multi-target selection, further optimizing the overall monitoring process and achieving the characteristics of non-contact, simple operation, and high degree of automation; 3. The structural vibration displacement monitoring method of this invention overcomes the drawbacks of traditional technologies and improves production and living efficiency. Attached Figure Description

[0019] Figure 1 is a background diagram of the present invention; (1a, 1b are schematic diagrams of existing contact monitoring tools;)

[0020] Figure 2 This is a schematic diagram of the video composition structure of the present invention;

[0021] Figure 3 This is a simplified schematic diagram of the truss structure model of the present invention;

[0022] Figure 4 This is a video image of structural vibration from the IC-SHM(2021) invention.

[0023] Figure 5 is a comparison of the local equalization of gray values ​​in this invention (5a grayscale image, 5b local equalization image);

[0024] Figure 6 is a comparison of grayscale histograms of the present invention (6a grayscale histogram of the grayscale image, 6b grayscale histogram of the local equalization image);

[0025] Figure 7 This is a node location diagram of the present invention;

[0026] Figure 8 is a comparison of corner feature detection in this invention (8a original image, 8b grayscale image corner points, 8c equalized grayscale image corner points);

[0027] Figure 9 This is a technical roadmap for the structural vibration video preprocessing scheme of the present invention;

[0028] Figure 10 This is a schematic diagram of the FAST corner detection method of the present invention;

[0029] Figure 11 is a comparison of the improved FAST corner detection effect of the present invention (11a image to be detected, 11b default threshold detection effect, 11c automatic threshold detection effect).

[0030] Figure 12 This is a schematic diagram of the pyramid of the present invention;

[0031] Figure 13 This invention involves capturing the first frame of the Damage1 video (after preprocessing).

[0032] Figure 14 This is a schematic diagram of the multi-target selection method of the present invention;

[0033] Figure 15 This is a schematic diagram of the region of interest in this invention;

[0034] Figure 16 This is a schematic diagram of feature point detection in the region of interest of this invention;

[0035] Figure 17 This is a technical roadmap for the structural displacement monitoring method based on the improved FAST corner detection method of this invention;

[0036] Figure 18 This is a schematic diagram of the numbering of the upper chord nodes of the truss structure of the present invention;

[0037] Figure 19 is a time history curve of the displacement of the undamaged node 7 of the present invention. Detailed Implementation

[0038] This invention comprises two parts: structural vibration video preprocessing and a structural displacement monitoring method based on improved FAST corner detection. The former acquires basic information and enhances the inherent features of the structure before monitoring begins, thereby improving the overall computational speed; the latter can complete the extraction of structural features and tracking of feature motion, thus realizing structural vibration and displacement monitoring under computer vision.

[0039] 1. Structural vibration video preprocessing

[0040] During video acquisition, the sampling size, frequency, and duration need to be pre-set, which plays a crucial role in subsequent image feature localization and data processing analysis. However, in practical applications, it is often impossible to pre-set or obtain relevant data. Therefore, obtaining basic information from video data is a necessary step in data preprocessing. Furthermore, video data acquisition is also subject to interference from factors such as shadow occlusion and overexposure during sampling. This can lead to problems such as unclear or even lost image features, affecting subsequent motion tracking performance. This is also an important aspect of data preprocessing.

[0041] To address the aforementioned issues, this invention establishes an automatic video basic information acquisition algorithm. This algorithm can automatically acquire basic information, including resolution, frame rate, and sampling duration, for video data of any encoding format. In addition, it establishes grayscale conversion and local histogram equalization processing for video frame images, simplifying image calculation complexity and enhancing image features, thus facilitating subsequent calculations.

[0042] As shown in Figure 1, this invention discloses a structural vibration displacement monitoring method based on computer vision, comprising the following steps:

[0043] Step 1: Structural vibration video preprocessing; The basic information of the video data is automatically obtained by an algorithm for automatic acquisition of basic video information, and the image grayscale conversion and local histogram equalization of video frames are used to simplify the image calculation complexity and enhance image features.

[0044] Step 2: Structural displacement monitoring based on improved FAST corner detection; introduce an automatic threshold calculation method based on maximum inter-class variance to form an improved FAST corner detection algorithm; combine the image features obtained by the improved FAST corner detection algorithm with the pyramid Lucas-Kanade optical flow method to establish a fully automated image feature detection and motion tracking scheme.

[0045] 1.1 Basic Concepts of Video

[0046] Video refers to a series of technologies that capture still images using electrical signals and process, store, and transmit them in chronological order. Therefore, video is composed of a series of still images. When continuous images change at a rate exceeding 24 frames per second (fps), the human eye cannot distinguish individual still images, resulting in a smooth and continuous visual effect. Currently, common video formats on the market include mp4, avi, and mov.

[0047] 1.2 Automatic Acquisition Scheme for Basic Video Data Information

[0048] The basic information of video data mainly includes resolution, frame rate, and video duration. Resolution refers to the precision of an image, representing the number of pixels in an image, usually expressed as "horizontal pixels × vertical pixels." In a two-dimensional image, the image coordinate system has its origin at the top left corner, with the positive x-axis pointing to the right and the positive y-axis pointing downwards. Therefore, obtaining the image resolution and combining it with the image coordinate system allows for quick location of any object in the image, providing a reference for subsequent feature tracking and calculation.

[0049] Image frame rate refers to the number of images transmitted per second during acquisition, i.e., the video sampling frequency. According to the Nyquist-Shannon sampling theorem, if the sampling frequency is more than twice the signal bandwidth, then the original continuous signal can be completely reconstructed from the sampled data. Therefore, the video frame rate determines the scope of data analysis and processing. Furthermore, the total duration of the video determines the temporal scope of data processing. Typically, in video data processing, it's not necessary to focus on the entire duration; only a specific time period needs to be considered. Therefore, acquiring the duration of the video data allows for segmentation and localization of the video data in the temporal domain.

[0050] This invention establishes an automatic method for acquiring basic information from structural vibration video data based on the OpenCV vision open-source library and related tools. The specific steps are as follows:

[0051] Step 10: Open the computer file management system using the visual interactive commands;

[0052] Step 11: Extract any frame of the image (here set to the first frame of the video), and use the video frame acquisition integration command in OpenCV (OpenSource Computer Vision Library) to obtain the width and height of the corresponding frame, and then obtain the image resolution;

[0053] Step 12: Obtain the video capture frame rate and the total number of frames in the video stream using CAP_PROP_FPS and CAP_PROP_FRAME_COUNT, and then obtain the total video duration;

[0054] Step 13: Save the obtained information to a document in the same directory as the video file.

[0055] Numerical Example Verification

[0056] The verification example used in this invention comes from the third project provided by the 2nd International Competition on Structural Health Monitoring (IC-SHM (2021)). This project provides vibration videos of a truss model synthesized from real photos based on finite element analysis results. Yasutaka Narazaki et al. synthesized vibration videos of a truss structure using a dense three-dimensional displacement estimation algorithm based on a finite element model, aiming to develop a civil structure health monitoring model applicable to multiple scenarios. All joints of the vibrating truss structure are subjected to three-dimensional band-limited white noise excitation. A schematic diagram of the synthesized truss structure model is shown below. Figure 3 As shown.

[0057] In the third project of the IC-SHM (2021) competition, the organizers, based on the aforementioned real photos, synthesized a truss model and provided video data of component damage under various working conditions, as well as a structural vibration video under an undamaged working condition, by reducing the stiffness of one of the components. Figure 4 As shown. The structural vibration displacement reference data for the first 10 seconds of each video was made public after the competition to facilitate researchers in developing independent civil structure health monitoring methods based on this data and verifying their feasibility.

[0058] The Damage1 video data was selected as an example to verify the video basic information acquisition method established in this section. The basic information of the video data was calculated from the video input, revealing a resolution of 1920×1080 pixels, a frame rate (sampling frequency) of 120fps, and a total video duration of 240s, as shown in Table 1. Comparison with directly viewing the video attributes on a computer system shows that the calculated results are consistent with the video information obtained through computer queries, indicating that the automatic video basic information calculation method proposed in this section has good performance.

[0059] Table 1. Comparison Table of Basic Video Information

[0060]

[0061] 1.2 Image grayscale set local histogram equalization

[0062] 1.2.1 Significance of Image Grayscale Conversion

[0063] With the continuous development of image acquisition devices such as cameras, the digital images and video data sampled by these devices are generally in color. The color of each pixel in a color image is determined by three components: R, G, and B. Each component has 255 possible values, resulting in a total of 16 million color variations for a single pixel.

[0064] In structural health monitoring, vibration monitoring is typically achieved by tracking the motion of features in images. If the target features need to be calculated based on color images, the sheer volume of computation can severely impact efficiency and speed. Therefore, converting various color image formats to grayscale images is a crucial method for reducing the computational burden of digital image processing. Grayscale images can be understood as a special type of color image where the R, G, and B components are identical; therefore, they have only one channel component, significantly reducing computational complexity and increasing speed. Furthermore, grayscale images, like color images, still reflect the overall and local distribution and characteristics of color and brightness levels within the entire image, ensuring accurate representation of image features.

[0065] 1.2.2 Basic Methods of Image Grayscale Conversion

[0066] In the RGB color space, there are many methods to achieve image grayscale, which can be basically divided into four categories: maximum value grayscale, average value grayscale, Gamma-corrected grayscale, and weighted average grayscale.

[0067] Maximum value grayscale uses the maximum value among the R, G, and B channels as the original pixel value. Unlike maximum value grayscale, average value grayscale uses the pixel values ​​from all three channels, averaging them to obtain the original pixel value. This method loses some image detail during grayscale conversion. Gamma-corrected grayscale is not a simple addition of the three channel pixel values; instead, it determines the specific calculation method for each component within each channel based on the different sensitivities of the human eye to different brightness levels. The human eye is more sensitive to dark areas than bright areas. Based on this difference in perception, the RGB values ​​are converted into physical light power, and the grayscale pixel value is calculated accordingly.

[0068] Weighted average grayscale conversion is a grayscale conversion method that conforms to the sensory characteristics of the human eye. The human eye has the lowest sensitivity to blue and the highest sensitivity to green. In the YUV color space, Y represents brightness. Based on the relationship between the RGB space and the YUV space, the correspondence between Y and the three color components R, G, and B can be established. The grayscale value of the image is established based on this brightness value, and the calculation principle is shown in Equation (1).

[0069] gray=0.299R+0.587G+0.144B (1)

[0070] In structural health monitoring, structural motion tracking based on computer vision principles requires the detection and recognition of features in the image. To preserve image features, brightness, and color distribution of the original color image to the greatest extent possible, this invention employs weighted average grayscale conversion and establishes a grayscale conversion algorithm.

[0071] 1.2.3 Principle of Local Histogram Equalization

[0072] Image features are the tracking targets for structural vibration monitoring proposed in this invention. During the sampling process, due to uncontrollable factors such as lighting and shadows, image features are often obscured, resulting in images with a concentrated grayscale distribution and poor contrast. For building structures, due to their complex shapes and other factors, these problems are frequently encountered. It is necessary to eliminate the influence of factors such as shadow obscuration before vibration monitoring begins.

[0073] Gray-level histogram equalization is a method in image processing that uses image histograms to enhance image contrast. This method allows brightness to be more evenly distributed across the image, enhancing contrast while effectively suppressing noise. The main idea is to transform the histogram distribution of an image into an approximately uniform distribution through its cumulative distribution function, thereby enhancing image contrast. The mapping principle of the cumulative distribution function in the histogram equalization process is shown in equation (2).

[0074]

[0075] Among them, s k This represents the value of the current gray level after being mapped by the cumulative distribution function, where n is the sum of the pixels in the image. j is the number of pixels at the current gray level, and L is the total number of gray levels in the image. Mapping pixel values ​​using the cumulative distribution function has two main characteristics:

[0076] (1) The order of pixel values ​​in the original image was not disrupted, and the relationship between the two sides in the image remained unchanged after the mapping was completed.

[0077] (2) The range of pixel values ​​after mapping is still between 0 and 255.

[0078] Because the sampling process is highly complex, the sampled image includes not only the target to be monitored but also other objects. Therefore, global image grayscale histogram equalization often fails to meet requirements and may even result in the loss of more detail. To address this issue, local histogram equalization is introduced.

[0079] The basic idea of ​​local histogram equalization is to divide the entire image into many sub-blocks and perform equalization processing on each sub-block. This method can maximize the enhancement of image details without increasing noise due to additional introduction.

[0080] 1.2.4 Grayscale Conversion and Local Histogram Equalization Scheme

[0081] This invention, based on the principles of image grayscale conversion and local histogram equalization, establishes an image grayscale conversion and local histogram equalization scheme during the video data preprocessing stage. The aim is to reduce computational load while enhancing image features and eliminating the effects of factors such as lighting and shadow occlusion. The specific steps of the image grayscale conversion and local histogram equalization scheme are as follows:

[0082] Step S1: Read video frame images; Since the method established in this invention processes video data, image grayscale conversion and local histogram equalization are not performed on a single static image, but on all video frame images in the video stream. Considering the convenience of the processing, grayscale conversion and local histogram equalization of the frame images in the video data are performed frame by frame, rather than performing all processing in advance and then proceeding with subsequent work.

[0083] Step S2: Image grayscale conversion; the image grayscale conversion method used is weighted grayscale conversion. Using the image coordinate system as a reference, first read the pixel at coordinate (0,0) and record the values ​​of each channel. Based on the weighted grayscale calculation principle, calculate the grayscale value and assign it to the current pixel. Read all pixels sequentially and convert them to grayscale. Finally, reassemble the grayscale pixels to obtain the grayscale image.

[0084] Step S3: Grid-divide the current image; to avoid loss of detail due to global grayscale histogram equalization when the image is too large, it is necessary to divide the image into a grid and perform histogram equalization on each grid sub-block separately. The grid division scale used in this invention is 20×20 (unit: pixels).

[0085] Step S4: Local grayscale histogram equalization; again, using the image coordinate system as the reference system, first perform histogram statistics on the grid sub-block in the upper left corner of the image, and calculate a new grayscale image based on the histogram equalization principle. Perform the above operation on all grid sub-blocks in sequence, and finally combine the obtained sub-blocks to form a new grayscale image.

[0086] 1.2.5 Numerical Example Verification

[0087] Using video data from IC-SHM (2021) as an example, this paper verifies the image grayscale and local histogram equalization scheme established in this invention. Taking Damage1 video data as an example, the first frame of the video data is extracted. Based on the above weighted grayscale principle, the grayscale value is calculated and a grayscale image is obtained.

[0088] After grayscale conversion, some image features may be lost due to factors such as lighting and sampling equipment. The grayscale image is divided into 20×20 pixel blocks, and then grayscale histogram local equalization is performed on each sub-block. As shown in Figure 5, the equalized grayscale image exhibits more pronounced contrast and other image details.

[0089] In a statistical sense, image grayscale histogram equalization, while preserving the original features of an image, distributes the grayscale values ​​more evenly, thereby enhancing image contrast. As shown in Figure 6, the equalized grayscale image exhibits a more uniform grayscale distribution, indicating more detailed image representation. This process is equivalent to performing a linear stretching and interpolation on the original grayscale histogram.

[0090] To further verify the beneficial effects of local equalization on image feature enhancement, the node at the upper left corner of the truss structure image was selected as the research target, and its position is as follows: Figure 7 As shown, the Shi-Tomas corner detection method was used to detect corners at this node. The resulting corners are a type of image feature, indicating that the grayscale variation at their location is relatively large.

[0091] The OpenCV vision open-source library provides a convenient and fast Shi-Tomas corner detector using the wrapper function `cv2.goodFeatureToTrack()`. It includes three parameters that affect detection performance: maximum number of corners returned; quality coefficient; and minimum Euclidean distance between corners. In this detection, the maximum number of corners returned was set to 100, the quality coefficient to 0.7, and the minimum Euclidean distance between corners to 10 pixels. The detection result is shown in Figure 8.

[0092] The above detection results show that the number of feature points obtained by the equalized grayscale image is higher than that of the ordinary grayscale image. This indicates that local equalization of grayscale images can enhance the features and local details of the image to a certain extent, providing richer computational information for subsequent image feature detection and tracking.

[0093] 1.3 Chapter Summary

[0094] To obtain structural response data more quickly and accurately, this invention establishes a video data preprocessing method, including automatic acquisition of basic video data information, and image grayscale conversion and local histogram equalization methods. The automatic acquisition method of basic video information plays a crucial role in the localization of image features and the scope of data processing and analysis; the image grayscale conversion and local histogram equalization methods reduce the computational complexity of image processing while enhancing local image details and features, thereby making subsequent image feature detection and tracking faster. The technical route of the structural vibration video preprocessing scheme is as follows: Figure 10 As shown.

[0095] 2. Structural Displacement Monitoring Method Based on Improved FAST Corner Detection

[0096] The goal of structural vibration monitoring is to measure structural vibration data and assess the structural vibration characteristics and their impact on the structure itself. In structural vibration video data, if vibration occurs, the coordinates of the corresponding components and parts will change. By obtaining the numerical values ​​of the image coordinates over time, the vibration displacement data of the structure can be obtained. In the field of computer vision, target tracking can utilize the contextual information of video or image sequences to model the appearance and motion information of a target, thereby predicting the target's motion or position. This method is of great significance for realizing structural vibration monitoring.

[0097] To achieve target tracking, image information must first be abstracted into data that computers can recognize. Image features refer to distinctive locations in computer vision, including color changes, shape changes, areas of grayscale variation, object corners, and line intersections, and are one of the main objects of target tracking. Using image features as the target for motion tracking can significantly reduce the computational load and improve computational speed. Furthermore, image features are unaffected by changes in position or environment. Therefore, achieving high-quality image feature detection is an important research area.

[0098] This invention proposes an improved FAST corner detection method. Based on the principle of maximum inter-class variance, it achieves automatic threshold calculation and applies it to the FAST corner detection method, overcoming the drawback of requiring manual threshold setting and adjustment. Furthermore, the image features obtained by this method are combined with the pyramid Lucas-Kanade optical flow method (hereinafter referred to as LK optical flow method) to establish a fully automated image feature detection and motion tracking scheme. Finally, the method is applied to a structural vibration video data example to verify its feasibility and accuracy.

[0099] 2.1 Improved FAST Corner Detection Method

[0100] 2.1.1 FAST Corner Detection Principle

[0101] FAST (Features from Accelerated Segment Test) is a corner detection method that can quickly calculate and extract image features. The main idea of ​​this algorithm can be summarized as follows: if a pixel has a certain number of surrounding pixels with different values, then that pixel is considered distinct from its surrounding pixels and can be regarded as a feature point. Due to its fast computation speed and high accuracy, the FAST corner detection algorithm is widely used in image feature detection.

[0102] The FAST corner detection method has several calculation methods, the most common being the calculation of 16 pixels within a circular window near the pixel. For example... Figure 10 As shown, consider whether a pixel P in the image is a corner point to be tested, and denote its pixel value as I. p Draw a discretized circle with a radius of 3 pixels centered on the pixel. There are 16 pixels on the boundary of this circle. Number these 16 pixels sequentially.

[0103] Let I denote the pixel value of the center point p. p Given the pixel values ​​of the 16 pixels on the ring, if the pixel values ​​of n consecutive points are all greater than or less than the pixel value of the center point P, then the center point P is a corner point. It can be seen that this method determines the corner point feature by measuring the difference between two pixel values. Therefore, in the actual comparison process, a threshold t is usually set as the standard for measuring the strength difference between pixel values. At this time, when the difference between pixel values ​​exceeds the threshold t (whether it is lower or higher), it will be defined as a corner point. Its calculation principle is shown in equation (3).

[0104]

[0105] As the calculation principle above shows, evaluating a single pixel requires detecting 16 pixels, which reduces the detection speed. To achieve faster detection, the FAST corner detection method employs an additional acceleration method: First, it tests each 90-degree pixel around the candidate point P, i.e., pixels at positions 1, 5, 9, and 13. Generally, pixels 1 and 9 are tested first, and if they meet the threshold requirements, pixels 5 and 13 are then tested. If P is a corner point, at least three of the four pixels must be greater than I. p +t or less than I p -t, if the above conditions are not met, it is definitely not a corner point.

[0106] Another problem encountered by the FAST corner detection method is that the detection process often leads to many feature points connecting together, and even feature overlap. In such cases, it is difficult to distinguish individual feature locations, and the method is also difficult to apply to other scenarios. Therefore, the FAST corner detection method introduces non-maximum suppression. Generally, a window of 3×3, 5×5, or other sizes is created around the corner point P. If other corner points Q appear within the window, the strengths of the two corner points are compared, and the corner point with the greater strength is retained. The strength of a corner point can then be represented as the maximum value that the corresponding threshold t can take when the point can be detected as a corner point.

[0107] The FAST corner detection method, strictly speaking, only involves feature detection and does not include image feature description. Therefore, it has advantages in terms of speed and accuracy. As a classic method in image feature detection, it is frequently used in real-time video processing. However, its computational principle reveals that this method is heavily influenced by the set threshold 't'. For example, if the threshold is set too small, it cannot filter out noise and weak features, leading to errors in subsequent tracking. If the threshold is too large, sufficient feature information cannot be obtained to support subsequent calculations. Correspondingly, the effectiveness of non-maximum suppression is also affected by the threshold.

[0108] In practical applications, the FAST corner detection method requires continuous threshold adjustment to eliminate weak and overlapping corners while retaining those with strong responses, thus achieving better detection results. However, this continuous trial and adjustment process can affect the efficiency and automation of the detection process.

[0109] 2.1.2 Automatic Thresholding Based on Maximum Inter-Class Variance

[0110] To address the drawback of FAST corner detection performance being affected by threshold setting, this invention introduces an automatic threshold calculation method based on the principle of maximum inter-class variance. The main idea of ​​this method is to utilize global image information to statistically analyze and classify all grayscale values, obtaining three indicators: high grayscale value, low grayscale value, and critical value. The adaptive threshold for FAST corner detection is then determined based on the differences between these three indicators.

[0111] The Otsu's method was initially used in image binarization segmentation. Statistically, variance characterizes the degree of data dispersion; a larger variance indicates greater data dispersion. For digital images, a larger variance between the gray values ​​of one class of pixels and the gray values ​​of another class of pixels allows for better differentiation between the two classes, i.e., the foreground and background in the image.

[0112] For a grayscale image, assume there exists a grayscale threshold k, dividing all pixels into two classes: class C1 (grayscale value greater than k) and class C2 (grayscale value less than k). The proportions of classes C1 and C2 in the total number of pixels are p1 and p2, respectively, and the mean grayscale values ​​of the two classes are m1 and m2, respectively. Let the global mean of the image be m. g The quantitative relationship between the two types of pixels is shown in equations (4) and (5).

[0113] p1×m1+p2×m2=m g (4)

[0114] p1 + p2 = 1 (5)

[0115] The inter-class variance is defined as the weighted variance of the mean of each of the two classes of data from the overall mean. Based on the concept of inter-class variance, the inter-class variance of the two classes of pixels can be calculated as shown in Equation (6).

[0116] σ 2 =p1(m1-m g ) 2 +p2(m2-m g ) 2 (6)

[0117] Substituting equations (4) and (5) into equation (6), we get:

[0118] σ 2 =p1p2(m1-m2) 2 (7)

[0119] The grayscale value ranges from 0 to 255. The threshold k that maximizes the inter-class variance of the current image is considered a good image segmentation threshold, capable of distinguishing the foreground and background of the current image to the greatest extent. Image segmentation using the maximum inter-class variance principle can effectively distinguish the foreground and background of an image and has been widely used.

[0120] Based on the principle of segmenting image pixels using the maximum inter-class variance, this invention proposes a novel automatic threshold calculation method for FAST corner detection. When the maximum inter-class variance is found, the entire image can be divided into three indices: the mean gray level m1 of class C1 pixels, the mean gray level m2 of class C2 pixels, and the segmentation threshold k, representing the foreground, background, and segmentation boundary of an image, respectively. Through the FAST corner detection principle, it can be found that a good image feature detection threshold should have the following two characteristics:

[0121] (1) The pixel value of the corner point should be different from the surrounding pixel value. Being stronger or weaker than the surrounding pixel value satisfies the condition.

[0122] (2) The threshold should be able to filter out weak features and retain strong features to the greatest extent.

[0123] Furthermore, the maximum inter-class variance principle utilizes all pixel information, making it more universal. Therefore, based on the aforementioned principles, characteristics, and overall image properties, this invention introduces an automatic threshold for the FAST corner detection process: when the inter-class variance of foreground and background pixel values ​​reaches its maximum, the mean gray level m1 of the foreground pixels and the mean gray level m2 of the background pixels are subtracted from the threshold k, and the difference with the larger absolute value is used as the automatic threshold k in FAST corner detection. auto The calculation formula is as follows:

[0124] k auto =max(|m1-k|,|m2-k|) (8)

[0125] 2.1.3 Improved FAST corner detection algorithm

[0126] Based on the principle of maximum inter-class variance and the FAST corner detection principle, this invention establishes an automatic threshold calculation method based on maximum inter-class variance and forms an improved FAST corner detection algorithm. The improved FAST corner detection algorithm mainly includes three steps:

[0127] (1) Statistical steps: To quickly access the distribution of each pixel value, a hash table is established, with the pixel value as the hash table index and the corresponding number of pixel blocks as the corresponding numerical value. At this point, you only need to count the pixel distribution and enter the hash table above to quickly look up the values ​​when calculating the local mean, the overall mean, and the inter-class variance, which greatly improves the calculation speed.

[0128] (2) Iterative steps: The above principle indicates that it is necessary to find the critical value k that maximizes the differentiation between the foreground and background within the range of 0 to 255. Therefore, the maximum and minimum values ​​of the statistical pixel values ​​are taken as the upper and lower limits of the value range, and the pixel segmentation threshold k is obtained by iteratively obtaining the value range within this range.

[0129] (3) Output step: The pixel segmentation threshold k calculated by the loop step is subtracted from the mean values ​​of the two types of pixels obtained based on the value to obtain the adaptive threshold for feature detection, and the result is output to the FAST corner detection algorithm.

[0130] This algorithm uses global information of the image for segmentation and obtains an automatic threshold, overcoming the drawback of the traditional FAST corner detection method, which requires continuous threshold adjustment to achieve good detection results. Furthermore, the improved FAST corner detection algorithm can retain strong features and remove weak features to a certain extent.

[0131] 2.1.4 Numerical Example Verification

[0132] Taking Figure 11 as an example, the images to be detected are tested using both the default threshold of FAST corner detection and the aforementioned automatic threshold. In the OpenCV vision open-source library, the default threshold for FAST corner detection is set to 10. The detection results show that using the default threshold often leads to "piggybacking," meaning overlapping features. This often requires repeatedly setting the threshold and performing repeated detections to obtain more ideal image features. Using the automatic threshold calculation method, weak features are automatically removed to a certain extent, while strong features are retained, thus improving the automation level of the detection program.

[0133] 2.2 Displacement Monitoring Method Based on Pyramid Optical Flow

[0134] 2.2.1 Basic Concepts of Optical Flow

[0135] Optical flow is the instantaneous velocity of pixels of a moving object in space on the observation imaging plane. Generally, the movement of the object itself, the camera, or the relative motion between the two in a scene will generate optical flow. In video streams or image sequences, optical flow can be obtained by utilizing the changes of pixels in the temporal domain and the correlation between adjacent frames, and the motion information of objects between adjacent frames can be calculated. As shown in the figure, the motion of an object in three-dimensional space is projected onto a two-dimensional imaging plane, resulting in a two-dimensional vector describing the change in position. When the motion interval is infinitesimally small, a two-dimensional vector describing the instantaneous velocity of that point can be obtained, which is the optical flow vector.

[0136] 2.2.2 Principle of Lucas-Kanade Optical Flow Method

[0137] Optical flow is a method that uses the temporal changes of pixels in an image sequence and the correlation between adjacent frames to find the correspondence between the previous and current frames, thereby calculating the motion information of objects between adjacent frames. The basic assumptions of optical flow are as follows:

[0138] (1) Brightness remains constant. This assumption means that when observing the same target, its brightness will not change as it moves between different frames. This assumption is a fundamental condition of optical flow methods, and any derivative method of optical flow methods must satisfy it to obtain the basic equations for the calculation process.

[0139] (2) The time is continuous or the motion is small. This assumption means that changes in time will not cause drastic changes in the target position, that is, the displacement between adjacent frames is relatively small.

[0140] Suppose there exists an observed pixel block in a digital image, whose light intensity in the first frame (or image sequence) is I(x,y,t). Relative to the current frame, in the next frame, this pixel block moves a distance of (dx,dy) over a time interval of dt. According to the first assumption above, the light intensity of the same pixel block remains unchanged before and after the movement, i.e.:

[0141] I(x,y,t)=I(x+dx,y+dy,t+dt) (9)

[0142] Expanding the right side of the equation using Taylor, and ignoring higher-order infinitesimals while retaining only the first-order terms in the Taylor expansion, we obtain:

[0143]

[0144] Let u and v represent the velocity vectors of the optical flow at the current position along the X and Y axes of the pixel coordinate system, respectively. At the same time, These represent the changes in pixel values ​​of pixels in the image along the X, Y, and T directions, respectively. Therefore, equation (10) can be written in the form of equation (11).

[0145] I x u+I y v+I t =0 (11)

[0146] In the formula, I x I y I t The optical flow vector (u, v) can be obtained from image data. There is only one constraint equation, but two unknowns, therefore additional constraints need to be introduced. Different constraints lead to different methods for calculating the optical flow field.

[0147] Gradient-based methods, also known as differential methods, utilize the spatiotemporal differential of time-varying image gray levels to calculate pixel velocity vectors. Compared to other optical flow calculation methods, including matching methods, energy methods, and phase methods, this method has been widely used and researched due to its computational simplicity and high accuracy. In extensive research, it is further divided into sparse optical flow and dense optical flow calculation methods. Among them, the Lucas-Kanade (LK) optical flow method is a typical representative of sparse optical flow.

[0148] The LK optical flow method adds a "spatial consistency" assumption to the two basic assumptions of the original optical flow method. This assumption stipulates that all adjacent pixels have similar behavior, that is, within an m×m region around the target pixel, each pixel has the same optical flow vector. In this case, there are N = m... 2 Each pixel block has the same motion, which allows us to obtain:

[0149]

[0150] Assumption:

[0151]

[0152] Then equation (12) can be expressed as:

[0153] AU=b (13)

[0154] Equation (12) is an overdetermined linear equation with no exact solution. The analytical solution can be obtained using the least squares method. Therefore, the motion of the pixel can be calculated as follows:

[0155] U = (A T A) -1 A T b (14)

[0156] The optical flow of a pixel can be calculated using the "spatial consistency" assumption in the LK optical flow method. The calculation process described above treats the weights of all pixel blocks within the window as uniform, i.e., evenly distributed. In practice, the LK optical flow method uses a window weight function W. 2 (x) makes the neighborhood closer to the center weighted more, to highlight the coordinates of the center point. Let W = diag(W1, W2, ..., W...). N ),Right now:

[0157]

[0158] Among them, W i (i = 1, 2, ..., N) represents the square root of the weight of each pixel block. Therefore, the movement of the pixel after adding window weights is:

[0159] U = (A T W 2 A) -1 A T W 2 b (16)

[0160] The three assumptions mentioned above are: constant brightness, small motion, and spatial consistency. Pixel optical flow can be calculated based on these assumptions. However, these assumptions are difficult to satisfy in practical applications. Especially the small motion assumption: if pixels move quickly, the motion between adjacent frames is no longer small, and the Taylor expansion in the solution process no longer meets the conditions, making subsequent solution derivation impossible.

[0161] 2.2.3 Pyramid LK Optical Flow Method

[0162] For problems involving large motion, the pyramidal Leyk-Königsberg optical flow method has been proposed. The main idea of ​​this method is to reduce the resolution of the entire image. For fast-moving pixels, when the resolution is reduced to a certain level, the motion becomes small enough to satisfy the Taylor expansion conditions, thus allowing the pixel optical flow to be calculated.

[0163] like Figure 12 As shown, the pyramid construction process is as follows: First, the original image is used as the base of the pyramid, i.e., the image with the highest resolution. Then, Gaussian convolution and downsampling are performed on the original image, and all even-numbered rows and columns are deleted to obtain the first layer image. The same operation is then performed on the first layer image to obtain the second layer image, and so on. In actual motion tracking, optical flow is first calculated on the top layer, and then the initial estimate of the pixel displacement of the lower layer is obtained based on the calculation results of the previous layer. Based on this estimate, optical flow continues to be calculated and passed to the next lower layer, and so on until the 0th layer. The number of pyramid layers needs to be determined based on the actual optical flow motion in the original image.

[0164] The significant advantage of the pyramid structure is that a large pixel displacement can be decomposed into multiple smaller pixel displacements for calculation. Each additional pyramid layer can handle twice the pixel displacement of the layer below it, and the overall motion processing capability of the pyramid structure is the sum of the values ​​of all its layers. Figure 12 It is a pyramid structure with a height of 3 and 4 layers, and it can handle a maximum pixel motion that is 15 times greater than that of the standard LK algorithm.

[0165] The pyramid LK optical flow method effectively increases the amplitude of optical flow motion that can be processed and analyzed without increasing the ensemble window, giving the LK optical flow method the ability to handle large pixel motions. Furthermore, introducing the pyramid LK optical flow method enhances the tracking accuracy while maintaining the algorithm's resistance to noise.

[0166] 2.3 Structural Displacement Monitoring Method and Flowchart

[0167] Based on the aforementioned principles of automatic threshold calculation, image feature detection, and optical flow tracing, this invention establishes a structural displacement monitoring method based on an improved FAST corner detection method, and implements an automatic algorithm. To improve the ease and convenience of operation, some functions have been optimized. The specific steps of this method are as follows:

[0168] Step 21: Read the structural vibration video data and extract the video frame at the start moment;

[0169] Step 22: Select the region of interest in the image;

[0170] Step 23: Region of Interest Feature Detection;

[0171] Step 24: Pyramid LK optical flow motion tracking;

[0172] Step 25: Output the displacement time history results for the region of interest. To further illustrate the algorithm's flow, the detailed methodological process is elaborated using Damage1 data from the third project of IC-SHM (2021). It should be noted that the video data here has already undergone the aforementioned video preprocessing.

[0173] First, the structural vibration video data is read and the video frame at the start time is extracted. This requires determining the start time and duration of the target vibration. Generally, a representative start time and duration are selected, such as the time the vibration began or the time of the start of a large-amplitude vibration. In the OpenCV vision open-source library, the `get` and `set` functions encapsulated in the `VideoCapture` module can read video data at any time. The former locates the video by arranging the frame sequence, while the latter is achieved by directly setting the time. Since the frame rate and duration of the video have already been obtained in the video data preprocessing stage, and to facilitate subsequent processing, the `get` function is chosen in the structural vibration monitoring method of this invention.

[0174] In the Damage1 dataset, the IC-SHM (2021) project provides the structural displacement data for the first 10 seconds (1200 frames) of the video data. Therefore, the first 10 seconds of video data are also selected as the research object here. The first frame image of the preprocessed Damage1 video data is shown below. Figure 14 As shown.

[0175] Secondly, after acquiring the starting frame of the video, a region of interest (ROI) needs to be selected for subsequent motion tracking and computation. Due to the complexity of the sampling environment, the information in digital images is often not entirely what is needed; therefore, selecting the ROI is essential.

[0176] Current research on structural vibration monitoring employs two main methods: one is by tracking a single pixel, and the other is by capturing and tracking a single region. Existing methods often only detect a single target point or object, and when multi-target detection is involved, repeated readings and calculations are required, making the process complex.

[0177] This invention develops a multi-target selection algorithm that enables visual selection of multiple tracking targets. After selection, the program automatically reads each tracking target and completes the preset structural vibration monitoring task without manual intervention.

[0178] The specific steps of the algorithm are as follows:

[0179] Step A1: Define a target class; this class mainly stores the coordinate information and image information of a single target, and multiple tracking targets form multiple classes with their respective information.

[0180] Step A2: Store the corresponding information of multiple targets sequentially;

[0181] Step A3: Implement the information output interface. During target motion tracking and calculation, it is only necessary to continuously read each storage unit and output the image information and coordinate information therein.

[0182] Due to the uncertainty of feature point distribution, for structural vibration videos, the selected region of interest should generally be a representative part of the structure's vibration, such as the joints of building components. Taking Damage1 video data as an example, the main region of interest for this truss structure is generally each joint of its upper or lower chord.

[0183] like Figure 14 As shown, to enhance ease of operation, the algorithm incorporates visual mouse interaction commands when selecting areas of interest, enabling mouse bounding selection and multi-target selection. The specific steps for the visual mouse bounding selection function are as follows:

[0184] Step Y1: Mouse selection command; This part allows you to drag a rectangular area in the digital image to be processed by holding down the left mouse button; when the left mouse button is released, the selection is complete.

[0185] Step Y2: Rectangular selection confirmation command; The purpose of the mouse selection function is to achieve multi-target selection. Therefore, after the selection is completed, the image information and coordinate information of the selected area should be saved. In this function, the white selection area represents the area being selected (e.g., ...). Figure 14 The area shown in the rightmost box is selected. Double-clicking the Enter key selects the area and saves the relevant image and coordinate information. The border of the selected area will turn black (as shown in the image). Figure 14 (As shown in the two selection boxes on the left). To exit, double-click the Esc key to finish selecting the target.

[0186] After multi-target selection is complete, the algorithm saves the image and coordinate information of the targets. During motion tracking, the stored targets are retrieved. The image information of the three nodes selected above is as follows: Figure 15 .

[0187] The next step in the structural displacement monitoring method based on the improved FAST corner detection method is to perform feature detection on the selected target area. The principle of the improved FAST corner detection method has been described in detail above. Taking the three truss structure nodes extracted from the Damage1 video data above as examples, the detection results for the three target areas are as follows: Figure 16 As shown.

[0188] Then, based on the image feature points obtained above, motion tracking is performed using the pyramid LK optical flow method. It's important to note that the feature points used for tracking should be those on the target structure; only then will the results contain data on structural vibration. Furthermore, the feature points obtained from FAST corner detection are of the KeyPoint type (a feature type based on feature information and descriptor information), which cannot be directly used for optical flow tracking. Therefore, KeyPoint type feature points need to be converted into a trackable feature type (such as Pint2f). For the specific circumstances of this example, the pyramid layer count is set to 2, and the height to 3. These settings should be adjusted according to the actual situation for different tracking targets.

[0189] After optical flow motion tracking, the coordinates of feature points in each video frame can be obtained. Subtracting the corresponding coordinates yields the displacement time history data of each feature point in the region. The displacement time history data obtained at this time is in pixels, which needs to be converted to meters (m).

[0190] In practical applications, if the physical dimensions of some actual components in the building structure are known, the conversion ratio from 3D space to 2D image can be obtained based on the actual dimensions of the components and their pixel dimensions in the sampled image (conversion ratio = actual size / pixel size). If the actual physical dimensions are unknown, pixel displacement time history data can be directly used for subsequent processing, as the only difference between it and the actual displacement time history data is the scaling factor. In the subsequent modal recognition process, the amplitude of the structural response data has no impact on the final recognition result.

[0191] The IC-SHM (2021) project data provides the actual dimensions of each component in the truss structure, allowing us to obtain the conversion ratio from pixels to actual meters (m). Through the aforementioned structural displacement monitoring process, pixel displacement time history data is obtained. Multiplying this pixel displacement time history data by its corresponding conversion ratio yields the final displacement time history data. The obtained displacement time history data for each target motion is stored as an array in the calculation program and output to a document.

[0192] This section provides a detailed breakdown and explanation of the algorithm flow for a structural displacement monitoring method based on improved FAST corner detection, utilizing structural vibration video data provided by IC-SHM (2021). The technical approach of this method is as follows: Figure 17 As shown.

[0193] 2.4 Numerical Example Verification

[0194] The organizers of the IC-SHM (2021) competition released reference values ​​for displacement time histories of 16 nodes (upper and lower chords) of the truss structure in the first 10 seconds and 1200 frames of all video data. Using the structural vibration monitoring method proposed in this invention, feature detection and motion tracking were performed on the IC-SHM (2021) video data. The monitoring results for the first 10 seconds and 1200 frames were compared with real data to verify the feasibility and accuracy of the proposed method.

[0195] In this verification example, displacement monitoring was performed only on the eight nodes of the upper chord of the truss structure, and the results were compared with the corresponding structural displacement time history reference values. The eight nodes of the upper chord of the truss structure are numbered as follows: Figure 18 As shown.

[0196] To further verify the accuracy of the data, the structural vibration data obtained by the proposed method of this invention are subjected to correlation analysis with the actual value of structural vibration. The correlation coefficient η is used to characterize the correlation. The value range of η is [-1, 1]. The closer the absolute value is to 1, the stronger the correlation. The calculation principle is shown in Equation (17).

[0197]

[0198] During monitoring, the method proposed in this invention calculates data in an image coordinate system, thus the measured vibration data is positive downwards and negative upwards. The actual vibration data provided by the competition project is based on the first frame, with positive upwards and negative downwards. The positive and negative directions are exactly opposite to those of the monitored data. Therefore, the correlation coefficient obtained in correlation analysis is usually negative.

[0199] Using the structural displacement monitoring method based on improved FAST corner detection proposed in this invention, feature detection and motion tracking were performed on the eight nodes of the upper chord of the undamaged video data to obtain the vibration data corresponding to the first 10 seconds and 1200 frames, which were then plotted as displacement time history curves. Similarly, the displacement time history data reference values ​​of the eight nodes of the upper chord of the truss structure provided by the project data were also plotted as displacement time history curves. By comparing the displacement time history curves of the corresponding nodes, the feasibility of the structural displacement monitoring method proposed in this invention and the certainty of the monitoring results can be verified. For ease of representation, only the displacement time history curve of node 7 of the upper chord of the truss structure is taken as a reference (as shown in Figure 19), as shown in Figure 19.

[0200] A comparison of the nodal displacement time history curves reveals that the nodal displacement time history curves of the truss structure obtained by using the structural displacement monitoring method proposed in this invention are basically consistent with the nodal displacement reference value time history curves of the truss structure provided by the project in terms of trend and amplitude.

[0201] Furthermore, the correlation analysis described above was performed on all eight nodes of the undamaged data, and the results are shown in Table 2. It can be observed that the absolute value of the smallest correlation coefficient among the eight nodes is 0.8808, indicating a strong correlation between the structural vibration data monitored by the method proposed in this invention and the actual structural vibration values.

[0202] Table 2 Time history correlation coefficients of undamaged node displacements

[0203]

[0204]

[0205] Through numerical examples, it can be found that the structural displacement monitoring method proposed in this invention can obtain highly reliable displacement time history data in actual structural vibration monitoring cases, and has strong feasibility.

[0206] 2.5 Chapter Summary

[0207] This chapter establishes a structural displacement monitoring method based on an improved FAST corner detection method by deeply studying the principles of maximum inter-class variance, FAST corner detection, and pyramid LK optical flow. The main improvements to the overall method flow are as follows:

[0208] (1) To address the drawback of FAST corner detection being affected by threshold, a new automatic threshold calculation method was developed by combining the principle of maximum inter-class variance, and an improved FAST corner detection method was formed.

[0209] (2) Develop a multi-target selection algorithm to automate the multi-target feature detection and motion tracking process. To address the shortcomings of single-target tracking in existing research, a multi-target selection algorithm was developed and visualized, further optimizing the structural displacement monitoring method.

[0210] (3) To realize the integration of the improved FAST corner detection method with the pyramid LK optical flow method, forming an automated structural vibration monitoring process.

[0211] Finally, the structural vibration monitoring method proposed in this chapter was applied to a practical example and it was found that the method is applicable to a variety of working conditions and can obtain relatively accurate and reliable results.

[0212] 3. Conclusion

[0213] Vibration monitoring is a crucial component of structural health monitoring. Traditional vibration monitoring methods involve deploying sensors on-site to collect data and monitor structural vibration. However, this approach requires extensive fieldwork, including system setup and data acquisition, resulting in high costs. To address the shortcomings of traditional methods, this invention establishes a structural displacement monitoring method based on computer vision principles, image features, and optical flow motion tracking. Furthermore, to ensure the accuracy and feasibility of structural displacement monitoring, this invention establishes a preprocessing scheme for structural vibration video data. The two algorithmic approaches are then combined to form a complete computer vision-based method for structural displacement monitoring and automatic system identification.

[0214] Regarding the preprocessing of structural vibration video data, this invention establishes a method for preprocessing structural video data, including automatic acquisition of basic video data information, and image grayscale conversion and local histogram equalization methods. This method can automatically and quickly acquire basic video data information, providing a basic framework for subsequent data analysis and processing; image grayscale conversion and local histogram equalization, while reducing image computational complexity, enhance image features and provide more information for subsequent displacement monitoring.

[0215] This invention relates to a structural vibration and displacement monitoring method based on improved FAST corner detection. Addressing the limitation of FAST corner detection being affected by threshold values, a new automatic threshold calculation method is developed based on the maximum inter-class variance principle, forming an improved FAST corner detection method. A multi-target selection algorithm is developed to automate the multi-target feature detection and motion tracking process. The improved FAST corner detection method is integrated with the pyramid LK optical flow method to form an automated structural vibration monitoring process. Finally, the proposed structural vibration monitoring method is applied to practical examples, demonstrating that it yields relatively accurate and reliable results.

[0216] In order to overcome the drawbacks of traditional technologies and improve production and living efficiency, this invention develops a structural vibration monitoring method based on computer vision.

[0217] This invention can be applied to vibration monitoring of building structures, extracting structural vibration response and providing data support for subsequent structural modal identification, structural damage identification, damage location, and other tasks.

[0218] The present invention also discloses a structural vibration displacement monitoring system based on computer vision, comprising: a memory, a processor, and a computer program stored in the memory, wherein the computer program is configured to implement the steps of the structural vibration displacement monitoring method of the present invention when called by the processor.

[0219] The present invention also discloses a computer-readable storage medium storing a computer program configured to implement the steps of the structural vibration displacement monitoring method of the present invention when invoked by a processor.

[0220] The advantages of this invention are: 1. The structural vibration displacement monitoring method of this invention uses a camera and imaging system as a data acquisition system, extracts structural vibration information using digital images as the processing object, and then outputs the actual vibration information of the structure in three-dimensional space according to the imaging principle; 2. The structural vibration displacement monitoring method of this invention has developed functions such as visualization operation and multi-target selection, further optimizing the overall monitoring process and achieving the characteristics of non-contact, simple operation, and high degree of automation; 3. The structural vibration displacement monitoring method of this invention overcomes the drawbacks of traditional technologies and improves production and living efficiency.

[0221] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A structural vibration displacement monitoring method based on computer vision, characterized in that, Includes the following steps: Step 1: Structural vibration video preprocessing; The basic information of the video data is automatically obtained by an algorithm for automatic acquisition of basic video information, and the image grayscale conversion and local histogram equalization of video frames are used to simplify the image calculation complexity and enhance image features. Step 2: Structural displacement monitoring based on improved FAST corner detection; introduce an automatic threshold calculation method based on maximum inter-class variance to form an improved FAST corner detection algorithm; combine the image features obtained by the improved FAST corner detection algorithm with the pyramid Lucas-Kanade optical flow method to establish a fully automated image feature detection and motion tracking scheme. In step 1, the specific steps of using the video frame image grayscale conversion and local histogram equalization processing method are as follows: Step S1: Read video frame images; Step S2: Image grayscale conversion; Using the weighted grayscale conversion method, with the image coordinate system as the reference system, first read the pixel at coordinate (0,0) and record the channel values. Based on the weighted grayscale conversion calculation principle, calculate the grayscale value after grayscale conversion and assign it to the current pixel. Read all pixels in sequence and convert them to grayscale. Finally, reassemble the grayscale pixels to obtain the grayscale image. Step S3: Divide the current image into grids; Step S4: Local grayscale histogram equalization; Using the image coordinate system as a reference, first perform histogram statistics on the grid sub-blocks in the upper left corner of the image, and calculate a new grayscale image based on the histogram equalization principle. Perform the above operation on all grid sub-blocks in sequence, and finally combine the obtained sub-blocks to form a new grayscale image. In step 2, the improved FAST corner detection algorithm includes the following steps performed sequentially: Statistical steps: To quickly access the distribution of each pixel value, a hash table is established, with the pixel value as the hash table index and the corresponding number of pixel blocks as the corresponding numerical value; Iterative steps: Take the maximum and minimum values ​​of the statistical pixel values ​​as the upper and lower limits of the value range, and continuously iterate within this range to obtain the pixel segmentation threshold k; Output steps: The pixel segmentation threshold k calculated in the loop step is subtracted from the mean values ​​of the two types of pixels obtained based on the value to obtain the adaptive threshold for feature detection, and the result is output to the FAST corner detection algorithm.

2. The structural vibration displacement monitoring method according to claim 1, characterized in that, In step 1, the basic information includes resolution, frame rate, and sampling duration. Step 1 employs an automatic video basic information acquisition algorithm to automatically obtain the basic information of the video data. The specific steps are as follows: Step 10: Open the computer file management system using the visual interactive commands; Step 11: Extract any frame of the image and use the video frame acquisition command in OpenCV to obtain the width and height of the corresponding frame, thereby obtaining the image resolution; Step 12: By integrating the video frame rate acquisition and video frame count acquisition commands, obtain the video capture frame rate and the total number of frames in the video stream, and then obtain the total video duration; Step 13: Save the obtained information to a document in the same directory as the video file.

3. The structural vibration displacement monitoring method according to claim 1, characterized in that, In step 2, the specific steps for structural displacement monitoring based on improved FAST corner detection are as follows: Step 21: Read the video data of structural vibration and extract the video frame at the start moment; Step 22: Select the region of interest in the image; Step 23: Perform feature detection on the selected region of interest; Step 24: Pyramid LK optical flow method motion tracking to obtain displacement time history data; Step 25: Output the displacement time history results of the region of interest.

4. The structural vibration displacement monitoring method according to claim 3, characterized in that, In step 21, a representative start time and duration are selected, including the time when vibration begins and the time when large vibration begins. The get function is selected to read video data at any time. In step 22, a multi-target selection algorithm is established to enable visual selection of multiple tracking targets. After selection, no manual intervention is required. The program automatically reads each tracking target and completes the preset structural vibration monitoring task. In step 24, the obtained feature points are first converted from feature types based on feature information and descriptor information into traceable feature types, and then used for pyramid LK optical flow motion tracking. After optical flow motion tracking, the coordinate positions of the feature points in each video frame are obtained. Subtracting the corresponding coordinates yields the displacement time history data of each feature point in the region.

5. The structural vibration displacement monitoring method according to claim 4, characterized in that, In step 22, the specific steps of the multi-target selection algorithm are as follows: Step A1: Define a target class, which mainly stores the coordinate information and image information of a single target. Multiple tracking targets form multiple classes with their respective information. Step A2: Store the corresponding information of multiple targets sequentially; Step A3: Implement the information output interface. During target motion tracking and calculation, it is only necessary to continuously read each storage unit and output the image information and coordinate information therein.

6. The structural vibration displacement monitoring method according to claim 4, characterized in that, In step S22, to increase the ease of operation, when selecting the region of interest, the multi-target selection algorithm incorporates a visual mouse interaction command to implement mouse bounding selection and complete the multi-target selection. The specific steps are as follows: Step Y1: Mouse selection command; Press and hold the left mouse button and drag to select a rectangular area. When you release the left mouse button, the selection is complete. Step Y2: Rectangular selection confirmation command; the white box indicates the area being selected. Double-clicking the Enter key selects the area and saves the relevant image and coordinate information. The border of the selected area will turn black. To end the target selection, double-click the Esc key to exit.

7. A structural vibration and displacement monitoring system based on computer vision, characterized in that, include: A memory, a processor, and a computer program stored on the memory, the computer program being configured to implement the steps of the structural vibration displacement monitoring method according to any one of claims 1-6 when invoked by the processor.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program configured to implement the steps of the structural vibration displacement monitoring method according to any one of claims 1-6 when invoked by a processor.

Images