Method for automatically extracting strong earthquake information in earthquake monitoring video based on computer vision

Abstract

The invention provides a method for automatically extracting strong earthquake information in an earthquake monitoring video based on computer vision. The method comprises the following steps: carrying out angular point detection on a video frame in earthquake monitoring to obtain an angular point on the video frame; extracting convolution features on each video frame; sampling the convolution features by using a bilinear interpolation algorithm to obtain initial track features of the angular points; and using a Forward Window window to process the initialized trajectory features to obtain waveform information of the earthquake. According to the invention, a corner detection algorithm is adopted to determine tracking points in a video; then, tracking the displacement change of the point in the video along with the seismic oscillation by combining a convolutional network and a transformer network; then, in combination with calibration of an object reference object in the video, a pixel-physical space mapping function is established, physical quantity conversion from video pixel displacement to real ground motion acceleration is realized, a seismic intensity estimation function is further completed, and a new technical means is provided for breaking through limitation of a traditional seismic observation system in the aspect of space coverage.

Description

Technical Field

[0001] This invention relates to the field of earthquake information extraction technology, and in particular to a method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision. Background Technology

[0002] In the fields of earthquake engineering and disaster assessment, accurately estimating strong earthquake information in the epicenter and surrounding areas is crucial for analyzing the source rupture process, assessing earthquake damage, responding to post-earthquake emergencies, predicting casualties, and making seismic fortification decisions. It directly impacts the efficiency of rapid post-earthquake response and disaster assessment. Currently, earthquake observation systems primarily rely on strong-motion networks located far from cities, acquiring core parameters such as seismic intensity and peak ground acceleration through instrumental observations. However, due to the highly localized nature of ground motion and insufficient station density, some moderate to strong earthquake events often result in missing instrumental intensity records or the inability to cover key strong ground motion parameters, particularly in densely populated cities. To compensate for this deficiency, researchers often have to rely on indirect methods for supplementary estimation, reconstructing the ground motion field in areas that cannot be directly observed to support subsequent disaster analysis and emergency deployment.

[0003] To overcome the limitations of instrumental observations in terms of spatiotemporal coverage, the academic community has successively developed a series of innovative methods to obtain or estimate seismic motion parameters. Early studies mainly relied on seismic geological markers to invert seismic parameters: Anooshehpoor et al. (1999) successfully reconstructed the peak horizontal acceleration (PGA) parameter based on the mechanical response of overturned train carriages in the 1906 San Francisco MW7.8 earthquake; similarly, earthquake-induced rockfall displacement, stone lantern overturning, and other geological relics have also been systematically applied to the extrapolation of historical seismic motion parameters. With the development of information technology, the citizen seismology platforms of the U.S. Geological Survey (USGS) and the European-Mediterranean Seismological Centre (EMSC) have achieved rapid assessment of macroscopic intensity through public perception data collection. These innovative methods effectively compensate for the lack of instrumental records and have certain application potential in historical earthquake surveys, but due to the unstructured nature of the data sources, they are difficult to meet the refined requirements of modern earthquake engineering for high-frequency dynamic parameters. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the objective of this invention is to provide a method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A computer vision-based method for automatically extracting strong earthquake information from earthquake monitoring videos includes:

[0007] Step 1: Obtain video frames from earthquake monitoring videos;

[0008] Step 2: Perform corner detection on the video frames to obtain the corner points on the video frames;

[0009] Step 3: Extract convolutional features from each video frame;

[0010] Step 4: Use bilinear interpolation to sample the convolutional features to obtain the initial trajectory features of the corner points;

[0011] Step 5: Use the Forward Window to process the initial trajectory features to obtain the earthquake waveform information.

[0012] Preferably, step 2: performing corner detection on the video frame to obtain corner points on the video frame includes:

[0013] Obtain the gradient of video frames;

[0014] Calculate the second-order matrix of the gradient within a local window of a video frame image;

[0015] Perform eigenvalue decomposition on a second-order matrix and find its eigenvalues;

[0016] Corner detection is performed on video frames based on feature values ​​to obtain the corner points on the video frames.

[0017] Preferably, calculating the second-order matrix of the gradient within a local window of a video frame image includes:

[0018] Formula used:

[0019]

[0020] ∑I x I y =∫ Ω I x (x,y)I y (x,y)w(x,y)dxdy

[0021] Calculate the second-order matrix of the gradient within a local window of a video frame image; where M represents the second-order matrix of the gradient, and I... x (x,y) is the horizontal gradient of the video frame at position (y,y), I y (x,y) is the vertical gradient of the video frame at position (x,y), ∑ represents the weighted summation within a neighborhood window, w(x,y) is the Gaussian weighted function, and Ω represents the local window region.

[0022] Preferably, the corner detection of the video frame based on feature values ​​to obtain the corner points on the video frame includes:

[0023] The point with the largest pixel change is selected as the corner point, and the formula for determining this is:

[0024]

[0025] in It is the trace of the matrix. It is the determinant of the matrix, where P represents the corner point, τ represents the threshold, and λ is the matrix determinant. 1,2 This represents the eigenvalues ​​of the second-order gradient matrix.

[0026] Preferably, step 3: extracting convolutional features on each video frame includes:

[0027] The video frame is input into the ConvBlock convolution module to extract the spatial features of the video frame; the ConvBlock convolution module is composed of two-dimensional convolution, InstanceNorm2D and a layer of ReLU activation function in sequence.

[0028] The extracted spatial features are input into a four-layer ResidualBlockLayer to obtain features at four scales; each ResidualBlockLayer contains two ResidualBlock modules; each ResidualBlock module contains two ConvBlock layers.

[0029] The features at the four scales are concatenated and then sequentially input into ConvBlock and Conv2D to obtain convolutional features.

[0030] Preferably, step 4: sampling the convolutional features using a bilinear interpolation algorithm to obtain the initial trajectory features of the corner points, includes:

[0031] Get the coordinates of the top corner of the video frame;

[0032] Based on the coordinates of the corner points, interpolate in the x-direction:

[0033]

[0034]

[0035] Then perform interpolation in the y-direction:

[0036]

[0037] in, For convolutional features φ(I) t The i-th tracking point in ) The corresponding feature value, Q, represents the initial trajectory feature obtained. Represents the x-coordinate of the corner point. Represents the ordinate of the corner point.

[0038] Preferably, step 5: processing the initial trajectory features using the Forward Window to obtain earthquake waveform information includes:

[0039] The convolutional features are input into the CorrBlock module to obtain features at different scales at the same time point, which form a feature pyramid.

[0040] The trajectory features are obtained by updating the convolutional features at each time step using the CorrBlock module;

[0041] Correlation features are calculated based on trajectory features and the feature pyramid surrounding the current trajectory position;

[0042] The tokens are formed by superimposing the feature pyramid, correlation features, trajectory features, and the feature pyramid corresponding to the current trajectory position.

[0043] The tokens are input into the Transformer layer for iterative optimization to obtain the seismic waveform extraction model;

[0044] The target video frame is processed using a seismic waveform extraction model to obtain the seismic waveform of the target video frame.

[0045] Preferably, the step of calculating the correlation features based on the trajectory features and the feature pyramid corresponding to the current trajectory position includes:

[0046] Formula used:

[0047]

[0048] Calculate the correlation features; where, Represents trajectory characteristics, φ s (I t ) represents the feature pyramid, The trajectory position is represented by S, the feature scale is k, the downsampling factor is k, and δ is δ. The neighborhood of Z 2 Let represent the integer field space from negative integers to positive integers, Δ represent the maximum range of the neighborhood (3 in this paper), and N represent natural numbers.

[0049] Preferably, the tokens are:

[0050]

[0051] in, This represents the trajectory position at time t. Indicates the starting position of the trajectory. Representing trajectory features, Indicates correlation characteristics, This represents the sinusoidal position code of the trajectory position at time t=1 relative to the trajectory position at time t. Indicates the starting trajectory position The sinusoidal position code, η′(t), represents the sinusoidal position code for time t.

[0052] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps in the above-described method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision.

[0053] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0054] This invention provides a method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision. Compared with existing technologies, this invention uses a corner detection algorithm to determine the tracking point in the video; then, it combines a convolutional network and a transformer network to track the displacement change of this point in the video with the earthquake motion; next, it combines the calibration of the reference objects in the video to establish a pixel-physical space mapping function, realizing the conversion of video pixel displacement into the physical quantity of real ground motion acceleration, and further completing the earthquake intensity estimation function, providing a new technical means to overcome the limitations of traditional earthquake observation systems in terms of spatial coverage. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 The flowchart of the automatic extraction method for strong earthquake information from earthquake monitoring videos based on computer vision provided by the present invention is shown below.

[0057] Figure 2 This is a schematic diagram of corner point selection provided by the present invention;

[0058] Figure 3 This is a tracer architecture diagram provided by the present invention;

[0059] Figure 4 This invention provides a schematic diagram of the feature extraction principle.

[0060] Figure 5 A schematic diagram of the bilinear interpolation algorithm provided by this invention;

[0061] Figure 6This is a schematic diagram of the Forward Window update mechanism provided by the present invention.

[0062] Figure 7 This is a schematic diagram of the automatically extracted waveform results provided by the present invention;

[0063] Figure 8 The pixel space-physical space calibration mapping diagram provided by this invention;

[0064] Figure 9 (a) Selected video tracking points; (b) Waveform of automatically tracked video key points; (c) Displacement waveform after filtering with exponential averaging algorithm; (d) Seismic acceleration data obtained after two differentiations;

[0065] Figure 10 (a) Waveform of manually tracked video key points; (b) Waveform of automatically tracked video key points; (c) Distance between key points in each frame tracked manually and automatically;

[0066] Figure 11 The first column in the figure represents earthquake data for region A; the second column represents earthquake data for region B (1); earthquake data for Yinchuan, Ningxia, region C; and the fourth column represents earthquake data for region B (2). Figure (b) shows earthquake data recorded by strong-motion stations; Figure (c) shows the displacement waveforms of video keypoints obtained using automatic tracking in the Y-axis direction within the same time period; Figure (d) shows histogram and kernel density estimation (KDE) analysis, reflecting the concentrated signal energy intervals; and Figure (e) shows the QQ plot (quantile-quantile plot).

[0067] Figure 12 This invention provides an instrumental seismic intensity distribution map for region A.

[0068] Figure 13 This invention provides an instrumental seismic intensity distribution map for region B.

[0069] Figure 14 (a) Near-field keypoint tracking waveform; (b) Mid-field keypoint tracking waveform; (c) Deep-field keypoint tracking waveform;

[0070] Figure 15 For intensity estimation of key points at different depths of field, the red numbers represent the intensity estimates;

[0071] Figure 16 (a) Keypoint tracking waveform with obvious features; (b) Keypoint tracking waveform with indistinct features;

[0072] Figure 17 The following waveforms are included: (a) Keypoint tracking waveform with obvious features and no occlusion; (b) Keypoint tracking waveform with obvious features and occlusion in the video; and (c) Keypoint tracking waveform with indistinct features and occlusion in the video. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0075] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, including a series of steps, processes, methods, etc., is not limited to the steps listed, but may optionally include steps not listed, or may optionally include other steps inherent to these processes, methods, products, or devices.

[0076] To make the above-mentioned objectives, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0077] Please see Figure 1 A computer vision-based method for automatically extracting strong earthquake information from earthquake monitoring videos includes:

[0078] Step 1: Obtain video frames from earthquake monitoring videos;

[0079] Step 2: Perform corner detection on the video frames to obtain the corner points on the video frames;

[0080] (1) For a given video frame I, calculate its gradient in the x and y directions. Usually, the Sobel operator or other edge detection operators are used for calculation, as shown in formula (1):

[0081]

[0082] Where I(x,y) represents the pixel value at position (x,y) of the image in the video frame; I x It is its horizontal gradient; Iy It is its vertical gradient.

[0083] (2) Calculate the second-order matrix (structure tensor) of the gradient in the local window of the image, as shown in formula (2):

[0084]

[0085] Here, the symbol ∑ represents a weighted summation within a neighborhood window (usually Gaussian weighted), which can be expressed as:

[0086]

[0087] ∑I x I y =∫ Ω I x (x,y)I y (x,y)w(x,y)dxdy, (5)

[0088] Where w(x,y) is a Gaussian weighted function, and Ω represents a local window region.

[0089] Perform eigenvalue decomposition on matrix M and find its eigenvalues, as shown in formula (6):

[0090]

[0091] in, It is the trace of the matrix.

[0092]

[0093] Formula (7) is the determinant of the matrix.

[0094] To determine whether a point is a corner, the formula is shown in (8):

[0095]

[0096] in,

[0097]

[0098] In the above (9), it means that the minimum feature value of each pixel is the largest, which means that its pixel change range is drastic. The corner point P is calculated according to the above process, such as Figure 2 As shown by the red dot.

[0099] This invention uses CNN and Transformer to construct a corner tracker to track the displacement of corner point P in subsequent video frames and plot it as a waveform. The main architecture of the tracker is as follows: Figure 3As shown: First, the video and corner points are input into the model; then, a convolutional neural network (CNN) is used to extract features from the video frames; based on the corner coordinates, the features are sampled using a bilinear interpolation algorithm to obtain the initial trajectory features; then, the correlation features are calculated through the CorrBlock layer; finally, the features are concatenated and input into the Transformer layer, and after M iterations and sliding window inference, the final trajectory coordinates are obtained.

[0100] Step 3: Extract convolutional features from each video frame;

[0101] In a CNN network, video frame I t After input, the data first passes through a ConvBlock convolutional module, which consists of a 2D convolution, an InstanceNorm2D layer, and a ReLU activation function. Next, to obtain multi-scale features, it passes through four ResidualBlockLayer layers, each containing two standard ResidualBlock modules. Within each module, the output is obtained after passing through two ConvBlock layers. When the stride is set to 2, the original input is first downsampled using Conv2D and InstanceNorm2D operations, then added to the output, and finally passed through a ReLU activation function. Finally, the features from the four scales are interpolated to the same size, concatenated, then passed through another ConvBlock layer, and finally adjusted for dimensionality using Conv2D to obtain the final convolutional feature φ(I). t ).

[0102] This invention employs a sliding window method to process video sequences, using a set starting position P to initialize the estimated trajectory. Its schematic diagram is as follows Figure 4 As shown.

[0103] From each video frame I t The mathematical expression for the process of extracting d-dimensional appearance features is as follows (10):

[0104]

[0105] Where H and W represent the height and width of the video frame, respectively, T represents the total number of video frames, and k=4 is a downsampling factor designed to improve computational efficiency. The final output is features with d=128 channels.

[0106] Step 4: Use bilinear interpolation to sample the convolutional features to obtain the initial trajectory features of the corner points;

[0107] Based on the coordinates of the starting position point P, the image feature φ(I) is processed using a bilinear interpolation algorithm. tSampling is performed to obtain the initialized trajectory features Q.

[0108] The principle of bilinear interpolation algorithm is as follows: Figure 5 As shown, the specific calculation process is as follows:

[0109]

[0110] Interpolate in the x-direction:

[0111]

[0112] Then perform interpolation in the y-direction:

[0113]

[0114] in For convolutional features φ(I) t The i-th tracking point in ) The corresponding feature value, where i=1 represents a tracking point. This yields the initialized trajectory feature Q. Further processing then proceeds into the Forward Windows window; see details of the sliding window for more information. Figure 6 .

[0115] Step 5: Use the Forward Window to process the initial trajectory features to obtain the earthquake waveform information.

[0116] (1) In the Forward Window processing, the operation of the CorrBlock module is involved first. Based on the above-calculated convolutional features φ(I t ), further generating multiple scaled versions of the features, the calculation formula of which is as follows (15):

[0117]

[0118] Where s = 1,...,S. Specifically, average pooling is applied to the basic features within the s×s neighborhood to obtain features of different scales in the S layers, forming a feature pyramid.

[0119] Then the trajectory features are calculated, from express, Time-dependent to adapt to trajectory changes. Initially initialized with the sampled image features described above, followed by trajectory features. It is updated by applying a neural network at each time step.

[0120] Next, the correlation features are calculated. The introduction of correlation features facilitates the matching of trajectories with images. These features are achieved by comparing trajectory features Q with the position around the current trajectory. Image features Obtain. Specifically, vector. It is obtained by stacking inner products, and the calculation process is shown in the following formula (16):

[0121]

[0122] S refers to the feature scale, and δ is the offset, representing the point. The neighborhood of φ. Image features φ are analyzed using bilinear interpolation and zero-padding. s (I t Sampling is performed at non-integer positions. In the specific implementation, S is set to 4 and Δ is 3, therefore the calculated... The dimension is ((2Δ+1)). 2 S = 196.

[0123] (2) The Transformer layer consists of two Linear layers and 18 AttnBlock attention modules. The aforementioned relevant features are superimposed with other features to form the input Transformer.

[0124]

[0125] These represent position, trajectory features, correlation features, and sinusoidal position coding, respectively. Except for the last feature, all other features have been described above. The last feature is derived from the estimated position: it is the sinusoidal position coding η of the trajectory position at time t=1 relative to the trajectory position at time t. Additionally, the starting position is also added. The encoding of time t, η′, helps the attention mechanism in Transformer to operate efficiently.

[0126] Transformer network output The content includes the updated location changes and appearance features.

[0127] To gradually obtain a more accurate estimated trajectory, the Forward Window is applied iteratively multiple times, with adjustments made to the predicted trajectory in each iteration to improve its accuracy. The Forward Window iterates a total of M times, and the output of the last iteration is taken as the final result. The update calculation for each iteration is shown in the following formula (18):

[0128]

[0129] Then the tracking trajectory and trajectory features are updated as shown in formula (19):

[0130]

[0131] Where m = 0, 1, ..., M, With Q (0) All are initialized by the starting position and time. The above formula applies to all trajectories i = 1, ..., N and t = 1, ..., T.

[0132] When there is a video V with a length T′>T, which exceeds the maximum window length T supported by the architecture, the video needs to be divided into segments in order to track points throughout the entire video V. Consider a window of length T, with T / 2 frames overlapping between windows. The Transformer will iterate M×J times: the output of the first window is used as the input of the second window, and so on, with the superscript (m,j) denoteing the iteration with respect to the m-th Transformer update of the j-th window. This will yield an M×J quantity. This spans the Transformer iteration and window. These quantities are initialized starting with m = 0 and j = 1. Then, the Transformer is iterated M times to obtain... From this state, initialize the second window. Repeat this process to achieve sliding window inference.

[0133] like Figure 7 As shown. Figure 7 (a) shows the coordinate changes of the tracking point in the x-direction in a two-dimensional image, while Figure 7 (b) reflects the coordinate changes of the tracking point in the y-direction. In these subplots, the x-axis represents the video frame index, and the y-axis represents the pixel position.

[0134] Since the tracking point data obtained from the video is pixel-level change, this invention uses object calibration to map the displacement changes in pixel space to displacement changes in physical space. In the subsequent sections of this invention, calibration and analysis are primarily focused on the vertical direction for the following reasons: First, the height characteristics of objects in the vertical direction are relatively clear, facilitating pixel calibration; second, compared to the horizontal direction, the vertical direction experiences less interference and can more closely approximate the waveform characteristics of an earthquake. The calibration method involves finding a reference object with standard dimensions in the video, identifying its pixel count, and completing the calibration task through comparative mapping. For example... Figure 8 As shown: The car model is Buick Verano Sedan 2017, 2018 or 2020, with a length of 1802mm × 4723mm × 1466mm (front width × side height × vehicle height). Two points were manually calibrated for the vehicle height and mapped to the original vehicle height to obtain 1 pixel ≈ 0.83cm. That is, each pixel in the video corresponds to a real-world size of approximately 0.83cm. The above waveform was calibrated and mapped to obtain an approximate physical waveform.

[0135] Because the camera uses a non-rigid fixing method, the motion captured in the earthquake video includes other motions (such as elastic motion between the camera and the wall, and vibration damping within the camera), which can interfere with acceleration calculations. Therefore, when estimating intensity, the first step is to consider... Figure 9 (b) The raw waveform data is automatically extracted from the video, filtered using an exponential averaging algorithm, and a window size of 50 is selected to obtain the filtered seismic motion displacement data. The results are as follows: Figure 9 As shown in (c). Then, differential calculations are performed. Seismic velocity data is obtained through the first differential, and seismic acceleration data is obtained through the second differential. The final result is presented in... Figure 9 (d)

[0136] According to the earthquake intensity calculation formula in GB / T 17742—2020 "China Earthquake Intensity Scale", the velocity and acceleration data obtained from the above steps are substituted into the following formulas (20), (21) and (22):

[0137] I A =3.17log 10 (PGA)+6.59, (20)

[0138] I V =3.17log 10 (PGV)+9.77, (21)

[0139]

[0140] Where PGA represents peak acceleration and PGV represents peak velocity. The calculation results are I... A =7.45,I V =6.45,I I =6.45. Therefore, the final intensity estimate is 6.45.

[0141] To evaluate the performance of the model, three evaluation methods were used to assess its performance: (1) comparison of the similarity between manually tracked and extracted seismic waveforms and automatically tracked and extracted seismic waveforms; (2) comparison of the similarity between automatically tracked and extracted seismic waveforms and strong earthquake waveforms provided by the strong earthquake network; and (3) comparison of the similarity between automatically extracted intensity information and intensity information provided by the strong earthquake network.

[0142] Figure 10 The results show a comparison between manual and automatic tracking of key points in earthquake videos of region A. In subfigure (a), the waveform of the Y-coordinate of the video key points obtained by manual tracking using this invention changes with the number of frames, showing that the key point coordinates fluctuate significantly throughout the sequence. In contrast, subfigure (b) shows the results obtained using the automatic tracking algorithm, whose waveform trend is basically consistent with that of manual tracking.

[0143] Subplot (c) details the differences between manual and automatic tracking results. The difference is calculated by directly subtracting the Y-coordinate values ​​of each frame. As seen in the difference plot, the difference for most frames is within ±3 pixels, indicating high reliability in tracking accuracy. However, a few frames show deviations, likely due to errors in keypoints from the automatic tracking algorithm and errors in pixel-level keypoints from manual tracking.

[0144] Table 1 Quantitative Assessment of Automatic Tracking and Manual Tracking

[0145] method MAE RMSE Automatic tracking algorithm 1.74 1.90

[0146] To quantitatively evaluate the performance of the automatic tracking algorithm in seismic video data processing, this study used two statistical metrics, mean absolute error (MAE) and root mean square error (RMSE), to measure the algorithm's accuracy. According to the experimental results, the automatic tracking algorithm achieved an MAE of 1.74 pixels and an RMSE of 1.90 pixels.

[0147] Mean Absolute Error (MAE) is a method for measuring the difference between predicted and actual values; a smaller MAE value indicates that the prediction is closer to reality. In this study, the MAE of the automatic tracking algorithm was 1.74 pixels, indicating that the algorithm has high accuracy in tracking the location of key points in earthquake videos. The low MAE proves that the algorithm can accurately capture the movement trajectory of key points in most cases, demonstrating its practical application value for earthquake monitoring and analysis.

[0148] The root mean square error (RMSE) measures the square root of the sum of squares of the errors in the predictions, and is typically used to quantify the variability or volatility of data. The automatic tracking algorithm achieved an RMSE of 1.90 pixels, further confirming its reliability and robustness in handling video tracking tasks. This relatively low RMSE indicates that, despite potentially large deviations in some frames, the automatic tracking algorithm effectively approximates the results of manual tracking overall.

[0149] In summary, when comparing the results of automatic tracking technology with those of manual tracking, both the MAE and RMSE indices are at a low level. This fully demonstrates that automatic tracking technology has good accuracy and stability in waveform extraction from earthquake monitoring videos, showcasing its great potential and practicality.

[0150] This invention employs an Autoregressive Model (AR Model) to model the features of vibration data extracted from video vibration measurement models and measured data from strong-motion stations. This method is applicable to situations where station signals have different shapes but consistent dynamic characteristics, such as propagation signals under different site conditions. The spatiotemporal matching degree of the two types of data is quantitatively evaluated by calculating three indices: Euclidean distance, cosine similarity, and Wasserstein distance.

[0151] Autoregressive (AR) models were built for automatically extracted seismic waveforms and strong-motion network data, respectively.

[0152]

[0153] Where x t a represents the observed value at the current time. i These are the AR model parameters, where p is the model order and ε is the parameter. t The noise term (usually assumed to be white noise) represents the unpredictable portion. This yields the video seismic waveform parameter vector A. x and strong earthquake network data parameter vector A y Calculate the Euclidean distance between the parameter vectors:

[0154]

[0155] Calculate cosine similarity:

[0156]

[0157] The closer the cosine similarity is to 1, the more similar the dynamic characteristics of the signals are.

[0158] Let's analyze an example of a magnitude 6.8 earthquake in region A, two magnitude 4.7 earthquakes in region B, and a magnitude 4.8 earthquake in region C:

[0159] First, we analyze the Euclidean distance. A smaller Euclidean distance indicates that the two signals have relatively similar numerical distributions after normalization, with small numerical differences, and is used to measure the absolute similarity of the signals. Analyzing the time-domain characteristics of the signals, the mean Euclidean distance of 1.2532 indicates that there is a limited deviation in the amplitude domain after normalization between the two types of signals. This reflects that ambient noise may slightly interfere with video vibration, but the overall amplitude distribution remains highly consistent.

[0160] At the dynamic characteristic analysis level, a cosine similarity of 0.8907 indicates a high degree of agreement between the two types of signals in the direction of the feature space vector. The cosine similarity value is between [0,1], and the closer the value is to 1, the more similar the directions of the two signals are. Cosine similarity focuses more on the trend of signal change rather than specific numerical differences. If the cosine similarity of two signals is high, it indicates that they are relatively consistent in their dynamic change trends, confirming that the video vibration measurement model can effectively capture the dominant modal characteristics of structural vibration. In particular, for vibration signals with significant site effects (such as the filtering effect of soft soil sites), the comparison of AR model parameters shows that the two have good consistency in core dynamic indicators such as system pole distribution and energy attenuation characteristics.

[0161] Finally, Wasserstein distance was introduced for probability distribution verification, further confirming the statistical similarity of the two types of signals in time-frequency energy distribution. Wasserstein distance (Earth Mover's Distance, EMD) measures the optimal transmission cost between two distributions, that is, the minimum "transport distance" required to transform one distribution into another. If the Wasserstein distance is close to 0.0, it indicates that the distributions of the two data points are almost identical. If the Wasserstein distance is large (e.g., above 0.5), it indicates that the distributions of the two data points differ significantly. This index effectively overcomes the sensitivity of traditional time-domain indices to phase differences, proving that observation systems based on different sensing principles have inherent consistency in characterizing the essential features of structural dynamic response. In this experiment, the Wasserstein distance was 0.0713, indicating a certain numerical deviation but overall trend similarity.

[0162] The analysis of the above results leads to the conclusion that there is a high correlation between the waveforms obtained by automatic tracking and the observation data from strong-motion stations. Although there are certain errors due to differences in sampling rates and between the actual vibrations transmitted to the monitoring camera and the observed data, vibration tracking through the monitoring camera can still accurately reflect the vibration situation in the area where the camera is located.

[0163] Based on the method described above, the estimated earthquake intensity for region A is I. I =6.45; the earthquake intensity in region C is I. I =5.50. (and below) Figure 12 and Figure 13 Qualitative analysis was performed using the instrumental seismic intensity distribution map to estimate the intensity value I. I Within a reasonable range, the intensity estimate is roughly within ±0.5 intensity units. This indicates that the algorithm for inverting seismic motion parameters through video motion features is explorable and provides a new approach for rapid seismic intensity assessment based on computer vision.

[0164] A corner detection algorithm was used to select the top 10 corners with the highest response values. These corners covered the far, middle, and near depths of field and were used for tracking and intensity estimation. Figure 14 As can be seen, under the same vibration conditions, the keypoint tracking waveforms at different depths of field exhibit certain differences. Although the overall waveforms are relatively stable and consistent, their amplitudes differ significantly, with the waveform amplitude at distant keypoints being slightly larger than that at near keypoints. This phenomenon is observed in... Figure 15 This is also confirmed in the intensity estimates, namely, the greater the depth of field, the greater the intensity estimate.

[0165] This phenomenon occurs because the calibration process uses the height of vehicles near point 5 as a reference for mapping pixels to physical space. Due to scale distortion caused by perspective projection—under the same vibration conditions—the pixel displacement in distant areas is amplified, and the projection ratio change in the depth direction of the fixed-height reference used in the calibration process is not compensated for. Therefore, in practical applications, the camera should be calibrated first to determine the true distance corresponding to each pixel; or a reference object should be selected near the tracking point to map pixel space to physical space, thereby improving the accuracy of intensity estimation.

[0166] Performance of keypoint tracking waveforms under different feature saliency levels:

[0167] Figure 16 (a) shows a keypoint tracking waveform with distinct features, characterized by large amplitude and high continuity. This indicates that high feature distinctiveness helps the tracking algorithm stably identify and track targets, thereby accurately capturing ground motion caused by earthquakes. In contrast, Figure 16 (b) shows the key point tracking waveform with indistinct features. Its amplitude is small and the waveform has many discontinuities, indicating that low feature distinctiveness reduces the accuracy and stability of tracking.

[0168] The impact of temporary occlusion on tracking performance:

[0169] In earthquake monitoring videos, the passing of pedestrians or other objects can obstruct the tracking point, a situation that is difficult to avoid in real-world scenarios. Figure 17 As shown. Therefore, this invention focuses on analyzing the impact of occlusion on tracking performance. Figure 17 (a) shows the keypoint tracking waveform with distinct features and no obstruction. The results show that the waveform is stable and has good continuity, indicating that the tracking algorithm can efficiently and accurately identify and track targets under unobstructed conditions, thus providing high-quality data for seismic analysis. This result highlights the importance of unobstructed line of sight for tracking accuracy. Figure 17 (b) shows the key point tracking waveform with obvious features but occlusion. Although tracking is still in progress, the continuity and stability of the waveform are affected to some extent, indicating that the occlusion affects the tracking performance. Figure 17 (c) The key point tracking waveform with indistinct features and occlusion performed the worst, with frequent breaks and large fluctuations, indicating that the tracking difficulty increased significantly under the combined effect of occlusion and low feature distinctiveness.

[0170] In conclusion, occlusion is one of the most significant factors affecting video tracking performance. In practical applications, when using corner detection algorithms, unoccluded keypoints should be considered as tracking targets to improve tracking accuracy and stability.

[0171] The present invention also provides an electronic device, including a bus, a transceiver, a memory, a processor, and a computer program stored in the memory and executable on the processor. The transceiver, the memory, and the processor are connected via the bus. The computer program, when executed by the processor, implements the steps in the above-described method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision. Compared with the prior art, the beneficial effects of the electronic device provided by the present invention are the same as those of the above-described method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision, and will not be elaborated upon here.

[0172] The present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the computer program is executed by a processor, it implements the steps in the above-described method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present invention are the same as the beneficial effects of the above-described method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision, and will not be elaborated here.

[0173] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Regarding the methods disclosed in the embodiments, since they correspond to the apparatus disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the apparatus description.

[0174] Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision, characterized in that, include: Step 1: Obtain video frames from earthquake monitoring videos; Step 2: Perform corner detection on the video frames to obtain the corner points on the video frames; Step 3: Extract convolutional features from each video frame; Step 4: Use bilinear interpolation to sample the convolutional features to obtain the initial trajectory features of the corner points; Step 5: Use the Forward Window to process the initial trajectory features to obtain the earthquake waveform information.

2. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 1, characterized in that, Step 2: Detecting corner points on the video frames to obtain the corner points on the video frames, including: Obtain the gradient of video frames; Calculate the second-order matrix of the gradient within a local window of a video frame image; Perform eigenvalue decomposition on a second-order matrix and find its eigenvalues; Corner detection is performed on video frames based on feature values ​​to obtain the corner points on the video frames.

3. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 2, characterized in that, Calculate the second-order matrix of the gradient within a local window of a video frame image, including: Formula used: ∑I x Yo y =∫ Ω Yo x (x,y)I y (x,y)w(x,y)dxdy Calculate the second-order matrix of the gradient within a local window of a video frame image; where M represents the second-order matrix of the gradient, and I... x (x,y) is the horizontal gradient of the video frame at position (x,y), I y (x,y) is the vertical gradient of the video frame at position (x,y), ∑ represents the weighted summation within a neighborhood window, w(x,y) is the Gaussian weighted function, and Ω represents the local window region.

4. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 3, characterized in that, The method of obtaining corner points on video frames by detecting corner points based on feature values ​​includes: The point with the largest pixel change is selected as the corner point, and the formula for determining this is: in, It is the trace of the matrix. It is the determinant of the matrix, where P represents the corner point, τ represents the threshold, and λ is the matrix determinant. 1,2 This represents the eigenvalues ​​of the second-order gradient matrix.

5. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 4, characterized in that, Step 3: Extracting convolutional features from each video frame, including: The video frame is input into the ConvBlock convolution module to extract the spatial features of the video frame; the ConvBlock convolution module is composed of two-dimensional convolution, InstanceNorm2D and a layer of ReLU activation function in sequence. The extracted spatial features are input into a four-layer ResidualBlockLayer to obtain features at four scales; each ResidualBlockLayer contains two ResidualBlock modules; each ResidualBlock module contains two ConvBlock layers. The features at the four scales are concatenated and then sequentially input into ConvBlock and Conv2D to obtain convolutional features.

6. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 5, characterized in that, Step 4: Using a bilinear interpolation algorithm to sample the convolutional features to obtain the initial trajectory features of the corner points, including: Get the coordinates of the top corner of the video frame; Based on the coordinates of the corner points, interpolate in the x-direction: Then perform interpolation in the y-direction: in, For convolutional features φ(I) t The i-th tracking point in ) The corresponding feature value, Q, represents the initial trajectory features obtained. Represents the x-coordinate of the corner point. Represents the ordinate of the corner point.

7. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 6, characterized in that, Step 5: Using the Forward Window to process the initialized trajectory features to obtain the earthquake waveform information, including: The convolutional features are input into the CorrBlock module to obtain features at different scales at the same time point, which form a feature pyramid. The trajectory features are obtained by updating the convolutional features at each time step using the CorrBlock module; Correlation features are calculated based on trajectory features and the feature pyramid surrounding the current trajectory position; The tokens are formed by superimposing the feature pyramid, correlation features, trajectory features, and the feature pyramid corresponding to the current trajectory position. The tokens are input into the Transformer layer for iterative optimization to obtain the seismic waveform extraction model; The target video frame is processed using a seismic waveform extraction model to obtain the seismic waveform of the target video frame.

8. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 7, characterized in that, The calculation of correlation features based on trajectory features and the feature pyramid corresponding to the current trajectory position includes: Formula used: Calculate the correlation features; where, Represents trajectory characteristics, φ s (I t ) represents the feature pyramid, The trajectory position is represented by S, the feature scale is k, the downsampling factor is k, and δ is δ. The neighborhood of Z 2 Let represent the integer field space from negative integers to positive integers, Δ represent the maximum range of the neighborhood (3 in this paper), and N represent natural numbers.

9. The method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in claim 8, characterized in that, The tokens are: in, This represents the trajectory position at time t. Indicates the starting position of the trajectory. Representing trajectory features, Indicates correlation characteristics, This represents the sinusoidal position code of the trajectory position at time t=1 relative to the trajectory position at time t. Indicates the starting trajectory position The sinusoidal position code, η′(t), represents the sinusoidal position code for time t.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the method for automatically extracting strong earthquake information from earthquake monitoring videos based on computer vision as described in any one of claims 1-8.

Images