A video micro-vibration signal correction method

Abstract

The application provides a video micro-vibration signal correction method and relates to the technical field of data processing.The method comprises the following steps: arranging a vibration sensor on a structure to be measured, capturing a video of the structure to be measured; based on the video capture result, extracting time domain features and frequency domain features of a vibration signal; using the time domain features and the frequency domain features to train and construct a multi-modal fusion calibration model, wherein the multi-modal fusion calibration model is a hybrid CNN-LSTM network architecture; introducing a spectral loss term based on a Fourier spectrum logarithmic amplitude difference, and using the constructed multi-modal fusion calibration model to obtain a dual correction target result of time domain waveform reconstruction and frequency domain key modal alignment, thereby completing the correction of the video micro-vibration signal.The application solves the problem of low precision caused by environmental factors, noise, on-site operation and the like when micro-vibration information is directly extracted from a monitoring video.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, and in particular relates to a method for correcting video micro-vibration signals. Background Technology

[0002] Hydraulic structures such as dams, spillways, and aqueducts play a crucial role in water resource management and infrastructure safety. During long-term operation, these structures are subjected to complex dynamic loads from continuous hydraulic impact, sediment erosion, and operational vibrations. Accumulated structural damage (such as concrete cracking and steel corrosion) and potential resonance phenomena caused by environmental excitations (such as flood discharge and traffic loads) can seriously jeopardize their operational safety. Structural health monitoring (SHM) systems are essential for early damage detection and risk mitigation, serving as a means to ensure the long-term safe operation of structures.

[0003] Traditional structural health monitoring systems primarily rely on point sensors (such as accelerometers, strain gauges, or displacement sensors) to measure responses at critical locations. However, deploying such sensors on large-scale or geometrically complex hydraulic structures often results in sparse spatial distribution due to economic considerations, and the limited number of measurement points cannot capture the fine vibration patterns across the entire field. In recent years, video-based non-contact vibration monitoring methods have emerged as a promising alternative, utilizing computer vision technology to extract full-field displacement data from monitoring videos. In particular, video vibration information extraction technology combining the phase-based optical flow method and phase-based video magnification offers advantages such as stronger robustness, no need for structural marking, sub-pixel displacement resolution, and magnification of micrometer-level vibrations to the naked eye. Theoretically, this achieves monitoring accuracy comparable to traditional point sensors, making it possible to apply this method to harsh or inaccessible environments and large structures. Over the past decade, this method has been continuously improved. The monitoring area has expanded from local motion areas to the entire shooting domain. Vibration information such as time history, spectrum, and structural modes, which previously required indirect conversion, can now be directly extracted from video. The computational efficiency of the extraction process has also been continuously improved. However, most current research is conducted in laboratory environments, using simplified scaled structural models for experimental verification and reliability assessment. The monitored objects are generally small or have clear textures, and the influencing factors of the monitoring environment are minimized. The main obstacle preventing this method from moving from theory (or carefully arranged experimental environments) to practical engineering is the reliability of the extracted structural vibration information under the influence of the above-mentioned combined factors. For example, existing technologies have studied the reasons why the algorithm itself may cause distortion in the extracted information and combined deep learning algorithms or denoising algorithms to further improve the accuracy of vibration information extraction. Existing technologies have also analyzed the possible influence of the monitoring environment and attempted to solve the data accuracy problem caused by camera vibration or instability in actual monitoring environments.

[0004] Previous studies have shown that distortion in vibration information extraction is present in both the time and frequency domains. In the time domain, the raw pixel motion extracted from video is easily affected by noise, scale drift, and low-frequency dominance, resulting in inaccurate time-domain waveforms. In the frequency domain, the spectrum is prone to distortion due to limitations in frame rate, resolution, and algorithmic approximations. Commonly used methods to improve the accuracy of video vibration measurements often only consider isolated domains. Time-domain methods (such as Kalman filtering and moving averages) struggle to recover high-frequency components lost during video acquisition. Frequency-domain methods (such as bandpass filtering) typically disrupt phase continuity, leading to waveform distortion. Summary of the Invention

[0005] To address the aforementioned shortcomings in existing technologies, this invention provides a video micro-vibration signal correction method that solves the problem of low accuracy caused by environmental factors, noise, and on-site operation when directly extracting micro-vibration information from monitoring videos.

[0006] To achieve the above objectives, the technical solution adopted by this invention is: a video micro-vibration signal correction method, comprising:

[0007] Video capture: Vibration sensors are placed on the structure under test to capture video of the structure under test;

[0008] Feature extraction: Based on the video capture results, extract the time-domain and frequency-domain features of the vibration signal;

[0009] Constructing a multimodal fusion calibration model: A multimodal fusion calibration model is constructed by training with time-domain and frequency-domain features. The multimodal fusion calibration model is a hybrid CNN-LSTM network architecture.

[0010] Correction result output: A spectral loss term based on the logarithmic amplitude difference of the Fourier spectrum is introduced, and the constructed multimodal fusion calibration model is used to obtain the dual correction target result of time-domain waveform reconstruction and frequency-domain key mode alignment, thus completing the correction of video micro-vibration signal.

[0011] The beneficial effects of this invention are as follows: This invention proposes a video micro-vibration signal correction method based on a hybrid CNN-LSTM-FFT deep learning architecture. This method uses signals acquired by point sensors as the target, combining the physical characteristics of structural vibration with the time-frequency domain information of the signal to achieve dual correction of the video micro-vibration signal in both the time and frequency domains. First, the time and frequency domain characteristics of the signal, such as displacement, velocity, acceleration, power spectral density (PSD), and short-time statistics, are extracted to construct a time-frequency feature matrix. Second, a hybrid CNN-LSTM model is trained. The CNN layer extracts local motion patterns and denoised signals, while the LSTM layer obtains long-term correlations and modal trends in the vibration sequence, learning the mapping relationship between the video-extracted signal and the sensor-acquired signal. Third, a spectral loss term based on the logarithmic amplitude difference of the Fourier spectrum is introduced to ensure the consistency of key modal frequencies in the corrected signal. This invention improves the data accuracy of non-contact video micro-vibration monitoring of structures through point sensors.

[0012] Furthermore, the time-domain features include:

[0013] Dynamic characteristics:

[0014]

[0015]

[0016] in, Indicates the vibration signal at the 1st n First-order difference of each sampling point Indicates the vibration signal at the 1st n Second-order difference of each sampling point This represents the displacement change of the vibration signal. Indicates the sampling time. , and These represent the vibration signals at... n , n -1、 n -2 displacement of sampling points Indicates the time step;

[0017] Sliding window statistics:

[0018]

[0019]

[0020]

[0021] in, Represents the moving average. Indicates the length of the sliding window. i This represents a local index sampling point within the sliding window. Indicates the first i The displacement values ​​of sampling points, i∈[1,n] Represents variance. Indicates window length The maximum value of the internal sampling point n;

[0022] Cumulative displacement:

[0023]

[0024] in, This represents the energy drift characteristics of a signal quantized by time accumulation.

[0025] Autocorrelation function:

[0026]

[0027] in, Indicates calculation lag q Signal similarity at each step Total signal length.

[0028] The beneficial effect of the above-mentioned further scheme is that, based on the calculation of the above time-domain characteristics, time-domain characteristic parameters are provided for signal calibration.

[0029] Furthermore, the frequency domain features include:

[0030] Fourier amplitude spectrum:

[0031]

[0032]

[0033]

[0034] in, Represents the eigenvectors of the amplitude spectrum. This represents the complex magnitude at the k-th frequency point of the Discrete Fourier Transform (DFT). To obtain the frequency domain amplitude spectrum of the complex modulus, K max This indicates the preset maximum effective frequency index. Indicates a windowing signal. j Represents the imaginary unit. The denot represents the Hamming window, N represents the total signal length, and n represents the sampling points.

[0035] Power spectral density:

[0036]

[0037] in, Represents the power spectral density. T Indicates the sampling time. Represents energy in the time domain. This represents the transpose of x after its Fourier transform.

[0038] The beneficial effect of the above-mentioned further scheme is that, through the above calculation of frequency domain characteristics, frequency domain characteristic parameters are provided for signal calibration.

[0039] Furthermore, the process of training and constructing a multimodal fusion calibration model using time-domain and frequency-domain features specifically involves:

[0040] Vibration signal depth features are automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN).

[0041] The depth features of the vibration signal are fused with time-domain and frequency-domain features;

[0042] Based on the fused feature vectors, a two-layer long short-term memory network (LSTM) is used to establish long-term time-dependent branches to model long-term dependencies, thereby learning the high-dimensional nonlinear mapping relationship between video extraction signals and point sensor acquisition signals, and completing the construction of a multimodal fusion calibration model.

[0043] The beneficial effects of the above-mentioned further scheme are: using 1D CNN to identify the local details and global trends of the signal respectively, and then using LSTM to solve the problem that traditional CNN is insufficient in modeling periodic features.

[0044] Furthermore, the vibration signal depth features automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN) specifically include:

[0045] For the input vibration displacement signal sequence, a symmetrically filled one-dimensional convolutional kernel is used in the short-time dynamic feature extraction layer to capture high-frequency components and transient impacts through local weighted summation and nonlinear mapping.

[0046] Based on the captured high-frequency components and transient impacts, batch normalization processing is performed to obtain the normalized high-frequency transient characteristics.

[0047] In the long-term trend modeling layer, low-frequency envelope and slow-varying drift features are extracted by diffusing convolution kernel and regularization to obtain low-frequency drift features. Among them, the vibration signal depth features include high-frequency transient features and low-frequency drift features.

[0048] The vibration signal depth features are concatenated with the extracted time-domain and frequency-domain features along the channel dimension to obtain a fused feature vector, which is then input into a two-layer long short-term memory network LSTM.

[0049] Furthermore, the expression for the high-frequency transient characteristics is as follows:

[0050]

[0051]

[0052]

[0053] in, Indicates high-frequency transient characteristics, and All represent affine transformation parameters. This represents the nth feature value output by the l-th convolutional layer. Represents variance. This represents the mean. Represents the numerical stability constant. d Indicates the feature index dimension. D Represents a dataset, Indicates the convolution kernel weights. Represents the linear rectification activation function. This represents the effective frequency index, and c represents the number of convolution channels. This represents the input vibration signal. Indicates time, This represents the bias term of layer 1; N represents the total signal length.

[0054] The expression for the low-frequency drift characteristic is as follows:

[0055]

[0056] in, Indicates low-frequency drift characteristics, This represents a regularization operation. This represents a trainable weight tensor. This represents the input features, specifically high-frequency transient features. This represents the bias term of the second layer.

[0057] The beneficial effect of the above-mentioned further scheme is that the accuracy of the high-frequency range is improved through the above calculations.

[0058] Furthermore, the expression for the fused feature vector is as follows:

[0059]

[0060] in, This represents the fused feature vector. Indicates the depth characteristics of the vibration signal. Representing time-domain features, Represents frequency domain characteristics, R Indicates feature splicing, , and Both represent feature dimensions.

[0061] The beneficial effect of the above-mentioned further scheme is that it fuses the time-domain and frequency-domain features of the signal and uses them as input for training the model.

[0062] Furthermore, the specific requirements for using a two-layer Long Short-Term Memory (LSTM) network to establish long-term dependency branches and model long-term dependencies are as follows:

[0063] The first-layer long short-term memory network LSTM is used to receive the fused dimensional feature vector, and key temporal patterns are filtered through a gating mechanism. The output is a hidden state sequence containing temporal context information. The first-layer long short-term memory network LSTM can learn and maintain the long-term spatiotemporal dependence of the current vibration sequence.

[0064] The second-layer Long Short-Term Memory (LSTM) network is used to perform high-order temporal evolution modeling with 256-dimensional hidden states, and the final high-order temporal feature representation is output. The processing results of the first and second-layer LSTM networks are further spliced ​​with the modal energy distribution information of the frequency domain power spectral density to obtain the high-dimensional nonlinear mapping relationship between the video extraction signal and the point sensor signal.

[0065] The beneficial effect of the above-mentioned further scheme is that, through the above design, the calibration model is trained to obtain more accurate calibration results.

[0066] Furthermore, the expression for the loss function of the multimodal fusion calibration model is as follows:

[0067]

[0068]

[0069]

[0070] in, This represents the loss function of the multimodal fusion calibration model. This represents the time-domain weighted mean square error loss. Indicates frequency domain spectral loss, and These represent the weighting coefficients for the time-domain loss and the frequency-domain loss, respectively. Represents the L2 regularization coefficient. This represents the L2 norm of the multimodal fusion calibration model. This represents the measured amplitude of the video micro-vibration signal spectrum. This represents the predicted amplitude of the video micro-vibration signal spectrum. and These represent the mean values ​​of the measured and predicted spectral amplitudes, respectively. This represents the total sample time. Indicates the sampling time. Represents time-domain weights, This represents the amplitude of the measured video micro-vibration signal spectrum. Indicates the amplitude of the predicted signal spectrum. k Indicates the valid frequency point index.

[0071] The beneficial effect of the above-mentioned further solution is that the video signal can be corrected through the calculation of the above loss function. Attached Figure Description

[0072] Figure 1 This is a flowchart of the method of the present invention.

[0073] Figure 2 This is a flowchart of the processing flow of the present invention. Detailed Implementation

[0074] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0075] Example

[0076] Based on the aforementioned background technology, this invention proposes a video micro-vibration signal correction method based on a hybrid CNN-LSTM-FFT deep learning architecture. This method uses signals acquired by point sensors as the target, combining the physical characteristics of structural vibration with the time-frequency domain information of the signal to achieve dual correction of the video micro-vibration signal in both the time and frequency domains. First, the time and frequency domain characteristics of the signal, such as displacement, velocity, acceleration, power spectral density (PSD), and short-time statistics, are extracted to construct a time-frequency feature matrix. Second, a hybrid CNN-LSTM model is trained. The CNN layer extracts local motion patterns and denoised signals, while the LSTM layer acquires long-term correlations and modal trends in the vibration sequence, learning the mapping relationship between the video-extracted signal and the sensor-acquired signal. Third, a spectral loss term based on the logarithmic amplitude difference of the Fourier spectrum is introduced to ensure the consistency of key modal frequencies in the corrected signal. Finally, the reliability of this method is verified using an old aqueduct in a field environment as the research object, aiming to lay the foundation for the application of accurate video micro-vibration monitoring in large-scale hydraulic structures.

[0077] like Figure 1 As shown, the present invention provides a video micro-vibration signal correction method, the implementation method of which is as follows:

[0078] Video capture: Vibration sensors are placed on the structure under test to capture video of the structure under test;

[0079] Feature extraction: Based on the video capture results, extract the time-domain and frequency-domain features of the vibration signal;

[0080] Constructing a multimodal fusion calibration model: A multimodal fusion calibration model is constructed by training with time-domain and frequency-domain features. The multimodal fusion calibration model employs a hybrid CNN-LSTM network architecture.

[0081] Correction result output: A spectral loss term based on the logarithmic amplitude difference of the Fourier spectrum is introduced, and the constructed multimodal fusion calibration model is used to obtain the dual correction target result of time-domain waveform reconstruction and frequency-domain key mode alignment, thus completing the correction of video micro-vibration signal.

[0082] In this embodiment, a multimodal fusion calibration model is constructed by training using time-domain and frequency-domain features, specifically as follows:

[0083] Vibration signal depth features are automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN).

[0084] The depth features of the vibration signal are fused with time-domain and frequency-domain features;

[0085] Based on the fused feature vectors, a two-layer long short-term memory network (LSTM) is used to establish long-term time-dependent branches to model long-term dependencies, thereby learning the high-dimensional nonlinear mapping relationship between video extraction signals and point sensor acquisition signals, and completing the construction of a multimodal fusion calibration model.

[0086] In this embodiment, the vibration signal depth features are automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN), specifically as follows:

[0087] For the input vibration displacement signal sequence, a symmetrically filled one-dimensional convolutional kernel is used in the short-time dynamic feature extraction layer to capture high-frequency components and transient impacts through local weighted summation and nonlinear mapping.

[0088] Based on the captured high-frequency components and transient impacts, batch normalization processing is performed to obtain the normalized high-frequency transient characteristics.

[0089] In the long-term trend modeling layer, low-frequency envelope and slow-varying drift features are extracted by diffusing convolution kernel and regularization to obtain low-frequency drift features. Among them, the vibration signal depth features include high-frequency transient features and low-frequency drift features.

[0090] The vibration signal depth features are concatenated with the extracted time-domain and frequency-domain features along the channel dimension to obtain a fused feature vector, which is then input into a two-layer long short-term memory network LSTM.

[0091] In this embodiment, a two-layer long short-term memory network (LSTM) is used to establish long-term dependency branches to model long-term dependency relationships. The specific requirements are as follows:

[0092] The first-layer long short-term memory network LSTM is used to receive the fused dimensional feature vector, and key temporal patterns are filtered through a gating mechanism. The output is a hidden state sequence containing temporal context information. The first-layer long short-term memory network LSTM can learn and maintain the long-term spatiotemporal dependence of the current vibration sequence.

[0093] The second-layer Long Short-Term Memory (LSTM) network is used to perform high-order temporal evolution modeling with 256-dimensional hidden states, and the final high-order temporal feature representation is output. The processing results of the first and second-layer LSTM networks are further spliced ​​with the modal energy distribution information of the frequency domain power spectral density to obtain the high-dimensional nonlinear mapping relationship between the video extraction signal and the point sensor signal.

[0094] In this embodiment, a two-layer Long Short-Term Memory (LSTM) network architecture is designed to model long-term temporal dependencies, as detailed below:

[0095] The first LSTM layer is used to capture basic long-term patterns and filter information. Its functions are: receiving the fused feature vector sequence; and a long-term dependency modeling mechanism: utilizing its cell states and gating mechanisms (forget gate, input gate, and output gate), this first LSTM layer can selectively retain, update, or forget historical information spanning long time steps. The forget gate determines which old information needs to be discarded, and the input gate determines which new information needs to be added to the cell state. This allows the first LSTM layer to learn and maintain long-term spatiotemporal dependency patterns that are crucial to the current prediction (e.g., the potential impact of specific vibration patterns in the early stages of equipment operation on the current fault state). The output of the first LSTM layer contains a sequence of hidden states containing temporal context information.

[0096] The second LSTM layer is used for high-order temporal evolution modeling and feature deepening. Its function is to receive the hidden state sequence output by the first LSTM layer. Long-term dependency modeling mechanism: Based on the temporal patterns extracted by the first layer, the second LSTM layer further learns more complex and abstract high-order temporal dynamics and evolutionary laws. It inherits the long-term dependency modeling ability of the first LSTM layer and captures dependencies spanning a longer time range in a higher-level feature space (e.g., capturing the feature evolution pattern of the entire gradual process from the inception to the development of a fault). Its hidden state dimension is set to 256, providing sufficient capacity to represent these complex long-term evolutionary features. The output of the second LSTM layer is the final high-order temporal feature representation (256-dimensional hidden state sequence).

[0097] Feature fusion enhancement: In order to construct a more comprehensive feature representation, the high-order temporal features output by the second-layer LSTM (representing long-term evolution modes) are concatenated with the modal energy distribution information extracted from the frequency domain power spectral density (PSD). The frequency domain features provide static distribution information of signal energy at different frequency components, which complements the dynamic long-term evolution features captured by the LSTM.

[0098] Final mapping relationship: The fused feature vector (time-series long-term dependent features + frequency domain energy features) is input into subsequent fully connected layers and other modules. This high-dimensional fused feature integrates the long-term dynamic evolution law of video vibration signal in the time dimension and the energy distribution characteristics in the frequency dimension, so as to more effectively learn the complex high-dimensional nonlinear mapping relationship between video extracted signal (containing spatial and temporal information) and point sensor signal (containing precise physical quantity information), providing strong feature support for the final diagnosis or prediction task.

[0099] In this embodiment, as Figure 2 As shown, the present invention mainly relates to the following:

[0100] (1) Video capture: The vibration sensor is placed on the structure to be tested. The video frame contains the location of the sensor. The sensor signal is used as a reference signal. The vibration signal extracted from the pixel point of the sensor location in the video is used as the signal to be corrected.

[0101] (2) Feature extraction: Extract time-domain and frequency-domain feature indicators of the signal and construct a feature indicator library. Among them, the time-domain indicators include displacement, velocity, acceleration, sliding window function (sliding average, variance, sum maximum, cumulative displacement) and autocorrelation function; the frequency-domain features include Fourier amplitude spectrum and power spectral density.

[0102] (3) Time-frequency domain joint depth analysis: Based on the vibration signal depth features automatically extracted by the hierarchical one-dimensional convolutional neural network 1D CNN, a feature expression that complements local motion details and global dynamic response is constructed; a long-term dependency relationship is modeled by a two-layer long short-term memory network LSTM, in which forward propagation captures vibration propagation and backward propagation models hysteresis effects such as structural damping oscillation; further, the modal energy distribution information of the frequency domain power spectral density (PSD) is spliced ​​to form a cross-domain joint feature matrix.

[0103] (4) Multimodal fusion calibration model: Design a hybrid CNN-LSTM network architecture, use the hierarchical convolution kernel of CNN to extract multi-scale spatiotemporal motion patterns, use LSTM layers to model the long-term spatiotemporal dependence and modal evolution law of vibration sequence, and establish a high-dimensional nonlinear mapping relationship between video extracted signal and point sensor signal.

[0104] (5) Time-frequency domain loss optimization: A spectrum consistency constraint term based on the logarithmic amplitude difference of the Fourier spectrum is introduced. By optimizing the time-frequency domain loss function together, the dual correction objectives of time-domain waveform reconstruction and frequency-domain key mode alignment are achieved.

[0105] In this embodiment, manual feature extraction is performed as follows: To achieve time-frequency domain co-representation of vibration signals, this invention extracts features from discrete signal sequences. Constructing joint eigenvectors It contains two complementary feature sets: one in the time domain and one in the frequency domain.

[0106] In this embodiment, the time-domain features characterize the physical behavior of the vibration signal through the dynamic evolution of the signal waveform and local statistical properties, and extract four key features: dynamic characteristics, sliding window statistics, integral features, and autocorrelation function.

[0107] In this embodiment, the dynamic characteristics are determined by the first-order difference (velocity) of the signal. With second-order difference (acceleration) Characterization, used to capture the changing trend of vibration waveforms and high-frequency impact effects, is calculated using the following formula:

[0108]

[0109]

[0110] in, Indicates the vibration signal at the 1st n First-order difference (velocity) of each sampling point Indicates the vibration signal at the 1st n Second-order difference (acceleration) of each sampling point This represents the displacement change of the vibration signal. Indicates the sampling time. , and These represent the vibration signals at... n , n -1、 n -2 displacement of sampling points Indicates the time step.

[0111] In this embodiment, the sliding window statistics are expressed through a length of... Sliding window calculation of moving average ,variance and maximum value Suppress noise interference and quantize transient response:

[0112]

[0113]

[0114]

[0115] in, Represents the moving average. Indicates the length of the sliding window. i This represents a local index within the sliding window. Indicates the first i The displacement values ​​of sampling points, i∈[1,n] Represents variance. Indicates window length Internal sampling points n The maximum value.

[0116] In this embodiment, the integral feature (cumulative displacement) Energy drift characteristics of signals quantized by time accumulation:

[0117]

[0118] in, This represents the energy drift characteristics of a quantized signal accumulated over time. Indicates the time step. i This represents a local index within the sliding window.

[0119] In this embodiment, the autocorrelation function Then through lag q Similarity analysis of steps reveals periodic components:

[0120]

[0121] in, Indicates calculation lag q Signal similarity at each step Total signal length (number of sampling points) i This represents a local index within the sliding window.

[0122] In this embodiment, to effectively characterize the frequency domain characteristics of non-stationary signals, the present invention extracts two types of features based on Fourier transform: power spectral density (PSD) and amplitude spectrum. Power spectral density reflects the frequency band distribution of signal energy through statistical averaging, suppressing noise interference; amplitude spectrum retains the amplitude details of instantaneous frequency components. The combination of the two can provide complementary frequency domain information for deep learning models.

[0123] In this embodiment, the windowing signal Calculate the Discrete Fourier Transform:

[0124]

[0125] Before keeping The amplitude spectrum eigenvectors of the effective frequency points are:

[0126]

[0127] The power spectral density was estimated using the Welch method:

[0128]

[0129] in, Represents the eigenvectors of the amplitude spectrum. Let X[k] represent the complex amplitude at the k-th frequency point of the Discrete Fourier Transform (DFT), and let X[k] be the frequency domain amplitude spectrum with the complex modulus. This indicates the preset maximum effective frequency index. Indicates a windowing signal. j Represents the imaginary unit. Indicates a Hamming window. Represents the power spectral density. T Indicates the sampling time. Represents energy in the time domain. This represents the transpose of x after its Fourier transform.

[0130] In this embodiment, deep feature analysis based on 1D-CNN is employed: To overcome the limitations of traditional temporal analysis in representing multi-scale vibration features, this invention designs a hierarchical one-dimensional convolutional neural network architecture. By integrating signal processing theory and deep learning methods, it achieves cross-scale feature extraction from transient impacts to low-frequency drifts. This design utilizes local convolution operations (… K =5) and extended convolutional kernel ( K Using =15 as the core, and combining regularization strategies, a two-level structure is constructed. Its mathematical form and physical meaning are as follows:

[0131] For the input vibration signal sequence The short-term dynamic feature extraction layer uses a symmetrically padded one-dimensional convolutional kernel (same). K =5, c =64), capturing high-frequency components (>50Hz) and transient impacts through local weighted summation and nonlinear mapping:

[0132]

[0133] Output features are processed using batch normalization (BN) to suppress covariate bias:

[0134]

[0135] in, , This represents a trainable weight tensor. The linear rectified activation function is defined as follows: , a This represents the linearly weighted sum of the results of the convolution operation. and All represent affine transformation parameters. This represents the nth feature value output by the l-th convolutional layer. Represents variance. This represents the mean. This represents the numerical stability constant, which is 10. -5 , d Indicates the feature index dimension. D Represents a dataset, This represents a trainable weight tensor. Represents the linear rectification activation function. The effective frequency index 'c' represents the number of convolution channels. This represents the input vibration signal. Indicates time, This represents the bias term of layer 1, where N represents the signal length. Ensure numerical stability. This layer is equivalent to 5 sampling intervals (Δ). t eff =5Δ t The differential filter enhances the sensitivity to transient responses such as mechanical collisions.

[0136] In this embodiment, the long-term trend modeling layer expands the convolutional kernel ( K =15, c =128) and Dropout regularization (ratio 0.2) are used to extract low-frequency envelopes (<10Hz) and slowly varying drift features:

[0137]

[0138] in, Indicates low-frequency drift characteristics, This represents a regularization operation. This represents a trainable weight tensor. This represents the input features, specifically high-frequency transient features. This represents the bias term of the second layer. W (2) ∈R 15×64×128 Dropout (ratio 0.2) enhances generalization ability by randomly masking neurons. The receptive field of this layer is expanded to 15 sampling intervals (Δ). t eff =3×5Δ t Its integral characteristics ( ) and first-level differential operations ( d X / dt This complements the existing technology, simulating multi-resolution analysis in vibration signal processing. The number of channels is increased to 128 dimensions to enhance feature representation capabilities, and Dropout prevents overfitting to noise by randomly masking 20% ​​of neurons.

[0139] Two-stage convolution achieves feature decoupling through a multi-scale collaborative mechanism: short kernels focus on local details (high-frequency transients), while long kernels model global trends (low-frequency drift), reducing the number of parameters (126,080) by two orders of magnitude compared to fully connected networks. This design achieves a balance between computational efficiency and feature completeness, providing high signal-to-noise ratio temporal embeddings for subsequent LSTM temporal modeling and significantly improving the physical consistency of vibration signal calibration.

[0140] In this embodiment, frequency domain feature concatenation is performed: To achieve multimodal representation synergy between temporal dynamic characteristics and frequency domain energy distribution, this invention proposes a channel-dimensional fusion strategy to combine the deep temporal features extracted by a hierarchical one-dimensional convolutional neural network (1D-CNN). , Represents the total sample time, manual time-domain statistical characteristics. and frequency domain characteristics Jointly splice along the channel dimension.

[0141] For each time The fused feature vectors are constructed by concatenation:

[0142]

[0143] in, This represents the fused feature vector. Indicates the depth characteristics of the vibration signal. Representing time-domain features, Represents frequency domain characteristics, R Indicates feature splicing, , and Both represent feature dimensions. In the formula, , as well as Joint representation after fusion Layer normalization and nonlinear projection ( , Indicates the dimension after projection. The weight matrix (representing the nonlinear projection layer) is then input into the bidirectional LSTM module.

[0144] In this embodiment, to accurately model the long-range spatiotemporal dependence of video vibration signals, the present invention employs a two-layer Long Short-Term Memory (LSTM) network to construct a temporal dynamic mapping model. The network input is a fused feature sequence. The network architecture employs a stacked LSTM design: the first LSTM layer receives a 256-dimensional feature vector and filters key temporal patterns through a gating mechanism; the second LSTM layer models higher-order temporal evolution using 256-dimensional hidden states, and the output gate uses the tanh activation function, which has better saturation characteristics. To enhance generalization ability, a random dropout rate of 0.2 is set between layers, and gradient pruning is used to ensure the stability of long sequence training.

[0145] In this embodiment, to jointly optimize the consistency of time-domain waveform matching and frequency-domain energy distribution, the present invention designs a composite loss function, which consists of time-domain weighted mean square error, frequency-domain spectral correlation loss, and L2 regularization term.

[0146] Time-domain weighted mean square error enhances calibration accuracy for transient shock events through dynamic weighting coefficients:

[0147]

[0148] Frequency domain spectral correlation loss constrains energy distribution consistency by normalizing the covariance:

[0149]

[0150] Total loss function integration with frequency domain monitoring signal and regularization constraints:

[0151]

[0152] in, This represents the loss function of the multimodal fusion calibration model. This represents the time-domain weighted mean square error loss. Frequency domain spectral loss, and These represent the weighting coefficients for the time-domain and frequency-domain losses, respectively. Represents the L2 regularization coefficient. This represents the L2 norm (weight decay term) of the multimodal fusion calibration model. k This represents the valid frequency index. This represents the measured amplitude of the signal spectrum. This represents the predicted signal spectral amplitude. and These represent the mean values ​​of the measured and predicted spectral amplitudes, respectively. This represents the total sample time. Indicates the sampling time. This represents the time-domain weight, which is dynamically adjusted according to the signal derivative, assigning a higher penalty weight in the vibration peak region (such as at the moment of mechanical collision). This represents the measured amplitude of the signal at time t. This represents the predicted amplitude of the signal at time t.

Claims

1. A method for correcting video micro-vibration signals, characterized in that, include: Video capture: Vibration sensors are placed on the structure under test to capture video of the structure under test; Feature extraction: Based on the video capture results, extract the time-domain and frequency-domain features of the vibration signal; Constructing a multimodal fusion calibration model: A multimodal fusion calibration model is constructed by training with time-domain and frequency-domain features. The multimodal fusion calibration model is a hybrid CNN-LSTM network architecture. The process of training and constructing a multimodal fusion calibration model using time-domain and frequency-domain features is as follows: Vibration signal depth features are automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN). The depth features of the vibration signal are fused with time-domain and frequency-domain features; Based on the fused feature vectors, a two-layer long short-term memory network LSTM is used to establish long-term time-dependent branches to model long-term dependencies, so as to learn the high-dimensional nonlinear mapping relationship between video extraction signals and point sensor acquisition signals, and complete the construction of the multimodal fusion calibration model. Correction result output: A spectral loss term based on the logarithmic amplitude difference of the Fourier spectrum is introduced, and the constructed multimodal fusion calibration model is used to obtain the dual correction target result of time-domain waveform reconstruction and frequency-domain key mode alignment, thus completing the correction of video micro-vibration signal.

2. The video micro-vibration signal correction method according to claim 1, characterized in that, The time-domain features include: Dynamic characteristics: in, Indicates the vibration signal at the 1st n First-order difference of each sampling point Indicates the vibration signal at the 1st n Second-order difference of each sampling point This represents the displacement change of the vibration signal. Indicates the sampling time. , and These represent the vibration signals at... n , n -1、 n -2 displacement of sampling points Indicates the time step; Sliding window statistics: in, Represents the moving average. Indicates the length of the sliding window. i This represents a local index sampling point within the sliding window. Indicates the first i The displacement values ​​of sampling points, i∈[1,n] Represents variance. Indicates window length The maximum value of the internal sampling point n; Cumulative displacement: in, This represents the energy drift characteristics of a signal quantized by time accumulation. Autocorrelation function: in, Indicates calculation lag q Signal similarity at each step Total signal length.

3. The video micro-vibration signal correction method according to claim 2, characterized in that, The frequency domain features include: Fourier amplitude spectrum: in, Represents the eigenvectors of the amplitude spectrum. This represents the complex magnitude at the k-th frequency point of the Discrete Fourier Transform (DFT). To obtain the frequency domain amplitude spectrum of the complex modulus, K max This indicates the preset maximum effective frequency index. Indicates a windowing signal. j Represents the imaginary unit. The denot represents the Hamming window, N represents the total signal length, and n represents the sampling points. Power spectral density: in, Represents the power spectral density. T Indicates the sampling time. Represents energy in the time domain. This represents the transpose of x after its Fourier transform.

4. The video micro-vibration signal correction method according to claim 1, characterized in that, The vibration signal depth features automatically extracted using a hierarchical one-dimensional convolutional neural network (1D CNN) are specifically as follows: For the input vibration displacement signal sequence, a symmetrically filled one-dimensional convolutional kernel is used in the short-time dynamic feature extraction layer to capture high-frequency components and transient impacts through local weighted summation and nonlinear mapping. Based on the captured high-frequency components and transient impacts, batch normalization processing is performed to obtain the normalized high-frequency transient characteristics. In the long-term trend modeling layer, low-frequency envelope and slow-varying drift features are extracted by diffusing convolution kernel and regularization to obtain low-frequency drift features. Among them, the vibration signal depth features include high-frequency transient features and low-frequency drift features. The vibration signal depth features are concatenated with the extracted time-domain and frequency-domain features along the channel dimension to obtain a fused feature vector, which is then input into a two-layer long short-term memory network LSTM.

5. The video micro-vibration signal correction method according to claim 4, characterized in that, The expression for the high-frequency transient characteristics is as follows: in, Indicates high-frequency transient characteristics, and All represent affine transformation parameters. Indicates the first The output of the first layer of convolution 1 eigenvalue, Represents variance. This represents the mean. Represents the numerical stability constant. d Indicates the feature index dimension. D Represents a dataset, Indicates the convolution kernel weights. Represents the linear rectification activation function. This represents the effective frequency index, and c represents the number of convolution channels. This represents the input vibration signal. Indicates time, This represents the bias term of layer 1; N represents the total signal length. The expression for the low-frequency drift characteristic is as follows: in, Indicates low-frequency drift characteristics, This represents a regularization operation. This represents a trainable weight tensor. This represents the input features, specifically high-frequency transient features. This represents the bias term of the second layer.

6. The video micro-vibration signal correction method according to claim 1, characterized in that, The expression for the fused feature vector is as follows: in, This represents the fused feature vector. Indicates the depth characteristics of the vibration signal. Representing time-domain features, Represents frequency domain characteristics, R Indicates feature splicing, , and Both represent feature dimensions.

7. The video micro-vibration signal correction method according to claim 1, characterized in that, The specific requirements for using a two-layer long short-term memory network (LSTM) to establish long-term dependency branches and model long-term dependencies are as follows: The first-layer long short-term memory network LSTM is used to receive the fused dimensional feature vector, and key temporal patterns are filtered through a gating mechanism. The output is a hidden state sequence containing temporal context information. The first-layer long short-term memory network LSTM can learn and maintain the long-term spatiotemporal dependence of the current vibration sequence. The second-layer Long Short-Term Memory (LSTM) network is used to perform high-order temporal evolution modeling with 256-dimensional hidden states, and the final high-order temporal feature representation is output. The processing results of the first and second-layer LSTM networks are further spliced ​​with the modal energy distribution information of the frequency domain power spectral density to obtain the high-dimensional nonlinear mapping relationship between the video extraction signal and the point sensor signal.

8. The video micro-vibration signal correction method according to claim 1, characterized in that, The expression for the loss function of the multimodal fusion calibration model is as follows: in, This represents the loss function of the multimodal fusion calibration model. This represents the time-domain weighted mean square error loss. Indicates frequency domain spectral loss, and These represent the weighting coefficients for the time-domain loss and the frequency-domain loss, respectively. Represents the L2 regularization coefficient. This represents the L2 norm of the multimodal fusion calibration model. This represents the measured amplitude of the video micro-vibration signal spectrum. This represents the predicted amplitude of the video micro-vibration signal spectrum. and These represent the mean values ​​of the measured and predicted spectral amplitudes, respectively. This represents the total sample time. Indicates the sampling time. Represents time-domain weights, This represents the amplitude of the measured video micro-vibration signal spectrum. Indicates the amplitude of the predicted signal spectrum. k Indicates the valid frequency point index.

Images