Structural health monitoring video restoration method based on flexible deep learning
By combining block-by-block compressed sampling with a flexible deep learning network, the problems of data loss and noise in structural health monitoring videos are solved, achieving high-precision video recovery and displacement estimation, adapting to different engineering monitoring scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANTONG MARINE ADVANCED RESEARCH INSTITUTE SOUTHEAST UNIVERSITY
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-29
AI Technical Summary
Existing methods for restoring structural health monitoring videos have shortcomings in terms of data storage, data loss, and data corruption. In particular, they fail to effectively mine the spatiotemporal correlation of 3D videos, resulting in inaccurate data recovery and sensitivity to environmental noise. Existing deep learning methods have also failed to effectively address different types and degrees of data corruption.
By employing a block-by-block compression sampling method and a flexible deep learning network, the sampling matrix and the restoration process are jointly optimized through convolutional layers. Combined with a pseudo-inverse matrix and a deep compensation network, environmental noise is eliminated and spatiotemporal correlation is enhanced to restore structural health monitoring videos.
Even under extreme conditions where the retention rate is less than one ten-thousandth, it can still reproduce videos with high fidelity, significantly reduce hardware dependence, adaptively remove complex environmental noise, achieve sub-pixel-level high-precision displacement estimation, and has good generalization performance.
Smart Images
Figure CN122115245A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of video restoration technology, and in particular to a method for restoring structural health monitoring videos based on flexible deep learning. Background Technology
[0002] Structural health monitoring has shown great potential in civil and infrastructure engineering, enabling damage identification and low-cost maintenance. Among structural health monitoring methods, vibration-based methods are widely used due to their non-destructive nature, convenience, and ability to diagnose overall structural damage. The core principle of these methods is to use measuring instruments to record the dynamic vibrations of the structure, and then perform qualitative and quantitative analysis on the changes in the physical properties of the structure caused by damage.
[0003] Compared to traditional vibration-based structural health monitoring methods that rely on physically attached sensors (such as displacement gauges and accelerometers), computer vision methods are less costly, easier to implement, and can remotely estimate high-resolution structural vibrations across the entire field simultaneously. In computer vision-based methods, the measured structure is recorded by a camera, and the structure is characterized by relevant pixels within the region of interest (ROI). Structural motion is reflected by the temporal changes of relevant pixels within the ROI. The estimated structural motion can be converted into displacement by multiplying the estimated structural motion by the ratio of the actual structural length to the corresponding number of pixels in the image. Computer vision-based structural health monitoring methods can be divided into target-based and target-free methods. Target-based methods (such as template matching, digital image correlation, and marker tracking) require attaching a high-contrast target to the structural surface as a reference. However, surface preprocessing and target installation can be challenging when the ROI is large or far away. Target-free methods (such as feature point matching, optical flow analysis, and phase-based methods) do not require a specific target, are easier to implement, and can use features such as image intensity changes, texture gradients, and phase shifts as virtual targets. Both methods have been successfully applied to indoor and outdoor structural health monitoring scenarios.
[0004] However, in practical engineering applications, three major problems may arise due to issues with video data transmission equipment: data storage, data loss, and data corruption. The recorded data from structural health monitoring videos is typically in three-dimensional space, including two-dimensional spatial information for each two-dimensional image and one-dimensional information representing the time series. Therefore, the data storage requirement increases by millions of times compared to one-dimensional signals. Furthermore, if images are downsampled to reduce storage, the measured structure is too far away, or the data transmission system is damaged, the region of interest in the structure in the video may have low sampling resolution, leading to data loss, and the calculated temporal structural motion may be inaccurate. On the other hand, because visual methods are highly dependent on indoor and outdoor lighting, they are inevitably susceptible to data corruption due to temporal environmental noise and subsequent calculation errors. Therefore, when assessing structural safety by estimating structural motion, the solutions may be ill-posed and sensitive to errors.
[0005] Three-dimensional structural health monitoring videos typically exhibit strong spatiotemporal correlations. However, existing deep learning-based compressed sensing methods applied to structural health monitoring have failed to exploit the temporal correlations within these videos. Consequently, data recovery may fail when data loss is severe. Furthermore, recorded structural health monitoring videos often contain significant environmental noise across both the temporal and spatial domains, and there is currently no specially designed compressed sensing network to address different types and degrees of data corruption. Summary of the Invention
[0006] Purpose of the invention: To address the above problems, the purpose of this invention is to provide a method for restoring structural health monitoring videos based on flexible deep learning.
[0007] Technical solution: One aspect of the present invention provides a method for restoring structural health monitoring videos based on flexible deep learning, comprising the following steps:
[0008] Step 1: Divide the structural health monitoring video into several frame groups. Each frame group contains one key frame and at least one non-key frame, with the first frame being the key frame and the rest being non-key frames. Use a block-by-block compression sampling method and achieve joint optimization of the sampling matrix and the recovery process through convolutional layers to generate key frame measurement values and non-key frame measurement values.
[0009] Step 2: After stitching the measured values with the corresponding noise level map, input them into the flexible deep network to eliminate environmental noise in the spatial and temporal domains and obtain the denoised measured values.
[0010] Step 3: Convolutional layers are constructed based on pseudo-inverse matrices to perform block-by-block initial recovery of the denoised key frame measurements and non-key frame measurements. Then, the recovery results of each block are stitched together through a combination layer to enhance the intra-frame spatial correlation.
[0011] Step 4: Construct a deep compensation network, utilize the multi-level features of the current frame group keyframes and the next frame group keyframes to compensate for the initial restoration results of non-keyframes, while enhancing the spatiotemporal correlation between frames, and output the final restored structural health monitoring video.
[0012] Furthermore, the specific implementation process of block-by-block compression sampling is as follows:
[0013] Divide each frame into Each size is The block, sampling matrix Each row corresponds to a size of The convolution kernel, through Each convolutional kernel Perform non-overlapping convolutions on the frames with a stride to generate The measured values are represented as follows:
[0014] ,
[0015] ,
[0016] in, For keyframes At sampling rate The measured values below, Non-keyframe At sampling rate The measured values below, and ; Indicates inclusion Each support domain is The filter, Indicates inclusion Each support domain is The filter, Let be the side length of the block.
[0017] Further, step 2 includes:
[0018] Step 21, with dimensions of The measured value frame was reshaped into four sizes. The downsampled frame;
[0019] Step 22, compare the downsampled frame with the noise level map. splicing together to form a size of tensor ;Noise level map Based on the standard deviation of environmental noise A uniform graph;
[0020] Step 23, convert the tensor The input is fed into a convolutional neural network for feature extraction; this convolutional neural network includes... The convolutional layer uses convolution and rectified linear unit operations in the first layer, convolution, batch normalization and rectified linear unit operations in the middle layer, and convolution operation in the last layer. Zero padding is performed after each convolution to keep the feature map size unchanged.
[0021] Step 24: Perform an upsampling operation on the feature map extracted in step 23 to generate a size of... The denoised measurement frames include denoised keyframe measurements and non-keyframe measurements.
[0022] Furthermore, step 3 includes:
[0023] The measurement values of each block in the denoised measurement frame are compressed into blocks of sizes respectively. and The vector is represented as:
[0024] ,
[0025] ,
[0026] in, For the first keyframe The initial recovery result of the block. In non-keyframes The initial recovery result of the block; For inclusion Each support domain is The filter, For inclusion Each support domain is Filter; For the first keyframe The denoising measurement value of the block. For the non-keyframe Denoising measurement of the block;
[0027] The initial recovery results of different blocks are concatenated and combined to generate the initial recovery result of each frame, represented as follows:
[0028] ,
[0029] in, Keyframe or non-keyframe , For size The vector, and for Spatial index; For vector-block reshaping functions, the function transforms the vector-block into a block of size. The vector is transformed into a vector of size. The block; This is a block concatenation function that concatenates all blocks to generate a complete frame. , These represent the number of blocks in the height and width directions of the frame, respectively.
[0030] Furthermore, step 4 includes:
[0031] A depth compensation network is constructed to reconstruct the keyframes. In the depth compensation network, features are first extracted through the input layer, as shown below:
[0032] ,
[0033] Then, deep feature mapping is performed in the intermediate layer, represented as:
[0034] ,
[0035] Finally, image reconstruction is performed at the output layer, and the reconstructed keyframes are represented as follows:
[0036] ,
[0037] in, It is the ReLU activation function. For inclusion Each support domain is The filter, For size The bias vector of the input layer; For iterative index, The maximum number of iterations; because Includes One channel, therefore Include Each support domain is The filter, For size The bias vector of the intermediate layer; since the recovered frame contains only one channel, therefore For a support domain containing only one The filter, Includes One channel, For size The bias vector of the output layer;
[0038] The non-key frames are depth-compensated and restored using key frames from the current frame group and the next frame group to generate the final restored non-key frames. This process first involves feature fusion through the input layer, as shown below:
[0039] ,
[0040] Then, depth-multi-level compensation is performed through intermediate layers, represented as follows:
[0041] ,
[0042] Finally, the image is reconstructed through the output layer, and the reconstructed non-keyframe representation is as follows:
[0043] ,
[0044] in, This is the key frame of the current frame group. This is the keyframe for the next frame group. For the filters of the first layer of the non-keyframe network, For non-keyframe networks Layer filters, For filters in the output layer of non-keyframe networks, The bias of the first layer of the non-keyframe network. This is the compensation bias from the current keyframe for layer 1. This is the compensation bias for the first layer from the next keyframe. For non-keyframe networks Layer bias, For the first The layer is the compensation bias from the current keyframe. For the first The layer is compensated for by the offset from the next keyframe. For the bias of the non-keyframe network output layer, This is the compensation filter for the first layer from the current keyframe. This is the compensation filter for layer 1 from the next keyframe. For the first The layer is a compensation filter from the current keyframe. For the first The layer is a compensation filter from the next keyframe. This is the feature map of the first layer of the current keyframe network. This is the feature map of the first layer of the network for the next keyframe. Non-keyframe networks Feature map of the layer For the current keyframe network Feature map of the layer For the next keyframe network Feature map of the layer For non-keyframe networks Feature map of the layer;
[0045] Reconstructed keyframes Non-keyframes The videos are spliced together in chronological order to create a complete and smooth video.
[0046] Furthermore, the flexible deep network is optimized using the alternating direction multiplier method to minimize the following loss function:
[0047] ,
[0048] in, This indicates the total number of image patches. This represents the mapping function of a flexible deep network, i.e., a neural network model used to perform denoising tasks. Indicates the first A noisy input image patch, i.e., the measurement value to be processed; Indicates the first Noise level map corresponding to each image patch Indicates module parameters, Indicates the first Measurements of each block after noise removal.
[0049] Furthermore, environmental noise includes Gaussian noise and Gaussian spatial variation noise.
[0050] Another aspect of the present invention provides a flexible video convolutional neural network for implementing a structural health monitoring video restoration method based on flexible deep learning, comprising a frame sampling module, a flexible error elimination module, a frame initial restoration module, and a depth compensation restoration module connected in sequence.
[0051] The frame sampling module contains two independent convolutional layers, which are used for block-by-block compressed sampling of key frames and non-key frames, respectively.
[0052] The flexible error elimination module includes a reversible downsampling layer, a tensor splicing layer, a multi-level convolutional layer, and an upsampling layer;
[0053] The frame initialization and recovery module includes a pseudo-deconvolutional layer and a combination layer. The convolutional kernel of the pseudo-deconvolutional layer corresponds to the row vector of the pseudo-inverse of the sampling matrix, and the combination layer is used to implement block concatenation.
[0054] The depth compensation and restoration module includes a keyframe depth restoration submodule and a non-keyframe depth compensation submodule, both of which are composed of convolutional layers. The non-keyframe depth compensation submodule also includes a feature fusion layer.
[0055] ,
[0056] Beneficial effects: Compared with the prior art, the significant advantages of this invention are:
[0057] This invention achieves joint optimization of sampling and restoration by constructing an interpretable deep learning framework. Even under extreme conditions where the data retention rate is less than one ten-thousandth, it can still restore videos with high fidelity and significantly reduce hardware dependence. Through a flexible error elimination network, it can adaptively remove various complex environmental noises. Combined with a multi-level feature compensation strategy for keyframes, it fully explores spatiotemporal correlations and ensures that it can still achieve sub-pixel-level high-precision displacement estimation even under severe data loss. In addition, the method has excellent generalization performance and can be accurately applied to various practical engineering monitoring with only a general dataset for training. Attached Figure Description
[0058] Figure 1 This is a flowchart of a health detection video recovery method based on flexible deep learning.
[0059] Figure 2 A flowchart illustrating the video recovery method;
[0060] Figure 3 A flowchart for block-by-block processing;
[0061] Figure 4 This is a flowchart of the processing of flexible deep networks;
[0062] Figure 5 Photographs of cold-formed steel wall structures;
[0063] Figure 6 A qualitative comparison chart of the recovery effects of different methods on cold-formed steel wall structures from video footage;
[0064] Figure 7 A quantitative comparison chart of PSNR evaluation of the denoising effect of cold-formed steel wall structure video under different sampling rates and Gaussian noise standard deviations;
[0065] Figure 8 Comparative graph of SSIM quantitative evaluation of the denoising effect of cold-formed steel wall structure video under different sampling rates and Gaussian noise standard deviations;
[0066] Figure 9 A qualitative comparison chart of the recovery effects of different methods on cold-formed steel wall structures from video footage;
[0067] Figure 10 A quantitative comparison chart of PSNR evaluation of the denoising effect of cold-formed steel wall structure video under different sampling rates and Gaussian spatially varying noise standard deviations;
[0068] Figure 11 A comparative chart of SSIM quantitative evaluation of the denoising effect of cold-formed steel wall structure video under different sampling rates and Gaussian spatial variation noise standard deviations. Detailed Implementation
[0069] The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the embodiments of the present invention, and not all structures.
[0070] In the following description, specific details such as target system architecture and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.
[0071] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.
[0072] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0073] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.
[0074] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include the target features, structures, or characteristics described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.
[0075] Combination Figure 1 and Figure 2 As shown in this embodiment, the structural health monitoring video recovery method based on flexible deep learning includes the following steps:
[0076] Step 1: Divide the structural health monitoring video into several frame groups. Each frame group contains one key frame and at least one non-key frame, with the first frame being the key frame and the rest being non-key frames. Use a block-by-block compression sampling method and achieve joint optimization of the sampling matrix and the recovery process through convolutional layers to generate key frame measurement values and non-key frame measurement values.
[0077] In one example, the structural health monitoring video is divided into several frame groups. Each frame group contains one keyframe and several non-keyframes, with the first frame in each group being the keyframe and the rest being non-keyframes. The keyframe has a higher sampling rate than the other non-keyframes, so the measurements of the keyframes contain more information and are used to compensate for the non-keyframes.
[0078] like Figure 3 As shown, the specific implementation process of block-by-block compression sampling is as follows:
[0079] Each frame is two-dimensional, and each frame is divided into... Each size is The block, It consists of 4 blocks. There are 6 blocks, and the sampling matrix is... Each row corresponds to a size of The convolution kernel, since the convolution of the kernel with the block generates a measurement, therefore each block needs... Each size is The nucleus is generated Each measurement value. Therefore, each frame passes through Each size is convolution kernel, with Perform non-overlapping convolutions on the frames with a stride to generate The measured values are represented as follows:
[0080] ,
[0081] ,
[0082] in, For keyframes At sampling rate The measured values below, Non-keyframe At sampling rate The measured values below, and ; Indicates inclusion Each support domain is The filter, Indicates inclusion Each support domain is The filter, Let be the block side length. To achieve non-overlapping sampling, the compensation for the sampling layer is set to . And it does not contain an activation layer.
[0083] Step 2: After stitching the measured values with the corresponding noise level map, input them into the flexible deep network to eliminate environmental noise in the spatial and temporal domains and obtain the denoised measured values.
[0084] After the frame sampling step, noise in the environment needs to be removed. In this example, a flexible deep network based on an adjustable noise level map P is used, such as... Figure 4 As shown, this method aims to eliminate different types and varying degrees of environmental noise in deep learning-based compressed sensing methods.
[0085] Furthermore, the flexible deep network is optimized using the alternating direction multiplier method to minimize the following loss function:
[0086] ,
[0087] in, This indicates the total number of image patches. This represents the mapping function of a flexible deep network, i.e., a neural network model used to perform denoising tasks. Indicates the first A noisy input image patch, i.e., the measurement value to be processed; Indicates the first Noise level map corresponding to each image patch Indicates module parameters, Indicates the first Measurements of each block after noise removal.
[0088] In one example, the flexible deep network needs to be trained and optimized before use. During training, an input-output pair training dataset is prepared: the input dataset consists of images with varying degrees of Gaussian noise added, and the output dataset consists of clean images. The large dataset of source images includes 400 Berkeley segmentation dataset images, 400 ImageNet database validation dataset images, and 4744 Waterloo Explorers dataset images. In each round, 1,024,000 images of size [missing information] are randomly cropped from these images. The blocks are used for training; cropping the image into smaller blocks reduces the impact of boundary artifacts. Noisy input blocks are added to clean blocks by adding a standard deviation ranging from [value missing]. Gaussian noise is generated; for noisy blocks, the noise level map is set to a uniform map. Furthermore, since the flexible deep network is a fully convolutional neural network with local connectivity, the output block is determined by the type and degree of local noise. Therefore, it can eliminate not only Gaussian noise but also Gaussian spatial variation noise. The loss function is... The initial learning rate is set to When the training error stops decreasing, it drops to When the training error remains constant for 5 consecutive epochs, the parameters of each batch normalization layer are merged into the adjacent convolutional filters, and then a smaller learning rate is applied. An additional 50 training epochs were performed to fine-tune the flexible deep network; the parameters for the alternating direction multiplier method were set to default. More specifically, the training dataset was largely unrelated to the structural health monitoring videos used in this example application, thus allowing for comprehensive verification of the generalization performance of the video restoration method described in this invention in structural health monitoring applications.
[0089] Further, step 2 includes:
[0090] Step 21, with dimensions of The measured value frame was reshaped into four sizes. Downsample the frames to improve efficiency;
[0091] Step 22, compare the downsampled frame with the noise level map. splicing together to form a size of tensor ;Noise level map Based on the standard deviation of environmental noise A uniform graph;
[0092] Step 23, convert the tensor The input is fed into a convolutional neural network for feature extraction; this convolutional neural network includes... The convolutional layer uses convolution and rectified linear unit operations in the first layer, convolution, batch normalization and rectified linear unit operations in the middle layer, and convolution operation in the last layer. Zero padding is performed after each convolution to keep the feature map size unchanged.
[0093] Step 24: Perform an upsampling operation on the feature map extracted in step 23 to generate a size of... The denoised measurement frames include denoised keyframe measurements and non-keyframe measurements.
[0094] Furthermore, environmental noise includes Gaussian noise and Gaussian spatial variation noise.
[0095] Step 3: Convolutional layers are constructed based on pseudo-inverse matrices to perform block-by-block initial recovery of the denoised key frame measurements and non-key frame measurements. Then, the recovery results of each block are stitched together through a combination layer to enhance the intra-frame spatial correlation.
[0096] Each 2D frame consists of independent blocks. Therefore, if each block is resampled and restored independently, the spatiotemporal correlation of the restored frame may be lost. To address this, this example employs a frame initial restoration and depth compensation restoration strategy: frame initial restoration is used to enhance spatial correlation, while depth compensation restoration is used to further enhance spatiotemporal correlation.
[0097] Furthermore, step 3 includes:
[0098] The measurement values of each block in the denoised measurement frame are compressed into blocks of sizes respectively. and The vector is represented as:
[0099] ,
[0100] ,
[0101] in, For the first keyframe The initial recovery result of the block. In non-keyframes The initial recovery result of the block; For inclusion Each support domain is The filter, For inclusion Each support domain is Filter; For the first keyframe The denoising measurement value of the block. For the non-keyframe Denoising measurement of the block;
[0102] To enhance the spatial correlation of blocks within each frame, it is necessary to compensate for the initial reconstruction results of different blocks. Therefore, the initial reconstruction results of different blocks are concatenated and combined to generate the initial reconstruction result of each frame, as follows:
[0103] ,
[0104] in, Keyframe or non-keyframe ,Right now This is the initial reconstruction result of the keyframes. This is the initial recovery result for non-keyframes. For size The vector, and for Spatial index; For vector-block reshaping functions, the function transforms the vector-block into a block of size. The vector is transformed into a vector of size. The block; This is a block concatenation function that concatenates all blocks to generate a complete frame. , These represent the number of blocks in the height and width directions of the frame, respectively.
[0105] Step 4: Construct a deep compensation network, utilize the multi-level features of the current frame group keyframes and the next frame group keyframes to compensate for the initial restoration results of non-keyframes, while enhancing the spatiotemporal correlation between frames, and output the final restored structural health monitoring video.
[0106] In one example, the deep compensation network needs to be trained before use. The training dataset consists of 200 training images and 200 test images from the Berkeley Segmentation Dataset and the Benchmark Dataset 500. These training images are cropped into 89,600 images of size [missing information]. Blocks were used to enhance the spatial correlation of structural health monitoring videos; then, two standard HEVC test video sequences (Cautus and Kimono, resolution) were employed. Training was performed. The frame group size was set to 8; the two training videos were divided into 128,000 frame groups, each block size was [size missing]. To further enhance spatial correlation, especially temporal correlation, the block size in frame sampling is set to... The filter size in the initial frame recovery is set according to different sampling rates; in the depth-compensated recovery, the parameter is set to... , and Both image and video training phases consist of 100 epochs, with learning rates set as follows for the first 50 epochs, epochs 51-80, and epochs 81-100: , and The images and videos used for training were drawn from a general dataset, rather than a specialized structural health monitoring dataset. Video restoration was performed using a trained deep compensation network.
[0107] Furthermore, step 4 includes:
[0108] A depth compensation network is constructed to reconstruct the keyframes. In the depth compensation network, features are first extracted through the input layer, as shown below:
[0109] ,
[0110] Then, deep feature mapping is performed in the intermediate layer, represented as:
[0111] ,
[0112] Finally, image reconstruction is performed at the output layer, and the reconstructed keyframes are represented as follows:
[0113] ,
[0114] in, It is the ReLU activation function. For inclusion Each support domain is The filter, For size The bias vector of the input layer; For iterative index, The maximum number of iterations; because Includes One channel, therefore Include Each support domain is The filter, For size The bias vector of the intermediate layer; since the recovered frame contains only one channel, therefore For a support domain containing only one The filter, Includes One channel, For size The bias vector of the output layer;
[0115] Use the keyframe of the current frame group and the keyframes of the next frame group Depth compensation is performed on non-key frames to generate the final restored non-key frames. This process begins with feature fusion through the input layer, as shown below:
[0116] ,
[0117] Then, depth-multi-level compensation is performed through intermediate layers, represented as follows:
[0118] ,
[0119] Finally, the image is reconstructed through the output layer, and the reconstructed non-keyframe representation is as follows:
[0120] ,
[0121] in, This is the key frame of the current frame group. This is the keyframe for the next frame group. For the filters of the first layer of the non-keyframe network, For non-keyframe networks Layer filters, For filters in the output layer of non-keyframe networks, The bias of the first layer of the non-keyframe network. This is the compensation bias from the current keyframe for layer 1. This is the compensation bias for the first layer from the next keyframe. For non-keyframe networks Layer bias, For the first The layer is the compensation bias from the current keyframe. For the first The layer is compensated for by the offset from the next keyframe. For the bias of the non-keyframe network output layer, This is the compensation filter for the first layer from the current keyframe. This is the compensation filter for layer 1 from the next keyframe. For the first The layer is a compensation filter from the current keyframe. For the first The layer is a compensation filter from the next keyframe. This is the feature map of the first layer of the current keyframe network. This is the feature map of the first layer of the network for the next keyframe. Non-keyframe networks Feature map of the layer For the current keyframe network Feature map of the layer For the next keyframe network Feature map of the layer For non-keyframe networks Feature map of the layer;
[0122] Reconstructed keyframes Non-keyframes The videos are spliced together in chronological order to create a complete and smooth video.
[0123] The method described in this invention is denoted as FVNet. By comparing FVNet with existing video convolutional neural network (VNet) methods, and through different types and degrees of data loss and data corruption problems, the superiority of FVNet in the resampling and restoration of three-dimensional structural health monitoring videos is further illustrated.
[0124] The test video is a shaking table test of a cold-formed steel wall structure, with a frame rate of 24Hz, a frame count of 1000, and a resolution of [missing information]. The cold-formed steel wall structure is a shear-gravity-shear wall, with tie rods at both ends of the shear wall section, fixed to the steel plate in a specific manner. The structure is 4880 mm long and 2740 mm high. Relevant test designs, reports, and data can be found in the reference (Hutchinson T, Schafer B, PK, CFS-NHERI: seismic resiliency of repetitively framed mid-rise cold-formed steel buildings, 2021). This shaking table test was conducted at the University of California, San Diego, to study the lateral mechanical properties of the cold-formed steel wall structure. The 1994 Northridge earthquake (magnitude [missing data]) was used. The CNP196 component, recorded at Canoga Park, served as the input motion for the shaking table test; a pull-wire potentiometer was installed on the side of the transfer beam at the top of the structure to record the reference displacement. A photograph of the cold-formed steel wall structure is shown below. Figure 5 As shown in the figure, the region of interest (ROI) marked in yellow is used to estimate the displacement of the cold-formed steel wall structure using computer vision methods. This is achieved by comparing the structural motion with the ratio of the actual length of the structure to the number of pixels in the video. Multiplying these components converts structural motion into displacement.
[0125] The first simulated environmental noise is Gaussian noise. To simulate different levels of data corruption, the standard deviation of the Gaussian noise is set to 0, 15, 30, 45, and 60, respectively; to simulate different levels of data loss, the sampling rates for non-critical frames are set to 0.01, 0.05, 0.1, and 0.2, respectively, and the sampling rate for critical frames is set to 0.5. Therefore, under the most severe data loss condition, after superimposing severe environmental noise, only data with a standard deviation far lower than 0.5 is retained. The original information. Qualitative recovery of the video, for example... Figure 6 The results are shown only when the standard deviation is 60 and the sampling rate is 0.1. Figure (a) shows the first frame of the original video. As can be seen from Figure (b), when the video is resampled at a sampling rate of 0.1 and Gaussian noise with a standard deviation of 60 is added, most of the information is lost and severely distorted. When using VNet to restore the video, most of the information can be recovered, but environmental noise cannot be eliminated. The restored video is shown in Figure (c), which has strong noise. In contrast, even when only much less than 1% of the information is retained, FVNet can not only restore most of the original information, but also eliminate most of the environmental noise, as shown in Figure (d). This is because FVNet's frame initial restoration and depth compensation restoration enhance the spatiotemporal correlation of non-key frames through key frames, thereby maximizing the recovery of video information; at the same time, FVNet's flexible depth network can eliminate severe environmental noise.
[0126] Quantitative comparison of video recovery and structural displacement estimation Figure 7 and Figure 8 As shown, Figure 7 and Figure 8 The sampling rates for figures (a), (b), (c), and (d) are 0.01, 0.05, 0.1, and 0.2, respectively. For comparative analysis, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) were used to measure the similarity between the restored image and the original image; higher PSNR and SSIM values indicate better restored video quality. Since a video consists of a series of images, the mean PSNR and SSIM values of all images in the video were used for comparison. Figure 7 and Figure 8 As can be seen, when the standard deviation is 0, the accuracy of FVNet and VNet is comparable, with both methods having a mean peak signal-to-noise ratio (PSNR) of approximately 32.8 and a mean structural similarity (SSIM) of approximately 0.95 (FVNet's value is slightly higher). On the other hand, as the standard deviation of Gaussian noise increases, the accuracy of both FVNet and VNet decreases, but the gap between them gradually widens. For example, when the standard deviation increases to 60, the mean PSNR of VNet drops to approximately 19, while that of FVNet drops to approximately 26; the mean SSIM of VNet drops to approximately 0.34, while that of FVNet drops to approximately 0.73. This indicates that as the noise level increases, FVNet exhibits significantly better noise resistance and robustness compared to VNet. This is because in FVNet, the Gaussian noise level can be extended to a large-scale adjustable noise level map M, thus effectively eliminating a wide range of severe environmental noise while enhancing temporal correlation to recover severely lost data.
[0127] The second type of simulated environmental noise is Gaussian spatial variation noise. Specifically, Gaussian noise follows a uniform Gaussian distribution across all image pixels, while Gaussian spatial variation noise is more general, with each pixel exhibiting a different noise level based on its spatial location. Therefore, Gaussian spatial variation noise is more common in practical engineering conditions and more difficult to eliminate. To simulate different degrees of data corruption, the standard deviation of Gaussian spatial variation noise was set to 0, 15, 30, 45, and 60, respectively; to simulate different degrees of data loss, the sampling rates for non-keyframes were set to 0.01, 0.05, 0.1, and 0.2, respectively, and the sampling rate for keyframes was set to 0.5. Qualitative comparisons of video recovery are shown below. Figure 9The results are shown in Figure (only the second non-keyframe result with a standard deviation of 60 and a sampling rate of 0.1 is displayed). Figure (a) shows the first frame of the original video. As can be seen from Figure (b), most information is lost and severely distorted. When using VNet to restore the damaged and resampled video, VNet cannot adaptively eliminate environmental noise. The restored video, as shown in Figure (c), still contains Gaussian spatial variation noise. In contrast, although the distribution of Gaussian spatial variation noise is different from that of Gaussian noise, FVNet can still eliminate this noise. The restored video, as shown in Figure (d), is highly similar to the original video. This is because although the noise level map is uniform, the flexible deep network is a fully convolutional neural network (CNN) with local connectivity characteristics, and the output pixels can be determined by the locally noisy input and the local noise level.
[0128] Quantitative comparison of recovered video, for example Figure 10 and Figure 11 As shown, Figure 10 and Figure 11 The sampling rates in Figures (a), (b), (c), and (d) are 0.01, 0.05, 0.1, and 0.2, respectively. Similar to the results in Gaussian noise scenarios, when the noise standard deviation is 0, the accuracy of FVNet and VNet is comparable, with both methods having a mean PSNR of approximately 32.8 and a mean SSIM of approximately 0.95. As the noise standard deviation increases, the accuracy of both FVNet and VNet decreases, but the difference between them gradually widens. This indicates that FVNet exhibits good accuracy and robustness under varying degrees of Gaussian spatial variation noise. This is because the adjustable noise level map encompasses a wide range of noise levels, thus FVNet can eliminate a large range of Gaussian spatial variation noise.
[0129] Based on the above verification and analysis, although the FVNet training dataset is completely unrelated to the video data from shaking table tests, FVNet can still accurately recover the video data and estimate structural displacements even under different types and degrees of severe data loss and corruption problems. Even with the original information retained being far less than 0.01%, FVNet can still recover structural health monitoring videos with the highest accuracy under different types and degrees of severe data loss and corruption problems, closely resembling the original videos. This further demonstrates that FVNet has good generalization performance in laboratory cases of structural health monitoring applications.
[0130] In other embodiments, a flexible video convolutional neural network is also provided to implement the above-described flexible deep learning-based structural health monitoring video restoration method, including a frame sampling module, a flexible error elimination module, a frame initialization restoration module, and a depth compensation restoration module connected in sequence.
[0131] The frame sampling module contains two independent convolutional layers, which are used for block-by-block compressed sampling of key frames and non-key frames, respectively.
[0132] The flexible error elimination module includes a reversible downsampling layer, a tensor splicing layer, a multi-level convolutional layer, and an upsampling layer;
[0133] The frame initialization and recovery module includes a pseudo-deconvolutional layer and a combination layer. The convolutional kernel of the pseudo-deconvolutional layer corresponds to the row vector of the pseudo-inverse of the sampling matrix, and the combination layer is used to implement block concatenation.
[0134] The depth compensation and restoration module includes a keyframe depth restoration submodule and a non-keyframe depth compensation submodule, both of which are composed of convolutional layers. The non-keyframe depth compensation submodule also includes a feature fusion layer.
Claims
1. A method for restoring structural health monitoring videos based on flexible deep learning, characterized in that, Includes the following steps: Step 1: Divide the structural health monitoring video into several frame groups. Each frame group contains one key frame and at least one non-key frame, with the first frame being the key frame and the rest being non-key frames. Use a block-by-block compression sampling method and achieve joint optimization of the sampling matrix and the recovery process through convolutional layers to generate key frame measurement values and non-key frame measurement values. Step 2: After stitching the measured values with the corresponding noise level map, input them into the flexible deep network to eliminate environmental noise in the spatial and temporal domains and obtain the denoised measured values. Step 3: Convolutional layers are constructed based on pseudo-inverse matrices to perform block-by-block initial recovery of the denoised key frame measurements and non-key frame measurements. Then, the recovery results of each block are stitched together through a combination layer to enhance the intra-frame spatial correlation. Step 4: Construct a deep compensation network, utilize the multi-level features of the current frame group keyframes and the next frame group keyframes to compensate for the initial restoration results of non-keyframes, while enhancing the spatiotemporal correlation between frames, and output the final restored structural health monitoring video.
2. The structural health monitoring video recovery method based on flexible deep learning according to claim 1, characterized in that, The specific implementation process of block-by-block compression sampling is as follows: Divide each frame into Each size is The block, sampling matrix Each row corresponds to a size of The convolution kernel, through Each convolutional kernel The stride is used to perform non-overlapping convolutions on the frames to generate The measured values are represented as follows: ， ， in, For keyframes At sampling rate The measured values below, Non-keyframe At sampling rate The measured values below, and ; Indicates inclusion Each support domain is The filter, Indicates inclusion Each support domain is The filter, Let be the side length of the block.
3. The structural health monitoring video recovery method based on flexible deep learning according to claim 2, characterized in that, Step 2 includes: Step 21, with dimensions of The measured value frame was reshaped into four sizes. The downsampled frame; Step 22, compare the downsampled frame with the noise level map. splicing together to form a size of tensor ;Noise level map Based on the standard deviation of environmental noise A uniform graph; Step 23, convert the tensor The input is fed into a convolutional neural network for feature extraction; this convolutional neural network includes... The convolutional layer uses convolution and rectified linear unit operations in the first layer, convolution, batch normalization and rectified linear unit operations in the middle layer, and convolution operation in the last layer. Zero padding is performed after each convolution to keep the feature map size unchanged. Step 24: Perform an upsampling operation on the feature map extracted in step 23 to generate a size of... The denoised measurement frames include denoised keyframe measurements and non-keyframe measurements.
4. The structural health monitoring video recovery method based on flexible deep learning according to claim 3, characterized in that, Step 3 includes: The measurement values of each block in the denoised measurement frame are compressed into blocks of sizes respectively. and The vector is represented as: ， ， in, For the first keyframe The initial recovery result of the block. In non-keyframes The initial recovery result of the block; For inclusion Each support domain is The filter, For inclusion Each support domain is Filter; For the first keyframe The denoising measurement value of the block. For the non-keyframe Denoising measurement of the block; The initial recovery results of different blocks are concatenated and combined to generate the initial recovery result of each frame, represented as follows: ， in, Keyframe or non-keyframe , For size The vector, and for Spatial index; For vector-block reshaping functions, the function transforms the vector-block into a block of size. The vector is transformed into a vector of size. The block; This is a block concatenation function that concatenates all blocks to generate a complete frame. , These represent the number of blocks in the height and width directions of the frame, respectively.
5. The structural health monitoring video recovery method based on flexible deep learning according to claim 4, characterized in that, Step 4 includes: A depth compensation network is constructed to reconstruct the keyframes. In the depth compensation network, features are first extracted through the input layer, as shown below: ， Then, deep feature mapping is performed in the intermediate layer, represented as: ， Finally, image reconstruction is performed at the output layer, and the reconstructed keyframes are represented as follows: ， in, It is the ReLU activation function. For inclusion Each support domain is The filter, For size The bias vector of the input layer; For iterative index, The maximum number of iterations; because Includes One channel, therefore Include Each support domain is The filter, For size The bias vector of the intermediate layer; since the recovered frame contains only one channel, therefore For a support domain containing only one The filter, Includes One channel, For size The bias vector of the output layer; The non-key frames are depth-compensated and restored using key frames from the current frame group and the next frame group to generate the final restored non-key frames. This process first involves feature fusion through the input layer, as shown below: ， Then, depth-multi-level compensation is performed through intermediate layers, represented as follows: ， Finally, the image is reconstructed through the output layer, and the reconstructed non-keyframe representation is as follows: ， in, This is the key frame of the current frame group. This is the keyframe for the next frame group. For the filters of the first layer of the non-keyframe network, For non-keyframe networks Layer filters, For filters in the output layer of non-keyframe networks, For the bias of the first layer of the non-keyframe network, This is the compensation bias from the current keyframe for layer 1. This is the compensation bias for the first layer from the next keyframe. For non-keyframe networks Layer bias, For the first The layer is the compensation bias from the current keyframe. For the first The layer is compensated for by the offset from the next keyframe. For the bias of the non-keyframe network output layer, This is the compensation filter for the first layer from the current keyframe. This is the compensation filter for layer 1 from the next keyframe. For the first The layer is a compensation filter from the current keyframe. For the first The layer is a compensation filter from the next keyframe. This is the feature map of the first layer of the current keyframe network. This is the feature map of the first layer of the network for the next keyframe. Non-keyframe networks Feature map of the layer For the current keyframe network Feature map of the layer For the next keyframe network Feature map of the layer For non-keyframe networks Feature map of the layer; Reconstructed keyframes Non-keyframes The videos are spliced together in chronological order to create a complete and smooth video.
6. The structural health monitoring video recovery method based on flexible deep learning according to claim 1, characterized in that, The flexible deep network is optimized using the alternating direction multiplier method, minimizing the following loss function: ， in, Indicates the total number of image patches. The mapping function represents a flexible deep network, i.e., a neural network model used to perform denoising tasks. Indicates the first A noisy input image patch, i.e., the measurement value to be processed; Indicates the first Noise level map corresponding to each image patch Indicates module parameters, Indicates the first Measurements of each block after noise removal.
7. The structural health monitoring video recovery method based on flexible deep learning according to claim 1, characterized in that, Environmental noise includes Gaussian noise and Gaussian spatial variation noise.
8. A flexible video convolutional neural network for implementing the structural health monitoring video restoration method based on flexible deep learning as described in any one of claims 1 to 7, characterized in that, It includes a frame sampling module, a flexible error elimination module, a frame initial recovery module, and a depth compensation recovery module connected in sequence; The frame sampling module contains two independent convolutional layers, which are used for block-by-block compressed sampling of key frames and non-key frames, respectively. The flexible error elimination module includes a reversible downsampling layer, a tensor splicing layer, a multi-level convolutional layer, and an upsampling layer; The frame initialization and recovery module includes a pseudo-deconvolutional layer and a combination layer. The convolutional kernel of the pseudo-deconvolutional layer corresponds to the row vector of the pseudo-inverse of the sampling matrix, and the combination layer is used to implement block concatenation. The depth compensation and restoration module includes a keyframe depth restoration submodule and a non-keyframe depth compensation submodule, both of which are composed of convolutional layers. The non-keyframe depth compensation submodule also includes a feature fusion layer.