A floating non-structural component earthquake damage identification method based on visual elements

Abstract

The application discloses a kind of based on visual element's floating non-structural member earthquake damage identification method, belong to the technical field of earthquake engineering;Including the following steps: with double flow length time sequence convolution video understanding model as foundation to build the earthquake damage identification system of floating non-structural member, the model is composed of three-channel convolutional neural network and bidirectional long time sequence prediction model;Spatial time sequence information is extracted using three-channel convolutional neural network, and the feature map and time sequence information are built using bidirectional long time sequence prediction model;Finally, the feature fusion of two kinds of ways is carried out, so as to realize the class prediction and damage state identification of floating non-structural member in earthquake damage video.The application method combines double flow video understanding model and long time sequence convolution processing model, utilizes the feature fusion of spatial information and time sequence information, strengthens the information sharing and fusion between each video frame, and makes more accurate judgment on the post-disaster situation of building.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of earthquake engineering, and particularly relates to a floating non-structural component earthquake damage identification method based on visual elements. BACKGROUND

[0002] Floating non-structural components such as tables, chairs, shelves, electrical appliances and furniture in buildings are vulnerable and sensitive due to their unique characteristics, and are easily damaged in earthquakes. In addition, such floating non-structural components are usually more expensive than structural components. Therefore, when an earthquake occurs, various floating non-structural components in the room will slide or collapse under the action of strong earthquakes, directly or indirectly causing a series of serious consequences such as property loss, loss of function and casualties. On the other hand, this also has a serious impact on the recovery of building functions after an earthquake. Therefore, real-time detection and damage state identification of various floating non-structural components in the room is an important link to protect life safety and control economic losses in engineering seismic resistance.

[0003] The current earthquake damage assessment technology usually involves structural health monitoring systems, unmanned aerial vehicle images, remote sensing quantitative assessment and other technical means. These technologies usually use the captured damage of the appearance of the structure or building to make a macro judgment of the post-disaster situation. However, in the case of a clear building outline, clear edges or no obvious damage to the structural components, the existing earthquake damage assessment technology cannot make a reasonable judgment on the damage state of the floating non-structural components inside the building.

[0004] With the development of deep learning, the computer vision technology has the obvious advantage of directly obtaining target dynamic information from images or videos, and has been widely concerned. This artificial intelligence algorithm based on visual elements combined with video monitoring inside the building provides a new direction for the current earthquake damage assessment technology research. The dynamic information captured by the video monitoring inside the building is used as the input data of the video understanding model for training, which can directly obtain the exact position and damage state of the floating non-structural components inside the building, so as to achieve the purpose of earthquake damage assessment, which can greatly make up for the shortcomings of the existing technology of earthquake damage assessment only through the changes of the structure or building appearance. However, this idea and technical path of using video understanding model to identify the damage state of non-structural components inside the building to achieve earthquake damage assessment lacks research and development due to the complexity and cross of the algorithm theory and technical field involved. SUMMARY

[0005] The application aims at overcoming the problems in the prior art and providing a floating non-structural component earthquake damage identification method based on visual elements, which combines a three-channel convolutional neural network and a bidirectional long time sequence prediction model, uses feature fusion of spatial information and time sequence information, strengthens information sharing and fusion between video frames, and makes a more accurate judgment on the post-disaster situation of buildings.

[0006] To achieve the above-mentioned purpose, the application adopts the following technical scheme: a floating non-structural component earthquake damage identification method based on visual elements, comprising the following steps:

[0007] S1, constructing an earthquake damage identification system for floating non-structural components:

[0008] The earthquake damage identification system for floating non-structural components is constructed based on a double-flow long time sequence convolutional video understanding model, wherein the double-flow long time sequence convolutional video understanding model is composed of a three-channel convolutional neural network and a bidirectional long time sequence prediction model, and the system comprises an identification module and a pre-training module;

[0009] S2, using the identification module in the system to collect and pre-process earthquake damage videos:

[0010] The identification module is composed of a monitoring video collection unit, a preprocessing unit, an input unit, a double-flow long time sequence convolutional video understanding model and an output unit, the monitoring video collection unit is used to collect in real time earthquake damage videos of various floating non-structural components in buildings during an earthquake, the collected videos are stored and transmitted to the preprocessing unit in real time, the preprocessing unit is used to sample and denoise the real-time collected earthquake damage video frames at two frame rates of fast and slow, and the input unit is used to transmit the video frames between the preprocessing unit and the double-flow long time sequence convolutional video understanding model in real time;

[0011] S3, using the pre-training module in the system to train the double-flow long time sequence convolutional video understanding model:

[0012] The pre-training module is composed of a data set preparation, a double-flow long time sequence convolutional video understanding model and a model pre-training weight, the earthquake damage video frames collected in step S2 are labeled and sorted as various floating non-structural component video data sets with damage state and category labels by the data set preparation, and then input to the double-flow long time sequence convolutional video understanding model for training, so as to obtain the best pre-training weight suitable for the model and having the minimum loss value;

[0013] S4, loading the best pre-training weight obtained from the pre-training module into the double-flow long time sequence convolutional video understanding model of the identification module, and classifying the floating non-structural components in the real-time collected earthquake damage video frames and identifying the damage state:

[0014] S41) Feature extraction of spatial and temporal information is performed using a three-channel convolutional neural network:

[0015] The video frame of slow frame rate processed by the preprocessing unit in step S2 is input into the low frame rate convolutional neural network to capture spatial semantic information, the video frame of fast frame rate is input into the high frame rate convolutional neural network to capture motion information with fine time resolution, and the features of the two branches are fused through lateral connection, and the fused high-level semantic features are used as a separate branch to forward propagate, and the effective feature layers containing spatial semantic information, motion information and fused high-level semantic features extracted by three channels are used to build spatial and temporal information together;

[0016] S42) Building of temporal relationship is performed using a bidirectional long time sequence prediction model:

[0017] The visual element feature extraction of the seismic damage video frame processed by the preprocessing unit in step S2 is performed by the ViT model, the extracted feature stream is processed by the bidirectional long time sequence prediction model, the residual connection is used to fine-tune the temporal information, and the past periodic information is used to form cross information to optimize the prediction result, and the bidirectional long time sequence prediction model Autoformer is used to model the visual feature layer;

[0018] S43) The effective feature layer and the visual feature layer are input into the feature fusion part of the double-flow long time sequence convolutional video understanding model for fusion of spatial and temporal features:

[0019] The visual feature layer with temporal relationship obtained is used to model the temporal relationship, and the effective feature layer extracted by the three channels is fused, and the fused features are connected to the full connection layer to classify various floating non-structural members and identify the damage state.

[0020] In step S2, the preprocessing unit first samples the seismic damage video frame at two frame rates of fast and slow, then adjusts it to a size of 416x416-1024x1024 suitable for model input, and uses a video denoising method of Gaussian denoising filter, bilateral filter or mean filter to denoise the video frame, eliminating information unrelated to floating non-structural members in the video frame.

[0021] In step S4, the classification of floating non-structural members includes tables and chairs, shelves, electrical appliances and furniture, and the damage state of floating non-structural members includes sliding, overturning, shaking and falling.

[0022] In step S41, the features of the two branches are fused through lateral connection. For the slow path branch with a video size of W×T, a larger step size τ is used to acquire video frames, and its number of channels is set to D. Taking the slow path as the standard, the fast path branch uses a channel size of γW×εT, and therefore uses a smaller step size. The network is kept lightweight by using the number of channels βD; the fusion path is a compromise between the two, with its channel size set to βW×αT and stride set to... The number of channels is set to γD; finally, global average pooling is performed on the outputs of the three paths respectively, and the three pooled effective feature layers are stacked together and propagated forward to jointly build spatial temporal information.

[0023] In step S42, the visual elements of the earthquake damage video frames are extracted using the ViT model. The extracted feature stream is then used to obtain the mapping relationship τ1 between the feature layer and the time frame through a bidirectional long-term series prediction model. The mapping relationship τ2 is obtained by fusing cross information from historical information using residual connections. The mapping relationships obtained using different branches are then fused to obtain a feature element P containing different time-series information. This feature element will be used for the decomposition of hidden sequences in the bidirectional long-term series prediction model.

[0024] In step S43, the spatial-temporal information obtained by the three-channel convolutional neural network is fused with the temporal features obtained by the bidirectional long-time series prediction model. For a feature layer X∈R with length L, width W, and depth H... l*w*h The processing formula is as follows:

[0025] y n =Avgpool(Concat{G1,G2,G3,...,G n})·SeriesDecomp{P1,P2,P3,...P n}, where y n This represents a fused feature containing both visual and temporal information. Avgpool represents average pooling, used to preserve high-dimensional information in the feature layer. Concat represents the feature fusion operation. SeriesDecomp represents time series decomposition, used to mine hidden temporal information in the data. G... 1,2,3,...,n P represents the visual feature layer at different time frames. 1,2,3,...,n It represents the timing information of different time frames in a video stream.

[0026] The beneficial effects of this invention are:

[0027] 1) The method of this invention combines a three-channel convolutional neural network and a bidirectional long-term series prediction model, and uses the feature fusion of spatial and temporal information to enhance information sharing and fusion between various video frames, so as to make a more accurate judgment on the post-disaster condition of buildings. Compared with existing earthquake damage assessment technologies such as UAV photography, structural health monitoring systems and satellite remote sensing, which mainly focus on damage to structural components and building appearance, the degree of earthquake damage inside the building can be effectively assessed simply by analyzing the monitoring video inside the building.

[0028] 2) The dual-stream long temporal convolutional video understanding model in this invention combines the dual-stream method with a temporal model. It uses optical flow and the Autoformer long-time series prediction model to capture the motion relationships and temporal information between video frames. It combines the method of feature extraction of video frames based on the ViT model with the processing of temporal information in the video using a bidirectional long temporal prediction model. This method can solve complex long temporal patterns to a certain extent, break through the bottlenecks of computational efficiency and information utilization, and improve the computational efficiency of the model.

[0029] 3) The method of the present invention combines the advantages of using optical flow maps in video to extract optical flow features in the dual-stream video understanding model with the advantages of using time series prediction models to model the temporal information between video frames. This can improve the accuracy of the video understanding model to a certain extent and enhance the understanding of the damage information of floating non-structural components in earthquake damage videos. Compared with existing video understanding models, it has a significant advantage in high precision.

[0030] 4) This invention can directly use earthquake damage videos captured in real time by indoor monitoring as input to the model, automatically analyze and process the damage information of floating non-structural components in the earthquake damage videos in real time, and make reasonable judgments on the post-disaster situation of buildings. The entire model training process does not require human intervention. Combined with existing earthquake damage assessment technologies that mainly focus on damage to structural components and building appearance, it can significantly reduce labor costs while making more accurate and efficient assessments of post-disaster damage, providing strong technical support for post-earthquake rescue work. Attached Figure Description

[0031] Figure 1 This is a block diagram of the earthquake damage identification system constructed in the method of the present invention;

[0032] Figure 2 This is a flowchart of the earthquake damage identification method of the present invention;

[0033] Figure 3 for Figure 1 Structure diagram of a dual-stream long-temporal convolutional video understanding model;

[0034] Figure 4 for Figure 1A structural diagram of a bidirectional long time sequence prediction model;

[0035] Figure 5 A model parameter diagram of a three-channel convolutional neural network in the embodiment of the present application. DETAILED DESCRIPTION

[0036] The present application will be further explained in conjunction with the accompanying drawings and specific embodiments.

[0037] Embodiment: As shown in the figure, the present application provides a floating non-structural component earthquake damage identification method based on visual elements, including the following steps: Figures 1-5 S1, constructing an earthquake damage identification system for floating non-structural components:

[0038] The earthquake damage identification system for floating non-structural components is constructed based on a double-flow long time sequence convolutional video understanding model, wherein the double-flow long time sequence convolutional video understanding model is composed of a three-channel convolutional neural network and a bidirectional long time sequence prediction model, as shown in the figure, the system includes an identification module and a pre-training module; the identification module is composed of a monitoring video acquisition unit, a preprocessing unit, an input unit, a double-flow long time sequence convolutional video understanding model and an output unit, and the pre-training module is composed of a data set preparation, a double-flow long time sequence convolutional video understanding model and a model pre-training weight.

[0039] Figure 1 S2, using the identification module in the system to collect and preprocess the earthquake damage video:

[0040] Through the monitoring video acquisition unit, the earthquake damage video of various floating non-structural components inside the building at the time of earthquake is collected in real time, the collected floating non-structural component earthquake damage video containing damage state pictures is stored and transmitted to the preprocessing unit in real time, the real-time collected earthquake damage video frames are sampled and denoised at fast and slow frame rates through the preprocessing unit, so as to facilitate the subsequent input into the feature extraction network designed for different frame rate video frames to extract the feature of the non-structural component damage state, then adjust it to the size of 416x416~1024x1024 suitable for model input, and use the video denoising method of Gaussian denoising filter, bilateral filter or mean filter to denoise the video frames, eliminate the information irrelevant to the floating non-structural component in the video frames, and enhance the detectability and reliability of the identification of the non-structural component; the input unit transmits the video frames between the preprocessing unit and the double-flow long time sequence convolutional video understanding model in real time.

[0041] S3, using the pre-training module in the system to train the double-flow long time sequence convolutional video understanding model:

[0042] As shown in the figure, the present application provides a floating non-structural component earthquake damage identification method based on visual elements, including the following steps:

[0043] As shown in the figure, the present application provides a floating non-structural component earthquake damage identification method based on visual elements, including the following steps: Figure 1 and​Figure 2 As shown, the video frames of earthquake damage collected in step S2 are marked and sorted into various floating non-structural component video datasets with damage state and category labels by dataset preparation, and input into the dual-flow long-time sequence convolution video understanding model for training to obtain the best pre-training weight suitable for the model and having the minimum loss value.

[0044] S4, load the best pre-training weight obtained in the pre-training module into the dual-flow long-time sequence convolution video understanding model of the recognition module, and perform category division and damage state recognition on the floating non-structural components in the real-time collected video frames of earthquake damage: the category division of floating non-structural components includes tables, chairs, shelves, electrical appliances and furniture, and the damage state of floating non-structural components includes sliding, overturning, shaking and falling.

[0045] S41) feature extraction of spatial and temporal information using a three-channel convolutional neural network:

[0046] As shown in Figure 3 , the video frames of slow frame rate processed by the preprocessing unit in step S2 are input into the low frame rate convolutional neural network to capture spatial semantic information, and the video frames of fast frame rate are input into the high frame rate convolutional neural network to capture motion information with fine time resolution, and the features of the two branches are fused by lateral connection, and the high-level semantic features of the fused spatial semantic and motion information extracted from different frame rates are forward propagated as separate branches, and the effective feature layers containing spatial semantic information, motion information and fused high-level semantic features extracted by three channels are used to jointly build spatial and temporal information;

[0047] As shown in Figure 5 , the features of the two branches are fused by lateral connection, and the three convolution channels all use the novel convolution structure in ConvNext to replace the original ResNet network structure to process the real-time input of the video stream of earthquake damage into the network. Specifically, a normalization layer is added at the beginning and end of each layer structure to stabilize the training process of the model, and a 2x2 convolution layer and a spatial downsampling layer with a step of 2 are added between each layer and the structure of the network to reduce the parameters of the model and reduce the calculation cost of the model; thus, the spatial semantic information obtained by the low frame rate convolutional neural network, the motion information obtained by the high frame rate convolutional neural network, and the three-channel fused features combining the two are used to jointly build spatial and temporal information.

[0048] S42) building of temporal relationship using a bidirectional long-time sequence prediction model:

[0049] As shown in Figure 4As shown, the visual element features of the video frames after the preprocessing unit in step S2 are extracted by the ViT model, the extracted feature stream is processed by the bidirectional long time series prediction model, the residual connection is used to fine-tune the time series information, and the past periodic information is used to form cross information to optimize the prediction result, and the bidirectional long time series prediction model Autoformer is used to model the time series relationship of the extracted visual feature layer; wherein A represents a self-correlation mechanism based on sequence periodicity, which is used to aggregate similar subsequences from the bottom layer period and explore the dependence between sequences, and S represents a decomposition operation of a time series, which is used as a separate internal structure for step-by-step decomposition of hidden sequences in the entire bidirectional long time series model, and is used as a preprocessing operation of a time series to preprocess the historical sequence.

[0050] The visual element features of the video frames are extracted by the ViT model, the mapping relationship τ1 of the feature layer and the time frame is further obtained by the bidirectional long time series prediction model using the extracted feature stream, and the mapping relationship τ2 of the cross information obtained by the residual connection from the historical information is fused, and the mapping relationships obtained by different branches are fused to obtain a feature element P containing different time series information, which will be used for the decomposition of the hidden sequence in the bidirectional long time series prediction model.

[0051] S43) input the effective feature layer and the visual feature layer into the feature fusion part of the double-flow long time series convolution video understanding model for spatial and time series feature fusion:

[0052] The acquired visual feature layer with time series relationship is used to model the time series relationship, and the effective feature layer extracted by the three-channel is used for feature fusion. After the fusion, the features are classified and the damage state is identified after passing through the full connection layer;

[0053] The spatial time series information obtained by the three-channel convolutional neural network is fused with the time series features obtained by the bidirectional long time series prediction model. For a feature layer X ∈ R l*w*h , the processing formula is:

[0054] y n = Avgpool(Concat{G1,G2,G3,......,G n})·SeriesDecomp{P1,P2,P3,......P n}, wherein y nrepresents the fusion feature containing visual and timing information, Avgpool represents average pooling, Concat represents feature fusion operation, SeriesDecomp represents time series decomposition for mining hidden timing information in data, G 1,2,3,...,n represents the visual feature layer of different time frames, P 1,2,3,...,n represents the timing information of different time frames in the video stream.

[0055] Finally, the damage state information of various floating non-structural components in the room is summarized to evaluate the loss degree of the building function.

[0056] In the method, a three-channel convolutional neural network and a bidirectional long time series prediction model are combined, feature fusion of spatial information and timing information is utilized, information sharing and fusion between video frames are strengthened, and more accurate judgment on the post-disaster situation of the building is made; compared with existing earthquake damage evaluation technologies such as unmanned aerial vehicle photography, structure health monitoring system and satellite remote sensing mainly based on structural components and building appearance damage, the earthquake damage degree inside the building can be effectively evaluated only by analyzing the monitoring video inside the building.

[0057] The above description is only used to illustrate the technical solutions of the present application but not limit the present application, and other modifications or equivalent replacements to the technical solutions of the present application made by those skilled in the art should be covered in the scope of claims of the present application as long as they do not depart from the spirit and scope of the present application.

Claims

1. A visual element-based floating non-structural component damage identification method, characterized in that: The method comprises the following steps: S1, constructing a seismic damage identification system for floating non-structural components: A seismic damage identification system for floating non-structural components is constructed based on a double-flow long-time convolution video understanding model, wherein the double-flow long-time convolution video understanding model comprises a three-channel convolutional neural network and a bidirectional long-time sequence prediction model, and the system comprises an identification module and a pre-training module; S2, collecting and preprocessing the seismic damage video by using the identification module in the system: The identification module comprises a monitoring video collection unit, a preprocessing unit, an input unit, a double-flow long-time convolution video understanding model and an output unit, the monitoring video collection unit is used to collect the seismic damage video of various floating non-structural components in the building in real time when an earthquake occurs, the collected video frames are stored and transmitted to the preprocessing unit in real time, the preprocessing unit is used to sample and denoise the video frames collected in real time at two frame rates of fast and slow, and the input unit is used to transmit the video frames between the preprocessing unit and the double-flow long-time convolution video understanding model in real time; S3, training the double-flow long-time convolution video understanding model by using the pre-training module in the system: The pre-training module comprises a data set preparation, a double-flow long-time convolution video understanding model and a model pre-training weight, the video frames collected in step S2 are labeled and arranged into various floating non-structural component video data sets with damage state and category labels by the data set preparation, and then input into the double-flow long-time convolution video understanding model for training, so as to obtain the best pre-training weight suitable for the model and having the minimum loss value; S4, loading the best pre-training weight obtained in the pre-training module into the double-flow long-time convolution video understanding model of the identification module, and classifying the floating non-structural components in the real-time collected seismic damage video frames and identifying the damage state: S41) using a three-channel convolutional neural network to extract spatial and temporal information: The video frames at the slow frame rate processed by the preprocessing unit in step S2 are input into the low frame rate convolutional neural network to capture spatial semantic information, the video frames at the fast frame rate are input into the high frame rate convolutional neural network to capture motion information with fine time resolution, and the features of the two branches are fused through horizontal connection, the fused high-level semantic features are used as a separate branch for forward propagation, and the effective feature layers containing spatial semantic information, motion information and fused high-level semantic features extracted by the three channels are used to build spatial and temporal information; S42) using a bidirectional long-time sequence prediction model to build a time sequence relationship: The ViT model is used to extract the features of the visual elements of the seismic damage video frames processed by the preprocessing unit in step S2, the extracted feature flow is processed by the bidirectional long-time sequence prediction model, the residual connection is used to fine-tune the time sequence information, the past periodic information is used to form cross information to optimize the prediction result, and the bidirectional long-time prediction model Autoformer is used to model the time sequence relationship of the extracted visual feature layer. S43) input the effective feature layer and the visual feature layer into the feature fusion part of the dual-flow long-time sequence convolution video understanding model for fusion of spatial and time sequence features: The visual feature layer with time sequence relationship obtained is used to model the time sequence relationship, and the effective feature layer extracted through the three channels is used for feature fusion, and the fused features are used for classification of various floating non-structural members and identification of damage states after passing through the full connection layer.

2. The visual element-based floating non-structural component seismic damage identification method according to claim 1, wherein: In the step S2, the pre-processing unit first samples the seismic damage video frames at two frame rates of fast and slow, then adjusts them to a size of 416*416~1024*1024 suitable for model input, and uses a video denoising method of Gaussian denoising filter, bilateral filter or mean filter to denoise the video frames, and eliminates the information irrelevant to the floating non-structural members in the video frames.

3. The visual element-based floating non-structural component seismic damage identification method of claim 1, wherein: In the step S4, the classification of the floating non-structural members includes tables and chairs, shelves, electrical appliances and furniture, and the damage state of the floating non-structural members includes sliding, overturning, shaking and falling.

4. The visual element-based floating non-structural component seismic damage identification method of claim 1, wherein: In the step S41, the features of the two branches are fused by transverse connection, wherein a larger step τ is used to collect video frames for the slow path branch with a video size of WxT, and the channel number thereof is set as D; the fast path branch uses a channel size of γWxεT as a standard of the slow path, and thus a smaller step is used to maintain the light weight of the network; the fusion path uses a compromise of both, the channel size thereof is set as βWxαT, and the step is set as The channel number is set as γD; finally, the outputs of the three paths are respectively subjected to global average pooling, and then the three pooled effective feature layers are stacked together and forward propagated to jointly build spatial timing information.

5. The visual element-based floating non-structural component seismic damage identification method of claim 1, wherein: In the step S42, the visual element feature of the seismic damage video frame is extracted by the ViT model, the mapping relationship τ1 between the feature layer and the time frame is further obtained by using the extracted feature flow through the bidirectional long-time sequence prediction model, and the mapping relationship τ2 obtained by using the residual connection to fuse the cross information from the historical information, and the mapping relationships obtained by using different branches are fused, thereby obtaining the feature elements P containing different time sequence information, which will be used for decomposition of the hidden sequence in the bidirectional long-time sequence prediction model.

6. The visual element-based floating non-structural component seismic damage identification method of claim 1, wherein: In the step S43, the spatial and timing information obtained by the three-channel convolutional neural network is fused with the timing characteristics obtained by the bidirectional long time series prediction model. For a feature layer Xe R l *w*h The processing formula is: y n =Avgpool(Concat{G1,G2,G3,......,G n })·SeriesDecomp{P1,P2,P3,......P n }, where y n represents the fusion feature containing visual and temporal information, Avgpool represents the average pooling for preserving high-dimensional information in the feature layer, Concat represents the feature fusion operation, SeriesDecomp represents the time series decomposition for mining the hidden temporal information in the data, G 1,2,3,...,n represents the visual feature layer of different time frames, P 1,2,3,...,n represents the temporal information of different time frames in the video stream.

Images