Earthquake building damage degree detection method and system based on multi-modal sensor data fusion

Abstract

The invention belongs to the technical field of earthquake disaster building detection, and particularly discloses an earthquake building damage degree detection method and system based on multi-modal sensor data fusion. Multi-modal sensor equipment is used for collecting multi-dimensional data, and signal preprocessing is performed on the sensor data; a data fusion module is used to enhance the feature representation capability, finally a detection model is constructed based on an improved YOLOv11 algorithm, a data set is used to train the model, and a final earthquake building damage degree detection model is obtained, and the model can detect and evaluate the damage degree of the building in real time. And results are divided into four grades: intact damage, slight damage, major damage and complete damage. The method can be used for rapidly extracting building damage information so as to assist rescue decision making and post-disaster reconstruction work.

Description

Technical Field

[0001] The present invention relates to the technical field of seismic disaster building detection, and specifically relates to a method and system for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion. Background Art

[0002] After a seismic disaster occurs, quickly and accurately assessing the damage degree of buildings is crucial for rescue decision-making and post-disaster reconstruction work. Traditional building damage detection methods mainly rely on a single data source, such as images or videos, which have problems such as low detection accuracy and incomplete information, and may not be able to fully reflect the actual damage situation of buildings. Therefore, there is a need for a system and method for detecting the damage degree of seismic buildings that can integrate multi-dimensional data and improve detection accuracy and efficiency. Summary of the Invention

[0003] To solve the problems existing in the prior art, the present invention provides a method and system for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion to solve the problem of large difficulty in data source matching and fusion in traditional methods.

[0004] To achieve the above object, the present invention provides the following technical solution: A method for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion, comprising the following steps:

[0005] Step S1: Collect image and video data of the earthquake-stricken area through a multi-modal sensor device to construct a multi-modal data set;

[0006] Step S2: Perform signal preprocessing on the data of the multi-modal sensor in a corresponding manner to generate a preprocessed multi-modal data set;

[0007] Step S3: Use the ASFF strategy to perform adaptive feature fusion on the preprocessed data set to obtain a data fusion result;

[0008] Step S4: Construct an improved building damage degree detection model;

[0009] Step S5: Building damage degree assessment: Input the data fused in step S3 into the improved building damage degree detection model for training to obtain a final seismic building damage degree detection model, and use the final seismic building damage degree detection model to perform real-time detection and evaluate the damage degree of the building.

[0010] Preferably, in step S1, the multi-modal sensor device includes:

[0011] An infrared thermal imaging camera for collecting thermal radiation image data of the earthquake area;

[0012] Multi - spectral camera, used to collect visible light and infrared spectral images and video data of the earthquake area;

[0013] Millimeter - wave radar, used to collect three - dimensional terrain information data of the earthquake area and its surrounding areas;

[0014] LiDAR, used to collect building structure point cloud data of the earthquake area.

[0015] Preferably, in step S2, the data of the multi - modal sensors are signal - pre - processed in corresponding ways, specifically including:

[0016] a. For the data of the infrared thermal imaging camera, the image is segmented by setting a temperature threshold to highlight the target of interest and remove noise interference;

[0017] b. For the data of the multi - spectral camera, radiometric correction is performed to eliminate the radiation error caused by the sensor itself and environmental factors, and geometric correction is performed to correct the geometric distortion of the image;

[0018] c. For the data of the millimeter - wave radar, radar signal processing and data association filtering are adopted to analyze the radar echo signal to extract target information, and at the same time, data association and filtering algorithms are used to remove redundant and incorrect data;

[0019] d. For the point cloud data generated by LiDAR, unsupervised clustering analysis is performed to group the point cloud data into different target objects to realize the recognition and detection of objects in the environment.

[0020] Preferably, in step S3, the ASFF strategy is adopted to perform adaptive feature fusion on the pre - processed data set to obtain the data fusion result, specifically including:

[0021] S31. For ASFF - 1, perform 3×3 max - pooling on the 3 - layer feature map with a stride of 2, and then perform 3×3 convolution with a stride of 2 to obtain X 3→1 ; perform 3×3 convolution on the 2 - layer feature map with a stride of 2 to obtain X 2→1 ;

[0022] S32. For ASFF - 2, perform 3×3 convolution on the 3 - layer feature map with a stride of 2 to obtain X 3→2 ; perform 1×1 convolution on the 1 - layer feature map and adjust the result to twice the original image resolution to obtain X 1→2 ;

[0023] S33. For ASFF - 3, perform 1×1 convolution on the 2 - layer feature map and adjust the result to twice the original image resolution to obtain X 2→3 ; perform 1×1 convolution on the 1 - layer feature map and adjust the result to four times the original image resolution to obtain X1→3 ;

[0024] S34. Multiply the features x from different layers with the weight parameters α, β, and γ and then sum them to generate the data fusion result, which is expressed by the formula as follows:

[0025]

[0026] where, represents the vector value at the position ij of the feature map after fusion in the l-th layer, is the weight coefficient corresponding to the l-th layer, respectively represent the vector values at the position ij after adjusting the feature maps from the first layer, the second layer, and the third layer to the resolution of the l-th layer through operations such as upsampling and interpolation.

[0027] Preferably, in step S4, the improved building damage degree detection model specifically refers to: first, introduce the depthwise separable convolutional DSCBAMConv with an attention mechanism into the C3k2 module in the backbone network of the YOLOv11 model to obtain the improved C3k2-DC module; then integrate the DAB dual attention block into the neck network, including the channel-spatial attention module CSAM and the parallel attention module PAM; finally, apply the DCNv4 deformable convolutional network to the detection layer and combine them to construct the improved building damage degree detection model.

[0028] Preferably, in the improved building damage degree detection model, the default SIoU loss function is replaced with the AReLU loss function, specifically: add the function of the ELSA attention mechanism as a residual structure to the original ReLU function to obtain the AReLU loss function;

[0029] The function of the ELSA attention mechanism is represented by a network layer with learnable parameters λ and μ as:

[0030]

[0031] The expression form of the ReLU function is:

[0032]

[0033] The expression form of the AReLU loss function is:

[0034]

[0035] For each input element xi, when the element is less than 0, it is scaled by C(λ); when the element is greater than or equal to 0, it is scaled by (1 + σ(μ)). The clipping operation C is used to limit the value of λ within [0.01, 0.99], and the sigmoid function is used to limit the value of μ within [0, 1].

[0036] Preferably, in the final earthquake building damage degree detection model of step S5, the detection process is as follows:

[0037] First, use the C3k2-DC module to extract the local features of damaged buildings. The input feature map is segmented into multiple channels, processed by two parallel convolutional layers to capture multi-scale features. After each layer of convolution extracts local features and splices them, then adjust the number of channels through 1x1 convolution. After adding the input feature map and the processed feature map element by element, output the first feature map;

[0038] Secondly, input the first feature map into the neck network. First, weight the channel and spatial dimensions by CSAM to highlight important features and regional information, and then enter PAM to integrate global, local channel and spatial attention, comprehensively capture key information, and enhance feature expression; fuse the feature maps processed by CSAM and PAM, and output the second feature map;

[0039] Then, in the detection layer, DCNv4 first pads the second feature map with zeros, calculates the position matrix combining the preset offset and the input offset to determine the sampling point positions, and performs boundary restrictions. According to the sampling point coordinates, calculate the weights and obtain the eigenvalue through bilinear interpolation, sum the weighted eigenvalues to get the offset feature map, and perform convolution through the ordinary convolutional layer, and the output is the final feature map.

[0040] Preferably, in step S5, based on the detection results of the final earthquake building damage degree detection model, evaluate the damage degree of the affected buildings, and divide the results into four levels, and use four iconic colors of red, yellow, blue, and green to represent the different damage degrees of the buildings; red represents complete damage, yellow represents major damage, blue represents minor damage, and green represents intact.

[0041] On the other hand, to achieve the above object, the present invention also provides the following technical solution: An earthquake building damage degree detection system, including the following modules:

[0042] Data collection module: Read the multi-modal data set collected by the multi-modal sensor device in real time, store and import it into the system;

[0043] Recognition parameter configuration module: Define the number of damaged building categories that the building damage degree detection model needs to recognize, count and record the number of each category, set the frame rate for the model to process video and image sequences, and specify the name of the weight file for model inference;

[0044] Recognition parameter adjustment module: Dynamically adjust the model weight file, threshold, confidence level, and recognition delay parameters used in real-time recognition tasks;

[0045] Data preprocessing module: Perform signal preprocessing on data from different sensor devices in corresponding ways to generate a preprocessed multi-modal dataset;

[0046] Data fusion module: Perform adaptive feature fusion on the preprocessed multi-modal dataset using the ASFF feature fusion strategy to obtain the data fusion result;

[0047] Building assessment module: Based on the detection results of the final seismic building damage degree detection model, evaluate the damage degree of the affected buildings, and divide the results into four levels: red - completely damaged, yellow - severely damaged, blue - slightly damaged, green - intact;

[0048] Data storage module: Use cloud computing resources for data storage and processing, and save the analysis results of the seismic building damage degree detection system to the database.

[0049] The beneficial effects of the present invention are as follows: The present invention collects multi-dimensional data through multi-modal sensor devices, performs signal preprocessing on the sensor data, then enhances the feature representation ability through the data fusion module, and finally based on the YOLOv11 algorithm, introduces the depthwise separable convolutional DSCBAMConv with an attention mechanism to optimize the C3k2 module in the backbone network, integrates the DAB double attention block into the neck network, applies the DCNv4 deformable convolutional network to the detection layer, and constructs an improved building damage degree detection model. This model can detect and evaluate the damage degree of buildings in real time, and divide the results into four levels: intact, slightly damaged, severely damaged, and completely damaged. The method of the present invention can achieve rapid and accurate detection of the damage degree of seismic buildings, provide strong support for rescue decision-making and post-disaster reconstruction work, and has important practical application value. Description of the Drawings

[0050] Figure 1 It is a schematic flow chart of the method for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion in the embodiment of the present invention;

[0051] Figure 2 It is a schematic diagram of the building damage degree detection model based on YOLOv11 in the embodiment of the present invention;

[0052] Figure 3 It is a schematic diagram of data fusion in the embodiment of the present invention;

[0053] Figure 4Schematic diagram of the earthquake building damage degree detection system module in the embodiments of the present invention. Detailed implementation manners

[0054] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0055] The present invention provides a technical solution: a method for detecting the damage degree of earthquake buildings based on multi-modal sensor data fusion, as Figure 1 shown, including the following steps:

[0056] Step S1: Collect image and video data of the earthquake-stricken area through multi-modal sensor devices to construct a multi-modal data set.

[0057] The multi-modal sensor devices include an infrared thermal imaging camera, a multi-spectral camera, a millimeter-wave radar, and a lidar, which are used to collect different types of data.

[0058] The infrared thermal imaging camera is used to collect thermal radiation image data of the earthquake area;

[0059] The multi-spectral camera is used to collect visible light and infrared spectral image and video data of the earthquake area;

[0060] The millimeter-wave radar is used to collect three-dimensional terrain information data of the earthquake area and its surroundings;

[0061] The lidar is used to collect building structure point cloud data of the earthquake area.

[0062] Step S2: Perform signal preprocessing on the data of the multi-modal sensors in a corresponding manner to generate a preprocessed multi-modal data set.

[0063] a. For the data of the infrared thermal imaging camera, segment the image by setting a temperature threshold to highlight the target of interest and remove noise interference;

[0064] b. For the data of the multi-spectral camera, perform radiometric correction to eliminate the radiometric error caused by the sensor itself and environmental factors, and perform geometric correction to correct the geometric distortion of the image;

[0065] c. For the data of the millimeter-wave radar, use radar signal processing and data association filtering to analyze the radar echo signal to extract target information, and at the same time use data association and filtering algorithms to remove redundant and incorrect data;

[0066] d. For the point cloud data generated by the lidar, perform unsupervised clustering analysis to group the point cloud data into different target objects, realizing the recognition and detection of objects in the environment.

[0067] Step S3: Use the ASFF strategy to perform adaptive feature fusion on the preprocessed data set to obtain the data fusion result.

[0068] As Figure 3 shown, use the ASFF strategy to perform adaptive feature fusion on the preprocessed data set to obtain the data fusion result, specifically including:

[0069] S31. For ASFF-1, perform 3×3 max pooling on the 3-layer feature map with a stride of 2, and then perform 3×3 convolution with a stride of 2 to obtain X 3→1 ; perform 3×3 convolution on the 2-layer feature map with a stride of 2 to obtain X 2→1 ;

[0070] S32. For ASFF-2, perform 3×3 convolution on the 3-layer feature map with a stride of 2 to obtain X 3→2 ; perform 1×1 convolution on the 1-layer feature map and adjust the result to twice the original image resolution to obtain X 1→2 ;

[0071] S33. For ASFF-3, perform 1×1 convolution on the 2-layer feature map and adjust the result to twice the original image resolution to obtain X 2→3 ; perform 1×1 convolution on the 1-layer feature map and adjust the result to four times the original image resolution to obtain X 1→3 ;

[0072] S34. Multiply and add the features x from different layers with the weight parameters α, β, and γ to generate the data fusion result. The formula is expressed as follows:

[0073]

[0074] Among them, represents the vector value at the position ij of the feature map after fusion of the l-th layer, is the weight coefficient corresponding to the l-th layer, respectively represent the vector values at the position ij after adjusting the feature maps of the first layer, the second layer, and the third layer to the resolution of the l-th layer through operations such as upsampling and interpolation.

[0075] Convert the unnormalized weight values to the normalized weight coefficients α, β, and γ through the softmax function, and their ranges are all in [0,1] and the sum is 1:

[0076]

[0077] Among them, is the unnormalized weight value.

[0078] Step S4: Construct an improved building damage degree detection model; the improved building damage degree detection model specifically refers to: First, introduce the depthwise separable convolutional DSCBAMConv with an attention mechanism into the C3k2 module in the backbone network of the YOLOv11 model to obtain the improved C3k2-DC module; then integrate the DAB dual attention block into the neck network, including the channel-spatial attention module CSAM and the parallel attention module PAM; finally, apply the DCNv4 deformable convolutional network to the detection layer, and after combination, construct the improved building damage degree detection model, as Figure 2 shown. Set the hyperparameters of the model, and select the AReLU loss function to replace the default SIoU loss function. The AReLU activation function dynamically adjusts its behavior through the adaptive parameter α, which is used to measure the activation degree of the input features in the positive and negative parts respectively.

[0079] Replace the default SIoU loss function with the AReLU loss function, specifically: Add the function of the ELSA attention mechanism as a residual structure to the original ReLU function to obtain the AReLU loss function;

[0080] The function of the ELSA attention mechanism is represented by a network layer with learnable parameters λ and μ as:

[0081]

[0082] The expression form of the ReLU function is:

[0083]

[0084] The expression form of the AReLU loss function is:

[0085]

[0086] For each input element xi, when the element is less than 0, it is scaled by C(λ); when the element is greater than or equal to 0, it is scaled by (1 + σ(μ)). The clipping operation C is used to limit the value of λ to the range [0.01, 0.99], and the sigmoid function is used to limit the value of μ to the range [0, 1].

[0087] Step S5: Building damage degree evaluation: Input the data fused in step S3 into the improved building damage degree detection model for training to obtain the final seismic building damage degree detection model, and use the final seismic building damage degree detection model to perform real-time detection and evaluate the damage degree of the building.

[0088] In the final seismic building damage detection model, the detection process is as follows:

[0089] First, use the C3k2-DC module to extract local features of damaged buildings. The input feature map is segmented into multiple channels, processed through two parallel convolutional layers to capture multi-scale features. After local features are extracted by each layer of convolution and then concatenated, the number of channels is adjusted through 1x1 convolution. The input feature map and the processed feature map are added element by element to output the first feature map;

[0090] Second, input the first feature map into the neck network. First, weight the channel and spatial dimensions by CSAM to highlight important features and regional information, and then enter PAM to integrate global, local channel, and spatial attention to comprehensively capture key information and enhance feature expression. The feature maps processed by CSAM and PAM are fused to output the second feature map;

[0091] Then, in the detection layer, DCNv4 first pads the second feature map with zeros, calculates the position matrix combining the preset offset and the input offset to determine the sampling point positions, and performs boundary restrictions. According to the sampling point coordinates, weights are calculated through bilinear interpolation and eigenvalues are obtained. The eigenvalues are weighted and summed to obtain the offset feature map, which is convolved through a common convolutional layer and the output is the final feature map.

[0092] Based on the detection results of the final seismic building damage detection model, evaluate the damage degree of the affected buildings, and divide the results into four levels, represented by four iconic colors: red, yellow, blue, and green, to indicate different damage degrees of the buildings; red indicates complete damage, yellow indicates major damage, blue indicates minor damage, and green indicates intact.

[0093] Based on the same inventive concept as the above method embodiment, the embodiment of the present application also provides a seismic building damage detection system, which can implement the functions provided by the above method embodiment, such as Figure 4 shown, the system includes the following modules:

[0094] Data collection module: Read the multi-modal data set collected by the multi-modal sensor device in real time, store and import it into the system;

[0095] Recognition parameter configuration module: Define the number of damaged building categories that the building damage detection model needs to recognize, count and record the number of each category, set the frame rate for the model to process videos and image sequences, and specify the name of the weight file used for model inference;

[0096] Recognition parameter adjustment module: Dynamically adjust the model weight file, threshold, confidence level, and recognition delay parameters used in real-time recognition tasks;

[0097] Data preprocessing module: Signal preprocessing is performed on data from different sensor devices in corresponding ways to generate a preprocessed multi-modal dataset;

[0098] Data fusion module: Adaptive feature fusion is performed on the preprocessed multi-modal dataset using the ASFF feature fusion strategy to obtain the data fusion result;

[0099] Building assessment module: Based on the detection results of the final seismic building damage degree detection model, the damage degree of the affected buildings is evaluated, and the results are divided into four levels: red - completely damaged, yellow - severely damaged, blue - slightly damaged, green - intact;

[0100] Data storage module: Cloud computing resources are used for data storage and processing, and the analysis results of the seismic building damage degree detection system are saved to the database.

[0101] In the embodiments provided by the present invention, it should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without further limitation, the element defined by the statement "including one..." does not exclude the existence of other identical elements in the process, method, article or device including the said element.

[0102] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "the" and "said" used in the embodiments of the present invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0103] It should be understood that the term " / and" used herein is only a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0104] Depending on the context, the word "if" as used herein can be interpreted as "when" or "while" or "in response to determining" or "in response to detecting". Similarly, depending on the context, the phrase "if determined" or "if detecting (stated condition or event)" can be interpreted as "when determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)".

[0105] The "first / second" mentioned in the embodiments is only used to distinguish similar objects and does not represent a specific order for the objects. It can be understood that the "first / second" can be interchanged in a specific order or sequence when permitted. It should be understood that the objects distinguished by the "first / second" can be interchanged under appropriate circumstances so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0106] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions recorded in the foregoing embodiments or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for detecting the damage degree of earthquake-damaged buildings based on multi-modal sensor data fusion, characterized in that, It includes the following steps: Step S1: Collect image and video data of the earthquake-stricken area through a multimodal sensor device to construct a multimodal dataset; Step S2: Perform signal preprocessing on the data of the multimodal sensor in a corresponding manner to generate a preprocessed multimodal dataset; Step S3: Use the ASFF strategy to perform adaptive feature fusion on the preprocessed dataset to obtain a data fusion result; Step S4: Construct an improved building damage degree detection model; Step S5: Building damage degree evaluation: Input the data fused in Step S3 into the improved building damage degree detection model for training to obtain a final earthquake building damage degree detection model, and use the final earthquake building damage degree detection model to perform real-time detection and evaluate the damage degree of the building.

2. The method for detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to claim 1, characterized in that: In Step S1, the multimodal sensor device includes: An infrared thermal imaging camera for collecting thermal radiation image data of the earthquake area; A multispectral camera for collecting visible light and infrared spectral image and video data of the earthquake area; A millimeter-wave radar for collecting three-dimensional terrain information data of the earthquake area and its surroundings; A lidar for collecting building structure point cloud data of the earthquake area.

3. The method for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion according to claim 1, characterized in that: In Step S2, the corresponding method for signal preprocessing of the multimodal sensor data specifically includes: a. For the data of the infrared thermal imaging camera, segment the image by setting a temperature threshold to highlight the target of interest and remove noise interference; b. For the data of the multispectral camera, perform radiometric correction to eliminate the radiometric error caused by the sensor itself and environmental factors, and perform geometric correction to correct the geometric distortion of the image; c. For the data of the millimeter-wave radar, use radar signal processing and data association filtering to analyze the radar echo signal to extract target information, and at the same time use data association and filtering algorithms to remove redundant and incorrect data; d. For the point cloud data generated by the lidar, perform unsupervised clustering analysis to group the point cloud data into different target objects to realize the recognition and detection of objects in the environment.

4. The method for detecting the damage degree of seismic buildings based on multi-modal sensor data fusion according to claim 1, characterized in that: In Step S3, the use of the ASFF strategy to perform adaptive feature fusion on the preprocessed dataset to obtain a data fusion result specifically includes: S31. For ASFF-1, perform 3×3 max pooling on the 3-layer feature map with a stride of 2, and then perform 3×3 convolution with a stride of 2 to obtain X 3→1 ; perform 3×3 convolution on the 2-layer feature map with a stride of 2 to obtain X 2→1 ; S32. For ASFF-2, perform a 3×3 convolution on the 3-layer feature map with a stride of 2 to obtain X 3→2 ; perform a 1×1 convolution on the 1-layer feature map and adjust the result to twice the original image resolution to obtain X 1→2 ; S33. For ASFF-3, perform 1×1 convolution on the 2-layer feature map and resize the result to twice the original image resolution to obtain X 2→3 ; perform 1×1 convolution on the 1-layer feature map and resize the result to four times the original image resolution to obtain X 1→3 ; S34: Multiply the features x from different layers by the weight parameters α, β, and γ and then add them to generate a data fusion result. The formula is expressed as follows: Among them, represents the vector value of the feature map after fusion at the l-th layer at the position ij, is the weight coefficient corresponding to the l-th layer, respectively represent the vector values at the position ij after the feature maps from the first layer, the second layer, and the third layer are adjusted to the resolution of the l-th layer through operations such as upsampling and interpolation.

5. The method for detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to claim 1, wherein: In Step S4, the improved building damage degree detection model specifically refers to: First, introduce a depthwise separable convolution with attention mechanism DSCBAMConv into the YOLOv11 model to optimize the C3k2 module in the backbone network to obtain an improved C3k2-DC module; then integrate the DAB dual attention block into the neck network, including a channel-spatial attention module CSAM and a parallel attention module PAM; finally, apply the DCNv4 deformable convolution network to the detection layer, and after combination, construct an improved building damage degree detection model.

6. The method for detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to claim 1 or 5, characterized in that: In the improved building damage degree detection model, the AReLU loss function is adopted to replace the default SIoU loss function. Specifically, the function of the ELSA attention mechanism is added as a residual structure to the original ReLU function to obtain the AReLU loss function; The function of the ELSA attention mechanism is represented by a network layer with learnable parameters λ and μ as follows: The expression form of the ReLU function is: The expression form of the AReLU loss function is: For each input element xi, when the element is less than 0, it is scaled by C(λ); when the element is greater than or equal to 0, it is scaled by (1 + σ(μ)).

7. The method for detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to claim 1, characterized in that: In the final earthquake building damage degree detection model in step S5, the detection process is as follows: First, the C3k2-DC module is used to extract the local features of damaged buildings. The input feature map is segmented into multiple channels, processed by two parallel convolutional layers to capture multi-scale features. After the local features are extracted by each layer of convolution and concatenated, the number of channels is adjusted by a 1x1 convolution. After the input feature map and the processed feature map are added element by element, the first feature map is output; Secondly, the first feature map is input into the neck network. First, the CSAM weights the channel and spatial dimensions to highlight important features and regional information, and then enters the PAM to integrate global, local channel and spatial attention, comprehensively capture key information, and enhance feature expression; The feature maps processed by CSAM and PAM are fused to output the second feature map; Then, in the detection layer, DCNv4 first zero-pads the second feature map, calculates the position matrix combining the preset offset and the input offset to determine the sampling point positions, and performs boundary limitation. According to the sampling point coordinates, the weights are calculated by bilinear interpolation and the feature values are obtained. The weighted sum of the feature values is used to obtain the offset feature map, which is convolved by a common convolutional layer, and the output is the final feature map.

8. The method for detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to claim 1, characterized in that: In step S5, based on the detection results of the final earthquake building damage degree detection model, the damage degree of the affected buildings is evaluated, and the results are divided into four levels, which are represented by four iconic colors: red, yellow, blue, and green. Red indicates complete damage, yellow indicates major damage, blue indicates minor damage, and green indicates intact.

9. A system for a method of detecting the damage degree of a seismic building based on multi-modal sensor data fusion according to any one of claims 1-8, characterized in that: It includes the following modules: Data collection module: Real-time reads the multi-modal data set collected by the multi-modal sensor device, stores it and imports it into the system; Recognition parameter configuration module: Defines the number of damaged building categories that the building damage degree detection model needs to recognize, counts and records the number of each category, sets the frame rate for the model to process video and image sequences, and specifies the name of the weight file used for model inference; Recognition parameter adjustment module: Dynamically adjusts the model weight file, threshold, confidence level, and recognition delay parameters used in the real-time recognition task; Data preprocessing module: Performs signal preprocessing on the data from different sensor devices in corresponding ways to generate a preprocessed multi-modal data set; Data fusion module: Adopts the ASFF feature fusion strategy to perform adaptive feature fusion on the preprocessed multi-modal data set to obtain the data fusion result; Building assessment module: Based on the detection results of the final seismic building damage detection model, assess the damage degree of the affected buildings, and divide the results into four levels: red - completely damaged, yellow - severely damaged, blue - slightly damaged, green - intact; Data storage module: Utilize cloud computing resources for data storage and processing, and save the analysis results of the seismic building damage detection system to the database.

Images