Building Damage Assessment Method Based on Intelligent Extraction and Fusion of Multi-Perspective Information

Abstract

The present invention discloses a method for building damage assessment based on intelligent extraction and fusion of multi-perspective information. Based on intelligent vision technology, an algorithm for extracting key video frames based on mutual information measurement and a single-stage lightweight object detection scheme are proposed to achieve intelligent extraction of visual damage information. Finally, a post-processing module based on the fusion of multi-perspective visual damage information is proposed to realize the division of damage levels of rooms in the building and give the inference result of the explosion point position. Through the algorithm for extracting key video frames based on mutual information measurement, the present invention effectively captures the key frame images before and after damage, improves the operation efficiency of the algorithm, efficiently extracts visual damage information, and realizes the reasonable assessment of overall damage and explosion point inference.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and particularly relates to a method for evaluating building damage. Background Art

[0002] With the continuous development of modern military technology and the wide application of missile weapon systems, the evaluation of missile damage effectiveness has become a research hotspot in the military field. By reasonably evaluating the damage effectiveness of ammunition, commanders can adopt the most effective methods to achieve the preset target damage level, thereby reducing the casualty risk of combatants and minimizing the damage to non-combatants and infrastructure. In future high-tech local wars, important ground buildings of the enemy, such as command buildings, will also be important targets for our army to strike. Therefore, in the rapidly changing battlefield environment, the rapid evaluation of building damage is conducive to combat commanders optimizing fire strike plans and efficiently allocating strike resources.

[0003] At present, the methods for effectively evaluating the damage of buildings under blast loads can be roughly divided into four categories. The first is the method based on image change detection: In the literature "Yang Yanping. Research on Damage Effect Evaluation Technology Based on Image Change Detection. Xidian University, 2013.", a change detection algorithm based on wavelet multi-scale decomposition was proposed, and a damage assessment grading model was constructed based on the measurement results of change detection. The second is the method based on Bayesian network: In the literature "Zeng Yawen. Research on Target Damage Effect Evaluation Based on Bayesian Network. China Shipbuilding Research Institute, 2014.", a simulation analysis of target damage level evaluation was carried out based on Bayesian network, and in the command and decision-making environment, a simulation calculation of the damage effect evaluation of typical targets was carried out. In the literature "Wang Minle, Fan Mingjun, Li Xiaoguang. Research on Damage Effect Evaluation of Building Based on Bayesian Network. Tactical Missile Technology, 27, 31-43, 2010.", based on the automatic strike effect evaluation system, a Bayesian network evaluation model was established according to the structure, damage mode and functional characteristics of the building. The third is the fuzzy comprehensive evaluation method: In the literature "Jiang Hao, Chen Haoguang, Xing Xuehua. Damage Effect Evaluation of Airport Targets Based on Second-Level Fuzzy Comprehensive Evaluation. Ordnance Industry Automation, 26, 1-2, 2007.", the airport target index system was divided by the analytic hierarchy process, and a second-level fuzzy comprehensive evaluation model was used to evaluate its damage effect. In the literature "Miao Qiguang, Liu Juan, Ning Shuting. Evaluation of Airport Strike Effect Based on Fuzzy Comprehensive Evaluation. Systems Engineering and Electronics, 34, 1395-1399, 2012.", according to the basic composition structure and characteristics of the airport, an airport damage assessment criterion based on geometric, texture and overall features and a knowledge reasoning fuzzy comprehensive evaluation mathematical model were proposed, and an airport runway strike effect evaluation system based on fuzzy comprehensive evaluation theory was realized. In the literature "Miao Qiguang, Ning Shuting, Zhang Ye. Research on Building Strike Effect Evaluation Based on Second-Level Fuzzy Comprehensive Evaluation. Systems Engineering Theory and Practice, 34, 2438-2445, 2014.", starting from the knowledge of fuzzy mathematics, a building strike effect evaluation model based on second-level fuzzy comprehensive evaluation theory was realized. The fourth is the manual judgment method. In the 1960s, the United States mainly evaluated the target damage effect by manually discriminating the target damage images obtained from different sensors.

[0004] Although the above damage assessment methods have achieved excellent results from different perspectives, in the face of diverse damage methods and the complex indoor environment of buildings, these methods still have some deficiencies. The traditional manual image judgment method is not real-time, and the evaluation results lag behind the decision-making requirements; the change detection method has high requirements for preprocessing such as image registration and correction; the Bayesian network method has problems in mining hidden information in sample data; the fuzzy comprehensive evaluation is relatively subjective and lacks generalization. Summary of the Invention

[0005] To overcome the deficiencies of the prior art, the present invention provides a method for building damage assessment based on intelligent extraction and fusion of multi-perspective information. Based on intelligent vision technology, an algorithm for extracting key video frames based on mutual information measurement is proposed, and a single-stage lightweight object detection scheme is used to realize the intelligent extraction of visual damage information. Finally, a post-processing module based on the fusion of multi-perspective visual damage information is proposed to realize the division of the damage levels of rooms in the building and give the inference result of the explosion point position. Through the algorithm for extracting key video frames based on mutual information measurement, the present invention effectively captures the key frame images before and after damage, improves the operation efficiency of the algorithm, efficiently extracts visual damage information, and realizes the reasonable assessment of overall damage and explosion point inference.

[0006] The technical solutions adopted by the present invention to solve its technical problems include the following steps:

[0007] Step 1: Extract key video frames through inter-frame mutual information measurement;

[0008] Step 2: Detect indoor targets in the building;

[0009] Adopt a single-stage lightweight object detection framework based on yolov5 to realize the extraction of the position and category information of indoor targets in the building;

[0010] Step 3: Information fusion post-processing module;

[0011] Adopt a dynamic weight strategy to assign weights to target categories with different strike values; obtain the damage information of each room through the fixed mapping relationship between the camera number and the building room; according to the damage conditions of different rooms, summarize the damage levels of different floors in turn, and then infer the explosion point position.

[0012] Further, the specific content of Step 1 is as follows:

[0013] Step 1-1: Define a video sequence F1, F2, F3, … F composed of n video frame images, use the background difference method to detect moving targets in the image frames, and record the starting frame F1 and the ending frame F n ,; n ;

[0014] Step 1-2: Analyze the video image frame sequence F1, F2, F3, … F n , and classify the key video segments through the mutual information I (t,t+1) <θ1, where θ1 is the change range threshold for extracting local minimum values, and a total of N types of video segments are obtained, that is, S = {S1, S2, S3, … S N}; The calculation formula for the mutual information between images F t and F t+1 is as follows:

[0015]

[0016] where: P t,t+1 (i,j) is the joint probability density of F t and F t+1 ; P t (i) and P t+1 (j) are the marginal probability densities of the images F t and F t+1 respectively; N is the number of gray levels of the image;

[0017] Step 1-3: Assume class C i :{S i}, calculate the difference in the mutual information between adjacent classes

[0018] Step 1-4: Calculate the average value I i of the mutual information in each class. Using the threshold θ2, where θ2 represents the change range threshold for extracting local minima in the images of each class, judge the similarity between classes: If then merge C i and C i+1 into a new class;

[0019] Step 1-5: Perform clustering analysis on the image frames reclassified by the threshold θ2, and at the same time extract key frames for each new class;

[0020] Step 1-6: Summarize the MAX and MIN image frames and the image frames close to the average value extracted from the key frames in each class again, and mark them as K;

[0021] Step 1-7: Calculate the difference between the MAX and MIN image frames in each class. If the difference is greater than the video frequency, select these two frames as key frames; if the interpolation is less than or equal to the video frequency, select the image frame closest to the average value in the class as the key frame.

[0022] Furthermore, the specific steps of step 2 are as follows:

[0023] Step 2-1: Image preprocessing;

[0024] Scale and normalize the pictures before and after damage to adapt to the input format of the network model;

[0025] Step 2-2: Network model inference;

[0026] Use the trained yolov5 network model to perform inference on the preprocessed pictures to obtain the object detection results;

[0027] Step 2-3: Post-processing;

[0028] Post-process the results obtained from the inference of the network model, including non-maximum suppression and confidence filtering operations, to obtain the category information and location information of each target before and after damage.

[0029] Further, step 3 is specifically as follows:

[0030] Step 3-1: Calculate the damage under a single camera view;

[0031] Access the key image frames before and after damage in each camera view. If the camera is damaged in the explosion, only the pre-damage image exists, and at this time, mark the camera in this view as damaged; if the camera is not damaged, calculate the damage based on the changes in the target category, location, and quantity extracted:

[0032]

[0033] Among them, is the quantity of target type c before damage, is the quantity of target type c after damage, w c is the strike value coefficient of target type c, d c is the damage quantification value of target type c, and D is the damage quantification value of this camera view, which is obtained by averaging the damage quantification values of various target types in this view;

[0034] Step 3-2: Calculate the room damage;

[0035] Establish a two-way mapping relationship between the camera and the room, and locate the camera views in each room through query and retrieval; based on this, obtain the overall damage quantification value of the room by calculating the average value of the damage quantification values of each camera view in the room;

[0036] Step 3-3: Calculate the floor damage;

[0037] From the obtained overall damage quantification value of the room, calculate the damage of each floor by averaging, that is, the floor damage;

[0038] Step 3-4: Blast point inference;

[0039] First, screen out the floor with the largest building damage quantification value as the blast point floor, and then take the room with the most serious damage on this floor as the blast point, so as to obtain the floor number of the blast point and the corresponding room number;

[0040] Step 3-5: Damage level assessment;

[0041] Divide the damage degree into three levels: mild, moderate, and severe, and give a fuzzy evaluation to the damage quantification values of each floor and room by dividing the threshold.

[0042] The beneficial effects of the present invention are as follows:

[0043] 1. By using a video key frame extraction algorithm based on mutual information measurement, the key frame images before and after damage are effectively captured, improving the operation efficiency of the algorithm.

[0044] 2. Through a single-stage lightweight object detection algorithm based on deep learning, the adaptive positioning and rapid recognition of indoor building targets are achieved, and visual damage information is efficiently extracted.

[0045] 3. According to the multi-view damage information of the indoor building extracted, the present invention designs a weight dynamic fusion algorithm to assign weights to targets with different strike values, realizing a reasonable evaluation of the overall damage and the inference of the detonation point. Description of the Drawings

[0046] Figure 1 Flow chart of building damage assessment based on intelligent extraction and fusion of multi-view information.

[0047] Figure 2 Schematic diagram of damage assessment of simulation data in the embodiment of the present invention. Detailed Embodiments

[0048] The present invention will be further described below in conjunction with the drawings and embodiments.

[0049] In order to improve the adaptability of the building damage assessment model to complex indoor environments and diverse damage methods, as well as the efficiency of damage assessment and detonation point location inference, the present invention proposes a new building damage assessment framework based on intelligent extraction and fusion of multi-view information. Its main contributions are threefold: First, in order to filter redundant visual information and effectively improve the operation efficiency, the present invention designs a video key frame extraction algorithm based on mutual information measurement; Second, aiming at the problems of large scale differences and uneven distribution of indoor building targets, the present invention uses a single-stage lightweight object detection algorithm based on deep learning to achieve adaptive and rapid positioning and recognition of targets; Third, according to the multi-view damage information of the indoor building extracted, the present invention proposes a post-processing module based on the fusion of multi-view visual damage information to achieve the division of the damage level of rooms in the building and give the inference result of the detonation point location. The intelligent visual damage information extraction strategy of the present invention efficiently and accurately captures the visual damage information of the target, while the post-processing module realizes the adaptive fusion and near-real-time comprehensive evaluation of the building damage information.

[0050] a) Video key frame extraction:

[0051] During the entire algorithm processing, if each frame of the monitored video stream is processed, it will bring a huge amount of computation to the extraction and post-processing of the target damage information. In addition, the background environment in the building interior is relatively complex except for the target. In order to filter out the interference of background noise on the key frames before and after damage, the present invention realizes the extraction of video key frames through the measurement of mutual information between frames.

[0052] b) Detection of indoor targets in buildings:

[0053] The position distribution of indoor targets in buildings is complex and there is a large scale difference. Traditional object detection algorithms usually adopt a two-stage detection process, which requires generating candidate regions and classifying images, and cannot meet the near real-time requirements of damage assessment. The present invention adopts a single-stage lightweight object detection framework based on yolov5 to realize the extraction of the position and category information of indoor targets in buildings. Since it introduces a Feature Pyramid Network (FPN), it can detect targets of different sizes at multiple scales. This multi-scale detection ability effectively improves the extraction accuracy of target information.

[0054] c) Information fusion and post-processing module:

[0055] Since the types of indoor targets are complex and their importance is inconsistent, the present invention proposes a dynamic weight strategy to assign weights to target categories with different strike values, which is conducive to fusing more reasonable room damage assessment results. In addition, through the fixed mapping relationship between the camera number and the building room, the damage information of each room can be obtained. Finally, according to the damage conditions of different rooms, the damage levels of different floors can be summarized in turn, and then the explosion point position can be inferred.

[0056] Refer to Figure 1 , and the specific implementation manners of the present invention will be described below from three stages: key frame extraction, visual damage information extraction, and damage information fusion and post-processing:

[0057] A) Video key frame extraction based on mutual information measurement:

[0058] Step 1, define a video sequence composed of n video frame images, namely F1, F2, F3, … F n , use the background difference method to detect moving targets in the image frames, and record the starting frame F1 and the ending frame F n .

[0059] Step 2, parse the video image frame sequence F1, F2, F3, … F n and classify the key video segments through the mutual information I (t,t+1) <θ1 (θ1 is the change range threshold for extracting local minima), and a total of N types of video segments are obtained, namely S = {S1, S2, S3, … S N}, Image F t and F t+1 The formula for calculating the mutual information between them is as follows:

[0060]

[0061] In the formula: P t,t+1 (i, j) is the joint probability density of F t and F t+1 ; P t (i) and P t+1 (j) are the marginal probability densities of image F t and F t+1 ; N is the number of gray levels of the image.

[0062] Step 3, assume class C i :{S i}, calculate the difference in mutual information between adjacent classes

[0063] Step 4, calculate the average value I i of the mutual information in each category, and judge the similarity between classes with a threshold θ2 (θ2 represents the change range threshold for extracting local minima in each category of images). If then merge C i and C i+1 into a new class.

[0064] Step 5, perform clustering analysis on the image frames reclassified by the threshold θ2, and at the same time extract key frames for each new category.

[0065] Step 6, summarize the MAX and MIN image frames and the image frames close to the average value extracted from the key frames in each category again, and mark them as K.

[0066] Step 7, calculate the difference between the MAX and MIN image frames in each category. If the difference is greater than the video frequency, select these two frames as key frames; if the interpolation is less than or equal to the video frequency, select the image frame closest to the average value in the class as the key frame.

[0067] B) Single-stage lightweight object detection based on yolov5 (visual damage information extraction):

[0068] Step 1, preprocess the pictures. Perform operations such as scaling and normalization on the pictures before and after damage to adapt to the input format of the network model.

[0069] Step 2, perform network model inference. Use the trained yolov5 network model to perform inference on the processed pictures to obtain the object detection results.

[0070] Step 3, post-processing. Post-process the results obtained from the inference of the network model, including operations such as non-maximum suppression and confidence filtering, to obtain the category information and location information of each target before and after damage.

[0071] C) Post-processing module based on multi-view visual damage information fusion:

[0072] Step 1, calculate the damage in a single camera view. Access the key image frames before and after damage in each camera view. If the camera is damaged in the explosion, there is only the pre-damage image. At this time, mark the camera in this view as damaged; if the camera is not damaged, calculate the damage based on the changes in the target category, location, and quantity extracted:

[0073]

[0074] Among them, is the quantity of c-type targets before damage, is the quantity of c-type targets after damage, w c is the strike value coefficient of c-type targets, d c is the damage quantification value of c-type targets, and D is the damage quantification value in this camera view, which is obtained by averaging the damage quantification values of various types of targets in this view.

[0075] Step 2, calculate the room damage. Since the structures and areas of different rooms in the building are different, resulting in different numbers of camera views in each room, a two-way mapping relationship of "camera-room" is established in this paper, so as to accurately locate the camera views in each room through query and retrieval. Based on this, the overall damage quantification value of the room can be obtained by calculating the average value of the damage quantification values of each camera view in the room.

[0076] Step 3, calculate the floor damage. Since the room damage has been obtained, the damage of each floor is then calculated by averaging, that is, the "floor damage".

[0077] Step 4, blast point inference. First, screen out the floor with the largest building damage quantification value as the blast point floor, and then select the room with the most serious damage on this floor as the blast point, so as to obtain the floor number of the blast point and the corresponding room number.

[0078] Step 5, damage level assessment. In this paper, the damage degree is divided into three levels: mild, moderate, and severe, and a fuzzy evaluation is given to the damage quantification values of each floor and room by dividing the threshold.

[0079] Example:

[0080] 1. Simulation conditions

[0081] The specific configuration of the experimental equipment of the present invention is: several high-definition surveillance cameras, switches, and central processors i7-6800K @ 3.40GHz, 64GB of memory, and an image processor GeForce GTX 1080Ti, operating system Ubuntu 2016, and used the PyTorch deep learning framework for simulation.

[0082] 2. Simulation Content

[0083] Indoor high-definition cameras were used to collect on-site data of the building interior. During the training phase, the collected image data was screened and then manually annotated. The specific target categories for annotation were: walls, tables, chairs, sofas, TVs, computers, radars, cabinets, and sand tables, etc. After annotation, 100 pieces of training data were used to train the proposed object detection algorithm. The maximum number of training epochs was set to 300, the initial learning rate was 0.01, and the batch size was 8. In the algorithm testing phase, since the damage assessment results mainly depend on the performance of object detection, in this experiment, the object detection algorithm in the proposed scheme was tested under 60 viewpoints (pictures) captured after damage. The best mAP50 obtained was 0.79362, and the recall was 0.76069. To verify the advantages of the proposed scheme, Table 1 lists the test results of this algorithm and other object detection algorithms:

[0084] Table 1

[0085] Method mAP50 Recall SPPNet 0.51 0.42 FasterRCNN 0.68 0.70 MaskRCNN 0.57 0.55 Ours 0.79 0.76

[0086] Based on the excellent object detection algorithm, in this experiment, further quantification of the damage value and multi-view fusion of the extracted target information were carried out. Figure 2 The evaluation results of simulated data damage are shown, and it is possible to clearly access the damage situation of each room and the location of the explosion point.

[0087] The present invention relates to a building damage assessment method based on intelligent extraction of multi-view information. By adaptively extracting, fusing, and evaluating damage information through intelligent vision technology, the problems of insufficient generalization and low efficiency of traditional damage assessment methods are effectively alleviated. Through experimental analysis, the object detection algorithm in the proposed scheme has obvious advantages compared with other widely used methods, which is beneficial to the extraction of target damage information; the proposed assessment method can efficiently and reasonably evaluate the damage situation of each room in the building and give the location of the explosion point.

Claims

1. A building damage assessment method based on intelligent extraction and fusion of multi - perspective information, characterized in that, It includes the following steps: Step 1: Extract video key frames through inter-frame mutual information measurement; Step 2: Detect indoor targets in buildings; Adopt a single-stage lightweight object detection framework based on yolov5 to extract the position and category information of indoor targets in buildings; Step 3: Information fusion post-processing module; Adopt a dynamic weight strategy to assign weights to target categories with different strike values; obtain the damage information of each room through the fixed mapping relationship between the camera number and the building room; according to the damage conditions of different rooms, summarize the damage levels of different floors in turn, and then infer the explosion point position; The specific content of Step 3 is as follows: Step 3-1: Calculate the damage under a single camera view; Access the key image frames before and after damage in each camera view. If the camera is damaged in the explosion, there is only the image before damage. At this time, mark the camera in this view as damaged; if the camera is not damaged, calculate the damage based on the changes in the detected target category, position, and quantity: (2) Among them, is the number of Class target before damage, is the number of Class target after damage, is the strike value coefficient of Class target, is the damage quantification value of Class target, is the damage quantification value under the camera view, which is obtained by averaging the damage quantification values of various Class targets under this view; Step 3-2: Calculate the room damage; Establish a two-way mapping relationship between the camera and the room, and locate the camera views in each room through query and retrieval; based on this, obtain the overall damage quantization value of the room by calculating the average value of the damage quantization values in each camera view in the room; Step 3-3: Calculate the floor damage; From the obtained overall damage quantization value of the room, calculate the damage of each floor through average calculation, that is, the floor damage; Step 3-4: Explosion point inference; First, select the floor with the largest building damage quantization value as the explosion point floor, and then select the room with the most serious damage on this floor as the explosion point, so as to obtain the floor number of the explosion point and the corresponding room number; Step 3-5: Damage level assessment; Divide the damage degree into three levels: mild, moderate, and severe, and give a fuzzy evaluation of the damage quantization values of each floor and room by dividing the threshold.

2. The method for evaluating building damage based on intelligent extraction and fusion of multi-perspective information according to claim 1, wherein, The specific content of Step 1 is as follows: Step 1-1: Define a video sequence consisting of video frame images, and use the background subtraction method to detect moving targets in the image frames, recording the starting frame and the ending frame ; ; Step 1-2: For the video image frame sequence perform parsing, and through the mutual information , which is the variation range threshold for local minimum extraction, classify the key video segments, and a total of categories of video segments are obtained, namely ; The calculation formula for the mutual information between images and is as follows: (1) In the formula: is and 's joint probability density; and are respectively the marginal probability densities of images and ; is the number of gray levels of the image; Step 1-3: Assume classes , calculate the difference in mutual information between adjacent classes ; Step 1-4: Calculate the average mutual information in each category , with a threshold , representing the change range threshold for local minimum extraction in each category of images, and judge the similarity between classes: If , then combine and into a new class; Step 1-5: Perform clustering analysis on the reclassified image frames, and at the same time extract key frames for each new category; ​ Step 1-6: Again, summarize the and image frames extracted from key frames of each type and the image frames close to the average value, and mark them as ; Step 1-7: Calculate the differences between and image frames in each category. If the difference is greater than the video frequency, select these two frames as key frames; if the interpolation is less than or equal to the video frequency, select the image frame closest to the average value in the category as the key frame.

3. The method for evaluating building damage based on intelligent extraction and fusion of multi-perspective information according to claim 2, wherein, The specific content of Step 2 is as follows: Step 2-1: Image preprocessing; Scale and normalize the images before and after damage to adapt to the input format of the network model; Step 2-2: Network model inference; Use the trained yolov5 network model to infer the preprocessed images to obtain the object detection results; Step 2-3: Post-processing; Perform post-processing on the results obtained from the network model inference, including non-maximum suppression and confidence filtering operations, to obtain the category information and position information of each target before and after damage.

Images