Unmanned aerial vehicle based building or structure damage assessment method, terminal and medium
By collecting multimodal data from drones to build a knowledge graph, and combining it with data fusion from visual and infrared sensors, deep learning algorithms are used to infer the physical performance indicators of buildings or structures. This solves the problem that drones cannot deeply identify internal safety performance, and achieves highly reliable safety assessment and risk identification.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN URBAN PUBLIC SAFETY & TECH INST CO LTD
- Filing Date
- 2023-03-03
- Publication Date
- 2026-07-03
AI Technical Summary
Existing drone technology cannot deeply identify the safety performance indicators inside buildings or structures, resulting in low reliability of hazard identification and making it difficult to meet the needs of urban drone supervision for comprehensive safety inspection of buildings or structures.
By collecting inspection and monitoring data of different modalities using drones, a multimodal knowledge graph is established. Combined with data fusion from visual and infrared sensors, and using a few-shot augmented deep learning algorithm, the physical performance indicators of buildings or structures are inferred, and a safety assessment is conducted.
It enables comprehensive safety assessments of buildings or structures, improving the reliability and accuracy of drone inspections, and is able to identify potential risks and output risk warning information.
Smart Images
Figure CN116229299B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and in particular to a method, terminal, and medium for assessing damage to buildings or structures based on UAVs. Background Technology
[0002] In the technical solutions for identifying potential hazards in urban buildings or structures using drones, drones collect image data of urban buildings or structures and urban infrastructure, identify whether there are any safety hazards, mark the location, and report it to the backend.
[0003] Currently, using drones for safety inspections and monitoring of urban construction projects such as buildings, structures, and infrastructure is an emerging regulatory approach. Using drones to monitor urban investment projects can improve the comprehensiveness and effectiveness of supervision and reduce labor costs.
[0004] However, due to the diversity of images of urban buildings, structures, or infrastructure, the current identification results only focus on the visual surface damage of buildings, structures, or facilities (such as damage to curtain walls or roofs), and cannot delve into the safety performance indicators inside the buildings or structures. This results in a deficiency of a single mode of information collection, making it difficult to meet the needs of urban drone monitoring projects for comprehensive safety inspection of buildings or structures, and leading to a problem of low reliability in identifying potential risks in buildings or structures.
[0005] The above content is only used to help understand the technical solution of the present invention and does not represent an admission that the above content is prior art. Summary of the Invention
[0006] The main objective of this invention is to provide a method for assessing damage to buildings or structures based on unmanned aerial vehicles (UAVs), aiming to solve the problem of improving the reliability of UAVs in identifying potential hazards in buildings or structures.
[0007] To achieve the above objectives, the present invention provides a method for assessing damage to buildings or structures based on unmanned aerial vehicles (UAVs), the method comprising:
[0008] Based on the inspection and monitoring data of different modes obtained by the UAV from the target building or structure, the appearance damage results of the target building or structure are determined;
[0009] Based on the results of the external damage, infer the physical performance indicators of the target building or structure;
[0010] Based on the appearance damage results and the physical performance indicators, the safety assessment results of the target building or structure are determined.
[0011] Optionally, before the step of inferring the physical performance indicators of the target building or structure based on the appearance damage results, the method further includes:
[0012] Acquire historical inspection and monitoring data of different buildings or structures collected by the UAV at historical moments;
[0013] Establish a multimodal knowledge graph between the historical inspection and monitoring data and the preset physical performance index template to form a mapping relationship between the inspection and monitoring data and the physical performance index;
[0014] The step of inferring the physical performance indicators of the target building or structure based on the appearance damage results includes:
[0015] Based on the multimodal knowledge graph, the physical performance indicators corresponding to the appearance damage results are matched.
[0016] Optionally, the step of establishing a knowledge graph based on the historical inspection and monitoring data and the historical physical performance indicators to form a mapping relationship between the inspection and monitoring data and the physical performance indicators includes:
[0017] Based on the physical performance index template, mark the areas to be marked in the historical inspection and monitoring data, and construct the multimodal knowledge graph based on the marked historical inspection and monitoring data; or,
[0018] The multimodal data items associated with the physical performance index template in the historical inspection and monitoring data are marked, so as to construct the multimodal knowledge graph based on the multimodal data items.
[0019] Optionally, the safety assessment result includes a safety rate, and the step of determining the safety assessment result of the target building or structure based on the appearance damage result and the physical performance indicators includes:
[0020] Determine the first classification accuracy of the appearance damage result in a preset building or structure appearance damage classification set, and determine the second classification accuracy of the physical performance index in a preset building or structure internal performance classification set;
[0021] The safety rate of the target building or structure is determined based on the first classification accuracy rate and the second classification accuracy rate.
[0022] Optionally, the safety assessment result includes a safety assessment value, and the step of determining the safety assessment result of the target building or structure based on the appearance damage result and the physical performance index includes:
[0023] Determine a first predicted value of the appearance damage result in a preset building or structure appearance damage prediction set, and determine a second predicted value of the physical performance index in a preset building or structure internal performance test set.
[0024] Determine a first prediction difference between the first predicted value and a preset first threshold, and determine a second prediction difference between the second predicted value and a preset second threshold;
[0025] The safety assessment value of the target building or structure is determined based on the first prediction difference and the second prediction difference.
[0026] Optionally, the drone includes a visual sensor and an infrared sensor. The step of determining the appearance damage result of the target building or structure based on the inspection and monitoring data of different modes obtained by the drone from the target building or structure includes:
[0027] The infrared inspection and monitoring data of the target building or structure collected by the infrared sensor within the same period and the image inspection and monitoring data of the target building or structure collected by the vision sensor are fused together to obtain fused data.
[0028] The fused data is input into a building or structure appearance diagnostic model trained using a few-shot augmentation deep learning algorithm to determine the appearance damage result.
[0029] Optionally, after the step of inputting the fused data into a building or structure appearance diagnostic model trained based on a few-shot augmentation deep learning algorithm to determine the appearance damage result, the method further includes:
[0030] The fused data is used as training samples for the building or structure appearance diagnostic model to achieve self-updating of the building or structure appearance diagnostic model.
[0031] Optionally, after the step of determining the safety assessment result of the target building or structure based on the appearance damage result and the physical performance index, the method further includes:
[0032] Determine whether the safety assessment results meet the preset safety assessment conditions for buildings or structures;
[0033] If the conditions are not met, output a risk warning message for the building or structure.
[0034] In addition, to achieve the above objectives, the present invention also provides a control terminal, the control terminal comprising: a memory, a processor, and a UAV-based building or structure damage assessment program stored in the memory and executable on the processor, wherein the UAV-based building or structure damage assessment program, when executed by the processor, implements the steps of the UAV-based building or structure damage assessment method as described above.
[0035] In addition, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a UAV-based building or structure damage assessment program, wherein the UAV-based building or structure damage assessment program, when executed by a processor, implements the steps of the UAV-based building or structure damage assessment method as described above.
[0036] This invention provides a method, control terminal, and storage medium for assessing damage to buildings or structures based on unmanned aerial vehicles (UAVs). The method uses UAVs to collect inspection and monitoring data of target buildings or structures to determine external damage results. Based on these external damage results, it infers the physical performance indicators of the buildings or structures. Finally, it determines the safety assessment result of the target building or structure based on both the external damage results and the physical performance indicators. This enables a comprehensive safety assessment of buildings or structures, improving the reliability of UAVs in urban inspections. Attached Figure Description
[0037] Figure 1 This is a schematic diagram of the hardware operating environment of the control terminal involved in an embodiment of the present invention;
[0038] Figure 2 This is a flowchart illustrating the first embodiment of the UAV-based building or structure damage assessment method of the present invention.
[0039] Figure 3 This is a flowchart illustrating a second embodiment of the UAV-based method for assessing damage to buildings or structures according to the present invention.
[0040] Figure 4 This is a flowchart illustrating the third embodiment of the UAV-based building or structure damage assessment method of the present invention.
[0041] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0042] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.
[0043] As one implementation scheme, Figure 1 This is a schematic diagram of the hardware operating environment of the control terminal involved in the embodiment of the present invention.
[0044] like Figure 1 As shown, the control terminal may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0045] Those skilled in the art will understand that Figure 1 The architecture of the control terminal shown does not constitute a limitation on the control terminal. It may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0046] like Figure 1 As shown, the memory 1005, serving as a storage medium, may include an operating system, a network communication module, a user interface module, and a UAV-based building or structure damage assessment program. The operating system is a program that manages and controls the hardware and software resources of the control terminal, the UAV-based building or structure damage assessment program, and the operation of other software or programs.
[0047] exist Figure 1 In the control terminal shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; the processor 1001 can be used to call the UAV-based building or structure damage assessment program stored in the memory 1005.
[0048] In this embodiment, the control terminal includes: a memory 1005, a processor 1001, and a UAV-based building or structure damage assessment program stored in the memory and capable of running on the processor, wherein:
[0049] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0050] Acquire historical inspection and monitoring data of different buildings or structures collected by the UAV at historical moments;
[0051] Establish a multimodal knowledge graph between the historical inspection and monitoring data and the preset physical performance index template to form a mapping relationship between the inspection and monitoring data and the physical performance index;
[0052] Based on the multimodal knowledge graph, the physical performance indicators corresponding to the appearance damage results are matched.
[0053] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0054] Based on the physical performance index template, mark the areas to be marked in the historical inspection and monitoring data, and construct the multimodal knowledge graph based on the marked historical inspection and monitoring data; or,
[0055] The multimodal data items associated with the physical performance index template in the historical inspection and monitoring data are marked, so as to construct the multimodal knowledge graph based on the multimodal data items.
[0056] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0057] Determine the first classification accuracy of the appearance damage result in a preset building or structure appearance damage classification set, and determine the second classification accuracy of the physical performance index in a preset building or structure internal performance classification set;
[0058] The safety rate of the target building or structure is determined based on the first classification accuracy rate and the second classification accuracy rate.
[0059] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0060] Determine a first predicted value of the appearance damage result in a preset set of predictions for appearance damage of buildings or structures, and determine a second predicted value of the physical performance index in a preset set of predictions for internal performance of buildings or structures.
[0061] Determine a first prediction difference between the first predicted value and a preset first threshold, and determine a second prediction difference between the second predicted value and a preset second threshold;
[0062] The safety assessment value of the target building or structure is determined based on the first prediction difference and the second prediction difference.
[0063] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0064] The infrared inspection and monitoring data of the target building or structure collected by the infrared sensor within the same period and the image inspection and monitoring data of the target building or structure collected by the vision sensor are fused together to obtain fused data.
[0065] The fused data is input into a building or structure appearance diagnostic model trained using a few-shot augmentation deep learning algorithm to determine the appearance damage result.
[0066] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0067] The fused data is used as training samples for the building or structure appearance diagnostic model to achieve self-updating of the building or structure appearance diagnostic model.
[0068] When processor 1001 calls the UAV-based building or structure damage assessment program stored in memory 1005, it performs the following operations:
[0069] Determine whether the safety assessment results meet the preset safety assessment conditions for buildings or structures;
[0070] If the conditions are not met, output a risk warning message for the building or structure.
[0071] Based on the hardware architecture of the control terminal based on UAV technology described above, an embodiment of the UAV-based building or structure damage assessment method of the present invention is proposed.
[0072] Reference Figure 2 In the first embodiment, the UAV-based building or structure damage assessment method includes the following steps:
[0073] Step S10: Based on the inspection and monitoring data of different modes obtained by the UAV from the target building or structure, determine the appearance damage result of the target building or structure;
[0074] In this implementation, the drone is first controlled to inspect and collect monitoring data on the target building or structure that needs to be assessed for damage. The results of the appearance damage to the target building or structure are then determined based on the inspection and monitoring data.
[0075] Optionally, the data modalities of inspection and monitoring data include, but are not limited to, visible light images, videos, infrared images, and other data information.
[0076] Visual damage results are characterized as the degree of difference between the appearance of a building or structure and its appearance data in a healthy state. Visual damage includes, but is not limited to, curtain wall glass breakage, wall cracking, and paint peeling. A damage diagnosis threshold is constructed using structural inspection and monitoring data of buildings or structures in a healthy state. By comparing the cumulative damage discriminant factor under different evaluation conditions with the damage diagnosis threshold, it is possible to determine whether the building or structure has suffered damage.
[0077] Step S20: Based on the appearance damage results, infer the physical performance indicators of the target building or structure;
[0078] In this embodiment, after obtaining the appearance damage results of the target building or structure, the UAV infers the physical performance indicators of the target building or structure based on the appearance damage results. The physical performance indicators are characterized as safety indicators of the internal structural layers of the building or structure. These physical performance indicators include, but are not limited to: building or structure safety level, building or structure seismic resistance, building or structure lightning protection capability, building or structure roof leak-proof capability, and building or structure electrical connectivity, etc.
[0079] For example, after the drone collects inspection and monitoring data of the target building or structure, and determines that the appearance damage of the target building or structure is as follows: the windows are not fitted with glass, the walls are severely weathered, the roof and wall cracks are severely cracked, and the roof is not fitted with a lightning rod, then the physical performance indicators of the building or structure are inferred to be: the building or structure has a low level of safety, the building or structure has poor or no lightning protection capability, and the building or structure has poor water leakage prevention capability.
[0080] It should be noted that this embodiment employs a few-shot augmentation deep learning algorithm, combined with data augmentation methods such as CutMix and Copy-Paste, enabling the model to be trained with augmented data even when based on a few-shot data source. Furthermore, data is accumulated during intelligent inspection in use, giving the model self-evolution capabilities. Through continuous self-evolution during training, the model achieves self-evolution and accuracy improvement.
[0081] Step S30: Determine the safety assessment result of the target building or structure based on the appearance damage results and the physical performance indicators.
[0082] In this embodiment, after the physical performance indicators of the target building or structure are inferred, the safety assessment result of the target building or structure is determined together based on the obtained appearance damage results and physical performance indicators.
[0083] Optionally, the safety assessment result can be the safety rate of a building or structure, i.e., the probability of a person being safe to move around inside the building or structure. When the safety rate is greater than a probability threshold, the building or structure is considered to have passed the assessment. In some specific embodiments, two classification sets are preset: one for the appearance damage of a building or structure, and the other for the internal performance of a building or structure. The appearance damage classification set contains multiple characteristic data representing the appearance damage of a building or structure, while the internal performance classification set contains multiple data representing the internal structural hierarchy of a building or structure. The appearance damage result is placed into the appearance damage classification set, and the matching similarity between the appearance damage result and each appearance damage characteristic data is determined. Features with a matching similarity greater than a similarity threshold are identified as features that match the appearance damage result. The more matching features, the higher the classification accuracy, meaning the higher the reliability of the result. The classification accuracy of the appearance damage result in the appearance damage classification set is used as the first classification accuracy. Similarly, the classification accuracy of the physical performance indicators of a building or structure within the internal performance classification set of the building or structure is used as the second classification accuracy. Further, based on the first and second classification accuracy rates, the safety rate of the target building or structure is determined. The higher the first and second classification accuracy rates, the higher the safety rate. When the safety rate exceeds a safety rate threshold, the target building or structure is deemed to have passed the safety assessment.
[0084] Optionally, the safety assessment result can also be a safety assessment value, which is a continuous value. In some specific embodiments, two classification sets are preset: one is a prediction set for the appearance damage of a building or structure, and the other is a prediction set for the internal performance of a building or structure. The prediction sets are obtained by using a linear regression function to obtain the predicted values for the appearance or interior of the building or structure. Since the predicted values usually have errors, they need to be subtracted from a preset threshold (also called the true value) after obtaining the predicted value. The result is the prediction difference. Among them, the prediction difference corresponding to the appearance damage result is called the first prediction difference, and the prediction difference corresponding to the physical performance index is called the second prediction difference. Finally, based on the first prediction difference and the second prediction difference, the safety assessment value of the target building or structure is determined. The smaller the first prediction difference and the second prediction difference (i.e., the closer the predicted value is to the true value), the larger the safety assessment value, which means that the safety assessment result is more accurate. When the safety assessment value is greater than the preset safety assessment threshold, the safety assessment of the target building or structure is deemed qualified.
[0085] Optionally, if the safety assessment results do not meet the preset safety assessment conditions for buildings or structures, the UAV will output a building or structure risk warning message to the control terminal indicating that the target building or structure has potential risks, so as to notify relevant personnel to carry out corresponding repairs to the target building or structure.
[0086] In the technical solution provided in this embodiment, after determining the appearance damage results based on the inspection and monitoring data of the target building or structure collected by the UAV, the physical performance indicators of the building or structure are inferred based on the appearance damage results. Finally, the safety assessment result of the target building or structure is determined by combining the appearance damage results and the physical performance indicators. This achieves a comprehensive safety assessment of the building or structure from the surface to the interior, improving the reliability of UAVs in urban inspections.
[0087] Reference Figure 3 In the second embodiment, based on the first embodiment, before step S20, the method further includes:
[0088] Step S40: Obtain historical inspection and monitoring data of different buildings or structures collected by the UAV at historical moments.
[0089] Step S50: Establish a multimodal knowledge graph between the historical inspection and monitoring data and the preset physical performance index template to form a mapping relationship between the inspection and monitoring data and the physical performance index.
[0090] Step S20 includes:
[0091] Step S21: Based on the knowledge graph, match the physical performance index corresponding to the appearance damage result.
[0092] Optionally, in this embodiment, a knowledge graph is used to establish a mapping relationship between the appearance damage results of the UAV and its physical performance indicators. In this embodiment, before the UAV performs the physical performance indicator prediction step, the UAV is pre-controlled to collect data from different buildings or structures as historical inspection and monitoring data collected at historical moments. Then, a multi-modal knowledge graph (MMKG) is established between the historical inspection and monitoring data and the preset physical performance indicator template. A knowledge graph (KG) describes knowledge resources and their carriers through visualization technology, mining, analyzing, constructing, drawing, and displaying knowledge and their interrelationships. Essentially, it is a large-scale semantic network with entities and concepts as nodes and various semantic relationships between concepts as edges. Most existing knowledge graphs are represented using pure symbols in text form, which weakens the machine's ability to describe and understand the real world. A multi-modal knowledge graph, however, can give the machine the ability to recognize specific entities in an image, enabling the machine to generate a more information-rich entity rather than a vague conceptual description. In this embodiment, a multimodal knowledge graph is applied to the safety assessment of buildings or structures by drones. Semantic connections are established between the image modal information of apparent damage to urban buildings or structures and infrastructure and the text and index concepts of building or structure damage assessment. A knowledge graph is established between the visual intelligent algorithm recognition results of apparent damage to urban buildings or structures and infrastructure and the structural performance index assessment, forming a mapping relationship between apparent damage and performance indicators, thereby achieving the matching of appearance to performance.
[0093] For example, in some implementations, semantic connections are established between the modal information of apparent damage images of urban buildings and infrastructure and the textual and indexal concepts of building damage assessment in existing knowledge and standards, so as to realize the establishment of a multimodal knowledge graph, thereby enriching the dimensions of the recognition results and providing a reliable knowledge base for safety hazard assessment.
[0094] Step S21 includes:
[0095] Step S211: According to the physical performance index template, mark the areas to be marked in the historical inspection and monitoring data, so as to construct the multimodal knowledge graph based on the marked historical inspection and monitoring data;
[0096] Alternatively, as a construction method, a multimodal knowledge graph can be built by labeling images. In this approach, the data modality of historical inspection and monitoring data is typically image data. Physical performance index templates are used as knowledge symbols to label specific areas in the historical inspection and monitoring image data. These areas need to be bounding boxes drawn and labeled by the workers. The multimodal knowledge graph is then constructed using the labeled images.
[0097] Step S212: Mark the multimodal data items in the historical inspection and monitoring data that are associated with the physical performance index template, so as to construct the multimodal knowledge graph based on the multimodal data items;
[0098] Alternatively, as another construction method, a multimodal knowledge graph can be constructed using symbolic localization. Symbolic localization involves using physical performance indicator templates to extract data from historical inspection and monitoring data, treating the extracted data as multimodal data items, and then constructing a multimodal knowledge graph based on these multimodal data items. In this approach, the data modalities of historical inspection and monitoring data typically include, but are not limited to, image data, video data, and audio data.
[0099] In the technical solution provided in this embodiment, by establishing a multimodal knowledge graph between the exterior and interior of a building or structure, when the UAV collects inspection and monitoring data of the target building or structure, the physical performance indicators corresponding to the inspection and monitoring data are matched based on the multimodal knowledge graph, thereby realizing a safety assessment of the building or structure from the surface to the interior during the UAV inspection process.
[0100] Reference Figure 4 In the third embodiment, based on any embodiment, step S10 includes:
[0101] Step S11: The infrared inspection and monitoring data of the target building or structure collected by the infrared sensor within the same period and the image inspection and monitoring data of the target building or structure collected by the vision sensor are fused together to obtain fused data.
[0102] Step S12: Input the fused data into the building or structure appearance diagnostic model trained by a deep learning algorithm based on few-shot augmentation to determine the appearance damage result.
[0103] Optionally, to improve the accuracy of UAV assessment of exterior damage to buildings or structures, fused data obtained by fusing infrared and visual data is used to determine the exterior damage results. In this embodiment, the UAV is equipped with infrared and visual sensors. At preset intervals, the UAV fuses infrared inspection and monitoring data and image inspection and monitoring data from the same period. The infrared data is used to optimize the image clarity of the image data collected by the UAV, resulting in more accurate fused data. The fused data is then input into a building or structure exterior diagnostic model trained using a few-shot augmentation deep learning algorithm to determine the exterior damage results of the target building or structure.
[0104] It should be noted that deep learning algorithms using few-sample augmentation can reduce the proportion of imbalanced samples, enabling deep learning methods to achieve efficient feature extraction and high-accuracy target recognition under few-sample conditions, and improving the generalization of the model.
[0105] Wherein, after S12, it also includes:
[0106] Step S60: Use the fused data as training samples for the building or structure appearance diagnostic model to achieve self-updating of the building or structure appearance diagnostic model.
[0107] Optionally, in order to enable the UAV to continuously update its deep learning model based on the data it collects during the inspection process, in this embodiment, fused data combining infrared and image data is used as training samples for the building or structure appearance diagnostic model. This allows the building or structure appearance diagnostic model to be trained, thereby achieving self-updating of the model. This enables the UAV to continuously accumulate data during the inspection process, thereby achieving self-evolution of the model and continuously improving the accuracy of appearance damage assessment during flight.
[0108] In the technical solution provided in this embodiment, infrared data and image data are fused to achieve higher accuracy in diagnosing the appearance damage results of the UAV. The fused data is then input into the appearance diagnosis model of the building or structure as a training sample to achieve self-evolution of the UAV's diagnosis model. This improves the accuracy of appearance damage assessment during UAV flight, thereby enhancing the reliability of UAV inspections in providing comprehensive safety assessments of buildings or structures.
[0109] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the control terminal to implement the process steps of the embodiments of the above methods.
[0110] Therefore, the present invention also provides a computer-readable storage medium storing a UAV-based building or structure damage assessment program, which, when executed by a processor, implements the various steps of the UAV-based building or structure damage assessment method as described in the above embodiments.
[0111] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.
[0112] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.
[0113] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0114] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.
[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0117] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0118] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.
[0119] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A method for assessing damage to buildings or structures based on unmanned aerial vehicles (UAVs), characterized in that, The method includes: Based on the inspection and monitoring data of different modes obtained by the UAV from the target building or structure, the appearance damage results of the target building or structure are determined; Based on the results of the external damage, infer the physical performance indicators of the target building or structure; Based on the appearance damage results and the physical performance indicators, the safety assessment results of the target building or structure are determined; Before the step of inferring the physical performance indicators of the target building or structure based on the appearance damage results, the method further includes: Acquire historical inspection and monitoring data of different buildings or structures collected by the UAV at historical moments; Based on the historical inspection and monitoring data and the physical performance indicators of buildings or structures, a multimodal knowledge graph is established to form a mapping relationship between the inspection and monitoring data and the physical performance indicators. Specifically, according to the physical performance indicator template, the areas to be marked in the historical inspection and monitoring data are marked so as to construct the multimodal knowledge graph based on the marked historical inspection and monitoring data. The step of inferring the physical performance indicators of the target building or structure based on the appearance damage results includes: Based on the multimodal knowledge graph, the physical performance indicators corresponding to the appearance damage results are matched; The drone includes a visual sensor and an infrared sensor. The step of determining the appearance damage result of the target building or structure based on the inspection and monitoring data of different modes obtained by the drone from the target building or structure includes: The infrared inspection data obtained by the infrared sensor from the target building or structure within the same period and the image inspection data obtained by the vision sensor from the target building or structure are fused together to obtain fused data. The fused data is input into a building or structure appearance diagnostic model trained by a deep learning algorithm based on few-shot augmentation to determine the appearance damage result; Furthermore, the deep learning algorithm based on few-shot augmentation uses data augmentation methods such as CutMix and / or Copy-Paste to augment the training samples.
2. The method as described in claim 1, characterized in that, The step of establishing a multimodal knowledge graph based on the historical inspection and monitoring data and the physical performance indicators of buildings or structures to form a mapping relationship between inspection and monitoring data and physical performance indicators includes: The multimodal data items associated with the physical performance index template in the historical inspection and monitoring data are marked, so as to construct the multimodal knowledge graph based on the multimodal data items.
3. The method as described in claim 1, characterized in that, The safety assessment result includes a safety rate. The steps for determining the safety assessment result of the target building or structure based on the appearance damage results and the physical performance indicators include: The accuracy rate of the first classification of the appearance damage results in a preset set of building or structure appearance damage classifications, and the accuracy rate of the second classification of the physical performance indicators in a preset set of building or structure internal performance classifications. The safety rate of the target building or structure is determined based on the first classification accuracy rate and the second classification accuracy rate.
4. The method as described in claim 1, characterized in that, The safety assessment result includes a safety assessment value. The step of determining the safety assessment result of the target building or structure based on the appearance damage result and the physical performance index includes: Determine a first predicted value of the appearance damage result in a preset building or structure appearance damage prediction set, and determine a second predicted value of the physical performance index in a preset building or structure internal performance test set. Determine a first prediction difference between the first predicted value and a preset first threshold, and determine a second prediction difference between the second predicted value and a preset second threshold; The safety assessment value of the target building or structure is determined based on the first prediction difference and the second prediction difference.
5. The method as described in claim 1, characterized in that, After the step of inputting the fused data into a building or structure appearance diagnostic model trained based on a few-shot augmentation deep learning algorithm to determine the appearance damage result, the method further includes: The fused data is used as training samples for the building or structure appearance diagnostic model to achieve self-updating of the building or structure appearance diagnostic model.
6. The method as described in claim 1, characterized in that, After the step of determining the safety assessment result of the target building or structure based on the appearance damage results and the physical performance indicators, the method further includes: Determine whether the safety assessment results meet the preset safety assessment conditions for buildings or structures; If the conditions are not met, output a risk warning message for the building or structure.
7. A control terminal, characterized in that, The control terminal includes: a memory, a processor, and a UAV-based building or structure damage assessment program stored in the memory and executable on the processor. When the UAV-based building or structure damage assessment program is executed by the processor, it implements the steps of the UAV-based building or structure damage assessment method as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a drone-based building or structure damage assessment program, which, when executed by a processor, implements the steps of the drone-based building or structure damage assessment method as described in any one of claims 1 to 6.