Unmanned aerial vehicle inspection exception cooperative decision method, device, equipment and medium

Abstract

The application belongs to the technical field of unmanned aerial vehicle inspection, and relates to a cooperative decision-making method, device, equipment and medium for unmanned aerial vehicle inspection abnormalities. The method comprises the following steps: acquiring information of an unmanned aerial vehicle inspection target, constructing a digital twin to serve as an offline twin; scoring the digital twin for inspection value to obtain a heat map, and performing three-dimensional flight planning to obtain a flight semantic path; acquiring flight data of the unmanned aerial vehicle performing inspection according to the flight semantic path, generating a dynamic three-dimensional model to serve as an online twin; comparing the offline twin and the online twin in the dimensions of geometry, vision and physical field to obtain a geometric shape comparison result, a vision texture comparison result and a physical field information comparison result, and generating a comprehensive abnormal data package; acquiring a current position, combining the comprehensive abnormal data package to generate a shared digital twin scene, and performing cooperative decision-making. The application can perform cooperative decision-making on abnormalities inspected by the unmanned aerial vehicle.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) inspection technology, and in particular to collaborative decision-making methods, devices, equipment, and media for UAV inspection anomalies. Background Technology

[0002] Currently, the inspection and maintenance of critical infrastructure such as bridges, power transmission networks, wind turbines, and large chemical plants are undergoing profound digital transformation.

[0003] Traditional operation and maintenance models rely heavily on manual labor, resulting in numerous pain points such as high operational risks, low efficiency, strong subjectivity, and ineffective data accumulation.

[0004] In recent years, drone inspections have been initially applied, solving some accessibility and safety issues. However, their application model remains rudimentary and suffers from the following problems: 1) Most drones still rely on flight camera solutions. Drone inspection is an isolated data collection process, with simple and haphazard path planning and a huge workload for data post-processing. 2) Although digital twin technology has emerged, it often remains at the level of visual display, and is mostly static, failing to drive business and thus failing to become a living brain that drives business. 3) Augmented reality applications are mostly isolated information displays. For example, AR collaboration is limited to general video calls, lacking spatial and data depth, and lacking precise linkage with real-time dynamic data.

[0005] The application of these individual technologies has created technological silos. In essence, the existing solutions are a shallow combination of various technologies, failing to form an organic whole and failing to build a complete and efficient closed loop from problem discovery to problem solving. This has led to a series of problems such as data processing lag, low decision-making efficiency, and the inability to effectively accumulate knowledge. Summary of the Invention

[0006] Therefore, it is necessary to provide a collaborative decision-making method, device, equipment, and medium for anomaly detection by unmanned aerial vehicles (UAVs) to address the above-mentioned technical problems, enabling collaborative decision-making on anomalies detected by UAVs.

[0007] Collaborative decision-making methods for anomaly detection by drones include: Acquire design and construction information, physical and mechanical information, and operation and maintenance history information of the targets inspected by drones, and construct a digital twin to serve as an offline twin; The digital twin is evaluated for inspection value to obtain a heat map; based on the heat map, 3D flight planning is performed to obtain a flight semantic path; flight data of the UAV inspecting according to the flight semantic path is acquired to generate a dynamic 3D model as an online twin; Based on offline and online twins, dynamic comparisons are performed in the dimensions of geometry, vision, and physical field to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and to generate comprehensive abnormal data packets. The system obtains the current location, combines it with comprehensive abnormal data packets, generates a shared digital twin scenario, and makes collaborative decisions.

[0008] In one embodiment, the design and construction information, physical and mechanical information, and operation and maintenance history information of the drone inspection target are obtained to construct a digital twin as an offline twin, including: The system acquires the identification, design parameters, material properties, and functional definitions of each component or area within the drone inspection target to obtain design and construction information. The stress distribution, vibration modes, and thermodynamic properties of each component or region in the UAV inspection target are obtained to acquire physical and mechanical information. Obtain maintenance records, historical fault points, anomalies found during routine inspections, and industry accident cases of the drone inspection targets to obtain historical operation and maintenance information; Based on design and construction information, physical and mechanical information, and operation and maintenance history information, a digital twin is constructed to serve as an offline twin.

[0009] In one embodiment, a heatmap is generated by scoring the inspection value of the digital twin, including: A weighted evaluation algorithm is used to score the inspection value of each component or region in the digital twin from multiple dimensions, resulting in a heat map.

[0010] In one embodiment, based on the heatmap, three-dimensional flight planning is performed to obtain a flight semantic path, including: An autonomous path planning algorithm is used to perform three-dimensional flight planning based on heat maps and with the goal of maximizing inspection value, thereby obtaining the flight semantic path.

[0011] In one embodiment, based on offline and online twins, dynamic comparisons are performed across geometric, visual, and physical dimensions to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and to generate a comprehensive anomaly data packet, including: Acquire real-time laser point cloud data generated by the UAV and perform real-time differential calculations with the original 3D geometric model of the offline twin to obtain geometric shape comparison results; High-resolution images captured in real time by drones are overlaid onto an online twin and compared with the baseline texture of the offline twin using a deep learning algorithm to obtain visual texture comparison results. Non-visible light data collected by drones is acquired, converted into a temperature field in real time, and superimposed on an online twin. Then, it is compared with the temperature distribution physical field of the offline twin to obtain the physical field information comparison result. Based on the comparison results of geometric morphology, visual texture, and physical field information, a comprehensive abnormal data packet is generated.

[0012] In one embodiment, obtaining the current location includes: Using real-time localization and mapping technology, the physical world is scanned and spatially registered with an offline twin to obtain the current location.

[0013] In one embodiment, the current location is obtained, and combined with comprehensive abnormal data packets, a shared digital twin scenario is generated, and collaborative decision-making is performed, including: Obtain the current location, combine it with comprehensive anomaly data packets, mark the anomaly information at the real location in the physical world, and generate a shared digital twin scene; Collaborative decision-making is carried out based on shared digital twin scenarios.

[0014] The collaborative decision-making device for anomaly detection by drones includes: The first module is used to acquire the design and construction information, physical and mechanical information, and operation and maintenance history information of the drone inspection target, and to construct a digital twin as an offline twin. The second module is used to score the inspection value of the digital twin and obtain a heat map; based on the heat map, three-dimensional flight planning is performed to obtain the flight semantic path; the flight data of the UAV inspecting according to the flight semantic path is acquired to generate a dynamic three-dimensional model as an online twin; The third module is used to perform dynamic comparisons between offline and online twins in the dimensions of geometry, vision, and physical field to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and generate comprehensive abnormal data packets. The fourth module is used to obtain the current location, combine it with comprehensive abnormal data packets, generate a shared digital twin scenario, and make collaborative decisions.

[0015] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the above method.

[0016] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above method.

[0017] The aforementioned collaborative decision-making method for anomaly detection by drones organically connects the autonomous perception of drones, the intelligent analysis of AI, the immersive interaction of AR, and the remote collaboration platform by constructing a unified framework centered on semantic digital twins. This drives the evolution of infrastructure operation and maintenance from a passive, responsive traditional model to a proactive, predictive, and data-driven intelligent new paradigm, thereby greatly improving the management efficiency, security, and decision-making level of assets throughout their entire lifecycle. It has broad application prospects in the fields of intelligent operation and maintenance and asset management technology, especially in the cross- and deep integration of cutting-edge information technologies such as digital twins, artificial intelligence, drone technology, augmented reality, and 5G communication in industrial applications. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating a collaborative decision-making method for detecting anomalies during drone inspections in one embodiment. Figure 2 This is a structural block diagram of a collaborative decision-making device for drone inspection anomalies in one embodiment; Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. In the description of this application, "multiple sets" means at least two sets, such as two sets, three sets, etc., unless otherwise explicitly specified.

[0021] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection, an electrical connection, a physical connection, or a wireless communication connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0022] Furthermore, the technical solutions of the various embodiments of this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by this application.

[0023] This application provides a collaborative decision-making method for anomaly detection by unmanned aerial vehicles (UAVs), such as... Figure 1 The flowchart shown, in one embodiment, includes: Step 101: Obtain the design and construction information, physical and mechanical information, and operation and maintenance history information of the drone inspection target, and construct a digital twin as an offline twin.

[0024] Specifically: The system acquires the identification, design parameters, material properties, and functional definitions of each component or area within the drone inspection target to obtain design and construction information. The stress distribution, vibration modes, and thermodynamic properties of each component or region in the UAV inspection target are obtained to acquire physical and mechanical information. Obtain maintenance records, historical fault points, anomalies found during routine inspections, and industry accident cases of the drone inspection targets to obtain historical operation and maintenance information; Based on design and construction information, physical and mechanical information, and operation and maintenance history information, a digital twin is constructed to serve as an offline twin.

[0025] More specifically: By importing the BIM model, the identification, design parameters, material properties and functional definitions of each component (such as each bolt) or area (such as each weld) in the drone inspection target (such as a generator) can be obtained, thus obtaining design and construction information; Through simulation technologies such as finite element analysis, the stress distribution, vibration modes, and thermodynamic characteristics of each component or region in the UAV inspection target are obtained, thus acquiring physical and mechanical information. This physical and mechanical information is then superimposed onto a three-dimensional geometric model (a three-dimensional geometric model of the inspection target created based on design data such as BIM (Building Information Modeling). Specifically, the stress, temperature, and other data calculated by finite element analysis are superimposed onto the three-dimensional geometric model in the form of visual layers (such as stress cloud maps), enabling the model to reflect its inherent stress state and internal physical state. The system acquires maintenance records, historical fault points, anomalies found during routine inspections, and industry accident cases of the drone inspection targets since their deployment, obtains operation and maintenance history information, and marks it as timestamp information in the corresponding position of the twin. Based on design and construction information, physical and mechanical information, and operation and maintenance history information, a digital twin is constructed to serve as an offline twin.

[0026] In this step, the constructed digital twin is rich in semantic information. It is not only a precise three-dimensional geometric replica of a physical object (such as a bridge or a wind turbine), but also a knowledge graph and information aggregator that deeply integrates multi-source and heterogeneous information. After the digital twin is constructed, the system has the ability to understand the inspection object, and the logic of inspection path planning also undergoes a qualitative change. At this time, path planning is no longer a geometric problem, but an intelligent decision-making process based on risk and importance assessment.

[0027] It should be noted that simulation technologies such as BIM models and finite element analysis, as well as how to construct digital twins, are all existing technologies and will not be elaborated upon here.

[0028] Step 102: Score the inspection value of the digital twin to obtain a heat map; perform three-dimensional flight planning based on the heat map to obtain the flight semantic path; acquire the flight data of the UAV inspecting according to the flight semantic path, and generate a dynamic three-dimensional model as an online twin.

[0029] Specifically: A weighted evaluation algorithm is used to score the inspection value of each component or region in the digital twin from multiple dimensions, resulting in a heat map. An autonomous path planning algorithm is used to perform three-dimensional flight planning based on heat maps and with the goal of maximizing inspection value, thereby obtaining a flight semantic path. The system acquires flight data from drones that perform inspections according to their flight semantic paths, and generates dynamic 3D models to serve as online twins.

[0030] More specifically: A weighted evaluation algorithm is used to score the inspection value of each component or region in the digital twin from multiple dimensions, resulting in a heat map. For example, a main load-bearing beam will score much higher than a decorative panel, a high-stress area shown in red in the stress cloud map will score higher than other areas, and a weld that has historically had cracks will be automatically prioritized. The heat map is a dynamically generated and visualized graph that represents three-dimensional risks and importance. The process of obtaining the heat map is essentially to transform the rich experience and judgment of in-service engineers and structural experts into machine-executable and quantifiable logical rules. An autonomous path planning algorithm is adopted (the core objective is no longer simple geometric full coverage, but has been transformed into an intelligent task generation guided by maximizing inspection value). Based on the heat map, three-dimensional flight planning is carried out with the maximization of inspection value as the guide, resulting in a non-uniform and focused flight semantic path. Using sensors such as lidar, high-definition cameras, and thermal imagers mounted on drones, the system acquires flight data of drones conducting inspections according to flight semantic paths, scans and perceives physical entities in real time, and dynamically generates dynamic 3D models to serve as online twins.

[0031] In this step, the obtained flight semantic path ensures that the UAV can conduct closer observations at lower speeds, in greater density, and from more angles in high-value areas (high risk, high importance). At the same time, it automatically generates a detailed sensor task list based on the inspection target (such as finding cracks or detecting temperature anomalies), instructing the UAV to automatically adjust the gimbal angle, camera focal length, or switch to infrared thermal imaging mode at specific locations. This ensures that the UAV has the deepest understanding of where to look, what to look for, and how to look before takeoff, thus providing the highest quality data input and the most efficient task execution guarantee for accurate comparison and anomaly detection in subsequent steps.

[0032] The process of generating dynamic 3D models based on flight data transforms the passive data acquisition and post-analysis mode into an active flight-in-analysis and real-time results mode.

[0033] It should be noted that the weighted evaluation algorithm and the autonomous path planning algorithm are both existing technologies and will not be elaborated upon here.

[0034] Step 103: Based on the offline twin and the online twin, perform dynamic comparison in the dimensions of geometry, vision and physical field to obtain geometric shape comparison results, visual texture comparison results and physical field information comparison results, and generate a comprehensive abnormal data packet.

[0035] Specifically: Acquire real-time laser point cloud data generated by the UAV and perform real-time differential calculations with the original 3D geometric model of the offline twin to obtain geometric shape comparison results; High-resolution images captured in real time by drones are overlaid onto an online twin and compared with the baseline texture of the offline twin using a deep learning algorithm to obtain visual texture comparison results. Non-visible light data collected by drones is acquired, converted into a temperature field in real time, and superimposed on an online twin. Then, it is compared with the temperature distribution physical field of the offline twin to obtain the physical field information comparison result. Based on the comparison results of geometric morphology, visual texture, and physical field information, a comprehensive abnormal data packet is generated.

[0036] More specifically: The system acquires real-time laser point cloud data generated by drones and performs real-time differential calculations with the original 3D geometric model of the offline twin to obtain geometric shape comparison results. This comparison can instantly identify any structural changes that are difficult to detect by traditional visual inspection or 2D images with millimeter-level precision, such as the slight settlement of bridge decks, the slight bulging of storage tank bodies, or the twisting deformation of wind turbine blades. These geometric anomalies are directly quantified, rather than subjectively judged. High-definition images captured in real time by drones are overlaid onto an online twin and compared with the baseline texture of the offline twin in a clean or healthy state using a deep learning algorithm to obtain visual texture comparison results. This comparison can efficiently and automatically identify surface defects such as cracks, rust, coating peeling, and oil leaks. Due to its precise positioning in three-dimensional space, it greatly eliminates the interference of environmental factors such as lighting and shadows. The system acquires non-visible light data such as infrared thermal imaging collected by drones, converts it into a temperature field in real time, and superimposes it onto an online twin. Then, it compares the temperature distribution physical field of the offline twin with the preset or historical normal temperature distribution physical field to obtain the physical field information comparison result. This comparison can quickly locate functional and hidden anomalies such as equipment overheating and insulation failure. Based on the comparison results of geometric morphology, visual texture, and physical field information, a comprehensive abnormal data packet is generated.

[0037] In this step, the offline twin represents the device's design state or absolute health state. It is a static but information-rich ideal reference. The offline twin and the online twin are registered at millisecond levels and subjected to continuous difference analysis in the spatiotemporal dimensions. The UAV, leveraging high-precision RTK positioning and SLAM technology, ensures that every frame of data it collects is accurately mapped to the corresponding position in the offline twin model, achieving precise alignment between the physical and digital worlds. Based on this, anomaly detection is no longer a single-dimensional image recognition but a multimodal, cross-dimensional fusion diagnostic.

[0038] Through this multimodal fusion detection of geometry, texture, and physical fields, the system can capture anomalous signals comprehensively and automatically, from surface to volume, from appearance to essence. Any subtle deviation from the ideal state is instantly flagged by the system, generating a comprehensive anomalous data package containing precise 3D coordinates, anomaly type, quantified severity, and multimodal raw evidence (point cloud fragments, high-resolution images, temperature values). This step signifies that the inspection process has evolved from a simple data collection activity into an intelligent, in-flight health diagnosis. The high-confidence, information-rich anomalous data it produces provides the most solid and reliable input for the next step of precise AR annotation and on-site decision-making.

[0039] It should be noted that real-time differential computation, how to attach to the online twin, deep learning algorithms, and how to convert to a temperature field in real time and superimpose it on the online twin are all existing technologies and will not be elaborated here.

[0040] Step 104: Obtain the current location, combine it with comprehensive abnormal data packets, generate a shared digital twin scenario, and make collaborative decisions.

[0041] Specifically: Using real-time positioning and mapping technology, the physical world is scanned and spatially registered with an offline twin to obtain the current location; Based on the current location and combined with comprehensive abnormal data packets, the abnormal information is marked at its real location in the physical world to generate a shared digital twin scene; Collaborative decision-making is carried out based on shared digital twin scenarios.

[0042] More specifically: The comprehensive anomaly data packet is pushed to a central cloud platform and distributed to on-site personnel wearing AR glasses (which could be inspection and verification personnel or subsequent maintenance teams). When on-site personnel activate the AR application, the AR device they wear will use its built-in camera and sensors, employing Simultaneous Localization and Mapping (SLAM) technology, to scan the physical world in real time and perform high-precision, real-time spatial registration with an offline twin to obtain the current location (including precise coordinates and orientation). Based on the current location and combined with comprehensive abnormal data packets, the abnormal information received from the cloud is accurately projected and anchored to the real location in the physical world in the form of three-dimensional virtual annotations, generating a shared digital twin scene. Collaborative decision-making is carried out based on shared digital twin scenarios.

[0043] Among them, the scenarios for generating shared digital twins include: First, a global coarse anchoring is performed during the startup phase to address the slow positioning issue: In traditional methods, after AR devices are started, they rely entirely on their own cameras and IMUs (Inertial Measurement Units) for SLAM (Simultaneous Localization and Mapping) to locate themselves in a large, unknown space (such as an entire bridge). This process is time-consuming and computationally intensive. In this application, when the AR device of the on-site personnel is started, in addition to running its own SLAM, it also performs a global position matching step. It uses its own low-precision sensors such as GPS and compass to obtain a rough initial position (e.g., within 5 meters of error). The system no longer blindly searches the entire offline twin, but directly calls the waypoint data in the UAV's flight path that is closest to the GPS coordinates. This waypoint data not only contains accurate 3D coordinates, but also includes wide-angle environmental photos or LiDAR point cloud slices taken by the UAV at that location. The AR device quickly matches the scene it sees in real time with the reference map provided by the UAV, thereby completing the initial spatial registration within a few seconds and transmitting itself to the correct approximate position in the digital twin, solving the "from 0 to 1" positioning problem and providing a reliable starting point for subsequent fine operations.

[0044] Then, local fine anchoring is performed during the task phase to solve the problem of low accuracy: the accuracy of traditional global SLAM is limited, and there may be accumulated errors of several centimeters or even higher on large structures, which is fatal for annotating millimeter-level cracks or tiny deformations. In this application, when on-site personnel approach an anomaly point, the system extracts high-resolution local feature information about the anomaly point from the comprehensive anomaly data packet, which is taken by the UAV at close range (e.g., a specific bolt combination, a unique weld texture, a special coating peeling). The AR system enters a high-precision feature search mode. The main task of its algorithm is no longer to maintain global positioning, but to prioritize finding and locking this unique feature fingerprint in the field of view. Once a match is successful, the system will use this stable and unchanging local feature in the physical world as the main anchor point to generate an ultra-high-precision local coordinate system. All 3D virtual annotation information is rendered relative to this main anchor point, improving the accuracy of virtual annotation information to the millimeter level, solving the "from 1 to 100" accuracy problem, and ensuring the absolute accuracy of information annotation.

[0045] Furthermore, a dynamic calibration mechanism based on environmental changes is adopted throughout the entire process to provide continuous anti-interference capabilities and solve the problem of easy drift: In traditional methods, environmental factors such as changes in lighting, personnel movement, and equipment vibration can interfere with the stability of SLAM, causing AR images to shake or drift. In this application, after positioning is established, multimodal data collected by a drone is used for dynamic calibration to combat environmental interference such as changes in lighting, occlusion, and vibration in real time. Once the confidence of the primary positioning method (such as visual SLAM) decreases, it immediately calls the backup data (geometric point cloud, heat map) provided by the drone for compensation and calibration, solving the drift and jitter problem of positioning "from 100 to 99", and ensuring the stability and reliability of the entire process. For example, when generating the online twin, the drone has already recorded the lighting direction and texture information at that time. If the lighting on site changes drastically (such as from cloudy to sunny), making visual feature matching difficult, the system can temporarily switch to geometric matching mode to compare the depth information perceived in real time by the AR device with the LiDAR point cloud of the drone to maintain positioning. For thermodynamic feature calibration, the surface temperature field of the operating device is a relatively stable physical feature. If both visual and geometric features are temporarily invalidated due to occlusion or changes, the AR system can call the thermal imaging camera to match the real-time temperature distribution perceived by it with the temperature field in the comprehensive abnormal data packet as an auxiliary positioning calibration method.

[0046] In summary, the process involves three steps: first, rapid global coarse anchoring to quickly enter the scene; second, local fine anchoring to achieve ultra-high precision at key points; and finally, a dynamic calibration mechanism to provide continuous anti-interference capabilities throughout the process. These three steps constitute a hierarchical and progressive anchoring process that moves from macro to micro and is continuously maintained dynamically. Their relationship is that they are activated sequentially, deepened step by step, and protected in parallel. Ultimately, based on prior data from UAVs, hierarchical anchoring and dynamic calibration are performed to guide and calibrate the spatial positioning of AR devices. This solves the inherent problems of slow AR positioning, low accuracy, and easy drift in large, complex, and dynamic industrial scenarios, and generates a shared digital twin scene.

[0047] For example, after virtual annotation, for a detected minor structural deformation, on-site personnel using AR glasses might see a dynamic, flashing 3D arrow steadily pointing to the deformation area that is almost imperceptible to the naked eye, with a virtual information panel floating next to it, displaying quantified deformation data (such as -5mm) in real time; for a tiny crack, the AR system will directly overlay a bright, eye-catching virtual light strip on the real wall, perfectly reproducing the direction and length of the crack; for an overheated device, its surface will be overlaid with a virtual pseudo-color thermal map from blue to red, making the invisible temperature immediately apparent. This spatial anchoring technology ensures that no matter how on-site personnel move or from what angle they observe, these virtual anomaly markers remain firmly and accurately attached to real objects, just like they are affixed to real objects. This completely solves the positioning problem under traditional methods and reduces the time cost of finding anomalies to almost zero. More importantly, the information carried by AR markers is multi-dimensional and interactive. On-site personnel can click on the virtual panel to view historical data of the anomaly point, relevant design drawings, and even multi-angle high-definition original photos and heat maps taken by drones, achieving deep penetration of information. This empowerment is revolutionary: on-site personnel are no longer just executors of instructions, but decision-makers endowed with X-ray vision and super memory, able to make more accurate judgments and preliminary handling plans on the spot based on the most comprehensive and intuitive information.

[0048] The generated shared digital twin scenario allows all stakeholders to immerse themselves in a shared workspace, completely resolving the pain points of spatiotemporal isolation in remote collaboration. The core carrier of this shared workspace is the digital twin that has been constructed and continuously updated in the previous steps. When an anomaly requiring collaborative decision-making is triggered, the system immediately sends a collaborative session invitation to all pre-defined stakeholders (such as the chief engineer at the command center, structural mechanics experts in remote locations, and representatives of overseas equipment manufacturers). These remote participants do not need to be physically present; they can instantly enter this shared digital twin scenario, completely synchronized with the physical site, through their respective terminals—whether providing a fully immersive VR headset or a convenient PC desktop client. In this immersive virtual space, groundbreaking collaborative methods are achieved. First, true spatial consensus is realized: all participants, including those on-site wearing AR glasses, are placed within the same 3D model environment. Remote experts can freely walk, fly, zoom, and rotate within the twin model, like playing a computer game, examining the anomaly highlighted by AR from any angle they need. The real-time position and field of view of on-site personnel are also presented as a virtual image in the model, allowing remote experts to... The expert can immediately understand where he is looking; this method instantly eliminates all communication barriers regarding spatial location, allowing the discussion to immediately focus on the problem itself; secondly, collaborative instructions become intuitive and executable, and remote experts no longer need to painstakingly describe complex repair procedures in words, but can directly perform virtual operations on the shared twin model; for example, a virtual pen can be used to precisely circle the area that needs to be polished on the surface of the equipment; a virtual wrench tool can be retrieved to demonstrate the correct sequence and torque for disassembling a bolt on the model; even an animation can be triggered to simulate the entire process of replacing a component; all these virtual operations and annotations are displayed in real time and synchronously in the form of AR instructions in the AR glasses field of view of the on-site personnel through the cloud; what the on-site personnel see is a transparent virtual guiding hand from the expert, guiding them step by step on the real equipment, realizing hands-on remote teaching, the accuracy and security of which are incomparable to traditional methods, realizing the knowledge accumulation and closed loop of the decision-making process. The entire collaborative session, including all participants' voice conversations, virtual annotations, operational demonstrations, and the finalized maintenance plan, is fully recorded by the system and automatically generates a structured multimedia maintenance log. This log is permanently attached as a new information layer to the lifecycle data archive of the malfunctioning component in the digital twin. When the system revisits this location in the future, it can immediately access this complete decision-making history. This not only solves the current problem, but more importantly, it solidifies valuable experience from solving a complex problem involving multiple experts into a traceable and reusable digital asset, achieving knowledge transfer and accumulation.In this way, the digital twin evolves and grows with each problem-solving process, providing a data foundation for more accurate predictive maintenance in the future. This constructs a complete and automated intelligent inspection loop encompassing perception, diagnosis, decision-making, and execution, from inspection and discovery to collaboration, execution, and knowledge accumulation. For example, updating component status: once a repair plan is determined and executed, the final conclusion in the log (such as "crack repaired" or "bolts tightened") can be parsed by the system, automatically updating the component's status attributes in the digital twin. It also dynamically adjusts risk weights. Expert discussions in collaborative sessions (which can be analyzed using natural language processing to convert speech to text) and final conclusions can be used to dynamically adjust the risk scoring model for that component or similar components. Another example: if experts confirm that a minor deformation at a certain location is due to material fatigue, the system can automatically increase the "inspection value score" of all components using the same materials and with similar service lives. Each repair record increases the "historical fault" weight of that location, allowing drones to inspect that area more frequently and thoroughly in future inspection plans.

[0049] The core improvement lies in the evolutionary closed loop from post-event data correlation to "causal relationship graph construction and generalized reasoning." This involves upgrading the accumulated knowledge from isolated "event-conclusion" logs into a dynamically learning and evolving causal relationship graph of equipment failures, and then using this graph for proactive, cross-asset risk prediction. Specifically: 1) Automatic Generation from “Text Logs” to “Structured Causal Nodes”: After a collaborative session ends, the system does not simply store a multimedia log; it initiates a causal relationship mining module. This module automatically performs entity identification, relationship identification, and node generation through Natural Language Processing (NLP) and data association analysis. Entity identification: Key entities are extracted from the session records, such as specific components (“No. 5 blade root bolt”), abnormal phenomena (“fretting wear”), environmental factors (“recent high humidity”), and operational behaviors (“over-torque operation”). The system analyzes the discussion logic of the relationship analysis experts to identify causal, correlation, and temporal relationships between entities. For example, the system might deduce that “high humidity” causes “bolt surface corrosion,” which exacerbates “fretting wear,” and “over-torque operation” is the root cause. Node generation: These entities and relationships are transformed into a structured “causal subgraph” in the graph database and permanently linked to the corresponding “No. 5 blade root bolt” in the digital twin.

[0050] 2) From "single-point weight adjustment" to "fusion and generalization of causal graphs": This newly generated "causal subgraph" does not exist in isolation. The system will execute a graph fusion algorithm: the pattern matching and fusion system will check whether there are similar failure modes in the entire knowledge graph. If it finds that another wind turbine has also caused "blade cracks" due to "high humidity" and "material fatigue", the system will compare and fuse the two subgraphs to extract a more universal and higher-confidence failure mode; cross-asset generalization reasoning: This is the key step of learning by analogy. The system will use this new fused and more powerful failure mode to actively perform hypothetical queries in the digital twins of all similar assets; the query logic is no longer simply checking historical failure points, but searching for all blade root bolts of "the same batch of materials" that are experiencing "high humidity" environment and "have been in service for more than 5 years", and temporarily upgrading their risk level to immediately generate preventive special inspection tasks.

[0051] The corresponding technical effects include: achieving deep abstraction of knowledge: going beyond simply recording "what broke," the system understands "why it broke" and "under what conditions it broke," representing a qualitative leap from data recording to knowledge modeling, surpassing simple database updates; achieving proactive generalization of predictive maintenance: the system no longer passively waits for the next failure, but proactively predicts and discovers potential risks across the entire equipment group based on learned causal relationships, resulting in an exponential increase in the scope and efficiency of predictive maintenance; and constructing a truly "thinking" digital twin: this mechanism endows the digital twin with preliminary reasoning abilities. After each collaborative decision, its "memory" is enhanced, and its "intelligence" is improved, evolving it from a digital archive into a truly intelligent decision-making brain.

[0052] In this step, AR technology seamlessly, intuitively, and accurately conveys the intelligent insights from the digital twin model to actors in the physical world, greatly shortening the cognitive gap from discovering a problem to understanding it, and laying a solid, immersive interactive foundation for subsequent efficient remote collaboration and final decision-making.

[0053] It should be noted that the central cloud platform, AR glasses, and real-time localization and mapping (SLAM) technology are all existing technologies and will not be elaborated upon here.

[0054] In this embodiment, the problem of intelligent guidance in path planning is solved, enabling UAVs to transform from blind traversal to semantic autonomous navigation based on prior knowledge of risk and importance; the problem of real-time accuracy in anomaly detection is solved, automatically and accurately identifying hidden and minor defects during flight through real-time twin comparison and multimodal data fusion; the problem of the virtual-real information transmission gap is solved, seamlessly and intuitively anchoring the anomaly information diagnosed in the digital space to the corresponding location in the physical world; and the problem of spatiotemporal isolation in collaborative decision-making is solved, constructing an immersive shared twin space that allows multiple experts to make efficient and closed-loop remote decisions in an immersive environment.

[0055] The aforementioned collaborative decision-making method for anomaly detection by drones organically connects the autonomous perception of drones, the intelligent analysis of AI, the immersive interaction of AR, and the remote collaboration platform by constructing a unified framework centered on semantic digital twins. This drives the evolution of infrastructure operation and maintenance from a passive, responsive traditional model to a proactive, predictive, and data-driven intelligent new paradigm, thereby greatly improving the management efficiency, security, and decision-making level of assets throughout their entire lifecycle. It has broad application prospects in the fields of intelligent operation and maintenance and asset management technology, especially in the cross- and deep integration of cutting-edge information technologies such as digital twins, artificial intelligence, drone technology, augmented reality, and 5G communication in industrial applications.

[0056] Specifically, it has the following beneficial effects: 1) Semantic Cognition: A semantic twin-driven UAV autonomous mission planning method based on multi-dimensional information fusion was designed. By quantitatively assessing risks and importance, non-uniform, key-focused inspection paths and sensor task lists are automatically generated. With a semantic digital twin that integrates BIM, physical field, and historical operation and maintenance data as the intelligent hub, the system has for the first time a deep understanding of the inspection objects, which is a prerequisite for achieving true autonomy and intelligence.

[0057] 2) Real-time Diagnostics: A multimodal anomaly detection system based on real-time comparison of online and offline twin models was designed. Real-time registration and geometric, texture and physical field difference operations are performed to automatically and quantitatively identify anomalies. Through real-time and multimodal (geometric, visual and physical field) comparison of online and offline twin models, anomaly detection is improved from lagging, manual two-dimensional interpretation to in-flight, automated, and accurate three-dimensional diagnosis.

[0058] 3) Immersive Collaboration: A closed-loop decision-making method integrating AR spatial anchoring and remote VR collaboration was designed. It accurately projects abnormal information to physical entities and supports multiple parties to conduct immersive interaction, command issuance and decision recording in a shared twin scene. By using AR / VR technology, it completely breaks down the time and space barriers between digital information and the physical world, and between remote experts and on-site personnel, and creates a shared decision-making space that can achieve precise guidance and knowledge accumulation.

[0059] In summary, this application constructs a full-process intelligent working paradigm from semantic understanding to closed-loop decision-making. It is not the application of a single technology, but a highly integrated and mutually empowering organic system.

[0060] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0061] This application also provides a collaborative decision-making device for anomaly detection by unmanned aerial vehicles, such as... Figure 2 As shown, in one embodiment, it includes: a first module 201, a second module 202, a third module 203, and a fourth module 204, wherein: The first module 201 is used to acquire the design and construction information, physical and mechanical information, and operation and maintenance history information of the UAV inspection target, and construct a digital twin as an offline twin. The second module 202 is used to score the inspection value of the digital twin and obtain a heat map; based on the heat map, three-dimensional flight planning is performed to obtain the flight semantic path; the flight data of the UAV inspecting according to the flight semantic path is acquired to generate a dynamic three-dimensional model as an online twin; The third module 203 is used to perform dynamic comparisons based on offline twins and online twins in the dimensions of geometry, vision, and physical field to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and generate comprehensive abnormal data packets. The fourth module 204 is used to obtain the current location, combine it with comprehensive abnormal data packets, generate a shared digital twin scenario, and make collaborative decisions.

[0062] Specific limitations regarding the collaborative decision-making device for drone inspection anomalies can be found in the limitations of the collaborative decision-making method for drone inspection anomalies described above, and will not be repeated here. Each module in the aforementioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0063] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a collaborative decision-making method for anomaly detection during UAV inspections. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0064] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0065] In one embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0066] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0067] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0068] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0069] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0070] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended application documents.

Claims

1. A collaborative decision-making method for anomaly detection by unmanned aerial vehicles (UAVs), characterized in that, include: Acquire design and construction information, physical and mechanical information, and operation and maintenance history information of the targets inspected by drones, and construct a digital twin to serve as an offline twin; The digital twin is evaluated for inspection value to obtain a heat map; based on the heat map, 3D flight planning is performed to obtain a flight semantic path; flight data of the UAV inspecting according to the flight semantic path is acquired to generate a dynamic 3D model as an online twin; Based on offline and online twins, dynamic comparisons are performed in the dimensions of geometry, vision, and physical field to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and to generate comprehensive abnormal data packets. The system obtains the current location, combines it with comprehensive abnormal data packets, generates a shared digital twin scenario, and makes collaborative decisions.

2. The collaborative decision-making method for anomaly detection during UAV inspections according to claim 1, characterized in that, Acquire design and construction information, physical and mechanical information, and operation and maintenance history information of the targets inspected by drones, and construct a digital twin to serve as an offline twin, including: The system acquires the identification, design parameters, material properties, and functional definitions of each component or area within the drone inspection target to obtain design and construction information. The stress distribution, vibration modes, and thermodynamic properties of each component or region in the UAV inspection target are obtained to acquire physical and mechanical information. Obtain maintenance records, historical fault points, anomalies found during routine inspections, and industry accident cases of the drone inspection targets to obtain historical operation and maintenance information; Based on design and construction information, physical and mechanical information, and operation and maintenance history information, a digital twin is constructed to serve as an offline twin.

3. The collaborative decision-making method for anomaly detection during UAV inspections according to claim 2, characterized in that, The digital twin is inspected and its value is scored, resulting in a heat map, including: A weighted evaluation algorithm is used to score the inspection value of each component or region in the digital twin from multiple dimensions, resulting in a heat map.

4. The collaborative decision-making method for anomaly detection during UAV inspections according to claim 3, characterized in that, Based on the heatmap, three-dimensional flight planning is performed to obtain the flight semantic path, including: An autonomous path planning algorithm is used to perform three-dimensional flight planning based on heat maps and with the goal of maximizing inspection value, thereby obtaining the flight semantic path.

5. The collaborative decision-making method for unmanned aerial vehicle (UAV) inspection anomalies according to any one of claims 1 to 4, characterized in that, Based on offline and online twins, dynamic comparisons are performed across geometric, visual, and physical dimensions to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results. A comprehensive anomaly data package is then generated, including: Acquire real-time laser point cloud data generated by the UAV and perform real-time differential calculations with the original 3D geometric model of the offline twin to obtain geometric shape comparison results; High-resolution images captured in real time by drones are overlaid onto an online twin and compared with the baseline texture of the offline twin using a deep learning algorithm to obtain visual texture comparison results. Non-visible light data collected by drones is acquired, converted into a temperature field in real time, and superimposed on an online twin. Then, it is compared with the temperature distribution physical field of the offline twin to obtain the physical field information comparison result. Based on the comparison results of geometric morphology, visual texture, and physical field information, a comprehensive abnormal data packet is generated.

6. The collaborative decision-making method for unmanned aerial vehicle (UAV) inspection anomalies according to any one of claims 1 to 4, characterized in that, Get the current location, including: Using real-time localization and mapping technology, the physical world is scanned and spatially registered with an offline twin to obtain the current location.

7. The collaborative decision-making method for unmanned aerial vehicle (UAV) inspection anomalies according to any one of claims 1 to 4, characterized in that, Obtain the current location, combine it with comprehensive abnormal data packets, generate a shared digital twin scenario, and make collaborative decisions, including: Obtain the current location, combine it with comprehensive anomaly data packets, mark the anomaly information at the real location in the physical world, and generate a shared digital twin scene; Collaborative decision-making is carried out based on shared digital twin scenarios.

8. A collaborative decision-making device for anomaly detection by unmanned aerial vehicles (UAVs), characterized in that, include: The first module is used to acquire the design and construction information, physical and mechanical information, and operation and maintenance history information of the drone inspection target, and to construct a digital twin as an offline twin. The second module is used to score the inspection value of the digital twin and obtain a heat map; based on the heat map, three-dimensional flight planning is performed to obtain the flight semantic path; the flight data of the UAV inspecting according to the flight semantic path is acquired to generate a dynamic three-dimensional model as an online twin; The third module is used to perform dynamic comparisons between offline and online twins in the dimensions of geometry, vision, and physical field to obtain geometric shape comparison results, visual texture comparison results, and physical field information comparison results, and generate comprehensive abnormal data packets. The fourth module is used to obtain the current location, combine it with comprehensive abnormal data packets, generate a shared digital twin scenario, and make collaborative decisions.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images