An unmanned aerial vehicle intelligent inspection and decision system and method based on digital twinning

Abstract

The application belongs to the technical field of unmanned aerial vehicle intelligent inspection, and particularly relates to an unmanned aerial vehicle intelligent inspection and decision system and method based on digital twinning. The system comprises a digital twinning environment construction module for collecting and fusing multi-source data, constructing a three-dimensional network model, and forming a digital twinning static environment base; a multi-source data fusion perception module for collecting and fusing processing multi-source data, generating structured situation information through identification and three-dimensional positioning; an artificial intelligence decision module for generating a recommended action plan when a potential threat is detected; a man-machine collaborative interaction module for visualizing the digital twinning environment, structured situation information, deduction path and recommended action plan, receiving and executing intervention instructions, and generating an optimized decision model. The application overcomes the shortcomings of existing automatic inspection schemes in terms of environmental adaptability, active early warning and real-time dynamic disposal, and realizes rapid, accurate and collaborative response to sudden threats in wide-area complex scenes.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) intelligent inspection technology, specifically relating to an UAV intelligent inspection and decision-making system and method based on digital twins. Background Technology

[0002] In areas such as critical infrastructure security, continuous, intelligent, and efficient inspection and monitoring of specific areas is of great significance. Traditional inspection methods mainly rely on manual patrols and fixed surveillance. However, manual patrols are inefficient, have limited coverage, and can endanger personnel safety in high-risk environments; fixed surveillance has blind spots and cannot flexibly respond to dynamically changing situations. These methods can no longer meet the urgent requirements of modern security for wide-area, real-time, and intelligent responses.

[0003] Conventional automation solutions, such as drone-based automated inspections using pre-planned flight paths, achieve basic automation and reduce operational burden. However, these systems have low intelligence, cannot perceive environmental changes, and cannot autonomously adjust their tasks for tracking or detailed investigation when faced with sudden threats. Operators still need to monitor and intervene manually throughout the process, resulting in slow response and potential missed opportunities for optimal action. Traditional inspection systems rely solely on single pre-planned flight paths and basic sensor data (such as visible light video), which not only fails to cope with interference from complex and dynamic mission environments (such as target concealment and weather changes), leading to missed detections of critical targets or mission interruptions, but also only passively records and alerts after abnormal events occur.

[0004] While inspection solutions with basic computer vision recognition can automatically detect preset targets such as personnel and vehicles, their functionality is limited to "perception." They lack the ability to "cognize" and "decide" about the target's behavioral intentions and the risk level of the event. The entire system is characterized by "rich data but scarce information, isolated perception but weak decision-making." At the same time, due to the lack of fusion and in-depth mining of multi-source information (such as geographic intelligence, electromagnetic spectrum, and historical events), it is impossible to achieve advanced situational awareness and proactive threat prediction.

[0005] More importantly, existing systems completely lack the ability to make intelligent decisions and dynamically reconstruct tasks based on real-time situational awareness. This results in slow emergency response, reliance on human experience for action plans, and an inability to effectively seize fleeting opportunities or windows for handling incidents. This poses a significant challenge for military and police forces in dealing with highly dynamic and confrontational emergencies. In scenarios such as border patrols and security for large-scale events, the suddenness and uncertainty are extremely high, making it difficult for traditional static monitoring and inspection systems to quickly perceive, accurately assess, and coordinate the response to dynamic threats. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide a digital twin-based intelligent inspection and decision-making system and method for unmanned aerial vehicles (UAVs).

[0007] This invention constructs a dynamic task planning and response system with multi-source information fusion, deep situational awareness, and autonomous intelligent decision-making capabilities to overcome the shortcomings of existing automated inspection solutions in terms of environmental adaptability, proactive early warning, and real-time dynamic handling, thereby enabling rapid, accurate, and collaborative responses to sudden threats in wide-area and complex scenarios.

[0008] This invention combines drones with digital twins, theoretically enabling the construction of an intelligent closed loop that moves from physical perception to virtual analysis and then back to guide physical actions. This provides a novel technical path for achieving truly autonomous intelligent inspection and decision-making in unmanned systems.

[0009] To achieve the above objectives, the technical solution of the present invention is as follows.

[0010] The first aspect of this invention provides a digital twin-based intelligent inspection and decision-making system for unmanned aerial vehicles (UAVs), comprising: The digital twin environment construction module is used to collect and fuse satellite imagery, UAV aerial photography, and LiDAR point cloud data to build a three-dimensional network model that carries real geographic coordinates and physical attributes, forming the foundation of the digital twin static environment.

[0011] The multi-source data fusion sensing module is used to simultaneously collect and fuse visible light images, infrared images, and lidar point cloud data from the UAV platform. Through identification and three-dimensional positioning, it generates structured situational information containing target type, location, and attitude in the digital twin static environment base.

[0012] The artificial intelligence decision-making module is used to define the structured situation information as an initial state based on the digital twin environment when a potential threat is detected, call multiple pre-set disposal plans that match the potential threat, perform parallel simulation and quantitative evaluation, and generate a recommended action plan based on a multi-objective optimization algorithm.

[0013] The human-machine collaborative interaction module is used to visualize the digital twin environment, structured situational information, inferred paths and recommended action plans, receive and execute intervention instructions, and record intervention data to generate optimized decision-making models.

[0014] Preferably, the digital twin environment construction module performs the following steps: Acquire multi-source remote sensing data of the target geographic area; the multi-source remote sensing data includes satellite imagery with sub-meter resolution and supplementary aerial photography data of key landforms and sensitive areas by UAVs.

[0015] Aerial triangulation and dense matching are performed on the multi-source remote sensing data to generate a three-dimensional point cloud. The generated three-dimensional point cloud is registered and fused with the lidar point cloud to construct a textureless three-dimensional mesh model that carries real geographic coordinates. Texture mapping is performed on the textureless three-dimensional mesh model to generate a three-dimensional digital twin model with centimeter-level accuracy.

[0016] Modular parameter modeling is performed on regular man-made facilities, and semantic attribute tags containing ID, type, and status are bound to generate facility models with semantic information.

[0017] The three-dimensional digital twin model and the facility model are subjected to geometric reduction and texture optimization to generate a multi-level detail model, which is then integrated in the same geodetic coordinate system to form a static environmental base for the digital twin.

[0018] Preferably, the multi-source data fusion sensing module performs the following steps: Simultaneously acquire visible light images, infrared images, and lidar point cloud data, and perform preprocessing including denoising, enhancement, correction, and filtering.

[0019] The temporally and spatially aligned visible light image is paired with the infrared image as input data and fed into a dual-branch encoder network to extract geometric texture features and thermal radiation features respectively. The geometric texture features and thermal radiation features are fused through a channel attention module to generate a multimodal feature map. The multimodal feature map is decoded and a pixel-level semantic segmentation map or object detection bounding box is output to obtain the two-dimensional detection result of the image.

[0020] Based on the camera-LiDAR extrinsic parameter matrix, the two-dimensional detection results of the image are back-projected onto the three-dimensional point cloud space of the LiDAR. The target point cloud clusters are extracted by clustering to obtain the three-dimensional centroid coordinates and principal component directions of the target point cloud clusters, so as to determine the geographical location and attitude of the target and obtain the recognition and localization results.

[0021] The identification and positioning results are used to generate structured situational information containing target type, location, and attitude in the digital twin static environment base, and encapsulated into a structured situational tuple. The structured situational tuple includes: target ID, target type, three-dimensional coordinates, attitude angle, timestamp, and confidence level.

[0022] Preferably, the artificial intelligence decision-making module performs the following steps: When a potential threat with a confidence level exceeding a preset threshold is detected, the structured situation information is defined as an initial state for parallel simulation, based on the current state of the digital twin environment; the initial state includes threat target information, resource status, and environmental parameters.

[0023] Multiple pre-set response plans matching the potential threat are retrieved from the contingency plan library. An independent simulation thread is created for each pre-set response plan in the digital twin environment and random perturbations are injected. The key performance indicators of each pre-set response plan are recorded.

[0024] A decision matrix is ​​formed based on the key performance indicators of the pre-set treatment plan. A multi-objective optimization algorithm is used for comprehensive evaluation and ranking to generate a list of recommended solutions and the results of advantages and disadvantages analysis.

[0025] Based on the recommended scheme list and the results of the advantages and disadvantages analysis, the final instructions are adjusted and confirmed before being issued to the physical execution unit. After the task is completed, the decision-making process data is stored in the database as experience samples, and the behavioral model in the simulation is iteratively optimized through online reinforcement learning.

[0026] Preferably, the key performance indicators include the estimated task time, the probability of task success, resource consumption, the risk of target loss, and the coordination conflict index.

[0027] Preferably, the human-machine collaborative interaction module performs the following steps: The digital twin environment, structured situational information, projected paths, and recommended action plans are visualized on the display screen. Users can also directly interact with the digital twin environment by selecting targets, drawing flight paths, and clicking decision buttons in a 3D scene.

[0028] Based on the received digital twin environment, structured situational information, simulation path, and recommended action plan, adjust the simulation parameters or modify the execution intervention instructions.

[0029] The system automatically switches information display modes based on the level of tension and pushes intervention information through multimodal alarms when potential threats are detected.

[0030] Intervention behaviors are recorded to form an intervention knowledge base, and the intervention data is fed back to the decision-making model for online learning and to generate an optimized decision-making model.

[0031] A second aspect of the present invention provides a digital twin-based intelligent inspection and decision-making method for unmanned aerial vehicles (UAVs), applied to the digital twin-based intelligent inspection and decision-making system for UAVs described in the first aspect, the method comprising: By utilizing the digital twin environment construction module to collect and integrate satellite imagery, UAV aerial photography, and LiDAR point cloud data, a three-dimensional network model carrying real geographic coordinates and physical attributes is constructed, forming the foundation of the digital twin static environment.

[0032] The system utilizes a multi-source data fusion sensing module to simultaneously collect and fuse visible light images, infrared images, and lidar point cloud data from the UAV platform. Through identification and 3D positioning, it generates structured situational information containing target type, location, and attitude within the digital twin static environment base.

[0033] When a potential threat is detected, the structured situation information is defined as an initial state based on the digital twin environment through the artificial intelligence decision-making module. Multiple pre-set handling schemes that match the potential threat are invoked, and parallel simulation and quantitative evaluation are performed. A recommended action plan is generated based on a multi-objective optimization algorithm.

[0034] The human-computer collaborative interaction module visualizes the digital twin environment, structured situational information, inferred paths, and recommended action plans. It receives and executes intervention instructions and records intervention data to generate optimized decision-making models.

[0035] A third aspect of the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the intelligent inspection and decision-making method for unmanned aerial vehicles based on digital twins as described in the second aspect.

[0036] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed, implements the UAV intelligent inspection and decision-making method based on digital twins as described in the second aspect.

[0037] The beneficial effects of this invention are: 1. This invention designs an intelligent inspection and decision support system for unmanned aerial vehicles (UAVs) based on digital twins and multi-source data fusion. By constructing a complete intelligent closed loop from physical perception to virtual simulation and precise execution, this invention overcomes the drawbacks of traditional inspection methods, such as isolated perception, delayed response, and reliance on manual decision-making. 2. This invention innovatively uses a digital twin environment as a decision-making simulation "sandbox" for drone inspections, synchronizing and rehearsing multiple handling schemes in real time in virtual space. Through multi-threaded parallel simulation and multi-objective optimization analysis, this invention upgrades traditional "human experience-based decision-making" to "data-driven intelligent decision-making," significantly improving the scientific rigor and foresight of command and control decisions. 3. This invention transforms the traditional static, pre-set UAV inspection mode into an intelligent response mode in which the system can autonomously adjust tasks and paths based on real-time dynamic situations. This ability to dynamically adapt to complex environments and sudden threats and to reconfigure tasks significantly improves the system's inspection success rate and robustness in highly dynamic scenarios such as border patrols and key area security. 4. The system and method proposed in this invention provide an efficient and reliable engineering approach for realizing autonomous intelligent inspection and collaborative command of unmanned systems. Its core technical framework is not only applicable to the military and police fields, but also has broad applicability potential for emergency command, urban security, and large-scale facility inspection, demonstrating significant promotional value and application prospects. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the intelligent inspection and decision-making method for unmanned aerial vehicles based on digital twins in an embodiment of the present invention.

[0039] Figure 2 This is a flowchart of the intelligent inspection and decision-making system for unmanned aerial vehicles based on digital twins, as described in this invention. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0041] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0042] Traditional patrol systems rely solely on pre-set flight paths and basic sensor data (such as visible light video). This not only fails to cope with interference from complex and dynamic mission environments (such as target concealment and weather changes), leading to missed detection of critical targets or mission interruptions, but also allows for passive recording and alerting only after anomalies occur. Furthermore, the lack of fusion and in-depth analysis of multi-source information (such as geographic intelligence, electromagnetic spectrum, and historical events) prevents proactive situational awareness and threat prediction. More importantly, existing systems completely lack the ability for intelligent decision-making and dynamic mission reconfiguration based on real-time situational awareness. This results in slow emergency response, reliance on human experience for action plans, and an inability to effectively seize fleeting opportunities or incident response windows, posing a significant challenge to military and police forces in dealing with highly dynamic and confrontational emergencies.

[0043] In scenarios such as border patrols and security for large-scale events, the suddenness and uncertainty are extremely high, and traditional static monitoring and inspection systems are difficult to achieve rapid perception, accurate assessment and coordinated handling of dynamic threats.

[0044] Based on this, the present invention proposes a digital twin-based intelligent inspection and decision support system and method for military and police drones, starting from improving the situational understanding and intelligent decision-making capabilities of drone inspection systems.

[0045] This invention constructs a digital twin that interacts synchronously with the physical drone and the operating environment in real time, and integrates multi-source sensing data and artificial intelligence analysis to achieve the technical effects of autonomous adaptation and intelligent response to dynamic environments, early perception and proactive warning of potential threats, and simulation and collaborative decision support for emergencies. This effectively overcomes the shortcomings of existing systems, such as lag, passivity, isolation and inefficiency.

[0046] To address the aforementioned problems, the main concept of this invention is described below.

[0047] This invention utilizes 3D modeling and simulation technology to panoramically recreate the physical inspection environment in virtual space, establishing a dynamic and interactive digital scenario. This lowers the barrier to understanding and operation of the system, enhancing its usability and promotional value. Based on multi-source data acquisition and AI cognitive technology, the fused inspection data undergoes in-depth mining and situational evolution analysis, constructing an intelligent reasoning mechanism to achieve a shift from passive response to proactive early warning. Through intelligent decision generation and command issuance mechanisms, the optimal inspection strategy is formed based on the deduction results and issued to the physical drones in real time, completing a fundamental upgrade from manual intervention to a human-machine collaborative operation mode. Finally, with the help of a visualized global view and auxiliary decision support, commanders are provided with intuitive global situational awareness and plan execution support, achieving the goals of intelligent inspection process, proactive early warning, and efficient collaboration. Through the above design, a new inspection paradigm of "system autonomous decision-making and efficient human-machine collaboration" is realized.

[0048] Compared with the prior art, the beneficial effects of the present invention are as follows: Existing technologies mostly employ rule-based or threshold-based automated response mechanisms, while this invention proposes an intelligent decision-making method based on parallel simulation and deduction using digital twins. This method pre-simulates the execution process and consequences of multiple tactical options in a virtual space and outputs quantified performance evaluation indicators for each option. This elevates the decision-making basis from a single threshold to multi-dimensional data comparison, achieving a shift from passive alerting to proactive, forward-looking decision-making.

[0049] Existing digital twins primarily serve as high-precision mirror images of physical entities for condition monitoring and predictive maintenance. The digital twin of this invention, however, functions as a dynamic tactical game space. It not only maps the physical world but, more importantly, carries and runs tactical simulation models, providing "sand table" capabilities for command and decision-making.

[0050] Unlike existing automated models that rely on a "human-machine hybrid" approach, this invention proposes a human-machine hybrid intelligent collaborative mechanism. The system combines AI's rapid computation and simulation capabilities with human experience, judgment, and command skills. Human commanders can intervene and take over throughout the entire decision-making process, forming an online learning and optimization loop based on human feedback.

[0051] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0052] The specific implementation scheme of this invention revolves around intelligent inspection and decision-making of military and police drones, and unfolds from four core aspects: "real-world panoramic reproduction, multi-source data fusion, AI intelligent decision-making, and human-machine collaborative operation".

[0053] A method for intelligent inspection and decision-making using unmanned aerial vehicles (UAVs) based on digital twins includes the following steps: Step 1: Collect and fuse satellite imagery, UAV aerial photography, and LiDAR point cloud data to construct a 3D network model that carries real geographic coordinates and physical attributes, forming a static environmental foundation for the digital twin.

[0054] The specific methods for forming a static environment foundation for a digital twin are as follows: Acquire multi-source remote sensing data of the target geographic area; the multi-source remote sensing data includes satellite imagery with sub-meter resolution and supplementary aerial photography data of key landforms and sensitive areas by UAVs.

[0055] Aerial triangulation and dense matching are performed on the multi-source remote sensing data to generate a three-dimensional point cloud. The generated three-dimensional point cloud is registered and fused with the lidar point cloud to construct a textureless three-dimensional mesh model that carries real geographic coordinates. Texture mapping is performed on the textureless three-dimensional mesh model to generate a three-dimensional digital twin model with centimeter-level accuracy.

[0056] Modular parameter modeling is performed on regular man-made facilities, and semantic attribute tags containing ID, type, and status are bound to generate facility models with semantic information.

[0057] The three-dimensional digital twin model and the facility model are subjected to geometric reduction and texture optimization to generate a multi-level detail model, which is then integrated in the same geodetic coordinate system to form a static environmental base for the digital twin.

[0058] The first step is to use 3D modeling and simulation technology to fully reproduce the physical inspection environment in a virtual space.

[0059] Step 1.1: Collect data from multiple sources.

[0060] To clarify the geographical scope of the digital twin model, three data sources were used in a coordinated manner: satellite remote sensing imagery, UAV aerial photography, and LiDAR point clouds.

[0061] Taking a 50-kilometer-long border line and a 5-kilometer-deep strip of land as an example, first, define the target area as a 50-kilometer-long border line and a 5-kilometer-deep strip of land. Then, utilize commercial or military satellite resources to acquire multispectral and panchromatic satellite imagery data with sub-meter resolution (e.g., 0.5 meters) covering the entire target area.

[0062] Based on this, high-density UAV flight routes are planned for undulating terrain areas, densely vegetated forest areas, river valleys and waterways, and other key landforms and sensitive areas to collect supplementary aerial photography data, ensuring comprehensive data coverage and obtaining high-resolution UAV aerial images. Simultaneously, LiDAR point cloud data is collected to obtain multi-source data with high-precision 3D terrain and land cover information.

[0063] Step 1.2, Multi-source data fusion and construction of 3D network model.

[0064] First, the satellite imagery, UAV aerial photography, and LiDAR point cloud data collected in step 1.1 are imported into reality 3D modeling software (such as ContextCapture, Pix4D, etc.) for automated processing. The reality 3D modeling software automatically identifies corresponding points in all images, performs aerial triangulation, and restores the accurate exterior orientation elements of all images. Then, a high-density 3D point cloud is generated based on a dense matching algorithm. The generated high-density 3D point cloud is registered and fused with the airborne LiDAR point cloud to construct a more accurate textureless 3D mesh model that reflects real geographic coordinates.

[0065] Based on the geometric model, the optimal texture information is further extracted and calculated from the oblique photogrammetric images. The optimal texture information is then accurately mapped to the corresponding surface of the 3D mesh model, ultimately generating a real-world 3D mesh model with centimeter-level precision and realistic texture, which serves as the base for the static environment of the digital twin.

[0066] Step 1.3, Modular modeling and semantic binding of critical facilities.

[0067] For regular man-made structures such as sentry posts, barbed wire fences, and camera towers, a modular modeling approach is adopted to improve efficiency and achieve semantic information binding. The specific process is as follows: First, in 3D modeling software, simplified geometric models of the man-made structures are constructed using basic geometric shapes such as cubes and cylinders. Then, each simplified geometric model is parametrically labeled and tagged to further generate a refined model that carries physical attributes. For example, taking a "camera tower" as an example, semantic attribute tags such as "ID: CAM_001", "Type: Infrared PTZ Camera", and "Status: Online" can be added to the model. This semantic attribute information will be used in subsequent simulation and deduction phases, supporting functional analysis and interactive operations.

[0068] Step 1.4, Model Optimization and Integration.

[0069] To ensure smooth system operation, the refined models of key facilities generated in step 1.3 need to be optimized. First, a model reduction tool is used to reduce the number of faces in the refined model while preserving key geometric features. Non-critical details are removed, and textures are compressed and merged to generate a Level of Detail (LOD) model. Based on the LOD mechanism, the system can dynamically load LOD models of different precision according to the viewpoint distance, achieving a balance between rendering efficiency and visual effects, resulting in a modular facility model.

[0070] After model optimization, the optimized realistic 3D terrain model and modular facility model are integrated into a professional visualization platform. All model elements are unified to the same geodetic coordinate system to ensure that the virtual environment and the real world correspond precisely in spatial location. Ultimately, a high-fidelity digital twin static environment base for border patrol is constructed, carrying real geographic coordinates, physical attributes, and scalable behavioral logic.

[0071] Step 2: Simultaneously collect and fuse visible light images, infrared images, and lidar point cloud data from the UAV platform. Through identification and 3D positioning, generate structured situational information containing target type, location, and attitude.

[0072] The specific method for generating structured situational information containing target type, location, and attitude is as follows: Simultaneously acquire visible light images, infrared images, and lidar point cloud data, and perform preprocessing including denoising, enhancement, correction, and filtering.

[0073] The temporally and spatially aligned visible light image is paired with the infrared image as input data and fed into a dual-branch encoder network to extract geometric texture features and thermal radiation features respectively. The geometric texture features and thermal radiation features are fused through a channel attention module to generate a multimodal feature map. The multimodal feature map is decoded and a pixel-level semantic segmentation map or object detection bounding box is output to obtain the two-dimensional detection result of the image.

[0074] Based on the camera-LiDAR extrinsic parameter matrix, the two-dimensional detection results of the image are back-projected onto the three-dimensional point cloud space of the LiDAR. The target point cloud clusters are extracted by clustering to obtain the three-dimensional centroid coordinates and principal component directions of the target point cloud clusters, so as to determine the geographical location and attitude of the target and obtain the recognition and localization results.

[0075] The identification and positioning results are used to generate structured situational information containing target type, location, and attitude in the digital twin static environment base, and encapsulated into a structured situational tuple. The structured situational tuple includes: target ID, target type, three-dimensional coordinates, attitude angle, timestamp, and confidence level.

[0076] Multi-source data fusion: This invention achieves in-depth understanding and evolutionary analysis of inspection situations by fusing data from multiple types of sensors carried by the drone and introducing AI cognitive technology. The specific steps are as follows: Step 2.1: Synchronous acquisition and preprocessing of multi-sensor data.

[0077] The control drone platform synchronously triggers the onboard visible light camera, infrared thermal imager, and LiDAR to collect data, ensuring time synchronization of data from different sensors via hardware signals or high-precision timestamps (such as the PTP protocol). The raw data is then transmitted back to the ground control station via 5G or a dedicated data link for decoding. For visible light images, Gaussian filtering for noise reduction and contrast-limited adaptive histogram equalization are performed to enhance image details. For infrared images, non-uniformity correction and temperature calibration are performed to generate an absolute temperature distribution map. For the airborne LiDAR point cloud, statistical outlier removal and voxel grid downsampling filtering are performed to eliminate noise and reduce data volume.

[0078] Step 2.2, visible light and infrared image fusion recognition.

[0079] Temporally and spatially aligned visible light images are paired with infrared images as input data. The paired images are then fed into a dual-branch encoder (the backbone network can be a pre-trained ResNet-50). The visible light branch focuses on extracting the target's geometric shape and texture features, while the infrared branch focuses on extracting the target's thermal radiation features and thermal contours. In the last convolutional layer of the encoder, a channel attention module is used to weightedly fuse the feature maps output from the two branches, generating a unified multimodal feature map. The fused feature map is then fed into a decoder (composed of transposed convolutional layers), outputting pixel-level semantic segmentation maps or target detection bounding boxes. This result integrates complementary information from visible light and infrared light.

[0080] Step 2.3, 3D localization and attitude estimation.

[0081] Using a pre-calibrated camera-LiDAR extrinsic parameter matrix, the two-dimensional detection box obtained in step 2.2 is back-projected onto the three-dimensional point cloud space of the airborne LiDAR. Within the projected three-dimensional spatial region, Euclidean clustering is used to extract the three-dimensional point cloud clusters belonging to the target. The three-dimensional centroid coordinates (X, Y, Z) of the point cloud clusters are calculated as the precise geographical location of the target. Principal component analysis is performed on the point cloud clusters, and the direction of the first principal component vector is taken as the target's orientation or attitude angle.

[0082] Step 2.4: Generate structured situation information.

[0083] The identification and localization results from the aforementioned steps are encapsulated into structured situational awareness tuples. The tuple data format for each target is: {Target ID, Target Type, 3D Coordinates (X, Y, Z), Attitude Angle, Timestamp, Confidence Score}. This list of tuples serves as a unified, semantical foundation for situational awareness, directly supplied to subsequent decision support modules.

[0084] Step 3: When a potential threat is detected, the structured situation information is defined as the initial state based on the digital twin environment. Multiple pre-set response plans that match the potential threat are invoked, and parallel simulation and quantitative evaluation are performed. A recommended action plan is generated based on a multi-objective optimization algorithm.

[0085] The specific method for generating recommended action plans is as follows: When a potential threat with a confidence level exceeding a preset threshold is detected, the structured situation information is defined as an initial state for parallel simulation, based on the current state of the digital twin environment. The initial state includes threat target information, resource status, and environmental parameters.

[0086] Multiple pre-set response plans matching the potential threat are retrieved from the contingency plan library. An independent simulation thread is created for each pre-set response plan in the digital twin environment and random perturbations are injected. The key performance indicators of each pre-set response plan are recorded. The key performance indicators include the expected task time, the probability of task success, resource consumption, the risk of target loss, and the coordination conflict index.

[0087] A decision matrix is ​​formed based on the key performance indicators of the pre-set treatment plan. A multi-objective optimization algorithm is used for comprehensive evaluation and ranking to generate a list of recommended solutions and the results of advantages and disadvantages analysis.

[0088] Based on the recommended scheme list and the results of the advantages and disadvantages analysis, the final instructions are adjusted and confirmed before being issued to the physical execution unit. After the task is completed, the decision-making process data is stored in the database as experience samples, and the behavioral model in the simulation is iteratively optimized through online reinforcement learning.

[0089] AI-powered intelligent decision-making: Building upon multi-source data fusion and deep situational awareness, this invention introduces an artificial intelligence decision-making model. This model transforms the fused and mined high-quality situational information into forward-looking and quantifiable action plans, thus achieving a closed loop from "perception and cognition" to "decision and command." The flowchart for this process is as follows: Figure 1 As shown. The specific steps are as follows: Step 3.1, Decision triggering and inference initialization.

[0090] When the system detects a potential threat (such as identifying a "lurking personnel" with a confidence level >90%), the AI ​​decision-making module is automatically activated. The system uses the current digital twin as a baseline to freeze the current battlefield situation, defining it as the initial state for parallel simulation. This state includes at least: the current location, type, and AI-predicted movement trajectories of all threat targets; the status of all available friendly resources (e.g., drone A's location, 80% battery; patrol vehicle B's location, 5 km away); and current environmental parameters (e.g., light intensity, weather conditions, terrain data).

[0091] Step 3.2: Perform parallel simulation and quantitative evaluation of multiple contingency plans.

[0092] The system retrieves multiple pre-defined response plans (e.g., "drone tracking," "forward identification," and "coordinated containment") from a contingency plan library that match the current threat. Within the digital twin environment, an independent simulation thread is created for each plan, injecting random disturbances (such as simulated target sudden turns or device communication delays) to simulate real-world uncertainties. In each parallel thread, a set of quantified key performance indicators is calculated and recorded in real time, including: estimated task time (T), task success probability (P_success), resource consumption (C), target loss risk (R_loss), and cooperative conflict index (I_conflict).

[0093] Step 3.3: Evaluation, ranking, and recommendation of the proposed solutions.

[0094] Collect quantitative indicators for all contingency plans to form a decision matrix. Use a multi-objective optimization algorithm to comprehensively evaluate and rank these plans, identifying the optimal solution set. Provide the commander with a ranked list of recommended plans, along with a comparative analysis of the advantages and disadvantages of each plan (e.g., "Plan A has the highest success rate but takes the longest; Plan C has the fastest response but carries the highest risk").

[0095] Step 3.4, Instruction issuance and model iterative optimization.

[0096] Commanders can adjust and confirm final instructions based on recommended plans. Instructions are directly issued to the corresponding physical units (UAVs, vehicles, etc.) for execution via the system. During mission execution, the system continuously compares real-world feedback data with simulation predictions in real time. After the mission, the system stores the entire decision-making process data (initial state, execution plan, and final result) as an experience sample in the database. Through online reinforcement learning technology, these new samples are used to periodically fine-tune and optimize the agent behavior model in the simulation, enabling the system's decision-making and prediction capabilities to continuously evolve.

[0097] The AI ​​intelligent decision-making module can be implemented based on, but is not limited to, one or more of the following artificial intelligence methods: 1) Rule-based reasoning expert systems are used to handle predefined typical scenarios.

[0098] 2) The case-based reasoning system generates decision-making suggestions by retrieving and adapting historical handling plans.

[0099] 3) Based on multi-objective optimization and parallel simulation, the consequences of various action plans are quantitatively evaluated and weighed.

[0100] 4) Policy networks based on reinforcement learning / multi-agent reinforcement learning learn optimal decision-making policies autonomously through interaction with the digital twin environment.

[0101] 5) A hybrid architecture of the above methods, such as using case reasoning to provide an initial solution, verifying and optimizing it through simulation, and using reinforcement learning models to fine-tune the inference parameters online.

[0102] Step 4: Visualize the digital twin environment, structured situational information, projected paths, and recommended action plans; receive and execute intervention instructions; and record intervention data to generate an optimized decision-making model.

[0103] The method for generating an optimized decision model is as follows: The system visualizes the digital twin environment, structured situational information, projected paths, and recommended action plans on a display screen. Users can directly interact with the system by selecting targets, drawing flight paths, and clicking decision buttons within the 3D scene of the digital twin. Based on the received digital twin environment, structured situational information, projected paths, and recommended action plans, the system adjusts projected parameters or modifies intervention commands. The system automatically switches information display modes based on the level of situational tension and pushes intervention information via multimodal alarms when potential threats are detected. Intervention behaviors are recorded to form an intervention knowledge base, and this intervention data is fed back to the decision-making model for online learning and the generation of an optimized decision-making model.

[0104] Human-machine collaborative operation: Combining AI's decision-making capabilities with human judgment to achieve efficient and natural command and control. Specific steps are as follows: Visualized decision-making is enabled. The command center's large screen directly displays the virtual environment, AI recognition results, simulated paths, and decision recommendations, while also displaying real-time drone footage, providing commanders with a clear overview of the situation. Commanders no longer need complex remote control operations; they can directly select targets, draw flight paths, and click decision buttons within the 3D scene. Instead of fragmented data interfaces, commanders now face a unified and intuitive panoramic view of the operational situation.

[0105] The system allows commanders to intervene and take over at any stage. Commanders can veto AI suggestions, adjust simulation parameters, or directly modify drone flight paths during mission execution. The system can automatically switch display modes based on the level of situational tension. During routine patrols, it displays concise information; once a threat is detected, the system automatically highlights the relevant area and proactively pushes the highest-level alarm information to the commander using multiple modalities, including visual (flashing) and audible (voice) alerts.

[0106] All human interventions, including modifications to AI suggestions, reasons for rejection, and manual data control, are meticulously recorded by the system and compiled into an "intervention knowledge base." This valuable data on human expert experience is fed back into the AI ​​decision-making model for supervised online learning and optimization, thereby making AI suggestions increasingly aligned with human thinking habits and leadership skills, achieving a spiral of hybrid intelligence.

[0107] This invention constructs a complete intelligent closed loop, from physical world perception to virtual space cognition, deduction, and decision-making, and then back to precise execution in the physical world. It elevates military and police drone patrols from a single "eye in the sky" to an intelligent combat system integrating comprehensive monitoring, intelligent analysis, scientific decision-making, and collaborative action, significantly improving the perception capabilities, decision-making efficiency, and handling effectiveness of tasks such as border patrols and key area security. The overall system architecture diagram of this invention is as follows. Figure 2 As shown.

[0108] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A digital twin-based intelligent inspection and decision-making system for unmanned aerial vehicles (UAVs), characterized in that, include: The digital twin environment construction module is used to collect and fuse satellite imagery, UAV aerial photography, and LiDAR point cloud data to build a three-dimensional network model that carries real geographic coordinates and physical attributes, forming the static environment base of the digital twin. The multi-source data fusion perception module is used to simultaneously collect and fuse visible light images, infrared images, and lidar point cloud data from the UAV platform. Through identification and three-dimensional positioning, it generates structured situational information containing target type, location, and attitude in the digital twin static environment base. The artificial intelligence decision-making module is used to define the structured situation information as the initial state based on the digital twin environment when a potential threat is detected, call multiple pre-set handling schemes that match the potential threat, perform parallel simulation and quantitative evaluation, and generate recommended action plans based on multi-objective optimization algorithms. The human-machine collaborative interaction module is used to visualize the digital twin environment, structured situational information, inferred paths and recommended action plans, receive and execute intervention instructions, and record intervention data to generate optimized decision-making models.

2. The UAV intelligent inspection and decision-making system based on digital twin as described in claim 1, characterized in that, The digital twin environment construction module performs the following steps: Acquire multi-source remote sensing data of the target geographic area; the multi-source remote sensing data includes satellite imagery with sub-meter resolution and supplementary aerial photography data from UAVs of key landforms and sensitive areas; Aerial triangulation and dense matching are performed on the multi-source remote sensing data to generate a three-dimensional point cloud. The generated three-dimensional point cloud is registered and fused with the lidar point cloud to construct a textureless three-dimensional mesh model that carries real geographic coordinates. Texture mapping is performed on the textureless three-dimensional mesh model to generate a three-dimensional digital twin model with centimeter-level precision. Modular parameter modeling is performed on regular man-made facilities, and semantic attribute tags containing ID, type, and status are bound to generate facility models with semantic information; The three-dimensional digital twin model and the facility model are subjected to geometric reduction and texture optimization to generate a multi-level detail model, which is then integrated in the same geodetic coordinate system to form a static environmental base for the digital twin.

3. The UAV intelligent inspection and decision-making system based on digital twin as described in claim 1, characterized in that, The multi-source data fusion sensing module performs the following steps: Simultaneously acquire visible light images, infrared images, and lidar point cloud data, and perform preprocessing including denoising, enhancement, correction, and filtering; The temporally and spatially aligned visible light image and infrared image are paired as input data and fed into a dual-branch encoder network to extract geometric texture features and thermal radiation features, respectively. The channel attention module fuses geometric texture features and thermal radiation features to generate a multimodal feature map; the multimodal feature map is decoded and pixel-level semantic segmentation map or object detection bounding box is output to obtain the two-dimensional detection result of the image; Based on the camera-LiDAR extrinsic parameter matrix, the two-dimensional detection results of the image are back-projected onto the three-dimensional point cloud space of the LiDAR. The target point cloud clusters are extracted by clustering to obtain the three-dimensional centroid coordinates and principal component directions of the target point cloud clusters, so as to determine the geographical location and attitude of the target and obtain the recognition and localization results. The identification and localization results are used to generate structured situational information containing target type, location, and attitude in the digital twin static environment base, and encapsulated into structured situational tuples. The structured situation tuple includes: target ID, target type, three-dimensional coordinates, attitude angle, timestamp, and confidence level.

4. The UAV intelligent inspection and decision-making system based on digital twin as described in claim 1, characterized in that, The artificial intelligence decision-making module performs the following steps: When a potential threat with a confidence level exceeding a preset threshold is detected, the structured situation information is defined as an initial state for parallel simulation, based on the current state of the digital twin environment; the initial state includes threat target information, resource status, and environmental parameters. Multiple pre-set response plans matching the potential threat are retrieved from the contingency plan library. An independent simulation thread is created for each pre-set response plan in the digital twin environment and random perturbations are injected. The key performance indicators of each pre-set response plan are recorded. A decision matrix is ​​formed based on the key performance indicators of the pre-set treatment plan. A multi-objective optimization algorithm is used for comprehensive evaluation and ranking to generate a list of recommended solutions and the results of advantages and disadvantages analysis. Based on the recommended scheme list and the results of the advantages and disadvantages analysis, the final instructions are adjusted and confirmed before being issued to the physical execution unit. After the task is completed, the decision-making process data is stored in the database as experience samples, and the behavioral model in the simulation is iteratively optimized through online reinforcement learning.

5. The UAV intelligent inspection and decision-making system based on digital twin as described in claim 4, characterized in that, The key performance indicators include estimated task time, task success probability, resource consumption, target loss risk, and coordination conflict index.

6. The UAV intelligent inspection and decision-making system based on digital twin as described in claim 1, characterized in that, The human-computer collaborative interaction module performs the following steps: The digital twin environment, structured situational information, deduced paths, and recommended action plans are visualized on the display screen, and users can directly interact by selecting targets, drawing flight paths, and clicking decision buttons in the 3D scene of the digital twin environment. Based on the received digital twin environment, structured situation information, simulation path, and recommended action plan, adjust the simulation parameters or modify the execution intervention instructions; The system automatically switches information display modes based on the level of tension and pushes intervention information through multimodal alarms when potential threats are detected. Intervention behaviors are recorded to form an intervention knowledge base, and the intervention data is fed back to the decision-making model for online learning and to generate an optimized decision-making model.

7. A method for intelligent inspection and decision-making using unmanned aerial vehicles (UAVs) based on digital twins, characterized in that, The method, applied to the unmanned aerial vehicle (UAV) intelligent inspection and decision-making system based on digital twins as described in any one of claims 1 to 6, comprises: By using the digital twin environment construction module to collect and integrate satellite imagery, UAV aerial photography, and LiDAR point cloud data, a three-dimensional network model carrying real geographic coordinates and physical attributes is constructed, forming the static environment base of the digital twin. The visible light image, infrared image and lidar point cloud data of the UAV platform are collected and processed simultaneously by the multi-source data fusion sensing module. Through identification and three-dimensional positioning, structured situation information containing target type, position and attitude is generated in the digital twin static environment base. When a potential threat is detected, the structured situation information is defined as an initial state based on the digital twin environment through the artificial intelligence decision-making module. Multiple pre-set handling schemes that match the potential threat are invoked, and parallel simulation and quantitative evaluation are performed. A recommended action plan is generated based on a multi-objective optimization algorithm. The human-computer collaborative interaction module visualizes the digital twin environment, structured situational information, inferred paths, and recommended action plans. It receives and executes intervention instructions and records intervention data to generate optimized decision-making models.

Images