Unmanned aerial vehicle image payload target real-time perception and intention speculation method and system

Abstract

The present application belongs to the technical field of real-time perception, in particular to a UAV image load target real-time perception and intention speculation method and system, the method comprising the following steps: S1, constructing a multi-modal UAV load image dataset; S2, constructing a lightweight target perception network PROYOLO based on the YOLO architecture; S3, designing a multi-scenario target intention speculation model; S4, deploying the optimized PROYOLO network and intention speculation model on a domestic AI chip platform. The present application realizes integrated perception and early warning functions covering multiple types of targets, can be efficiently deployed on a UAV embedded platform, meets the demand for real-time processing, and significantly improves the perception reliability, decision-making intelligence and deployment adaptability of the UAV intelligent body system in real complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of real-time perception technology, and in particular to a method and system for real-time perception and intent inference of targets in UAV image payloads. Background Technology

[0002] In recent years, drones have rapidly evolved from traditional, single-function aerial photography and mapping platforms into intelligent systems integrating perception and understanding, decision-making and execution. They are playing an increasingly important role in key national intelligent transportation and public safety scenarios such as forest fire emergency response, maritime vessel monitoring, urban low-altitude security, battlefield reconnaissance, and energy inspection. However, with the increasing complexity of missions, drones often need to operate autonomously in environments with limited communication, strong interference, low light, or highly dynamic and complex resistance. This poses a dual challenge to drone onboard systems: on the one hand, they need to achieve robust real-time detection under conditions of low signal-to-noise ratio, dense small targets, and cluttered backgrounds; on the other hand, they need to accurately predict what the target will do, i.e., have the ability to dynamically infer intent, in order to support high-level decision-making such as early warning, avoidance, and collaboration.

[0003] Current mainstream UAV vision systems still face significant bottlenecks in both of these aspects. At the perception layer, traditional target detection models (such as the YOLO series and RetinaNet) exhibit significantly reduced detection performance in scenarios involving smoke obscuration, high-speed motion blur, or small targets at long distances. They suffer from high false negative rates and unstable localization, making it difficult to reliably capture critical targets (such as fire extinguishers and firefighters, suspicious vessels, and hostile UAVs) in disaster scenarios. More critically, at the cognitive layer, existing systems generally lack the ability to model the semantics of target behavior. Most work remains at the level of static detection or simple tracking, failing to accurately infer the target's dynamic intent from trajectory dynamics, motion patterns, and spatial context in real time. This results in the system being able to see the target but struggling to understand it, severely restricting the high-order autonomy of UAV agents.

[0004] Meanwhile, the deployment of high-performance vision systems heavily relies on NVIDIA's CUDA ecosystem-based single AI chip. Amidst a tense international environment and escalating supply chain risks, Huawei's AI chips, representing a non-CUDA platform, have elevated algorithm deployment from an engineering adaptation issue to a core path for building secure, reliable, efficient, autonomous, and diversified intelligent systems. While domestically produced AI chips (such as Huawei Ascend and Cambricon MLU) possess advantages in energy efficiency and self-control, the lack of specific collaborative optimization frameworks tailored to their hardware characteristics hinders the efficient deployment of advanced algorithms, resulting in a disconnect between "advanced algorithms and difficult implementation."

[0005] Therefore, building an edge-side intelligent vision system that is compatible with domestically produced AI chips and has both high-precision perception and lightweight intent reasoning capabilities has become a strategic requirement. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a method and system for real-time perception of target and intent inference of UAV image payload.

[0007] The method for real-time target perception and intent inference of UAV image payload proposed in this invention includes the following steps:

[0008] S1. Construct a multimodal UAV payload image dataset. The dataset covers a variety of typical intelligent transportation and emergency scenarios and includes bounding boxes, small target labels, and behavioral intent annotations.

[0009] The multimodal drone payload image dataset is the DUT-UAV dataset, which contains image data from three major sub-scenarios: forest fire emergency response, maritime vessel monitoring, and urban aerial inspection. The categories include at least: cars, trucks, buses, people, boats, birds, drones, fire extinguishers, fire, and buildings.

[0010] S2. Construct a lightweight target perception network, PROYOLO, based on the YOLO architecture. The PROYOLO network includes:

[0011] S2.1 Improved backbone network, in which the spatial pyramid pooling module is replaced with the SPPFCSPC module to enhance the multi-scale feature fusion capability; the SPPFCSPC module fuses spatial features of different scales and cross-stage features by performing multi-scale max pooling in parallel and combining cross-stage partial connection mechanism.

[0012] S2.2 Improved Feature Pyramid Network: The Feature Pyramid Network includes a new P2 feature layer, which is used to fuse shallow spatial details and deep semantic features to enhance the feature representation of small objects. The P2 feature layer outputs a high-resolution feature map, which is fused with deep features through upsampling, and then purified by the MobileNetV3_BLOCK module.

[0013] S2.3 Introduces a deformable convolution module into the P2 feature layer to improve the robustness of feature extraction for deformable targets;

[0014] S2.4 uses the Wise-IoU loss function as the bounding box regression loss to dynamically adjust the optimization weights for samples of different difficulty.

[0015] S3. Design a multi-scenario target intent inference model. By integrating Kalman filter trajectory prediction, dense optical flow motion analysis and principal component analysis of spatial activity regions, dynamic behavior intent inference and early warning modeling of targets across scenarios can be achieved.

[0016] The intent inference model performs differentiated inference for different target types:

[0017] For aerial targets, a three-dimensional Kalman filter is used to establish a state model for trajectory smoothing and prediction, and the collision time is calculated to classify the threat level.

[0018] For fire targets, the direction and speed of fire spread are analyzed based on PROYOLO network detection boxes and Farneback dense optical flow, and graded alarms are triggered by combining area thresholds.

[0019] For vessel targets, the spatial relationship between the center point of the detection frame and the preset channel area is used to determine whether the vessel is deviating from its course and to trigger an early warning.

[0020] S4. Deploy the optimized PROYOLO network and intent inference model on a domestic AI chip platform to perform model compression, operator adaptation and inference acceleration, and achieve real-time processing on the edge.

[0021] Specifically, it includes:

[0022] S4.1 performs quantization-aware training and channel pruning on the PROYOLO network to compress the model size;

[0023] S4.2 converts the optimized model into ONNX format, and further converts it into an OM model adapted to domestic AI chips;

[0024] On a domestically developed AI chip development board, S4.3 uses a custom operator fusion framework to fuse multiple independent operators in the network, reducing operator call overhead.

[0025] S4.4 employs block loading, double buffering, and a full-thread call strategy to schedule memory and computing resources, thereby improving inference efficiency.

[0026] The UAV image payload target real-time perception and intent inference system proposed in this invention is used to implement the above-mentioned UAV image payload target real-time perception and intent inference method. The system includes:

[0027] The data building module is used to build and store multimodal UAV payload image datasets;

[0028] The model training and optimization module is used to train and optimize the PROYOLO object perception network;

[0029] The intent inference module is used to execute a multi-scenario target intent inference model to perform behavioral understanding and early warning of detected targets.

[0030] The deployment and inference module is used to adapt and deploy the trained and optimized model to the domestic AI chip platform to perform real-time image perception and intent inference.

[0031] Furthermore, the deployment and inference module is specifically used to: integrate multiple independent operators in the PROYOLO network into composite operators based on a custom operator development framework to reduce computational overhead, and improve inference efficiency on domestic AI chip platforms through block loading and double-buffered memory scheduling mechanisms.

[0032] The beneficial effects of this invention are as follows:

[0033] 1. This invention proposes PROYOLO, a lightweight detection model for dynamic multi-scene real-time perception, recognition, and intent inference based on an improved LeYOLO architecture. It constructs a novel UAV dataset encompassing multiple angles, modalities, orientations, and scenarios, overcoming the limitations of traditional single-frame detection. By innovatively integrating temporal analysis techniques, PROYOLO not only achieves high-precision target detection but also possesses the ability to infer the intent of dynamic targets. This enables integrated perception and early warning functions covering multiple target types, allowing for efficient deployment on UAV embedded platforms. It meets real-time processing requirements and significantly improves the perception reliability, decision-making intelligence, and deployment adaptability of UAV intelligent agent systems in real-world complex scenarios. Attached Figure Description

[0034] Figure 1 This is an overall framework diagram of the UAV image payload target real-time perception and intent inference method proposed in this invention;

[0035] Figure 2 This is a comparison chart of the number of categories in VisDrone and the DUT-UAV dataset proposed in this invention;

[0036] Figure 3 This is a diagram of the multi-target perception network architecture based on PROYOLO proposed in this invention;

[0037] Figure 4 This is a structural diagram of the SPPFCSPC module proposed in this invention;

[0038] Figure 5 This is a schematic diagram of the newly added P2 feature layer proposed in this invention;

[0039] Figure 6 This is a schematic diagram of the MobileNetV3_BLOCK structure proposed in this invention;

[0040] Figure 7 This is a structural diagram of the intent reasoning module proposed in this invention;

[0041] Figure 8 This is a flowchart of the edge deployment process of the intent inference model proposed in this invention;

[0042] Figure 9 This is a visualization of the comparison of operators before and after optimization and the inference of intent in the model proposed in this invention;

[0043] Figure 10 This is a confusion matrix diagram of the datasets proposed in this invention;

[0044] Figure 11 These are precision curves for the datasets proposed in this invention.

[0045] Figure 12 Here are the recall rate curves for each dataset proposed in this invention;

[0046] Figure 13 The following are F1 score curves for each dataset proposed in this invention;

[0047] Figure 14 This is a comparison chart of the detection results of the method of this invention and YOLOv8 under different scenarios proposed in this invention. Detailed Implementation

[0048] Reference Figures 1-14 This paper proposes a fully autonomous and controllable real-time intelligent perception system. Based on the existing YOLO model, a novel PROYOLO dynamic multi-scene real-time perception, recognition, and intent inference model is constructed. The system mainly includes dataset construction, multimodal perception network design and optimization, multi-scene target intent inference algorithm design, algorithm compression and deployment, etc. The overall framework of the system is as follows: Figure 1 As shown.

[0049] I. Establishment of DUT-UAV Multimodal Dataset: To address the issues of limited categories, unbalanced small target annotations, and single modality in existing UAV datasets, this invention constructs the DUT-UAV dataset.

[0050] like Figure 2 As shown, the DUT-UAV dataset, based on public datasets such as VisDrone, supplements scene data such as water vessels, ground fires, fire hydrants, and birds in the air. It adopts a "main dataset + supplementary data" approach, integrating multi-source data such as anti-drone, bird recognition, and ground personnel / vehicles. It uses LabelImg for fine annotation and uses Python scripts for random rotation, cropping, and splicing to increase diversity.

[0051] The dataset contains a total of 15,931 images, divided into 10 categories: car, truck, bus, people, ship, bird, drone, fire extinguisher, fire, and building. Specifically, it is divided into: aerial recognition dataset (8,523 images, 2 categories), fire rescue dataset (2,969 images, 3 categories), and ground recognition dataset (5,174 images, 5 categories). The dataset is divided into 70% (training), 20% (validation), and 10% (testing). In addition to bounding box annotations, the dataset also includes labels adapted to infer target intent (e.g., danger level 1 - approach - reconnaissance, danger level 2 - hovering - surveillance, danger level 0 - leave quickly - avoidance).

[0052] Specifically:

[0053] 1.1 Dataset Construction and Statistics:

[0054] We adopted a "master dataset + supplementary data" approach to construct three different datasets for different scenarios. We used open-source datasets from Roboflow and Kaggle, and used Python code to randomly rotate, crop, and stitch images. We used LabelImg annotation software to manually draw bounding boxes around flames, fire extinguishers, and buildings. By integrating anti-drone datasets, bird recognition datasets, and multi-source data such as ground personnel / vehicles / buildings / ships / fire smoke, we constructed a new dataset DUT-UAV that is multi-angle, multi-modal, multi-directional, and multi-scenario.

[0055] The DUT-UAV dataset contains a total of 15,931 images, categorized into 10 classes: car, truck, bus, people, ship, bird, drone, extinguisher, fire, and building.

[0056] Referring to Table 1, the aerial recognition inference dataset (2 classes: bird, drone) has 8523 images, the fire rescue scene dataset (3 classes: extinguisher, fire, building) has 2969 images, and the ground recognition dataset (5 classes: car, truck, bus, people, ship) has 5174 images.

[0057] The samples are divided into three parts: training, validation, and testing; the training part contains 70% of the samples, the validation part contains 20% of the samples, and the testing part contains 10% of the samples.

[0058] Accurate bounding box annotations were performed on small objects in the dataset, including common categories such as pedestrians, vehicles, and fire extinguishers. The annotation information included the location, size, and category of the target, providing a precise reference for small object detection experiments. The distribution of targets in the images is mainly concentrated slightly below the center, with vehicles being the most abundant category.

[0059] Table 1 Basic Information of DUT-UAV Dataset

[0060]

[0061] 1.2 Comparison with other datasets:

[0062] While existing datasets have made significant progress in the field of general object detection, they have significant limitations in low-altitude recognition tasks in urban scenes. They lack annotations for typical urban scenes such as cities, dense building clusters, and low-altitude flight corridors, and do not fully incorporate association data between low-altitude targets and small ground objects (traffic signs, fire hydrants). The VisDrone dataset contains complex scenes such as urban roads, squares, transportation hubs, and farmland, and identifies 10 types of traffic-related targets such as pedestrians, vehicles, and bicycles. This dataset covers complex scenes such as urban roads, squares, transportation hubs, and farmland, and includes dense traffic flow and pedestrian interaction scenes, providing training samples with dynamic occlusion and scale changes for the model.

[0063] However, by analyzing the characteristics of the VisDroneDET2021 dataset, it was found that the main reasons for the low detection performance were the small size of objects, the large scale span, the long tail distribution, and the confusion of similar classes. There was also a lack of benchmarks for validating detection models in scenarios such as water vessels, ground fires, fire hydrants, and birds in the air. To address this, based on the DUT-UAV dataset and benchmarks of large public drone datasets such as VisDrone and UAVDT, target image data of drones, birds, fire smoke, and ships at sea were collected from platforms such as Roboflow and Kaggle, filling the gaps in existing drone datasets for professional scenarios such as low-altitude security and disaster relief.

[0064] The DUT-UAV dataset uses LabelImg software to perform fine-grained bounding box annotations on key targets (such as the boundaries of flames and the specific locations of fire extinguishers). Multiple rounds of checks and corrections were conducted, especially for small and deformable targets, to ensure annotation quality. It annotates 15,931 images, covering 10 target categories in scenes including boats on water, ground fires, fire hydrants, and birds in the air. To increase data diversity, Python scripts were used to randomly rotate, crop, and stitch the images, simulating various perspectives and occlusions during drone flight, greatly increasing data diversity. Although currently dominated by visible light data, the data architecture has reserved interfaces for incorporating multimodal data such as infrared and radar, ensuring future scalability.

[0065] The DUT-UAV dataset not only adds data benchmarks for scenarios such as surface vessels, ground fires, fire hydrants, and birds in the air, but also constructs a new small-scale dataset for real-time target perception and intent inference adapted to domestic UAV payloads. This dataset includes labels such as Hazard Level 1 (Approach - Reconnaissance), Hazard Level 2 (Hovering - Surveillance), and Hazard Level 0 (Rapid Departure - Avoidance), aiming to support the application of UAVs in disaster relief and national defense. A comparison of the number of categories in each category between VisDrone and the DUT-UAV dataset is shown below. Figure 2 As shown.

[0066] II. Design of a Multi-Target Perception Network Based on PROYOLO: When using LeYOLO for real-time target detection and inference, the detection process is often affected by factors such as angle changes and complex environmental backgrounds, leading to problems such as false detections and low accuracy. Therefore, a PROYOLO model structure is proposed, which improves the backbone network, neck network, head network, and loss function of LeYOLO. The network architecture is as follows: Figure 3 As shown.

[0067] 2.1 Improved backbone network feature aggregation module:

[0068] To address the issue of detail loss during multi-scale feature fusion in the original LeYOLO backbone network's SPPF module, the SPPF module was replaced with the SPPFCSPC module. While the SPPF module achieves feature aggregation through multi-scale pooling, pooling operations can lead to the loss of fine-grained features of small targets, limiting its performance, especially in scenarios with dense small targets such as drone aerial photography. The SPPFCSPC module combines the multi-scale receptive field advantage of SPPF with the cross-stage feature fusion mechanism of CSPC (CrossStage Partial Connection). For details, see [link to details]. Figure 4 ;

[0069] First, feature channels are compressed using 1×1 convolutions to reduce computation. Then, multi-scale max pooling (such as 5×5, 9×9, and 13×13) is performed in parallel to preserve spatial features at different scales. Finally, shallow detail features and deep semantic features are fused through cross-stage partial connections and integrated and output by 1×1 convolutions. This module enhances global context awareness while reducing the loss of fine-grained features and improving the backbone network's ability to express features of targets at different scales (especially small targets), providing richer feature inputs for subsequent detection heads.

[0070] 2.2 Improved Feature Pyramid Structure (Added P2 Feature Layer):

[0071] like Figure 5 As shown, to address the issue that the original LeYOLO feature pyramid relies solely on layers P3 and above (such as P3, P4, and P5), resulting in insufficient extraction of small target features, an enhanced feature pyramid structure including a P2 layer was designed. In the original feature pyramid, after multiple downsamplings, the feature maps of layers P3 and above have significantly attenuated the detailed information of small targets, making it difficult to meet the detection requirements of small targets (such as distant pedestrians and small vehicles) in UAV scenarios.

[0072] The improved architecture enhances small target feature extraction in the following ways: a new P2 feature layer is added to the feature pyramid, outputting a high-resolution feature map of size 160×160, directly preserving small target details (such as edges and textures) in the shallow network; deep features (such as the P3 layer) are fused with P2 layer features through upsampling, using deep semantic information to correct shallow noise; the fused features are further purified by MobileNetV3_BLOCK to enhance the discriminative power of small target features, significantly improving the ability to capture features of tiny targets; and it detects tiny targets of 4x4 pixels, preserving more details.

[0073] 2.3 Introducing a deformable convolution module:

[0074] To address the issue that fixed receptive field convolutions in LeYOLO are ill-suited to non-rigid deformations of targets (such as pedestrians with distorted postures or irregularly shaped objects), deformable convolutions are introduced in the feature purification stage of the P2 feature layer to replace the ordinary convolutions in the original MobileNetV3_BLOCK.

[0075] like Figure 6As shown, ordinary convolution has a fixed sampling position, which limits its adaptability to target deformation (such as occlusion or bending), easily leading to incomplete feature extraction of deformed targets. Deformable convolution dynamically adjusts the sampling point position by learning offsets, enabling the convolution kernel to "focus" on the deformed region of the target. An offset mapping is generated through 1×1 convolution to predict the offset of each sampling point; based on the offsets, the sampling position of the original convolution kernel is adjusted to perform targeted feature extraction of the deformed region; for example... Figure 3 By incorporating GhostConv's lightweight design, the module introduces deformability while controlling the growth of parameters. This module improves the model's robustness in feature extraction of deformable targets in complex scenes, especially in drone scenarios where the target is partially occluded or has variable poses.

[0076] 2.4 Improved Detector Loss Function:

[0077] The original LeYOLO detector lacked sufficient accuracy in bounding box regression. This invention replaces the loss function with Wise-IoU (WIoU) to improve the optimization capability for difficult bounding boxes. The original loss function (such as CIoU) has a fixed penalty mechanism for low-overlap bounding boxes, which can easily lead to slow convergence or localization errors, especially in dense small object scenes where the overlap of bounding boxes varies greatly, making it difficult for traditional loss functions to balance the optimization weights of samples with different difficulty levels. Wise-IoU achieves accurate regression by dynamically adjusting the penalty factor: it dynamically allocates loss weights based on the overlap (IoU) between the predicted box and the ground truth box, imposing a greater penalty on difficult bounding boxes with low IoU; it introduces a regression path optimization term to guide the bounding boxes to converge to the ground truth position along a better path; and it keeps the classification loss (BCEloss) unchanged, only replacing the bounding box regression loss to ensure model compatibility.

[0078] Specifically, Wise-IoU achieves accurate regression by dynamically adjusting the penalty factor, and its calculation includes three core parts:

[0079] (1) Loss calculation: final loss value From the regression quality term With focus weight Multiplying them together gives: ;

[0080] (2) Focus weighting mechanism: The focus weight β is dynamically adjusted based on the anchor frame quality. Where Q is the quality metric between the anchor box and the ground truth box (usually negatively correlated with IoU), a is a dynamically adjusted scale parameter, 15 is the scaling factor, and the square operation makes the weight changes more sensitive; the key to this design is that when the anchor box quality Q is low, the β value increases significantly, imposing a stronger penalty on difficult samples; when the anchor box quality Q is high, the β value decreases, preventing the model from overfitting to simple samples.

[0081] (3) Dynamic benchmark update: The model maintains a dynamic benchmark μ through momentum update, which reflects the average regression level in the current training phase. ;in, The mean IoU in the current batch is λ, and the momentum coefficient is usually taken as 0.9 or higher. This benchmark μ serves as a reference for the scale parameter a in the focus weight mechanism formula or directly participates in the calculation, ensuring that the adjustment of the focus weight can adapt to the changes in the regression level throughout the training process and avoid optimization imbalance caused by a fixed threshold.

[0082] WIoU loss, through its dynamic focusing mechanism, significantly enhances the optimization of low-quality samples (such as severely occluded, small targets, or targets with complex backgrounds) while reducing excessive attention to high-quality samples. This guides the model to learn bounding box regression more robustly and in a balanced manner, comprehensively improving localization accuracy and generalization ability, and effectively reducing the false negative rate of small targets.

[0083] III. Lightweight Design and Deployment of Multi-Scenario Target Intent Inference Model: (e.g.) Figure 7 As shown, the intent inference module integrates multiple technologies to achieve behavior understanding based on PROYOLO real-time detection.

[0084] 3.1 Intent Inference Model Design: The system deploys a lightweight PROYOLO model based on domestically produced edge computing devices, processes 1080P / 30fps video streams in real time, and converts pixel coordinates into spatial coordinates through camera calibration and pose fusion (positioning error ≤ ±0.5 meters), providing a unified spatial reference for multiple scenarios. For different target tasks, the system employs differentiated inference logic:

[0085] For aerial targets, an optimized feature pyramid is used to enhance small target detection capabilities. A 6-dimensional state model containing position (x / y / z) and velocity (vx / vy / vz) is established through 3D Kalman filtering. Combined with the motion characteristics of aerial targets, process noise parameters are optimized to reduce the multi-target tracking ID switching rate to 5.2%. This enables smooth trajectory prediction and target correlation modeling. Furthermore, a 3D protected airspace (configurable radius and altitude) is defined, the collision time (TTC) between the target and the UAV is calculated, and two threat levels are divided into "warning (TTC>5s)" and "emergency (TTC≤5s)" and a graded response is triggered (track log is recorded in the safe state, airborne acoustic wave drive-away is triggered in the warning state, and vertical climb obstacle avoidance is performed in the emergency state).

[0086] In fire target processing, the system sets fire alarm thresholds and warning levels based on the size of the PROYOLO bounding box. It analyzes pixel motion in continuous video frames using the Farneback dense optical flow algorithm to predict the direction and speed of fire spread. The specific process includes generating candidate fire area bounding boxes by detecting fire in each frame of the image, and then using non-maximum suppression (NMS) to eliminate duplicate recognition (the initial nms_threshold parameter is set to 0.5, candidate boxes are sorted in descending order of confidence and the intersection-over-union ratio (IoU) is calculated to suppress overlapping boxes, and NMS is executed twice at key nodes to ensure optimal detection boxes). The area of ​​the fire target bounding box is extracted as a quantitative indicator of fire scale. The average spread speed is calculated by combining optical flow field analysis and principal component analysis to determine the dominant direction. Based on the area threshold (e.g., 500 pixels is threshold 1, 2000 pixels is threshold 2), a three-level alarm is triggered. The parameters support dynamic optimization (the alarm area threshold is scaled proportionally according to the camera installation height, the NMS threshold is adjusted between 0.4 and 0.6, and the optical flow neighborhood radius poly_n is adapted to the flame texture between 15 and 25 pixels).

[0087] For vessel targets, the system detects vessel positions based on PROYOLO (confidence threshold 0.3, NMS IOU threshold 0.5 to handle overlapping boxes), extracts the coordinates of the center point of the detection box, automatically initializes a trapezoidal channel region (converts pixel coordinates through proportional coordinates and expands to form a polygon detection area with a width of 2%), and uses the pointPolygonTest function to determine the spatial relationship between the center point and the channel (marking the center point in green within the region and accumulating channel flow, marking off-center with red "OffChannel" text and triggering a full-screen red "WARNING: VESSELSOFFCHANNEL" alarm), while continuously tracking channel flow statistics.

[0088] 3.2 Deployment of Multi-Scenario Target Intent Reasoning Model: In order to deploy more efficiently and improve the overall performance of the model on the Atlas200IDKA2 platform and Ascend NPU, this invention optimizes model volume compression, operator adaptation and compatibility, and inference efficiency.

[0089] like Figure 8 As shown, quantization deployment uses PyTorch integrated quantization-aware training (QAT), inserting pseudo-quantization nodes to simulate INT8 operations; pruning uses the pruner module of Huawei toolchain, evaluating channel importance based on L1 / L2 norm or BN layer scaling factor, avoiding key YOLO operators (such as the Detect layer); the optimized PyTorch model is converted to ONNX format using the official export.py tool: after loading .pt weights, eval mode is enabled, specifying opset_version=12 (adapting to Ascend chips), and after verifying operator compatibility, an ONNX model in NCHW format is generated.

[0090] This invention adopts the "PyTorch→ONNX→OM" process to adapt to the Ascend chip. To avoid the resource limitations of edge devices (memory / heat dissipation), the OM conversion is completed on the PC Ubuntu 22.04 system, and the CANN toolkit is configured. Before ONNX input, the operator version is set to 12, the dynamic axis batch_size / image_size is configured, and pre-optimization (QAT / pruning) is performed. After the OM model is transferred to the Atlas200IDKA2 development board, a debugging environment is established via SSH, the inference engine is initialized using the AscendCL interface, and HugePage memory is allocated using aclrtMalloc to achieve zero-copy transfer. At the same time, this invention designs a double-buffered pipeline: Thread1 processes image acquisition and preprocessing (filling BufferA), and Thread2 executes model inference and result output (processing BufferB).

[0091] Final deployment results verification shows that, as Figure 9 As shown: After distillation and compression, the model size was reduced from 7.4MB to 4.0MB, the 1080P single-frame processing latency was reduced to 71ms, the video stream throughput was increased to 13FPS, and the HBM2E memory usage was reduced by 40%.

[0092] IV. Experimental Results and Analysis:

[0093] 4.1 Experimental Environment:

[0094] Training environment: This experimental platform uses Windows 11 as the operating system, equipped with a 13th Gen Intel(R) Core(TM) i7-13700H with a main frequency of 2.40GHz, 16GB of system memory, an NVIDIA GeForce RTX 4060 graphics card with 7GB of video memory, PyTorch framework version 2.4.1+cu118, Python version 3.8.20, Batch size set to 32, image size set to 640×640, epoch set to 100, the network does not use the weights of the pre-trained model, and all experiments use the same hyperparameter configuration during training.

[0095] The hyperparameters were set as follows: input image size 640×640, batch size 32, training epochs 150, SGD optimizer used, initial learning rate 0.01, and cosine annealing strategy for decay. The evaluation metric was the mean average accuracy (mAP), which is common in the field of object detection.

[0096] Core computing chip: This experiment uses Huawei Atlas200IDKA2 Ascend AI accelerator chip, which integrates the Ascend 310 processor (16nm process), providing 8 TOPSINT8 computing power and supporting edge-end collaborative architecture; it adopts 3DCube matrix operation unit, completing 4096 multiply-accumulate operations in a single cycle, with a computing density 1.8 times that of international chips of the same level. It is manufactured entirely with domestic 28nm process, and the instruction set and packaging technology are 100% independently controllable, completely avoiding the risks of US export controls. At the same time, it supports Huawei HiLens edge computing platform to realize dynamic model offloading (such as edge execution of feature extraction layer and local operation of detection head). This platform has a fully independent intellectual property system and military-grade security protection.

[0097] Evaluation Criteria: To further quantify the classification performance of the model, this invention introduces three typical evaluation metrics: Precision, Recall, and F1-score; these metrics can be calculated from the statistical data in the confusion matrix.

[0098] Precision is a key metric for evaluating the performance of object detection models. It represents the percentage of true positives (i.e., detected targets) among all predictions made by the model; in other words, it indicates the proportion of truly effective detections. Its mathematical expression is: As can be seen from the formula, the higher the accuracy, the fewer false positives there are among the targets predicted as positive by the model, and the higher the reliability.

[0099] Recall is another important metric for evaluating the performance of object detection models. Recall represents the percentage of true positive samples that the model successfully detects; in other words, it indicates the model's ability to avoid missing any targets. Its mathematical expression is: The higher the recall rate, the stronger the model's coverage of positive class samples, meaning it can detect more real targets.

[0100] The F1 score is a comprehensive metric that combines precision and recall to measure a model's overall performance in balancing false positives and false negatives. It provides a relatively objective performance measure, especially when there is an imbalance between these two metrics. Its mathematical expression is: The F1 score is essentially the harmonic mean of precision and recall. It is biased towards the lower term compared to the arithmetic mean. Therefore, if there is a significant difference between precision and recall, the F1 score will be closer to the lower term, thus revealing the potential weaknesses of the model.

[0101] Average Displacement Error (ADE) measures the degree of deviation between the predicted trajectory and the actual trajectory. Its mathematical expression is: Where N is the number of trajectories, and T is the prediction time step. )and( , ) represent the predicted position and the actual position of the i-th trajectory at time t, respectively; ADE reflects the algorithm's overall ability to grasp the trend of ship motion, and the smaller the value, the higher the degree of agreement between the predicted trajectory and the actual trajectory.

[0102] Final Displacement Error (FDE) measures the reliability of long-term predictions and is crucial for applications such as collision warning and path planning; its mathematical expression is: Where N is the number of trajectories, and T is the prediction time step. , )and( , ) represent the predicted position and the actual position of the i-th trajectory at time T, respectively.

[0103] Mismatch Rate (MR): The false match rate assesses the stability of identity maintenance in multi-target tracking. A low false match rate ensures the continuity of target identity and is the foundation of a reliable monitoring system.

[0104] IntentAccuracy (IA) = Number of correct predictions / Total number of samples.

[0105] Taking the intention inference of maritime vessels as an example, the accuracy evaluation model for intention inference assesses the correctness of the model in recognizing the behavioral patterns of vessels and other targets. Based on the model's recognition accuracy, the accuracy of the model's inference is evaluated. First, video sequences of vessel behavior are acquired and labeled to generate real trajectory data for each frame, including target ID and bounding box information. The improved PROLEYOLO model and multi-target tracking algorithms such as Kalman filtering are run to process the video, outputting the tracking results for each frame. Then, each frame is compared frame by frame, and the IoU between the real bounding box and the predicted bounding box is calculated. When the IoU is greater than 0.5, it is considered a match. The number of false detections, the number of missed detections, and the ID are counted for each frame. The number of switching counts; summarizing the total number of false detections, missed detections, and ID switching counts for all frames, as well as the total number of real targets; finally, substituting into the MOTA formula; by comparing the tracking results and real annotations for each frame, the number of false detections, missed detections, and ID switching counts for each frame is calculated, the parameters for all frames are summarized, and the accuracy is calculated using the motmetrics tracking algorithm; the inference time per target (InferenceTimeperTarget) represents the time difference from the start of a single target's behavior to the system's correct recognition (the smaller the better), the inference time per target = the total time taken from the start of target detection to the output of its intent ÷ the number of targets.

[0106] 4.2 Test Results: PRO-LeYOLO was tested on a self-built dataset. The confusion matrices for three different scenarios are as follows: Figure 10 As shown; by Figure 10 It can be seen that in the three scenarios, the recognition accuracy of various targets reached over 80%, and the average accuracy reached over 90%, indicating that the model has a strong main class recognition capability.

[0107] like Figure 11 As shown, in three different scenarios, when the confidence level is set to 0.875, 0.947, and 0.963 respectively, the precision value can reach 1. This indicates that under this numerical condition, all bounding boxes predicted as targets by the model are correct targets, that is, there are no false detections, and the model has extremely high prediction accuracy.

[0108] like Figure 12 As shown, in the current detection tasks for three different scenarios, based on the image data results, when the confidence threshold is set to 0, the recall rates of the model can reach 0.98, 0.96 and 0.96 respectively. This indicates that when the detection standard is at the lowest level, the model can detect more than 96% of the real targets. Although there is a certain degree of missed detection, it can "identify" potential targets to the greatest extent.

[0109] like Figure 13As shown, in the detection tasks of three different scenarios, according to the model performance graph results, when the confidence values ​​are set to 0.479, 0.518, and 0.372 respectively, the average F1 score of the model on all categories reaches 0.93, 0.89, and 0.89 respectively. This indicates that under the current settings, the model achieves a relatively balanced performance between false detections and false negatives, which is a reasonable parameter setting point and suitable for application as the default or recommended threshold in real-world scenarios. In addition, this invention was tested on the VisDrone dataset against PROYOLO, and the test results show that the improved model still maintains 76% accuracy for vehicles and pedestrians.

[0110] 4.3 Performance Comparison of Different Target Detection Algorithms: Experiments were conducted in three different scenarios: ground scene, aerial scene, and fire scene. The specific results of various indicators (P, R, mAP@0.5 and mAP@0.5: 0.95, etc.) are shown in Table 2. It is worth noting that the model accuracy mAP@0.5 can reach more than 92% in the ground scene, aerial scene, and fire scene, and reaches 95.5% mAP in the aerial scene. This shows that the model of the present invention has good robustness and meets the requirements of accurate perception of multiple targets in multiple scenarios.

[0111] Table 2 Model metrics under different scenarios

[0112]

[0113] Furthermore, to further verify the superiority of the model of this invention, the accuracy of testing each target category on the constructed multi-scene dataset is presented, and the specific results are shown in Table 3. All experiments were conducted under the same environment and dataset conditions, using different YOLO models to test the accuracy of each category. The self-built dataset contains targets of various scales, from small targets (such as birds and dragons) to large targets (such as buildings and ships), covering challenging categories (such as fires with varied shapes).

[0114] As can be clearly observed from Table 3, PROYOLO achieves AP values ​​of 0.92 and 0.93 for small object categories (such as Bird and Drone), which is slightly better or on par with YOLOv8 (0.94 and 0.91), but significantly better than YOLOv5 (0.71 and 0.83). This verifies the enhancement effect of the method of this invention in the recognition of small objects. At the same time, in large object categories (such as Building and Ship), PROYOLO (0.86 and 0.87) also outperforms YOLOv5 (0.31 and 0.51) and remains competitive with YOLOv8 (0.81 and 0.74), reflecting the balance of the model.

[0115] Because this invention employs a more refined and adaptive multi-scale feature learning mechanism, it enhances the detail perception capability of the feature pyramid to improve the accuracy of small targets and ensures the stability of large target recognition through the synergistic effect of global semantic information and multi-scale training. This mechanism not only strengthens the ability to capture details of small targets, but also ensures that the performance of large targets is not affected by maintaining the integrity of deep semantic features and adopting a balanced optimization objective.

[0116] Table 3 shows the AP results of each method in the main target category test on the self-built dataset.

[0117]

[0118] The improved model was tested and its performance was compared with other models, as shown in Table 4. This invention selected representative YOLO models, such as YOLOv8, YOLOv10, and YOLOv11. Additionally, the latest YOLOv12 model from 2025 was added. Different versions of the YOLO algorithm showed clear performance gradients on this dataset, indicating that the dataset has good discriminative power and can effectively identify the merits of different algorithms. Compared with the state-of-the-art object perception method YOLOv12s, the improved model has only 0.24 times the number of parameters (2.2M), a higher map size (8.7 percentage points), and a single image processing time of 5.7 milliseconds. This demonstrates that the method of this invention can achieve real-time segmentation with high accuracy at a lower model scale, meeting the needs of UAV mobile device model deployment.

[0119] Table 4 shows the performance comparison results of the method of the present invention and various models.

[0120]

[0121] To further verify the advantages of the joint optimization of perception and intent inference in the method of the present invention, a quantitative comparison was made between trajectory tracking and intent inference models for different targets. Compared with the current most similar target trajectory method, the target tracking and intent inference accuracy of the present invention is also the highest in a shorter time when inferring intent for a single target, as shown in Table 5.

[0122] Table 5 Comparison of accuracy and time consumption for single target trajectory tracking and intent inference

[0123]

[0124] To more intuitively demonstrate the effectiveness of the proposed algorithm, images show the detection results of the YOLOv8 algorithm and the improved algorithm in different scenarios (YOLOv8 on the left, PROYOLO on the right). The images compare the inference detection results of dense targets and small targets in low-light environments and under light pollution conditions. Figure 14 As shown in the image comparison, the improved algorithm can better detect small and dense targets from the drone's perspective, reducing missed and false detections. Furthermore, the improved algorithm can still detect targets even in poor lighting conditions or with severe light pollution, indicating that it is less affected by lighting conditions, has stronger anti-interference capabilities, and is more robust. It can be seen that compared to the traditional YOLOv8 algorithm, the improved algorithm enhances the detection capabilities of small, dense, and complex targets, effectively reducing the number of missed and false detections.

[0125] 4.4 Ablation Experiment:

[0126] To fully verify the performance improvement effect of each improved module on the model, ablation experiments were conducted on the proposed improved model on the VisDrone2019 dataset. These experiments were based on the LeYOLO model and the improved methods were integrated step by step according to the modules to quantify the contribution of each module to the detection accuracy and efficiency.

[0127] The specific experimental process is as follows: First, the SPPF in the network is replaced with the proposed SPPFCSPC structure; then, the MobileNetV3_BLOCK in the backbone network is replaced with GhostConv; and a P2 detection layer is added; finally, the original detection head is replaced with the detection head Wise-IoU; as shown in Table 6, the table shows the detailed results of the ablation experiment, with the baseline being the LeYOLO-small model.

[0128] Table 6 Ablation experiments of each component

[0129]

[0130] After introducing the SPPFCSPC module separately, the model's mAP@0.5 and mAP@0.5:0.95 were improved, while the number of parameters and computational cost increased only slightly. This indicates that the improved spatial pyramid pooling module, through its multi-branch and CSP structure, can more efficiently fuse contextual features at different scales, providing the detection head with a feature foundation that has stronger representational capabilities and a wider receptive field, and preliminarily verifying its effectiveness in enhancing the model's global perception capability.

[0131] Building upon SPPFCSPC, the P2 feature layer and GhostConv module are further introduced, resulting in a continued steady improvement in model accuracy. At the same time, the number of parameters and computational cost even slightly decreases. It not only enhances the ability to represent features by generating more diverse feature maps, but also successfully reduces the complexity and redundancy of the network, demonstrating its accuracy-efficiency trade-off and meeting the stringent requirements of deployment on mobile and embedded devices.

[0132] Finally, the Wise-IoU (WIoU) loss function was adopted, which further improved the model's performance on mAP@0.5 and mAP@0.5:0.95. WIoU is a plug-and-play optimizer-level improvement that does not introduce any additional inference time computation overhead. Through its dynamic focusing mechanism, WIoU reduces regression noise caused by low-quality samples (such as targets with severe occlusion or complex backgrounds), guiding the model to more robustly optimize the bounding boxes of ordinary quality samples, thereby comprehensively improving localization accuracy and generalization ability.

[0133] When the four modules are integrated, the improved model achieves an mAP@0.5 that is 0.031 higher than the baseline, and an mAP@0.5 that is 0.018 higher than the baseline. Despite the increase in the number of parameters and computational cost compared to the original model, the improved model still performs better in real-time detection.

[0134] Table 7. End-to-end experimental comparison results of target perception and intent inference.

[0135]

[0136] To further verify the advantages of the lightweight module system designed in this invention that directly drives intention reasoning based on perception results, an intention-labeled dataset containing 10 target trajectories was constructed, covering three typical behaviors (labels such as approach-reconnaissance, hovering-monitoring, and quick departure-avoidance).

[0137] The intent module adopts an end-to-end scheme jointly optimized by Kalman filter trajectory, dense optical flow motion and principal component analysis rule method and lightweight classifier PROYOLO, with the kinematic features of the tracking trajectory as the input. Based on the detection results of YOLOv8 and PROYOLO, the target intent inference module designed in this invention is used. The results are shown in Table 7. The experimental results show the advantages of the target perception and intent joint optimization scheme designed in this invention.

[0138] Compared to YOLOv8+ intent inference, the proposed method achieves a target intent inference accuracy of 87.3% within 53.2ms, which is 13.3 percentage points higher than that of YOLOv8+. Furthermore, the method achieves faster single-target intent inference speed and consumes less time, only 0.6 times the time of the YOLOv8+ method. The improved Proyolo detection model of this invention not only improves intent inference accuracy compared to YOLOv8 but also optimizes inference time. This demonstrates that using the improved detection model, the system can not only more accurately infer the target's intent but also complete the entire process from target detection to intent inference in a shorter time, thereby improving the system's real-time performance and efficiency.

[0139] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for real-time target perception and intent inference in UAV image payloads, characterized in that, Includes the following steps: S1. Construct a multimodal UAV payload image dataset. The dataset covers a variety of typical intelligent transportation and emergency scenarios and includes bounding boxes, small target labels, and behavioral intent annotations. S2. Construct a lightweight target perception network, PROYOLO, based on the YOLO architecture. The PROYOLO network includes: S2.1 Improved backbone network, in which the spatial pyramid pooling module is replaced with the SPPFCSPC module to enhance multi-scale feature fusion capability; S2.2 Improved Feature Pyramid Network: The feature pyramid network includes a new P2 feature layer, which is used to fuse shallow spatial details and deep semantic features to enhance the feature representation of small targets. S2.3 Introduces a deformable convolution module into the P2 feature layer to improve the robustness of feature extraction for deformable targets; S2.4 The Wise-IoU loss function is used as the bounding box regression loss to dynamically adjust the optimization weights for samples of different difficulty. S3. Design a multi-scenario target intent inference model. By integrating Kalman filter trajectory prediction, dense optical flow motion analysis and principal component analysis of spatial activity regions, dynamic behavior intent inference and early warning modeling of targets across scenarios can be achieved. S4. Deploy the optimized PROYOLO network and intent inference model on a domestic AI chip platform to perform model compression, operator adaptation and inference acceleration, and achieve real-time processing on the edge.

2. The method for real-time target perception and intent inference of UAV image payload according to claim 1, characterized in that, In step S1, the multimodal drone payload image dataset is the DUT-UAV dataset, which contains image data from three major sub-scenarios: forest fire emergency response, maritime vessel monitoring, and urban aerial inspection. The categories include at least: cars, trucks, buses, people, boats, birds, drones, fire extinguishers, fire, and buildings.

3. The method for real-time target perception and intent inference of UAV image payload according to claim 1, characterized in that, In step S2.1, the SPPFCSPC module performs multi-scale max pooling in parallel and combines it with a cross-stage partial connection mechanism to fuse spatial features at different scales and cross-stage features.

4. The method for real-time target perception and intent inference of UAV image payload according to claim 1, characterized in that, In step S2.2, the P2 feature layer outputs a high-resolution feature map, which is then fused with deep features through upsampling, and finally purified by the MobileNetV3_BLOCK module.

5. The method for real-time target perception and intent inference of UAV image payload according to claim 1, characterized in that, In step S3, the intent inference model performs differentiated inference for different target types: For aerial targets, a three-dimensional Kalman filter is used to establish a state model for trajectory smoothing and prediction, and the collision time is calculated to classify the threat level. For fire targets, the direction and speed of fire spread are analyzed based on PROYOLO network detection boxes and Farneback dense optical flow, and graded alarms are triggered by combining area thresholds. For vessel targets, the spatial relationship between the center point of the detection frame and the preset channel area is used to determine whether the vessel is deviating from its course and to trigger an early warning.

6. The method for real-time target perception and intent inference of UAV image payload according to claim 1, characterized in that, Step S4 specifically includes: S4.1 performs quantization-aware training and channel pruning on the PROYOLO network to compress the model size; S4.2 converts the optimized model into ONNX format, and further converts it into an OM model adapted to domestic AI chips; On a domestically developed AI chip development board, S4.3 uses a custom operator fusion framework to fuse multiple independent operators in the network, reducing operator call overhead. S4.4 employs block loading, double buffering, and a full-thread call strategy to schedule memory and computing resources, thereby improving inference efficiency.

7. A real-time target perception and intent inference system for UAV image payloads, characterized in that, The system for implementing the real-time target perception and intent inference method for UAV image payload as described in any one of claims 1 to 6 includes: The data building module is used to build and store multimodal UAV payload image datasets; The model training and optimization module is used to train and optimize the PROYOLO object perception network; The intent inference module is used to execute a multi-scenario target intent inference model to perform behavioral understanding and early warning of detected targets. The deployment and inference module is used to adapt and deploy the trained and optimized model to the domestic AI chip platform to perform real-time image perception and intent inference.

8. The method for real-time target perception and intent inference of UAV image payload according to claim 7, characterized in that, The deployment and inference module is specifically used to: integrate multiple independent operators in the PROYOLO network into composite operators based on a custom operator development framework to reduce computational overhead, and improve inference efficiency on domestic AI chip platforms through block loading and double-buffered memory scheduling mechanisms.

9. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes a computer program, it implements the real-time target perception and intent inference method for UAV image payloads as described in any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the real-time target perception and intent inference method for UAV image payload as described in any one of claims 1 to 6.

Images