An unmanned aerial vehicle intelligent perception and intention reasoning system and device

Abstract

The application discloses an unmanned aerial vehicle intelligent sensing and intention reasoning system and device, which comprises image acquisition, target detection, knowledge graph, multi-modal large model, airborne computing power and edge-cloud cooperation module. The system is guided by target detection, enhances scene knowledge through the knowledge graph, improves the analysis ability of the multi-modal large model, and realizes the intelligent monitoring of the unmanned aerial vehicle based on the RK3588 platform. Real-time monitoring is realized by using the maneuverability of the unmanned aerial vehicle, which can not only analyze the scene, but also infer and predict the behavior intention and situation, thereby realizing early warning. The local and cloud systems are equipped with a historical database, a visual interface and a large parameter model, and the returned information is deeply analyzed. The system can be applied to the scenes of border patrol, city security, animal protection and the like.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring and edge computing technology for unmanned aerial vehicles (UAVs), specifically to an intelligent perception and intent reasoning system and device for UAVs, which is particularly suitable for complex scenarios with high requirements for real-time performance, intelligence and autonomous controllability, such as border patrol and security inspection. Background Technology

[0002] Currently, drones are widely used in border patrols and regional security, but the intelligence level of existing systems still has certain shortcomings. Most drone systems only utilize onboard computing units to achieve basic target detection functions, making it difficult to perform semantic understanding and intent analysis of target behavior in complex environments. Related high-level reasoning tasks typically rely on remote cloud servers, resulting in high system response latency and difficulty meeting real-time application requirements in border areas with poor communication conditions. Furthermore, existing systems mostly rely on single visible light sensors, leading to a significant decrease in detection performance at night or in low-visibility environments. Although some systems have introduced infrared sensors to achieve multimodal perception, achieving efficient multimodal fusion processing on embedded platforms remains a technical challenge. Therefore, developing a drone collaborative system based on a domestically developed computing platform capable of multimodal perception and intelligent intent reasoning is of great significance. Summary of the Invention

[0003] (a) Technical problems to be solved This invention aims to address the technical problems of insufficient intelligence and weak autonomous controllability in existing drone monitoring systems in complex scenarios such as border patrol and urban security. Currently, most systems remain at the basic target detection level, unable to perform real-time semantic understanding and intent judgment of target behavior in complex environments. Higher-order inference tasks rely excessively on cloud servers, resulting in high response latency and significantly reduced practicality in communication-constrained scenarios. Simultaneously, their perception modality is singular, heavily reliant on visible light cameras, leading to a sharp performance degradation under low-visibility conditions such as nighttime or fog. Existing multimodal fusion algorithms have high computational overhead, making efficient real-time processing difficult to achieve on resource-constrained embedded systems.

[0004] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: an intelligent perception and intent reasoning system and apparatus for unmanned aerial vehicles (UAVs), comprising: The image acquisition module is used to acquire image data. It is equipped with two types of cameras, one for visible light and one for infrared light, to acquire and enhance images for subsequent processing. The target detection module is used to identify targets in surveillance images. It uses a visible light detection model based on the YOLOv11 series architecture and a multimodal target detection model based on the Oriented R-CNN architecture for targeted processing during the day and at night, respectively. The knowledge graph module uses the target category and confidence level detected by the target detection module to make preliminary inferences about the target's possible intentions. The multimodal large model module uses a fine-tuned Qwen2-VL-2B-Instruct large model to perform reasoning analysis based on the original image and the preliminary reasoning results of the knowledge graph to predict the target intent; The airborne computing module, based on the RK3588 computing platform and equipped with the Galaxy Kylin operating system, lightweightly deploys and system-integrates the aforementioned visible light detection model, multimodal detection model, multimodal large model, and knowledge graph. The edge-cloud collaboration module selectively deploys a visual interface, historical database, complete knowledge base, and multimodal large model with significantly more parameters than the lightweight model on the drone end on the local end and the cloud end. It is used for data statistics, in-depth analysis and other purposes, and is also convenient for relevant personnel to view. Preferably, the image acquisition module is equipped with both visible light and infrared light cameras. To achieve all-weather cognitive capabilities, dual-modal acquisition using both visible light and infrared light will be performed when acquiring image information. Specifically, multimodal image acquisition includes: Step S1: Acquire one or more frames of visible light images and determine whether the current scene is in a situation of sufficient light and good visibility, sufficient light but limited field of vision (such as rainy or foggy days), or low light (such as night). Step S2: Collect image / video data in a targeted manner. If it is the first two types, turn on the visible light camera; if it is the latter type, turn on both the visible light and infrared cameras. Step S3: Apply image enhancement algorithms to the three cases accordingly; Step S1 uses computer vision to distinguish between three cases, and the specific steps include: Step S11: Transfer the acquired image from RGB space to HSV color space and YUV color space, and extract the V channel (luminance), Y channel (luminance), and S channel (saturation). Step S12: Calculate the mean value of V in the HSV space and extract the global average brightness. To determine whether it is a low light condition; Step S13: Calculate the transmittance map of the image using the Dark Channel Prior algorithm and calculate the mean. ; Step S14: Perform Laplacian convolution on the grayscale image and calculate the variance. ,according to and Two indicators are used to determine whether the weather is characterized by low visibility, such as rain or fog. Step S15: If it does not fall under the two conditions of low light and low visibility mentioned above, it is determined to be a case of sufficient light and good visibility.

[0005] Preferably, in the target detection module, the model is selected based on the lighting conditions determined by the previous module; for good lighting conditions, YOLOv11 based on a series architecture is used to achieve accurate detection of small targets; for low-light and night scenes, a series architecture model based on Oriented R-CNN is used.

[0006] Preferably, the knowledge graph module is structured knowledge for enhanced reasoning. It constructs a triplet knowledge graph tailored to the application scenario, performs preliminary reasoning on detected on-site targets, and the preliminary reasoning results are used to enhance the subsequent multimodal large-scale model reasoning effect. Specifically, constructing the knowledge graph includes: Step T1: Define the schema layer; Step T2: Multi-source data acquisition and preprocessing; Step T3: Structured triple extraction; Step T4: Graph storage and visualization.

[0007] Preferably, the multimodal large model module adopts a pre-trained Qwen2-VL-2B-Instruct large model, constructs a targeted image-text pair dataset for specific application scenarios, and applies LoRA technology to achieve lightweight fine-tuning, resulting in a large model with specialized and enhanced scene knowledge; the specific fine-tuning steps include: Step R1: Prepare the fine-tuning dataset; Step R2: Prepare the environment, load the model and word segmenter; Step R3: Configure LoRA using a library (such as Hugging Face's PEFT library); Step R4: Perform training and check the fine-tuning effect after training; Step R5: Merge the LoRA weights back into the base model and export them; Among these steps, the fine-tuning dataset needs to be a dataset that fits the application scenario. The specific steps for constructing the border patrol dataset include: Step R11: Collect raw data, either from public datasets, public news or documentaries, or synthesize using generative models such as StableDiffusion; Step R12: Perform data preprocessing, desensitize the data (watermark, face, license plate, etc.), and fix the image resolution to be consistent with Qwen2-VL-2B-Instruct (392x392). Step R13: Generate subtitles, preset text templates for border patrol scenarios, use CLIP to calculate image-text similarity, and filter highly relevant image-text pairs; Step R14: Data augmentation. Utilize open-source large models or call APIs to enrich caption expressions, simulate real monitoring conditions for visual enhancement, and annotate after review. Step R15: Format conversion, adapt to the Qwen2-VL-2B-Instruct chat template, and export the dataset.

[0008] Preferably, the onboard computing module first converts the visual encoder portions of YOLOv11, Oriented R-CNN, and Qwen2-VL-2B-Instruct into .ONNX format, then quantizes the parameters into INT8 type and converts them into .rknn format; the language decoder portion of Qwen2-VL-2B-Instruct is directly quantized into INT8 type and converted into .rkllm format; both converted formats are file formats that can be directly deployed on the NPU unit of the RK3588 platform; specifically, the visual part quantization includes the following steps: Step A1: Select a subset from the training (fine-tuning) dataset to use as the quantization calibration dataset; Step A2: Load the model weights and export them in ONNX format with the original parameter precision (FP32); Step A3: Build a quantization model using the rknn-toolkit2 provided by Rockchip. The quantization process uses a calibration dataset to reduce accuracy loss. Converting to ONNX requires exporting the model. For example, when converting the visual encoder part of Qwen2-VL-2B-Instruct, the following steps are required: Step A21: The dimensions of the input image are changed from the original dimensions [1,3,392,392] to [784,1176]; Step A22: Rewrite the forward propagation function and replace it; Step A23: Output the projection using identity mapping and export it in ONNX format; The language decoder for the Qwen2-VL-2B-Instruct large model also needs to quantize the weights and activation values ​​and convert them into a file format that can be directly executed by RK3588; specifically, the quantization of the language part requires the following steps: Step B1: Prepare the calibration dataset; Step B2: Load the weights of the model to be quantized; Step B3: Call rkllm.api, configure the quantization type, quantize both weights and activation values ​​to INT8 type, and use the dataset generated in step B1 for calibration during quantization; Step B4: Export the model and convert it to .rkllm format; Step B1 requires preparing calibration data for the language decoder, which is data after both images and text are embedded and concatenated. The specific generation steps include: Step B11: Extract data from the constructed fine-tuning dataset and add it to the general data to form the text-image pair data for quantization calibration; Step B12: Load the fine-tuned model weights, and input the image and text into the visual encoder and projection layer of the large model and the text segmenter, respectively, to obtain the embedded image and text tensors; Step B13: Convert to a format supported by the rkllm library and output.

[0009] Preferably, in the edge-cloud collaboration module, a visual early warning platform is deployed on the local server and serves as a hub for information transmission, connecting the drone terminal, the local server terminal, and the cloud server terminal. The visual early warning platform displays the early warning information, images, and video information transmitted back by the drone terminal and records historical early warning information.

[0010] Preferably, the edge-cloud collaboration module can compensate for the performance loss caused by deploying a small-parameter model due to limited computing power on the drone side. It can choose to deploy a large-parameter model on a local server or a cloud server for in-depth analysis. If the cloud computing power is sufficient, the cloud server deployment is selected. If there are data confidentiality requirements, the local server deployment is selected.

[0011] Preferably, the unmanned aerial vehicle (UAV) device, used as a hardware carrier for constructing the aforementioned UAV intelligent perception and intent reasoning system, specifically includes: The drone flight platform adopts a four / six-rotor drone with a built-in flight control computer; The mission payload compartment is equipped with an embedded computing platform. The core board has heat sinks and is packaged in a lightweight metal box. It is connected to the drone through a shock-absorbing device. The image acquisition cabin integrates visible light and infrared cameras, which are connected to the embedded computing platform via a USB cable. The communication link unit is a 2.4 / 5G dual-band WIFI module, integrated on the core board of the computing platform, responsible for receiving and sending data; a mobile WIFI device can also be added as a relay station to extend the communication distance between the local server and the embedded terminal of the drone. The power management unit uses dual power supply lines to power the drone device. The large-capacity battery powers the drone's drive and flight control circuits, while the small-capacity battery powers the computing platform separately.

[0012] (III) Beneficial Effects Firstly, in terms of intelligence, this invention deeply integrates multimodal target detection, structured knowledge graphs, and lightweight multimodal large models, forming a continuous processing capability from pixel-level perception to semantic-level reasoning. The system can identify targets such as vehicles and personnel, and, by combining their movement trajectories, geographical locations, time information, and rules in the knowledge base, infer behavioral intentions such as "abnormal loitering" and "close reconnaissance" in real time and assess risk levels. This achieves a leap from "perception" to "cognition," significantly improving the intelligence and foresight of early warning systems.

[0013] Secondly, regarding system performance and adaptability, the multimodal fusion detection framework and deep model optimization technology for the domestic RK3588 chip NPU proposed in this invention effectively solve the problem of small target detection under complex imaging conditions and ensure the real-time running efficiency of the algorithm on embedded devices. Combined with a dynamic task scheduling mechanism that integrates "edge-cloud" three-level collaboration, the system places the initial perception and inference tasks with high real-time requirements on the edge, while placing sensitive or complex context analysis tasks on the local server. Only a small amount of non-sensitive data is uploaded to the cloud for in-depth analysis, significantly reducing end-to-end response latency, ensuring basic operational capabilities in weak network environments, and enhancing system data security through data layering processing.

[0014] Finally, regarding technological independence and controllability, this invention achieves a closed-loop domestic technology system, encompassing everything from underlying hardware and operating systems to core algorithm models. This not only clears the way for the large-scale deployment of the system in highly sensitive areas such as national defense and border defense, but also provides an independent and reliable technological paradigm for building an intelligent border governance system, possessing significant strategic importance and broad prospects for wider application. Attached Figure Description

[0015] Figure 1 This is a diagram of the overall system architecture of the present invention; Figure 2 This is a flowchart of the adaptive all-weather perception process of the image acquisition module and the target detection module of the present invention; Figure 3 The following is an internal flowchart of the process by which the data from the image acquisition module and the target detection module are further input into the knowledge graph module and the multimodal large model module. Figure 4 This is a schematic diagram of the data flow of the edge-cloud collaboration module of the present invention. Detailed Implementation

[0016] To better understand the purpose, structure, and function of this invention, a more detailed description is provided. A drone intelligent perception and intent reasoning system and device utilizes the high mobility and wide monitoring range of drones to achieve intelligent global monitoring of scenarios such as border patrol, urban security, and animal protection. The system employs multiple models—target detection model, knowledge graph, and multimodal large-scale model—in deep collaboration to achieve intelligent perception of the on-site situation and prediction of the behavioral intent of on-site targets. The system is based on the Rockchip RK3588 platform, with lightweight deployment and system integration of various modules. The "edge-cloud" three-terminal system works together, balancing real-time performance, computing power utilization, and data security, enabling multi-level processing and dynamic allocation of drone inspection data.

[0017] Example 1, according to Figure 1 This invention provides an intelligent perception and intent reasoning system and apparatus for unmanned aerial vehicles (UAVs), comprising: The image acquisition module is used to acquire image data. It is equipped with both visible light and infrared light cameras to acquire and enhance images for subsequent processing. The target detection module is used to identify targets in surveillance images. It uses a visible light detection model based on the YOLOv11 series architecture and a multimodal target detection model based on the Oriented R-CNN architecture for targeted processing during the day and at night, respectively. The knowledge graph module uses the target category and confidence level detected by the target detection module to make preliminary inferences about the target's possible intentions. The multimodal large model module uses a fine-tuned Qwen2-VL-2B-Instruct large model to perform reasoning analysis based on the original image and the preliminary reasoning results of the knowledge graph to predict the target intent; The airborne computing module, based on the RK3588 computing platform and equipped with the Galaxy Kylin operating system, lightweightly deploys and system-integrates the aforementioned visible light detection model, multimodal detection model, multimodal large model, and knowledge graph. The edge-cloud collaboration module selectively deploys a visual interface, historical database, complete knowledge base, and multimodal large models with significantly more parameters than the lightweight large models on the drone side on both the local end and the cloud. These are used for data statistics, in-depth analysis, and other purposes, while also making it convenient for relevant personnel to view them.

[0018] Example 2, this example is based on Example 1, according to Figure 2 In this example, the image acquisition module is used to acquire dual-modal image and video information in visible and infrared light to support the system's all-weather operation. The system hardware platform connects to the visible and infrared cameras, uses the hardware platform's built-in hardware decoder to receive information, and implements image processing algorithms executed by the computing platform, specifically including the following: Step S1: Since multimodal detection and analysis are more time-consuming, the system prioritizes visible light detection. In order to determine the subsequent running direction, one or more frames of visible light images are first acquired to determine whether the current scene is in a situation of sufficient light and good visibility, sufficient light but limited field of vision (such as rainy or foggy days), or low light (such as night). Step S2: Collect image / video data in a targeted manner. If the current lighting conditions are good, turn on the visible light camera. If the lighting conditions are low and it is difficult to achieve accurate perception with a single visible light mode, turn on both the visible light and infrared light cameras. Step S3: Apply image enhancement algorithms to the three situations. For example, a defogging network can be used for preprocessing in scenarios with limited field of view, such as rainy or foggy days. Specifically, step S1 mainly includes the following steps: Step S11: To distinguish the above three cases, the acquired image is transferred from the RGB space to the HSV color space and the YUV color space; Step S12: Illumination intensity is related to image brightness. Calculate the mean value of V in the HSV space and extract the global average brightness. Set a brightness threshold to determine whether the situation is low light. Step S13: Visibility is related to image transmittance. Rain and fog can affect this metric. The Dark Channel Prior algorithm is used to calculate the image transmittance map and the mean value is calculated. ; Step S14: Rainy or foggy weather can cause object boundaries in images to become blurred. Therefore, a Laplacian convolution will be performed on the grayscale image, and the variance will be calculated. Thus, the degree of boundary clarity is obtained, based on and Two indicators are used to determine whether the weather is characterized by low visibility, such as rain or fog. Step S15: If it does not fall under the two conditions of low light and low visibility mentioned above, it is determined to be a case of sufficient light and good visibility.

[0019] Example 3, according to Figure 2 This example is based on Embodiment 1. In this example, the target detection module is used to initially perceive and identify targets present in the scene. It uses two models, YOLOv11 and Oriented R-CNN, to achieve all-weather detection. Specifically, it includes the following: Since visible light detection has lower time complexity and can better meet the system's real-time requirements, the system defaults to using YOLOv11 for visible light target detection; in low-light scenarios, such as at night, the system uses Oriented R-CNN for target detection to achieve higher accuracy; the switching between the two models depends on the judgment result of Embodiment 2.

[0020] Example 4, according to Figure 3 This example is based on Implementation Example 1. In this example, after the knowledge graph module detects targets in the scene in the previous module, it uses its output to explore the relationships between targets and make a preliminary judgment on the target's intent. The knowledge graph reasoning results are converted into natural language and concatenated into the large model Prompt of the next example in text form. The construction of the knowledge graph specifically includes the following: Step T1: Define the schema layer. For specific application domains (such as border security), pre-design the ontology structure of the knowledge graph. This includes clearly defining entity categories (such as "border inspection officers", "patrol vehicles", "monitoring blind spots", etc.), attributes of each entity (such as personnel identity, equipment deployment location, and time of behavior occurrence), and the types of relationships between entities (such as "personnel – execution – behavior" and "equipment – ​​deployment at – location"). This schema layer constitutes the semantic skeleton of the knowledge graph, ensuring the consistency and reasonability of subsequent knowledge. Step T2: Multi-source data collection and preprocessing, integrating structured data (such as law enforcement registration forms and equipment logs) and unstructured text (such as border regulations, incident reports, and monitoring descriptions), and after deduplication, noise reduction, and format standardization, constructing a high-quality domain corpus to provide reliable input for knowledge extraction; Step T3: Structured triple extraction. Using natural language processing techniques, especially named entity recognition and relation extraction models based on pre-trained language models (such as BERT), entities are automatically identified from the text and their semantic relationships are mined to generate triples that conform to the pattern layer specification, such as: "infrared monitoring instrument – ​​deployed at – border outpost" and "abnormal trajectory – occurred in – surveillance blind spot". Implicit relationships can also be supplemented by combining image context or common sense, such as "soldier – holding – gun". Step T4: Graph storage and visualization. Import the extracted triples into a graph database (such as Neo4j), represent entities with nodes, and represent relationships or attributes with directed edges to construct a graph-like knowledge network; support the visualization of the relationship paths between entities, which is convenient for manual verification and interactive analysis; The knowledge graph module is deployed on the embedded terminal of the drone, the local server, and the cloud server. The local and cloud servers deploy the original knowledge graph, which contains rich structured knowledge. To enable lightweight operation on the embedded computing platform, the lightweight deployment of the knowledge graph can adopt, but is not limited to, the following methods: subgraph extraction based on the importance of topological structure, targeted pruning based on rule templates, or semantic compression based on embedded representation. In specific implementation, one or a combination of the above methods can be selected according to the storage capacity and inference latency requirements of the embedded terminal. After extraction, these triples are hard-coded and string matching is performed. They are then directly deployed on the embedded computing platform, and inference is performed using the CPU to achieve lightweight real-time inference on the edge.

[0021] Example 5, according to Figure 3 This example is based on Implementation Example 1. In this example, the multimodal large model module is used to perform higher-level semantic understanding and behavioral intent judgment based on target detection and preliminary reasoning of the knowledge graph. The system uses Alibaba Cloud's pre-trained Qwen2-VL-2B-Instruct as its foundation and receives structured context information output from the border security knowledge graph based on key frames of raw images or videos collected by UAVs on-site. The construction of this example specifically includes the following: Step R1: To further enhance the model's professional understanding and reasoning ability in specific task scenarios (such as border patrol and intelligent security), it is necessary to construct a clear image-text pair dataset, which includes typical real-world scene images and accurate descriptions, to guide the model in learning professional semantic associations in border security tasks. Step R2: Prepare the environment, pull the weight file of the pre-trained distillation model, and load the model and word segmenter; Step R3: The system uses LoRA (Low-Rank Adaptation) technology to efficiently fine-tune the parameters of the distilled Qwen2-VL-2B-Instruct model, inserting only a small number of trainable low-rank matrices into the key attention layers of the original model, effectively adapting to downstream tasks and achieving lightweight fine-tuning; LoRA is configured using libraries (such as Hugging Face's PEFT library); Step R4: Perform fine-tuning training. During the fine-tuning process, a learning rate decay strategy is used, and the training process is visualized and monitored using the SwanLab tool to ensure that the model converges stably. After training, check the fine-tuning effect. Step R5: Merge the LoRA weights back into the base model and export them. By integrating visual perception and structured prior knowledge, the model can not only accurately describe the current scene content, but also make reasonable predictions about the potential behavior of the target based on the massive amount of general knowledge it has learned in the pre-training stage. Specifically, the steps of R1 include: Step R11: Fine-tuning the dataset requires data that matches the application scenario. To build the border patrol dataset, it is necessary to collect data from public datasets, public news or documentaries that match the border patrol scenario, or synthesize it using generative models such as StableDiffusion. Step R12: Perform data preprocessing and desensitize the data, such as blurring the watermarks in the four corners, using the detection model to find sensitive information such as faces and license plates, then blurring them, and fixing the image resolution to be consistent with Qwen2-VL-2B-Instruct (392x392). Step R13: Generate captions, preset text templates for border patrol scenarios (e.g., "Suspicious people are trying to climb over the fence", "Multiple armed people are gathering", "Wild animals are approaching the border", etc.), use CLIP to calculate the similarity between images and text, filter out highly relevant image-text pairs, and manually review those with high uncertainty. Step R14: Data augmentation is performed on both text and images. For text, open-source large models or API calls can be used to input image-text pairs and generate detailed descriptions. Simulate real monitoring conditions by performing geometric transformations, color transformations, or weather degradation simulations on the images. Then, the image-text pairs are automatically labeled. Step R15: Format conversion, adapt to the Qwen2-VL-2B-Instruct chat template, and export the dataset; The multimodal large model is divided into two parts. One part is a small-parameter, lightweight large model that is quantized by the onboard computing power module and deployed on a resource-constrained UAV embedded computing power platform. The other part is a large model with a larger number of parameters that is deployed on local or cloud servers according to the actual situation for high-precision inference and prediction.

[0022] Example 6: This example is based on Example 1. In this example, the airborne computing module integrates the above software algorithms using the RK3588 platform equipped with the Galaxy Kylin operating system, and deploys the neural network model locally in a lightweight manner. During the lightweighting process, the visual part and the language part are quantized separately. The quantization of the visual part specifically includes the following: Step A1: The visual part includes the YOLOv11 after transfer learning, the Oriented R-CNN model, and the fine-tuned Qwen2-VL-2B-Instruct visual part. A subset is selected from the training or fine-tuning dataset to be used as the quantization calibration dataset. The calibration data of the large model's visual part also needs to be included in some general data. Step A2: Load the model weights, first using the original parameter precision, which is 32-bit floating-point (FP32) by default, and then export the intermediate format (ONNX format) without quantization. Step A3: Build a quantization model using the rknn-toolkit2 provided by Rockchip. The quantization process uses a calibration dataset to calibrate the parameter changes of each layer and find a reasonable mapping to map the original 32-bit floating-point (FP32) precision parameters to 8-bit signed integers (INT8) to achieve quantization with the lowest precision loss. Specifically, step A2 includes the following: Step A21: In the quantization of the visual part, the visual part of the Qwen2-VL-2B-Instruct large model, namely the visual ViT encoder and projection part, needs to be quantized according to the special processing method of the model itself. Due to the limited resources on the edge, the dynamic resolution is changed to a fixed input image size. The input image needs to be changed to a series of dimensional changes according to the structure of the pre-trained model, from the original dimensions [1,3,392,392] to [784,1176]. Step A22: In order to reduce resource consumption, the rotation position encoding matrix is ​​no longer calculated after the resolution is fixed, and the calculated matrix is ​​used directly. Therefore, the forward propagation function needs to be rewritten and replaced. Step A23: Output the projection using identity mapping and export it in ONNX format; The specific steps of dimensional transformation are derived from the structural constraints of the pre-trained model. These steps include: The input image resolution is fixed at 392x392. According to the requirements of the pre-trained model, the original input [1,3,392,392] is repeated in the frame dimension (temporal patch) to become [2,3,392,392]. It is then divided into 14x14 blocks, and four adjacent blocks are merged into one (merge_size=2), becoming [1,2,3,14,2,14,14,2,14]. The dimensions are, in order, temporal grid (set to 1 for image task), temporal patch, number of RGB channels, height grid after merging, height merging factor (default is 2), block height, width grid after merging, width merging factor (default is 2), and block width. According to the requirements of the pre-trained model, the transformation dimension index order is [0,3,6,4,7,2,1,5,8], and the output shape is [1,14,14,2,2,3,2,14,14]; The flattening is done in two dimensions. The first dimension is the product of the time grid, the merged height grid, and the merged width grid. The second dimension is the product of the number of RGB channels, the time patch, the block height, and the block width. The output shape is [784, 1176]. In step A22, the forward propagation function is rewritten. The specific steps are as follows: Align the visual space dimension to the semantic space, perform linear projection on the input, and project the input shape [784,1176] onto [784,1536]; Rotational position encoding is performed, and the output shape is [784,24,64,2]. Since the resolution is fixed, the rotational position encoding matrix is ​​saved, thus avoiding the NPU computation adaptation problem. Because the resolution is fixed, the cumulative sequence length is calculated, and the fixed tensor is saved to avoid the NPU computation adaptation problem. The output shape is restored to [784, 1536]. Continue rewriting the forward propagation logic, with the current output being input into each Transformer block in sequence; the outputs are merged and projected, and the shape is preserved using an identity mapping. The language part (Qwen2-VL-2B-Instruct language decoder) is quantized, converting both its weight values ​​and activation values ​​into 8-bit signed integers (INT8) type. Specifically, this includes the following: Step B1: Prepare the calibration dataset, calibrate the mapping of the language part, and ensure that the model accuracy decreases little when compressing parameter accuracy; Step B2: Load the model weights to be quantized. For the fine-tuned model, the weights trained after fine-tuning need to be used. Step B3: Call rkllm.api, configure the quantization type, weigh the deployment hardware limitations and accuracy requirements, and choose to quantize both weights and activation values ​​to INT8 type. Use the dataset generated in step B1 to calibrate during quantization. Step B4: Export the model and convert it to .rkllm format; Specifically, step B1 includes the following: Step B11: Extract data from the constructed fine-tuning dataset. To prevent overfitting, general data is included. The fine-tuning data accounts for 90% of the calibration data, forming the graph-text pair data for quantitative calibration. Step B12: Since the fine-tuning object is the language decoder, it is necessary to simulate the situation before the data is input into the language decoder, that is, the tensor data after the text and image are embedded and concatenated according to the chat template; load the fine-tuned model weights, and input the image and text into the visual encoder and projection layer and text segmenter of the large model respectively to obtain the embedded image and text tensors; Step B13: Convert to a format supported by the rkllm library and output it, using JSON format.

[0023] Example 7, according to Figure 4 This example is based on Implementation Example 1. In this example, the edge-cloud collaboration module has a visual early warning platform deployed on the local server, which serves as the core hub for information uploading and downloading in the entire system, effectively connecting the drone terminal, the local server terminal, and the cloud server terminal. Specifically, the visualization early warning platform receives and displays key information such as detection results, intent reasoning conclusions, original images and video streams transmitted back from the drone in real time; at the same time, the platform stores and classifies all historical early warning events in a structured manner, which facilitates subsequent retrospective analysis, strategy optimization and command decision-making. To compensate for the performance loss caused by the limitation of computing power, power consumption, and memory on the drone side, which can only deploy lightweight models with small parameters, the system supports secondary analysis on the local or cloud server side: if the task scenario has high requirements for data security and the local server has sufficient computing power, the local server is used first, and the medium-parameter Qwen large model deployed on it completes the refined intent reasoning and risk assessment; if the task scenario does not have strict confidentiality restrictions and the cloud has sufficient computing power, the large-parameter or even full-version large model deployed on the cloud can be used to perform more complex multimodal reasoning; the fusion of the edge / cloud reasoning results can adopt conventional multi-source information fusion methods in this field, such as weighted fusion based on confidence weights, voting fusion based on rule logic, or manual judgment. Given that the fusion strategy is a general technology in the field of information fusion, this invention does not limit it, and implementers can flexibly choose according to the requirements of the actual scenario and the security level. Example 8: In this example, the drone device is used to integrate the modules of Examples 1 to 7 above as a hardware carrier; specifically, the device mainly includes: The drone flight platform adopts a four / six-rotor drone with a built-in flight control computer. When selecting a model, it is necessary to consider the payload weight and endurance requirements, and select a suitable model that ensures stability while also having maneuverability and wind resistance. The mission payload compartment is equipped with an embedded computing platform. The core board has heat sinks and is packaged in a lightweight metal box, which protects the core board and allows for air cooling on the drone platform. The payload compartment is connected to the drone via a shock-absorbing device. The image acquisition cabin includes visible light and infrared cameras, or uses an integrated dual-light optoelectronic pod, connected to an embedded computing platform via a USB cable; it is also connected to the drone via a shock-absorbing device to reduce the interference of the drone's high-frequency vibration and movement on image quality. The communication link unit is a 2.4 / 5G dual-band WIFI module, integrated on the core board of the computing platform. It is responsible for receiving and sending data, and uses a glue rod antenna to expand the transmission range. It is fixed on the metal protective shell of the core board. A mobile WIFI device can also be added as a relay station to connect the local server and the drone's onboard WIFI module, extending the communication distance between the local server and the drone's embedded end. The power management unit uses dual power supply lines to power the drone device. A large-capacity battery powers the drone drive and flight control circuits. To avoid voltage fluctuations caused by the drone's motor rotation from interfering with the power supply of the computing platform, and considering the low power consumption of the computing platform, a small-capacity battery is used to power the computing platform separately. A DC voltage regulator module is used to stabilize the power supply within the power supply requirements of the computing platform. It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. An unmanned aerial vehicle intelligent perception and intention reasoning system, characterized in that, include: The image acquisition module is used to acquire image data. It is equipped with both visible light and infrared light cameras to acquire and enhance images for subsequent processing. The target detection module is used to identify targets in surveillance images. It uses a visible light detection model based on the YOLOv11 series architecture and a multimodal target detection model based on the Oriented R-CNN architecture for targeted processing during the day and at night, respectively. The knowledge graph module uses the target category and confidence level detected by the target detection module to make preliminary inferences about the target's possible intentions. The multimodal large model module uses a finely tuned Qwen2-VL-2B-Instruct large model to perform reasoning analysis based on the original images collected by the image acquisition module and the preliminary inference of the knowledge graph to predict the target's intention. The airborne computing module, based on the RK3588 computing platform and equipped with the Galaxy Kylin operating system, lightweightly deploys and system-integrates the aforementioned visible light detection model, multimodal detection model, multimodal large model, and knowledge graph. The edge-cloud collaboration module connects the embedded terminal, local terminal, and cloud terminal of the drone. It selectively deploys a visual interface, historical database, complete knowledge base, and multimodal large model components with significantly more parameters than the lightweight model on the drone terminal on the local terminal and cloud terminal for data statistics, in-depth analysis, and other purposes, while also making it convenient for relevant personnel to view. The output of the image acquisition module is connected to the input of the target detection module, and is used to input the enhanced image data; the output of the target detection module is connected to the knowledge graph module. The knowledge graph module injects the preliminary reasoning results into the input context of the multimodal large model module in the form of text prompts; The inference results of the multimodal large model module are processed by the airborne computing power module and then used for information interaction and enhanced inference through the edge-cloud collaboration module.

2. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The image acquisition module is used to acquire image data and is equipped with both visible light and infrared cameras. Specifically, it includes the following steps: Step S1: Acquire one or more frames of visible light images to determine the current scene's lighting and visibility conditions; Step S2: Targeted acquisition of image / video data, including visible light and multimodal data; Step S3: Apply targeted image enhancement algorithms for the three scenarios: low light, low visibility, and normal light and normal visibility.

3. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The target detection module uses two detection models: a visible light target detection model based on the YOLOv11 series architecture; and a multimodal target detection model based on the Oriented R-CNN architecture.

4. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The knowledge graph module, which constructs a triplet knowledge graph based on frequently occurring targets and behaviors in the scene, specifically includes the following steps: Step T1: Define the schema layer; Step T2: Multi-source data acquisition and preprocessing; Step T3: Structured triple extraction; Step T4: Graph storage and visualization.

5. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The multimodal large model module includes the Qwen2-VL-2B-Instruct multimodal large model, which is fine-tuned for the scene, and specifically includes the following steps: Step R1: Prepare the fine-tuning dataset; Step R2: Prepare the environment, load the model and word segmenter; Step R3: Configure LoRA (Low-Rank Adaptation); Step R4: Perform training and check the fine-tuning effect after training; Step R5: Merge the LoRA weights back into the base model and export them; Step R1 specifically includes the following steps: Step R11: Collect raw data; Step R12: Perform data preprocessing and de-identify the data; Step R13: Preset text templates for border patrol scenarios and set thresholds to filter highly relevant image-text pairing data; Step R14: Data augmentation, followed by manual review and annotation; Step R15: Format conversion, export dataset.

6. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The airborne computing module deploys the aforementioned visible light detection model, multimodal detection model, and large multimodal model on the RK3588 platform, quantizing the original floating-point precision to INT8 precision and converting it into a .rknn or .rkllm format that the platform can directly execute. The lightweight deployment of the visual model specifically includes the following steps: Step A1: Select a subset to be used as the quantization calibration dataset; Step A2: Export the ONNX format according to the original parameters; Step A3: Construct a quantization model and use a calibration dataset to reduce accuracy loss; The lightweight deployment of the language model specifically includes the following steps: Step B1: Prepare the calibration dataset; Step B2: Load the weights of the model to be quantized; Step B3: Configure the quantization type and calibrate using the dataset generated in step B1 during quantization; Step B4: Export the model and convert its format.

7. The UAV intelligent perception and intent reasoning system according to claim 1, characterized in that: The edge-cloud collaboration module connects the embedded terminal of the drone, the local server terminal, and the cloud server terminal. It selectively deploys a visual interface, historical database, complete knowledge base, and multimodal large model components with a significantly larger number of parameters than the lightweight large model on the drone terminal on the local terminal and the cloud terminal.

8. A drone device, characterized in that, It includes an unmanned aerial vehicle (UAV) flight platform, a mission payload compartment, an image acquisition compartment, a communication link unit, and a power management unit; the mission payload compartment is equipped with an embedded computing platform, and the embedded computing platform is deployed with the image acquisition module, target detection module, knowledge graph module, multimodal large model module, and airborne computing module as described in claim 1, which is an airborne deployment example.

Images