An adaptive multi-modal unmanned aerial vehicle target tracking system and method

Abstract

The application discloses a kind of self-adapting multimode unmanned vehicle target tracking system and method, system is embedded in unmanned vehicle, including: distance perception module, real-time calculation unmanned vehicle and target distance, speed and predict trajectory;Intelligent decision engine, according to distance selection corresponding complexity visual perception model, and optimal combination is sought in three-dimensional space with platform computing power as constraint, simultaneously preloading next model;Multi-modal perception execution layer, drive variable parameter camera data acquisition according to optimal resolution and frame rate, and by multi-model inference engine distributes task to corresponding hardware;When resolution switches, FPA-Net aligns different resolution features, maintains target ID continuous.The application realizes the whole process millisecond-level adaptive switching of far-middle-near target tracking, greatly improves detection rate and hardware utilization, solves edge computing power limited, poor dynamic scene adaptability, real-time-precision contradiction and resolution switching target loss and other problems.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target tracking technology, and in particular to an adaptive multimodal UAV target tracking system and method. Background Technology

[0002] Unmanned aerial vehicle (UAV) aerial target tracking technology has demonstrated significant application value in fields such as reconnaissance, public safety monitoring, wildlife conservation, and disaster relief. However, deploying advanced visual algorithms on edge computing platforms like UAVs typically faces three major challenges: 1. Drone platforms are extremely sensitive to power consumption, weight, and cost. The edge computing chips (such as RK3588) in existing technologies have limited computing power (usually at the level of 6 TOPS), making it difficult to continuously run high-complexity, high-precision deep learning models. Currently, there is a lack of collaborative scheduling algorithms specifically designed for tracking tasks, resulting in generally low utilization of hardware resources.

[0003] 2. In actual missions, the relative distance and attitude between the UAV and the target are constantly changing. The target changes from a blurry small dot at a distance (occupying less than 1% of the frame) to a clear large target at close range (occupying more than 30% of the frame). This process places drastically different demands on the resolution, model complexity, and frame rate of the perception algorithm. Most existing systems adopt fixed perception modes, which are not adaptable to dynamic scenes and are difficult to adapt to such drastic dynamic changes.

[0004] 3. High-resolution input (such as 4K) can provide richer details, which is beneficial for the detection of distant targets, but it will significantly increase the computational burden, causing the processing frame rate to drop to 10-15 FPS; high frame rate (such as 120 FPS) is beneficial for the accurate tracking of high-speed maneuvering targets, but this usually requires sacrificing input resolution and model accuracy.

[0005] In the field of UAV target tracking, facing the aforementioned challenges, patent number US20160266579A1 proposes using GPS to achieve route planning and autonomous flight. However, in its solution, GPS information is only used for flight control and is not coupled with the dynamic adjustment of the visual perception algorithm, resulting in a disconnect between GPS guidance and visual perception. While patent number US20170144756A1 achieves target acquisition under GPS guidance, its perception system uses a fixed mode and lacks the ability to adapt to targets at different distances. Patent number US20210081794 can dynamically switch neural network models based on network conditions or computing resources, but its decision-making criteria do not include the key variable of physical distance to the target, and the model switching mechanism lacks distance perception. Patent number US10956807B1 switches models based on macro-environmental factors such as weather and geographical location, but similarly fails to achieve refined and real-time adjustments driven by target distance. Furthermore, existing layered or multi-model tracking methods require unloading the current model and loading the new model when switching models, which introduces a delay of 200-500 milliseconds. This can easily lead to target loss in high-speed dynamic scenes. Although patent number EP2751984A1 can achieve resolution switching in streaming media, it does not solve the problem of inconsistent target features caused by drastic resolution changes. This can easily lead to target loss or identity (ID) exchange, disrupting the continuity of tracking. Summary of the Invention

[0006] To address the aforementioned issues, this invention provides an adaptive multimodal UAV target tracking system and method, which enables efficient adaptive switching of the UAV tracking system at different distances while ensuring the continuity and real-time performance of the tracking.

[0007] This invention provides an adaptive multimodal unmanned aerial vehicle (UAV) target tracking system, applied in a UAV, wherein the UAV is embedded with a heterogeneous edge computing platform including a multi-core NPU and a GPU. The specific technical solution is as follows: The system includes a distance perception module, an intelligent decision engine, and a multimodal perception execution layer; The distance sensing module acquires the UAV's position and the target waypoint's position in real time, and calculates the distance and relative speed between the UAV and the target; The intelligent decision engine is communicatively connected to the distance perception module, and the intelligent decision engine includes a perception model selector and a three-dimensional parameter co-optimizer. The perception model selector selects a visual perception model of corresponding complexity from a pre-set model library based on the real-time distance; the three-dimensional parameter co-optimizer solves and outputs the optimal combination of three-dimensional parameters in a predefined three-dimensional parameter space, with the total computing load not exceeding the upper limit of the computing power of the edge computing platform as a constraint. The multimodal perception execution layer is communicatively connected to the intelligent decision engine, and the multimodal perception execution layer includes a variable parameter camera module and a multi-model inference engine; like Figure 6 As shown, the variable parameter camera module receives three-dimensional parameters and acquires video stream data based on the optimal parameter combination; the multi-model inference engine drives the currently selected visual perception model, takes the received video stream data as input, and outputs target detection results, which include the target bounding box coordinates (x, y, w, h), confidence score, and target category label; then, the target detection results are post-processed, including: Non-maximum suppression (NMS) is used to filter overlapping redundant detection boxes; Target association matching is used for multi-target tracking based on intersection-over-union (IoU) and feature similarity. Track smoothing is used to filter jittery tracks; Coordinate transformation is used to transform the image coordinate system to the world coordinate system; Based on the target location obtained from post-processing, execute control logic tasks, including: PID controller parameter calculation is used to calculate the control quantity based on the target position deviation; Heading angle calculation is used to determine the amount of heading adjustment for the UAV; Speed ​​planning is used to dynamically adjust flight speed based on distance.

[0008] When the multi-model inference engine is driven, each task is assigned to the most suitable hardware unit for execution, as follows: The inference task of the visual perception model is assigned to the NPU for execution; the NMS calculation and target association matching tasks in post-processing are assigned to the GPU for execution; and the control logic tasks and flight control command generation tasks are assigned to the CPU for execution. The allocation is performed based on a pre-established task-hardware mapping table, which is obtained offline based on the computational characteristics of each task and the computing power characteristics of each hardware unit.

[0009] Furthermore, the distance sensing module also includes: predicting the future trajectory of the target over a set time period based on Kalman filtering, and outputting the predicted trajectory information.

[0010] Furthermore, the intelligent decision engine also includes a parallel preloading scheduler, which receives the predicted trajectory information and preloads the visual perception model to be switched to in the next stage to the idle computing core of the edge computing platform.

[0011] Furthermore, the multimodal perception execution layer also includes a cross-resolution feature alignment network, which maps the feature spaces of different resolutions when the camera resolution is switched.

[0012] Furthermore, the cross-resolution feature alignment network adopts the FPA-Net network structure, including a Siamese dual branch, each branch sequentially containing a spatial pyramid pooling layer and a channel-space dual attention module.

[0013] Furthermore, the feature spaces at different resolutions are mapped, as follows: The video stream data collected during the resolution switching period is input into the alignment network model to extract image features; After the resolution switching is completed, the video stream data acquired is input into the alignment network model to extract image features; Image features extracted from images acquired at different resolutions are fused using an attention fusion module to obtain aligned, unified features.

[0014] This invention also provides an adaptive multimodal UAV target tracking method, based on the aforementioned adaptive multimodal UAV target tracking system, the method comprising: S1: Real-time calculation of the distance and relative speed between the UAV and the target waypoint; S2: Select a visual perception model of corresponding complexity from the preset model library according to the distance; S3: Based on the model complexity corresponding to the selected visual perception model, in the three-dimensional parameter space composed of model complexity, camera resolution, and frame rate, and constrained by the upper limit of the total computing power of the UAV's edge computing platform, solve for the optimal combination of three-dimensional parameters. The solution steps are as follows: Define a three-dimensional parameter space Ω={(M,R,F)}, where M is the model complexity parameter, R is the camera resolution parameter, and F is the frame rate parameter; the model complexity dimension M contains multiple preset complexity levels; the resolution dimension R contains multiple resolution levels supported by the camera. Establish a single-frame processing latency model:

[0015] in, The single-frame inference latency of the selected model M on the NPU; Image preprocessing single-frame latency (as opposed to resolution) (related) Fixed delay for post-processing; Establish real-time constraints: ≤

[0016]

[0017] in, For target frame rate The corresponding maximum processing time per frame, Target frame rate; Construct the optimization objective function:

[0018] in, The average detection accuracy under different model and resolution combinations is pre-calibrated on a standard dataset through offline testing and stored as a lookup table; This is a normalized frame rate continuity metric. This comes at the cost of normalized power consumption; , , These are weighting coefficients, pre-set according to task requirements; The optimal parameter combination is found by traversing all parameter combinations that satisfy the constraints. Calculate the corresponding objective function value Return the combination of parameters that maximizes the objective function value as the optimal three-dimensional parameter combination. .

[0019] S4: Acquire video streams based on the optimal camera resolution and frame rate, perform selected model inference on the video streams, and allocate post-processing and control logic to the corresponding hardware units.

[0020] Furthermore, step S1 also includes using Kalman filtering to predict the future trajectory of the target over a preset time period to obtain predicted trajectory information, as follows: Define the target state vector , where (x,y,z) are the three-dimensional position coordinates of the target in the world coordinate system, and (vx,vy,vz) are the velocity components of the target in the three directions; Establish discrete-time linear state transition equations ,in, The state transition matrix is ​​constructed based on a uniform motion model. The process noise has the following covariance matrix: ; Establish observation equations ,in, For the observation matrix, The relative position observations of the UAV and the target obtained via GPS. To measure noise, its covariance matrix is: ; Based on historical observation data, the prediction and update steps of Kalman filtering are performed to obtain the optimal state estimate at the current time k. and its covariance matrix ; Based on optimal state estimation Iteratively perform N forward prediction steps to output the predicted trajectory sequence for the future preset time period. N is determined based on the latency required for model switching, and its value ranges from 2 to 10 seconds, corresponding to the number of frames.

[0021] Furthermore, the method also includes: obtaining the visual perception model to be switched to in the next stage based on the predicted trajectory information, and loading it into an idle computing core.

[0022] Furthermore, the method also includes mapping features at different resolutions when the camera resolution changes.

[0023] The beneficial effects of this invention are as follows: 1. This invention uses GPS real-time distance as the decision variable. In the three-dimensional parameter space composed of model complexity M, resolution R, and frame rate F, it solves the optimal parameter combination with 6 TOPS hard constraints and simultaneously adjusts camera parameters and inference model. This solves the current problems of insufficient edge computing power, algorithm inability to adapt or long switching delay caused by drastic changes in far-to-medium-to-near scale, real-time-accuracy contradiction, and target loss caused by resolution switching. It achieves continuous tracking of 30–120 FPS through a single chip, improving detection rate and hardware utilization.

[0024] 2. The intelligent decision engine of this invention first uses Kalman filtering to output a predicted trajectory for a preset duration, and then loads the next model to be switched into an idle NPU core in advance, thereby reducing the model switching latency and thus reducing the target loss rate in high-speed maneuvering scenarios.

[0025] 3. This invention uses the FPA-Net cross-resolution feature alignment network to map features from different resolutions to a unified space at the instant of resolution switching, thus ensuring tracking continuity. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the system framework of the present invention.

[0027] Figure 2 This is a schematic diagram of the dual-core switching timing of the present invention.

[0028] Figure 3 This is a schematic diagram of the internal structure of the intelligent decision engine of the present invention.

[0029] Figure 4 This is a schematic diagram of the FPA-Net network structure of the present invention.

[0030] Figure 5 This is a schematic diagram of the method flow of the present invention.

[0031] Figure 6This is a schematic diagram of the task allocation process of the multi-model inference engine of the present invention.

[0032] Figure 7 This is a schematic diagram of the three-dimensional parameter optimization solution of the present invention.

[0033] Figure 8 This is a schematic diagram of the data flow for the Kalman filter prediction trajectory of the present invention. Detailed Implementation

[0034] The technical solutions in the embodiments of the present invention are clearly and completely described in the following description. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0035] In the description of the embodiments of the present invention, it should be noted that the indicated orientation or positional relationship is based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of the invention is conventionally placed during use, or the orientation or positional relationship in which those skilled in the art conventionally understand it during use. This is only for the convenience of describing the present invention and simplifying the description, and is not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of the present invention. Furthermore, the terms "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0036] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set" and "connection" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0037] Example 1 Embodiment 1 of the present invention discloses an adaptive multimodal unmanned aerial vehicle (UAV) target tracking system, which is applied in a UAV, wherein the UAV is embedded with a heterogeneous edge computing platform including a multi-core NPU and a GPU; Specifically, the core chip of the heterogeneous edge computing platform can be the RK3588 (3×2 TOPSNPU + Mali-G610 GPU + quad-core A76 / A55 CPU), with a total power consumption of <8W. It can be directly mounted on the MIPI-CSI2 camera, UART / GPS module, and PWM / MAVLink flight control interface to achieve real-time inference at the data source without the need to transmit back to the cloud.

[0038] like Figure 1 As shown, the system includes a distance perception module, an intelligent decision engine, and a multimodal perception execution layer; The distance sensing module acquires the UAV's position and the target waypoint's position in real time, and calculates the distance d between the UAV and the target and the relative speed v. The distance sensing module further includes: predicting the future trajectory of the target over a set time period based on Kalman filtering, and outputting the predicted trajectory information.

[0039] like Figure 3 As shown, the intelligent decision engine is communicatively connected to the distance perception module. The intelligent decision engine includes a perception model selector, a three-dimensional parameter co-optimizer, and a parallel preloading scheduler. The perception model selector selects a visual perception model of corresponding complexity from a pre-set model library (containing models of different complexities such as YOLOv8-L, YOLOv8-M, and YOLOv8-N) based on the real-time distance d. Specifically, the mapping relationship established in the perception model selector is: M=f(d), where f(d) is a piecewise linear function or a lookup table used for model selection based on d; the mapping of model selection can be achieved by setting different thresholds; for example: YOLOv8-L is selected when d is less than the first threshold, YOLOv8-M is selected when d is between the first and second thresholds, and YOLOv8-N is selected when d is greater than the second threshold.

[0040] The three-dimensional parameter co-optimizer solves and outputs the optimal combination of three-dimensional parameters in a predefined three-dimensional parameter space, with the constraint that the total computing load does not exceed the upper limit of the computing power of the edge computing platform; in this embodiment, the constraint is that the total computing power of the edge computing platform does not exceed 6 TOPS.

[0041] The parallel preloading scheduler receives the predicted trajectory information and loads the visual perception model to be switched in the next stage into the idle computing core of the edge computing platform in advance. like Figure 2 As shown, the timing process of model switching in the intelligent decision engine is as follows: After the perception model selector selects a visual perception model based on the real-time distance d, it runs the currently selected model through the default NPU core in the activity, which is designated as the first computing core. The parallel preloading scheduler obtains the visual perception model to be switched in the next stage based on the predicted trajectory information, and then preloads the visual perception model to be switched through the standby NPU core, which is designated as the second computing core. When switching models, if the model being switched to is the same as the pre-loaded model, the first computing core stops its computing task, the second computing core is activated, the data flow is switched, the pre-loaded visual perception model is run, the first computing core is released, and the first computing core is set up as a standby NPU core to execute the next round of pre-loaded tasks.

[0042] The multimodal perception execution layer is communicatively connected to the intelligent decision engine. The multimodal perception execution layer includes a variable parameter camera module, a multi-model inference engine, and a cross-resolution feature alignment network. The variable parameter camera module receives three-dimensional parameters and acquires video stream data based on the optimal parameter combination; The multi-model inference engine includes a model-driven module, a post-processing module, a control logic module, and a task scheduler. The model-driven module drives the currently selected visual perception model, takes the received video stream data as input, performs forward inference on each frame of the image, and outputs the target detection result. The target detection result includes the bounding box coordinates (x, y, w, h) of each detected target, the confidence score of each detected target, and the category label of each detected target. Where (x,y) are the coordinates of the center point of the bounding box, and (w,h) are the width and height of the bounding box; the confidence score of each detected target ranges from [0,1].

[0043] The post-processing module receives the target detection results and performs the following processing sequentially: (a) Non-maximum suppression (NMS) processing: For multiple overlapping detection boxes of the same target, filtering is performed based on the confidence score and the intersection-over-union (IoU) threshold, and the detection box with the highest confidence score is retained. In this embodiment, the IoU threshold is set to 0.5; (b) Target association matching: Based on the IoU overlap and appearance feature similarity between the current frame detection result and the historical trajectory, the Hungarian algorithm is used to perform multi-target association and maintain the continuity of target identity ID; (c) Trajectory smoothing: The target trajectory is smoothed using a moving average filter to eliminate detection jitter; (d) Coordinate transformation: Based on the current attitude of the UAV and the intrinsic and extrinsic parameters of the camera, the position of the target in the image coordinate system is transformed into a three-dimensional position in the world coordinate system.

[0044] The control logic module receives the target position after coordinate transformation and executes the following control tasks: (a) Calculate the deviation between the target position and the current position of the UAV; (b) Calculate the control parameters for pitch angle, roll angle and yaw angle based on the PID control algorithm; (c) Dynamically plan flight speed based on target distance; (d) Generate flight control commands in MAVLink format and send them to the flight control system.

[0045] The task scheduler allocates the above modules to the corresponding hardware units for execution according to the pre-defined task-hardware mapping table: the NPU core is responsible for performing tensor operations and convolution operations of deep learning models, with high energy efficiency; the GPU is responsible for performing post-processing algorithms with high parallelism such as NMS and feature matching; and the CPU core is responsible for performing serial control logic such as PID control, heading calculation, and flight control command generation.

[0046] The cross-resolution feature alignment network maps different resolution feature spaces when the camera resolution switches, such as... Figure 4 As shown, in this embodiment, the FPA-Net network structure is adopted, including a Siamese dual branch, each branch containing a spatial pyramid pooling layer and a channel-space dual attention module in sequence; the video stream data collected during the resolution switching period is input into the alignment network model to extract image features; the video stream data collected after the resolution switching is completed is input into the alignment network model to extract image features. Image features extracted from images acquired at different resolutions are fused using an attention fusion module to obtain aligned, unified features, which are then used as input to the model.

[0047] Example 2 Embodiment 2 of the present invention discloses an adaptive multimodal unmanned aerial vehicle (UAV) target tracking method, based on the adaptive multimodal UAV target tracking system described in Embodiment 1 above, such as... Figure 5 As shown, the method includes: S1: The distance perception module calculates the distance and relative speed between the UAV and the target waypoint in real time, and uses Kalman filtering to predict the target's trajectory in the next 5-10 seconds to obtain the predicted trajectory information; like Figure 8 As shown, the process for obtaining the predicted trajectory information is as follows: S101: Define the target state vector , where (x,y,z) are the three-dimensional position coordinates of the target in the world coordinate system (such as the ENU coordinate system), in meters, and (vx,vy,vz) are the velocity components of the target in the east, north, and sky directions, in meters per second; S102: Establish the discrete-time linear state transition equation ,in, The state transition matrix is ​​constructed based on a uniform motion model. The process noise follows a Gaussian distribution with a mean of zero, and its covariance matrix is: Reflecting model uncertainty; State transition matrix It is a 6×6 matrix, expressed in the following form:

[0048] in, In this embodiment, the sampling time interval is... =1 / F (F is the current frame rate).

[0049] S103: Establish the observation equation Among them, the observation matrix It is a 3×6 matrix, expressed as follows:

[0050] The relative position observations of the UAV and the target obtained via GPS. To measure the noise, it follows a Gaussian distribution with a mean of zero, and its covariance matrix is: Reflecting the accuracy of GPS measurements; S104: Filter Initialization: Initial State of the Kalman Filter Based on the initial GPS positioning results, the initial velocity is set to zero; the initial covariance matrix... Let it be a diagonal matrix, with each element reflecting the uncertainty of the initial state.

[0051] S105: Based on historical observation data, perform the prediction and update steps of Kalman filtering to obtain the optimal state estimate at the current time k. and its covariance matrix ; Specifically as follows: For each time step k, execute the following prediction-update loop: Prediction steps: State prediction: ; Covariance prediction: ; Update steps: Calculate the Kalman gain: ; Status Update: ; Covariance update: ; S106: Optimal state estimation based on the current time k Iteratively perform N forward prediction steps to output the predicted trajectory sequence for the future preset time period:

[0052]

[0053]

[0054] The output predicted trajectory sequence is determined by the delay required for model switching, and the value ranges from 2 to 10 seconds, corresponding to the number of frames.

[0055] FOR i = 1 TO N: (k+i|k) = F· (k+i-1|k) P(k+i|k) = F·P(k+i-1|k)·F + Q Output predicted trajectory sequence Where N corresponds to a preset time period, which in this embodiment is 2-10 seconds. , For the preset time period, The current frame rate. The preset time period is dynamically determined based on the latency required for the current model switching to ensure that a sufficiently long prediction trajectory can be obtained before the model switching is completed.

[0056] S2: The intelligent decision engine selects a visual perception model of corresponding complexity from the pre-set model library based on the distance; S3: Based on the model complexity corresponding to the selected visual perception model, in the three-dimensional parameter space consisting of model complexity M, camera resolution R and frame rate F, with the total computing power of the edge platform 6TOPS as a constraint, solve for the optimal parameter combination. like Figure 7 As shown, the specific process is as follows: S301: Define a three-dimensional parameter space Ω={(M,R,F)}, where M is the model complexity parameter, R is the camera resolution parameter, and F is the frame rate parameter; The model complexity dimension M includes multiple preset complexity levels. In this embodiment, the model complexity dimension... These correspond to lightweight models (such as YOLOv8-N), standard models (such as YOLOv8-M), and high-precision models (YOLOv8-L), respectively; the resolution dimension R includes multiple resolution levels supported by the camera. In this embodiment, the resolution dimension R = {640×480, 1280×720, 1920×1080, 3840×2160}. The frame rate dimension F includes multiple frame rate levels. In this embodiment, the frame rate dimension F = {15fps, 30fps, 60fps, 120fps}. S302: Establish a single-frame processing delay model:

[0057] in, The single-frame inference latency of the selected model M on the NPU, in milliseconds, was obtained through offline testing. Image preprocessing single-frame latency (as opposed to resolution) (related), that is , The processing time coefficient per unit pixel. Reference frame rate; To ensure a fixed post-processing delay, this embodiment sets it to 2ms; Establish real-time constraints: ≤

[0058]

[0059] in, For target frame rate The corresponding maximum processing time per frame, Target frame rate; S303: Constructing the optimization objective function:

[0060] in, The average detection precision under different model and resolution combinations is pre-calibrated on the standard dataset (COCO) through offline testing and stored as a two-dimensional lookup table; As a normalized frame rate continuity metric, higher frame rates correspond to better tracking continuity; This comes at the cost of normalized power consumption; , , The weighting coefficients are preset according to task requirements. In this embodiment, =0.5、 =0.3、 =0.2.

[0061] S304: The optimal parameter combination is found using a traversal search algorithm, which iterates through all parameter combinations that satisfy the constraints. Calculate the corresponding objective function value Return the combination of parameters that maximizes the objective function value as the optimal three-dimensional parameter combination. .

[0062] S4: The parallel preloading scheduler obtains the visual perception model to be switched in the next stage based on the predicted trajectory information and loads it into the idle computing core. S5: The variable parameter camera acquires the video stream according to the optimal (R,F) and the multi-model inference engine performs the selected model inference on the video stream on the heterogeneous edge platform, and allocates the post-processing and control logic to the most suitable hardware unit.

[0063] In the above process, when the camera resolution changes, the cross-resolution feature alignment network FPA-Net is used to map the features of different resolutions to maintain the continuity of the target identity ID; the specific mapping process executes the mapping logic described in Embodiment 1 above.

[0064] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. An adaptive multimodal unmanned aerial vehicle (UAV) target tracking system, characterized in that, When applied to drones, the drones are equipped with a heterogeneous edge computing platform that includes a multi-core NPU and a GPU. The system includes a distance perception module, an intelligent decision engine, and a multimodal perception execution layer. The distance sensing module acquires the UAV's position and the target waypoint's position in real time, and calculates the distance and relative speed between the UAV and the target; The intelligent decision engine is communicatively connected to the distance perception module, and the intelligent decision engine includes a perception model selector and a three-dimensional parameter co-optimizer. The perception model selector selects a visual perception model of corresponding complexity from a pre-set model library based on the real-time distance. The three-dimensional parameter co-optimizer outputs the optimal combination of three-dimensional parameters within a predefined three-dimensional parameter space, with the total computational load not exceeding the upper limit of the edge computing platform's computing power. The multimodal perception execution layer is communicatively connected to the intelligent decision engine, and the multimodal perception execution layer includes a variable parameter camera module and a multi-model inference engine; The variable parameter camera module receives three-dimensional parameters and acquires video stream data based on the optimal parameter combination; The multi-model inference engine drives the currently selected visual perception model, takes the received video stream data as input, outputs the target detection result, and performs post-processing on the target detection result to obtain the target position. Based on the target position, it executes control tasks to generate flight control commands and sends them to the flight control system.

2. The adaptive multimodal UAV target tracking system according to claim 1, characterized in that, The distance sensing module further includes: predicting the future trajectory of the target over a set time period based on Kalman filtering, and outputting the predicted trajectory information.

3. The adaptive multimodal UAV target tracking system according to claim 2, characterized in that, The intelligent decision engine also includes a parallel preloading scheduler, which receives the predicted trajectory information and loads the visual perception model to be switched to in the next stage into the idle computing core of the edge computing platform in advance.

4. The adaptive multimodal UAV target tracking system according to claim 2, characterized in that, The multimodal perception execution layer also includes a cross-resolution feature alignment network, which maps the feature spaces of different resolutions when the camera resolution is switched.

5. The adaptive multimodal UAV target tracking system according to claim 4, characterized in that, The cross-resolution feature alignment network adopts the FPA-Net network structure, which includes a Siamese dual branch. Each branch contains a spatial pyramid pooling layer and a channel-space dual attention module in sequence.

6. The adaptive multimodal UAV target tracking system according to claim 5, characterized in that, The feature spaces at different resolutions are mapped as follows: The video stream data collected during the resolution switching period is input into the alignment network model to extract image features; After the resolution switching is completed, the video stream data acquired is input into the alignment network model to extract image features; Image features extracted from images acquired at different resolutions are fused using an attention fusion module to obtain aligned, unified features.

7. An adaptive multimodal unmanned aerial vehicle (UAV) target tracking method, characterized in that, Based on the adaptive multimodal UAV target tracking system according to any one of claims 1-6, the method includes: S1: Real-time calculation of the distance and relative speed between the UAV and the target waypoint; S2: Select a visual perception model of corresponding complexity from the preset model library according to the distance; S3: Based on the model complexity corresponding to the selected visual perception model, in the three-dimensional parameter space composed of model complexity, camera resolution and frame rate, and with the upper limit of the total computing power of the drone's edge computing platform as a constraint, solve for the optimal combination of three-dimensional parameters. S4: Acquire video streams based on the optimal camera resolution and frame rate, perform inference on the selected model on the video stream, and output target detection results.

8. The adaptive multimodal UAV target tracking method according to claim 7, characterized in that, Step S1 also includes using Kalman filtering to predict the future trajectory of the target over a preset time period to obtain predicted trajectory information.

9. The adaptive multimodal UAV target tracking method according to claim 8, characterized in that, The method also includes: obtaining the visual perception model to be switched to in the next stage based on the predicted trajectory information, and loading it into an idle computing core.

10. The adaptive multimodal UAV target tracking method according to claim 7, characterized in that, The method also includes mapping features at different resolutions when the camera resolution changes.

Images