Flight trajectory tracking method and device and storage medium

Abstract

The invention discloses a flight trajectory tracking method and device and a storage medium. The flight trajectory tracking device obtains image data and point cloud data of a tracking target; wherein the image data comprises images acquired by a visible light camera and an infrared thermal imaging camera, and the point cloud data represents a point cloud acquired by a radar; preprocessing the image data and the point cloud data to obtain processed data; using the trained tracking model to analyze the flight trajectory of the tracking target in the processed data to obtain target trajectory data; wherein the trained tracking model is obtained by training the initial tracking model based on a preset loss function; the preset loss function comprises detection loss, tracking loss and shielding compensation loss; the detection loss is used for constraining the matching degree of the prediction frame and a real target, the tracking loss is used for constraining the continuity of trajectory prediction, and the shielding compensation loss is used for performing historical trajectory compensation under the condition that the target confidence is lower than a preset threshold value.

Description

Technical Field

[0001] This application relates to the field of airspace monitoring technology, specifically to a flight trajectory tracking method, device, and storage medium. Background Technology

[0002] Currently, when performing real-time trajectory tracking of small targets such as drones in low-altitude environments, most methods rely on data from a single mode. This approach is not only susceptible to interference from complex environments but also has limitations in detecting small targets, making it difficult to achieve high-precision and continuous trajectory tracking in complex low-altitude environments. Summary of the Invention

[0003] This application provides a flight trajectory tracking method, apparatus, and storage medium, which can achieve high-precision and continuous trajectory tracking in complex low-altitude environments.

[0004] To achieve the above objectives, this application provides the following technical solution: In a first aspect, embodiments of this application provide a flight trajectory tracking method, including: Acquire image data and point cloud data of the target being tracked; wherein, the image data includes images acquired by visible light cameras and infrared thermal imaging cameras, and the point cloud data represents point clouds acquired by radar; Preprocessing is performed on image data and point cloud data to obtain processed data; The flight trajectory of the tracked target in the processed data is analyzed using the trained tracking model to obtain the target trajectory data; The trained tracking model is obtained by training the initial tracking model based on a preset loss function. The preset loss function includes detection loss, tracking loss, and occlusion compensation loss. The detection loss is used to constrain the matching degree between the predicted bounding box and the real target, the tracking loss is used to constrain the continuity of the trajectory prediction, and the occlusion compensation loss is used to compensate for the historical trajectory when the target confidence is lower than a preset threshold.

[0005] In some embodiments of this application, the method further includes: Obtain the training dataset; wherein, the training dataset represents a dataset constructed based on data collected by visible light cameras, infrared thermal imaging cameras, and radar; The initial tracking model was used to perform object detection and trajectory prediction on the training dataset to obtain the first result; Based on the difference between the first result and the labeled data in the training dataset, the loss function value corresponding to the preset loss function is determined, and the gradient information is determined using the loss function value. The gradient information is then used to iteratively update the parameters of the initial tracking model to obtain the trained tracking model.

[0006] In some embodiments of this application, the loss function value corresponding to the preset loss function is determined based on the difference information between the first result and the labeled data in the training dataset, including: Based on the target detection bounding boxes in the first result and the true target bounding boxes in the labeled data, the detection loss value is determined; Based on the predicted trajectory in the first result and the real trajectory in the labeled data, determine the tracking loss value; Based on the confidence level of the target under test in the analysis results, it is determined whether the target under test is occluded, and the occlusion judgment result is obtained. Based on the occlusion judgment result, the occlusion compensation loss value is determined. The loss function value is determined based on the detection loss value, tracking loss value, occlusion compensation loss value, and weight information.

[0007] In some embodiments of this application, the detection loss value is determined based on the target detection bounding box in the first result and the ground truth target bounding box in the labeled data, including: The area of ​​the target to be measured is determined based on the target detection box; The detection loss value is determined based on the improved cross-union ratio, target weight, area of ​​the target to be tested, and the image area corresponding to the target to be tested. Specifically, if the area of ​​the target to be measured is smaller than the preset area, the target weight is set to the first value; otherwise, the target weight is set to the second value.

[0008] In some embodiments of this application, the tracking loss value is determined based on the predicted trajectory in the first result and the true trajectory in the labeled data, including: Based on the first result, determine the velocity difference between adjacent moments of the target under test, and determine the norm of the velocity difference; The tracking loss value is determined based on the norm of the position error between the predicted trajectory and the actual trajectory, as well as the norm of the velocity difference.

[0009] In some embodiments of this application, the presence or absence of occlusion of the target is determined based on the confidence level corresponding to the target in the analysis results, resulting in an occlusion judgment result. An occlusion compensation loss value is then determined based on the occlusion judgment result, including: If the confidence level of the target under test is less than the preset confidence level in the analysis results, the occlusion judgment result is determined to be that the target under test is occluded, and the occlusion compensation loss value is determined to be the third value; otherwise, the occlusion judgment result is determined to be that the target under test is not occluded, and the occlusion compensation loss value is determined to be 0. The third value is determined by fitting the first position based on historical trajectories and the current predicted position.

[0010] In some embodiments of this application, the trained tracking model includes a first preset network and a second preset network; the processed data includes processed image data and processed point cloud data; The trained tracking model is used to analyze the flight trajectory of the tracked target in the processed data to obtain target trajectory data, including: The image features of the processed image data are determined by a first preset network; wherein, the image features include the texture features and bounding boxes of the tracking target; The point cloud features of the processed point cloud data are determined by a second preset network; wherein, the point cloud features include the three-dimensional position features and velocity features of the tracking target; Spatial alignment is performed on image features and point cloud features to obtain aligned image features and aligned point cloud features; The aligned image features and aligned point cloud features are fused based on preset weights to obtain fused feature data; wherein, the preset weights include a first preset weight corresponding to the visible light features in the aligned image features, a second preset weight corresponding to the infrared features in the aligned image features, and a third preset weight corresponding to the aligned point cloud features. Target trajectory data is determined based on the fused feature data.

[0011] In some embodiments of this application, determining target trajectory data based on fused feature data includes: Target detection is performed based on the fused feature data to obtain target detection results, which include target confidence scores. If the target confidence level is less than the first confidence level but greater than or equal to the second confidence level, the third preset weight is adjusted, and the feature fusion process is executed again based on the adjusted third preset weight to determine the target trajectory data based on the updated fused features. If the target confidence level is less than the second confidence level, the trajectory of the tracked target is predicted to determine the target trajectory data; When the target confidence level is greater than the third confidence level, the target trajectory data is determined based on the target detection results; wherein the first confidence level is greater than the second confidence level and less than the third confidence level.

[0012] Secondly, embodiments of this application provide a flight trajectory tracking device, including a visible light camera, an infrared thermal imaging camera, a radar, a processor, and a memory storing processor-executable instructions; when the instructions are executed by the processor, the above-mentioned flight trajectory tracking method is implemented.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned flight trajectory tracking method. Attached Figure Description

[0014] To more intuitively illustrate the prior art and this application, several exemplary figures are provided below. It should be understood that the specific shapes and structures shown in the figures should not generally be regarded as limiting conditions for implementing this application; for example, based on the technical concept disclosed in this application and the exemplary figures, those skilled in the art are able to easily make conventional adjustments or further optimizations to the addition / reduction / classification, specific shapes, positional relationships, connection methods, size ratios, etc. of certain units (components).

[0015] Figure 1 Schematic diagram of the implementation process of the flight trajectory tracking method provided in the embodiments of this application Figure 1 ; Figure 2 Schematic diagram of the implementation process of the flight trajectory tracking method provided in the embodiments of this application Figure 2 . Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Any combination of different embodiments is possible.

[0017] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0018] To address the challenges of real-time trajectory tracking of small targets in low-altitude environments, this application provides a flight trajectory tracking method, apparatus, and storage medium. The flight trajectory tracking apparatus acquires image data and point cloud data of the target being tracked. The image data includes images captured by a visible light camera and an infrared thermal imaging camera, while the point cloud data represents point clouds acquired by radar. The image data and point cloud data are preprocessed to obtain processed data. A trained tracking model is used to analyze the flight trajectory of the target in the processed data to obtain target trajectory data. The trained tracking model is obtained by training an initial tracking model based on a preset loss function. The preset loss function includes detection loss, tracking loss, and occlusion compensation loss. The detection loss constrains the matching degree between the predicted bounding box and the real target, the tracking loss constrains the continuity of trajectory prediction, and the occlusion compensation loss compensates for historical trajectories when the target confidence level is below a preset threshold. Based on this solution, interference in complex low-altitude environments can be effectively overcome, improving the detection reliability of small targets and enabling high-precision, continuous trajectory tracking in complex low-altitude environments.

[0019] This application provides a flight trajectory tracking method, such as... Figure 1 As shown, the flight trajectory tracking method of the flight trajectory tracking device may include the following steps: Step 101: Acquire image data and point cloud data of the target being tracked; wherein, the image data includes images acquired by a visible light camera and an infrared thermal imaging camera, and the point cloud data represents the point cloud acquired by radar.

[0020] In embodiments of this application, the flight trajectory tracking device can acquire image data and point cloud data of the tracked target; wherein, the image data includes images acquired by a visible light camera and an infrared thermal imaging camera, and the point cloud data represents point clouds acquired by radar.

[0021] In the embodiments of this application, a visible light camera can be used to capture target texture features (adapted to normal daytime lighting scenes), and an infrared thermal imaging camera can be used to capture target thermal features (adapted to nighttime and low-light scenes). The two complement each other to overcome the limitations of a single lighting condition.

[0022] In some embodiments of this application, three visible light cameras with a frame rate of 30fps and a field of view of 60° can be set; two infrared thermal imaging cameras with a resolution of 1280×720 and a temperature measurement range of -20~150℃ can be set; and one 24GHz millimeter-wave radar with a detection range of 0~3km and a ranging accuracy of ±0.5m can be set to provide three-dimensional position and velocity information, unaffected by illumination or occlusion, and provide spatial dimension support for trajectory tracking.

[0023] Step 102: Preprocess the image data and point cloud data to obtain the processed data.

[0024] In the embodiments of this application, after acquiring image data and point cloud data of the tracked target, the flight trajectory tracking device can preprocess the image data and point cloud data to obtain processed data.

[0025] In embodiments of this application, preprocessing may include timestamp alignment and coordinate system mapping; wherein, timestamp alignment may be timestamp alignment of multi-source image data and point cloud data, and coordinate system mapping may be mapping to a unified WGS84 geographic coordinate system.

[0026] Step 103: Analyze the flight trajectory of the tracked target in the processed data using the trained tracking model to obtain target trajectory data; wherein, the trained tracking model is obtained by training the initial tracking model based on a preset loss function; the preset loss function includes detection loss, tracking loss and occlusion compensation loss; the detection loss is used to constrain the matching degree between the predicted box and the real target, the tracking loss is used to constrain the continuity of trajectory prediction, and the occlusion compensation loss is used to compensate for historical trajectories when the target confidence is lower than a preset threshold.

[0027] In the embodiments of this application, after preprocessing the image data and point cloud data to obtain the processed data, the flight trajectory tracking device can use the trained tracking model to analyze the flight trajectory of the tracked target in the processed data to obtain the target trajectory data.

[0028] In the embodiments of this application, the trained tracking model refers to the model used for trajectory analysis. It can be a model that has been pre-trained on a large-scale dataset and then fine-tuned in combination with the characteristics of low-altitude small targets, and has the ability to identify and track low-altitude small targets.

[0029] For example, the tracking model was pre-trained based on the COCO and VisDrone datasets and then fine-tuned for 50 rounds using 100,000 real-world airport images and 50,000 radar point cloud frames.

[0030] In the embodiments of this application, the detection loss, tracking loss, and occlusion compensation loss are designed for target recognition accuracy, trajectory continuity, and occlusion scene robustness, respectively, which solves the problem that traditional loss functions have poor adaptability to small targets and occluded targets.

[0031] In the embodiments of this application, through the design of multimodal data fusion and a dedicated loss function, three main technical effects are achieved. First, multimodal data acquisition breaks through the environmental limitations of a single sensing device. The combination of visible light, infrared, and radar can cover complex scenarios such as strong daylight, low light at night, and occlusion, solving the problem of poor environmental adaptability of traditional single-modal methods. Second, targeted preprocessing operations can unify data formats, optimize data quality, and reduce the interference of noise and coordinate deviations on model inference, providing a reliable data foundation for accurate trajectory analysis. Third, the preset loss function can comprehensively cover the three core requirements of accurate recognition, stable trajectory, and resistance to occlusion. Compared with traditional single loss functions, it has stronger constraints on the recognition of small targets, better trajectory continuity, and higher robustness to occlusion scenarios.

[0032] In some embodiments of this application, the trained tracking model may include a first preset network and a second preset network.

[0033] In some embodiments of this application, the processed data includes processed image data and processed point cloud data.

[0034] In some embodiments of this application, when the flight trajectory tracking device analyzes the flight trajectory of the tracked target in the processed data using a trained tracking model to obtain target trajectory data, it can determine the image features of the processed image data through a first preset network; determine the point cloud features of the processed point cloud data through a second preset network; perform spatial alignment processing on the image features and point cloud features to obtain aligned image features and aligned point cloud features; fuse the aligned image features and aligned point cloud features based on preset weights to obtain fused feature data; and determine the target trajectory data based on the fused feature data.

[0035] In the embodiments of this application, the first preset network refers to a deep learning network used to extract image features; the first preset network can be an improved YOLOv8 based on transfer learning; after training and fine-tuning, this application enables the first preset network to extract small target texture features and bounding boxes.

[0036] In the embodiments of this application, the first preset network may be the removal of the last 3 convolutional layers of YOLOv8 and the addition of 2 3×3 convolutional layers (256 channels).

[0037] In the embodiments of this application, the second preset network refers to a deep learning network used to extract point cloud features; the second preset network may be an improved PointNet. This application enables the second preset network to extract the three-dimensional position and velocity features of the tracking target from the radar point cloud through operations such as voxelization and attention mechanism.

[0038] In the embodiments of this application, spatial alignment processing refers to converting the two-dimensional coordinates of image features into three-dimensional geographic coordinates through the camera intrinsic parameter matrix, and matching them with the three-dimensional coordinates of point cloud features to ensure the consistency of multimodal features in spatial dimensions, thus laying the foundation for feature fusion.

[0039] In embodiments of this application, image features include texture features and bounding boxes of the tracking target.

[0040] In embodiments of this application, point cloud features include the three-dimensional position and velocity features of the tracking target.

[0041] In the embodiments of this application, the preset weights include a first preset weight corresponding to the visible light features in the aligned image features, a second preset weight corresponding to the infrared features in the aligned image features, and a third preset weight corresponding to the point cloud features in the aligned image features.

[0042] For example, the feature fusion method can be expressed as the following formula: (1); in, Indicates the characteristics of visible light. Indicates infrared characteristics, Representing point cloud features; Indicates the first preset weight. This indicates the second preset weight. This indicates the third preset weight.

[0043] In some embodiments of this application, the first preset weight, the second preset weight, and the third preset weight can be dynamically adjusted according to the environment; for example, in a nighttime environment, It can be 0.2. It can be 0.5. It can be 0.3; in strong light environments, It can be 0.5. It can be 0.2. It can be 0.3.

[0044] In the embodiments of this application, the complementary advantages of multimodal data can be maximized through the complete process of multimodal feature extraction, alignment, and fusion. The first preset network (improved YOLOv8) and the second preset network (improved PointNet) are optimized for image data and point cloud data, respectively, and can accurately extract the core features of their respective modalities. Compared with general networks, they have stronger feature capture capabilities for low-altitude small targets. Spatial alignment processing solves the problem of spatial coordinate differences of multimodal features, ensures the effectiveness of feature fusion, avoids the distortion of fused features caused by coordinate deviation, and fully leverages the complementarity of multimodal data. The adaptive preset weights support dynamic adjustment according to the environment, strengthening the infrared feature weights at night, strengthening the visible light feature weights under strong light, and strengthening the radar feature weights in complex scenes, so that the model can obtain the optimal fused features in different environments. Compared with fixed weight fusion, the environmental adaptability is significantly improved.

[0045] In some embodiments of this application, when determining target trajectory data based on fused feature data, the flight trajectory tracking device can perform target detection based on the fused feature data to obtain a target detection result; wherein, the target detection result includes a target confidence level; if the target confidence level is less than a first confidence level but greater than or equal to a second confidence level, a third preset weight is adjusted, and the feature fusion process is executed again based on the adjusted third preset weight to determine the target trajectory data based on the updated fused features; if the target confidence level is less than the second confidence level, trajectory prediction is performed on the tracked target to determine the target trajectory data; if the target confidence level is greater than the third confidence level, the target trajectory data is determined based on the target detection result; wherein, the first confidence level is greater than the second confidence level and less than the third confidence level.

[0046] In the embodiments of this application, adjusting the third preset weight means increasing the third preset weight; for example, adjusting the third preset weight from 0.3 to 0.4.

[0047] In the embodiments of this application, the target confidence level can reflect the degree of trust the model has in the detection results and is the core basis for judging the target state (normal, partially occluded, completely occluded).

[0048] In the embodiments of this application, the specific values ​​of the first confidence level, the second confidence level, and the third confidence level are not limited; for example, the first confidence level can be 0.5, the second confidence level can be 0.3, and the third confidence level can be 0.6.

[0049] In some embodiments of this application, such as Figure 2 As shown, the flight trajectory tracking method may also include the following steps: Step 201: Obtain the training dataset; wherein, the training dataset represents a dataset constructed based on data collected by visible light cameras, infrared thermal imaging cameras, and radar.

[0050] In embodiments of this application, the flight trajectory tracking device can acquire a training dataset; wherein, the training dataset represents a dataset constructed based on data collected by a visible light camera, an infrared thermal imaging camera, and radar.

[0051] Understandably, the data source of the training dataset is consistent with the data type of the actual flight trajectory tracking scenario, making the model training more targeted.

[0052] Step 202: Use the initial tracking model to perform object detection and trajectory prediction on the training dataset to obtain the first result.

[0053] In the embodiments of this application, after acquiring the training dataset, the flight trajectory tracking device can use the initial tracking model to perform target detection and trajectory prediction on the training dataset to obtain a first result.

[0054] In the embodiments of this application, the initial tracking model refers to the basic model that has not undergone targeted fine-tuning, that is, the model pre-trained based on a large-scale general dataset (COCO, VisDrone), which has general target recognition and tracking capabilities, but is not adapted to the characteristics of low-altitude small targets.

[0055] In the embodiments of this application, the first result refers to the preliminary result output by the initial tracking model after performing object detection and trajectory prediction on the training dataset, including information such as prediction boxes, prediction trajectories, and confidence scores, which is the basis for calculating the loss function value.

[0056] Step 203: Based on the difference information between the first result and the labeled data in the training dataset, determine the loss function value corresponding to the preset loss function, and use the loss function value to determine the gradient information, so as to use the gradient information to iteratively update the parameters of the initial tracking model and obtain the trained tracking model.

[0057] In the embodiments of this application, after the flight trajectory tracking device performs target detection and trajectory prediction on the training dataset using the initial tracking model and obtains the first result, it can determine the loss function value corresponding to the preset loss function based on the difference information between the first result and the labeled data in the training dataset, and use the loss function value to determine the gradient information, so as to use the gradient information to iteratively update the parameters of the initial tracking model and obtain the trained tracking model.

[0058] In the embodiments of this application, the difference information is the quantitative difference between the first result and the labeled data (real target boxes, real trajectories, etc.) in the training dataset, which provides a basis for calculating the loss function value.

[0059] In the embodiments of this application, gradient information refers to the direction and magnitude of model parameter updates calculated based on the loss function value, which is the core basis for iterative optimization of model parameters.

[0060] In the embodiments of this application, iterative update refers to repeatedly adjusting the parameters of the initial tracking model based on gradient information until the model performance reaches the optimal level; for example, iterating for 50 rounds, with an initial learning rate of 0.001 and a decay of 10% every 10 rounds.

[0061] In the embodiments of this application, the multimodal training dataset is fully matched with the data type of the actual tracking scenario, avoiding the disconnect between model training and actual application, and ensuring that the features learned by the model can be directly adapted to the multimodal characteristics of low-altitude small targets; the loss function value is calculated based on the difference information and the parameters are iteratively updated to achieve precise optimization of the model parameters, so that the model gradually adapts to the size, speed and flight characteristics of low-altitude small targets. Compared with the initial model without fine-tuning, the accuracy of small target recognition is significantly improved; during the iterative update process, strategies such as learning rate decay (decay by 10% every 10 rounds) are used to avoid model overfitting and slow convergence, ensuring that the model reaches optimal performance after 50 rounds of iteration, providing core support for subsequent accurate tracking, and shortening the adaptation cycle after model deployment.

[0062] In some embodiments of this application, when the flight trajectory tracking device determines the loss function value corresponding to the preset loss function based on the difference information between the first result and the labeled data in the training dataset, it can determine the detection loss value based on the target detection box in the first result and the real target box in the labeled data; determine the tracking loss value based on the predicted trajectory in the first result and the real trajectory in the labeled data; determine whether the target to be tested is occluded based on the confidence level corresponding to the target to be tested in the analysis result, obtain the occlusion judgment result, and determine the occlusion compensation loss value based on the occlusion judgment result; and determine the loss function value based on the detection loss value, the tracking loss value, the occlusion compensation loss value, and the weight information.

[0063] In the embodiments of this application, the weight information refers to the weight coefficients assigned to the detection loss value, the tracking loss value, and the occlusion compensation loss value. The weight information includes a first weight coefficient assigned to the detection loss value, a second weight coefficient assigned to the tracking loss value, and a third weight coefficient assigned to the occlusion compensation loss value.

[0064] For example, the loss function value can be expressed as the following formula: (2); in, To detect the loss value, To track the loss value, To compensate for the loss value due to occlusion, As the first weighting coefficient, This is the second weighting coefficient. This is the third weighting coefficient.

[0065] In the embodiments of this application, the detection loss value, tracking loss value, and occlusion compensation loss value correspond to the three core requirements of "accurate positioning", "stable trajectory", and "anti-occlusion". The separate calculation ensures that the error of each dimension can be accurately captured, avoiding the problem of some requirements being ignored due to the "one-size-fits-all" approach of the traditional single loss function. The occlusion judgment result is dynamically determined based on confidence, making the calculation of the occlusion compensation loss value more targeted. Compensation is only activated when the target is occluded, which ensures the robustness of the occluded scene without affecting the model training efficiency in the normal scene. The introduction of weight information can flexibly adjust the importance of different losses.

[0066] In some embodiments of this application, when the flight trajectory tracking device determines the detection loss value based on the target detection box in the first result and the real target box in the annotation data, it can determine the area of ​​the target to be tested based on the target detection box; and determine the detection loss value based on the improved intersection-union ratio, target weight, area of ​​the target to be tested, and the image area corresponding to the target to be tested; wherein, if the area of ​​the target to be tested is less than a preset area, the target weight is a first value; otherwise, the target weight is a second value.

[0067] In the embodiments of this application, the area of ​​the target to be tested refers to the target pixel area calculated based on the target detection box in the first result, which is the core basis for distinguishing between small targets and ordinary targets; for example, 32×32 pixels can be used as the boundary, and when the area of ​​the target to be tested is less than 32×32 pixels, it can be considered as a small target.

[0068] In the embodiments of this application, the improved intersection-over-union ratio refers to the intersection-over-union ratio (CIoU) optimized for small targets. Compared with the traditional IoU, it can more accurately quantify the differences in position, size, and shape between the predicted bounding box and the ground truth bounding box, thus adapting to the localization needs of small targets.

[0069] In the embodiments of this application, the target weight refers to the loss weight allocated according to the area of ​​the target to be tested, which is used to strengthen the loss constraint of small targets and solve the problem of insufficient weight of small targets due to low pixel ratio during training.

[0070] In the embodiments of this application, the first value refers to the target weight value corresponding to the small target, which can be 2; by increasing the target weight, the penalty for the detection error of the small target can be strengthened; the second value refers to the target weight value corresponding to the ordinary target, which can be 1, which is the conventional loss penalty weight.

[0071] In the embodiments of this application, the image area corresponding to the target to be tested refers to the overall pixel area of ​​the acquired image (e.g., 3840×2160 pixels for a 4K image), which can be used to quantify the proportion of small targets in the image and help strengthen the loss constraint.

[0072] In the embodiments of this application, the preset area refers to the area threshold that distinguishes small targets from ordinary targets. For example, the preset area can be 32×32 pixels, which is the criterion for determining the target weight allocation.

[0073] For example, the determination of the detection loss value can be expressed as the following formula: (3); in, This represents the improved intersection-union ratio. Indicates the target weight. This represents the area of ​​the target to be measured. This represents the image area corresponding to the target being measured; when When the pixel size is less than 32×32, It is 2 (the first value), otherwise It is 1 (the second value).

[0074] In the embodiments of this application, the problem of low accuracy in small target recognition can be solved by specifically optimizing the calculation of the detection loss value; the improved CIoU (Intersection over Union) compared to traditional IoU can more comprehensively quantify the difference between the predicted bounding box and the ground truth bounding box, avoiding the defect of traditional IoU being insensitive to small target localization errors, and improving the localization accuracy of small targets; the target weight is dynamically allocated based on the target area, and the weight of small targets is increased to 2. By strengthening the loss penalty, the model pays more attention to the features of small targets during training, solving the problem of missed detection and false detection caused by the low pixel proportion and indistinct features of small targets; the proportion of small targets is quantified by image area, through... Further strengthen the loss constraint on extremely small targets to make the model more adaptable to small targets of different sizes.

[0075] In some embodiments of this application, when the flight trajectory tracking device determines the tracking loss value based on the predicted trajectory in the first result and the real trajectory in the labeled data, it can determine the velocity difference between adjacent moments of the target under test based on the first result and determine the norm of the velocity difference; and determine the tracking loss value based on the norm of the position error between the predicted trajectory and the real trajectory, and the norm of the velocity difference.

[0076] In the embodiments of this application, velocity difference refers to the difference between the velocities of the target at adjacent moments.

[0077] In the embodiments of this application, the norm of the velocity difference is a quantized value obtained by calculating the L2 norm of the velocity difference, which is used to convert the multidimensional velocity difference into a single value, so as to facilitate loss quantization.

[0078] In the embodiments of this application, the norm of the position error is a quantized value obtained by calculating the L2 norm of the position difference between the predicted position and the actual position, and is used to quantify the spatial error of target positioning.

[0079] For example, the tracking loss value can be calculated using the following formula: (4); in, , , Indicates the actual position at time t. , , Indicates the predicted position at time t; Represents the velocity at time t. This represents the velocity at time t-1.

[0080] In the embodiments of this application, the tracking loss value is calculated by fusing position error and velocity difference, which can effectively ensure trajectory continuity. The norm of position error directly quantifies the spatial deviation between the predicted trajectory and the actual trajectory, ensuring the positioning accuracy of the trajectory. The norm of velocity difference is associated with the motion state of adjacent frames, constraining abrupt changes in velocity, avoiding discontinuities such as trajectory jumps and breaks, and solving the problem of poor trajectory smoothness caused by traditional tracking loss focusing only on position and ignoring velocity. The velocity difference norm is weighted with a weight of 0.1 to balance position accuracy and velocity smoothness, ensuring trajectory accuracy while avoiding trajectory lag caused by excessive constraint on velocity.

[0081] In some embodiments of this application, when the flight trajectory tracking device determines whether the target to be tested is obstructed based on the confidence level corresponding to the target to be tested in the analysis results, obtains the obstruction judgment result, and determines the obstruction compensation loss value based on the obstruction judgment result, it can determine that the target to be tested is obstructed if the confidence level corresponding to the target to be tested in the analysis results is less than the preset confidence level, and determine that the obstruction compensation loss value is a third value; otherwise, it determines that the obstruction judgment result is that the target to be tested is not obstructed, and determines that the obstruction compensation loss value is 0.

[0082] In the embodiments of this application, the preset confidence level refers to the confidence threshold value used to determine whether the target is occluded; the specific value of the preset confidence level is not limited in this application, for example, the preset confidence level can be 0.5.

[0083] In the embodiments of this application, the third value refers to the value of the occlusion compensation loss when the target is occluded; the third value is determined based on the first position determined by fitting the historical trajectory and the current predicted position.

[0084] In some embodiments of this application, the third value can be determined based on the error quantization value between the current predicted position and the first position at the current moment determined by fitting the trajectory of the previous 3 frames (historical trajectory).

[0085] For example, the occlusion compensation loss value can be determined by the following formula: (5); When there is no obstruction, the occlusion compensation loss value The value is 0, and the occlusion compensation loss value is the third value when occlusion exists. , This indicates the predicted position (first position) at frame t, determined by the trajectory fitting of the first 3 frames. This indicates the current predicted position.

[0086] In the embodiments of this application, the robustness of the model in occlusion scenarios is significantly improved by dynamically determining the occlusion state and calculating the compensation loss value. Using confidence as the occlusion determination criterion, it can quickly and accurately identify whether the target is in an occluded state, avoiding the subjectivity and lag of manual judgment and adapting to real-time tracking requirements. In the occluded state, a third value is introduced to correct the current prediction result by using the continuity characteristics of historical trajectories, effectively reducing tracking drift caused by occlusion and solving the problem that traditional models are prone to losing targets and interrupting trajectories when occluded. In the unoccluded state, the occlusion compensation loss value is 0 to avoid interference of invalid compensation with model training and ensure training efficiency in normal scenarios.

[0087] This application provides a flight trajectory tracking method. The flight trajectory tracking device acquires image data and point cloud data of the target being tracked. The image data includes images acquired by a visible light camera and an infrared thermal imaging camera, and the point cloud data represents point clouds acquired by radar. The image data and point cloud data are preprocessed to obtain processed data. A trained tracking model is used to analyze the flight trajectory of the target in the processed data to obtain target trajectory data. The trained tracking model is obtained by training an initial tracking model based on a preset loss function. The preset loss function includes detection loss, tracking loss, and occlusion compensation loss. The detection loss constrains the matching degree between the predicted bounding box and the real target, the tracking loss constrains the continuity of trajectory prediction, and the occlusion compensation loss compensates for historical trajectories when the target confidence level is below a preset threshold. Therefore, this application, through multimodal data acquisition, can overcome the environmental limitations of a single sensing device. The combination of visible light, infrared, and radar covers complex scenarios such as strong daylight, low nightlight, and occlusion, solving the problem of poor environmental adaptability of traditional single-modal methods. Targeted preprocessing operations unify data formats, optimize data quality, and reduce the interference of noise and coordinate deviations on model inference, providing a reliable data foundation for accurate trajectory analysis. The design of the preset loss function can comprehensively cover the three core requirements of accurate recognition, stable trajectory, and resistance to occlusion. Compared with traditional single loss functions, it has stronger constraints on the recognition of small targets, better trajectory continuity, and higher robustness to occlusion scenarios. Thus, it can achieve high-precision and continuous trajectory tracking in complex low-altitude environments.

[0088] Based on the above embodiments, in another embodiment of this application, an exemplary scheme for tracking the flight trajectories of small low-altitude targets is proposed. This involves constructing a deep learning network that fuses multimodal data (visible light images, infrared images, and radar point clouds), and fine-tuning it for the characteristics of small low-altitude targets (drones, birds, etc.) using transfer learning after pre-training on a large-scale dataset. A dedicated loss function is designed to improve the model's ability to identify and continuously track small and occluded targets. Application scenarios include low-altitude airspace monitoring systems, which can provide accurate target identification and tracking information, detect illegal intrusions and abnormal flight behavior, and ensure airspace safety.

[0089] For example, using a monitoring task in a 5km × 5km low-altitude area (0~1000m altitude) around an airport as a background, the effectiveness of the low-altitude flight trajectory tracking method based on small targets is verified. This area requires real-time tracking of two typical small targets: small drones (0.3m × 0.5m in size, flight speed 5~20m / s, possibly carrying camouflage or low-reflection coatings); and migratory birds (wingspan 0.2-0.8m, flight speed 3~15m / s, prone to occlusion when flying in flocks). The scenario includes complex interference factors: strong daytime light (illuminance 10000~50000 lux), low nighttime light (<50 lux), building / tree occlusion (occlusion rate up to 60%), and multiple target cross-flight (up to 10 targets appearing simultaneously). The monitoring targets are: small target identification accuracy ≥90%, trajectory tracking error ≤5m, and continuous tracking interruption rate <5% under occlusion conditions.

[0090] Sensing devices for multimodal data acquisition: 3 visible light cameras (4K resolution, 30fps frame rate, 60° field of view), 2 infrared thermal imaging cameras (1280×720 resolution, temperature range -20~150℃, suitable for night / low light environments), and 1 millimeter-wave radar (24GHz, detection range 0~3km, ranging accuracy ±0.5m, point cloud sampling rate 10Hz); Processing unit: adopts NVIDIA Jetson AGX Orin edge computing platform (8-core CPU + 2048-core GPU), supporting real-time multimodal data fusion and model inference (latency ≤100ms).

[0091] Multi-source data synchronization: Visible light images, infrared images, and radar point clouds are mapped to a unified geographic coordinate system (WGS84) through timestamp alignment (error <10ms). Small target sample augmentation: scaling (0.5~2 times), rotating (±30°), and adding noise (Gaussian noise) to the collected raw data. =0.02), occlusion simulation (randomly occluding 20%~50% of the region), expanding the training sample to 100,000 images.

[0092] The overall network architecture adopts a three-stage architecture of "feature extraction - cross-modal fusion - target tracking"; among which, single-modal feature extraction includes visible light / infrared branch, radar point cloud branch and radar point cloud branch.

[0093] Visible / Infrared Branch: An improved YOLOv8 based on transfer learning, using a model pre-trained on the COCO and VisDrone datasets as a foundation, removes the last 3 convolutional layers and adds 2 new 3×3 convolutional layers (256 channels) to focus on extracting texture / thermal features of small targets.

[0094] Radar point cloud branch: Improve PointNet, voxelize the radar point cloud (512 points per frame) into a 0.5m×0.5m×0.5m grid, highlight the target reflection points (points with reflection intensity > 10dB) through an attention mechanism, and output 3D position features (x,y,z) and velocity features (vx, vy, vz).

[0095] Furthermore, the cross-modal feature fusion module includes: spatial alignment, which converts the 2D coordinates (u,v) of the visible light / infrared image into 3D geographic coordinates (X,Y,Z) through the camera intrinsic parameter matrix and matches them with the radar point cloud coordinates (error ≤0.3m); feature fusion, which adopts an adaptive weight fusion strategy and the calculation formula is as shown in the aforementioned formula (1).

[0096] Furthermore, the tracking head includes target detection and trajectory prediction; the output of target detection includes target bounding boxes (visible light / infrared) and 3D bounding boxes (radar), predicts the target category (drone / bird) and confidence level; trajectory prediction can be implemented based on an LSTM network, taking the position of the past 5 frames (Xt-4,Yt-4,Zt-4)~(Xt,Yt,Zt) as input, and predicting the position of the next 2 frames (Xt+1,Yt+1,Zt+1) and (Xt+2,Yt+2,Zt+2).

[0097] Furthermore, in order to improve the tracking capability of small targets and occluded targets, a multi-task loss function is designed, and its calculation method is shown in the aforementioned formula (2); among which, the detection loss is calculated as shown in the aforementioned formula (3), the tracking loss is calculated as shown in the aforementioned formula (4), and the occlusion compensation loss is calculated as shown in the aforementioned formula (5).

[0098] For example, during the offline training phase, the network was fine-tuned using an airport real-world dataset (containing 100,000 images and 50,000 radar point clouds) for 50 iterations. The initial learning rate was 0.001, and it decreased by 10% every 10 iterations. On the validation set, the core metric for small object detection, mAP@0.5, was 89%, and the trajectory prediction error was <3m. The optimal model was then saved.

[0099] For example, the online tracking process includes: real-time data acquisition, synchronously acquiring visible light / infrared images and radar point clouds at 30Hz, with each frame containing approximately 20MB of data; multimodal fusion: preprocessing the input data (denoising, coordinate transformation), and outputting fused features through the fusion module; target detection and trajectory update: the model infers and outputs the target position, category, and confidence level, combines the Hungarian algorithm to perform target association (ID matching), and updates the trajectory database.

[0100] In some embodiments of this application, an occlusion handling strategy is designed for situations where the target is occluded: if the target is partially occluded (confidence level 0.3~0.5): increase the weight corresponding to the radar feature (third preset weight). =0.4. If the target is completely occluded (confidence < 0.3), start the trajectory prediction mode, predict the position based on the LSTM output, and mark "suspected occlusion" at the same time. After the occlusion is removed (confidence > 0.6), realign the trajectory.

[0101] For example, the effectiveness of the flight trajectory tracking method proposed in this application is verified by the following test scenario: continuous 24-hour monitoring, including 100 small drones (including 5 camouflaged drones), 200 birds (including 30 instances of swarm occlusion), and 10 instances of low-light nighttime environments (<50 lux). Key processes include: Scenario 1 (daytime small target tracking): A 0.3m × 0.3m drone flies at an altitude of 300m, partially occluded by trees (occlusion rate 30%). The target in the visible light image is only 20 × 20 pixels (confidence level 0.45). The system automatically increases the weights of infrared (significant thermal features) and radar, and through feature fusion, the confidence level is increased to 0.72, with the trajectory error stabilizing at 2.3m; Scenario 2 (nighttime swarm occlusion): 10 birds (wingspan 0.5m) fly in a flock (occlusion rate 50%). Visible light is ineffective at night (confidence level 0.2). The system relies on infrared (thermal imaging to distinguish individuals) and radar point clouds (3D position separation), through... The loss constraint is that the trajectory interruption rate is 0% during the 3-second occlusion period, and the ID matching accuracy is 90% after recovery.

[0102] For example, Table 1 below shows a comparison of the implementation effects of the flight trajectory tracking method of this application with related traditional tracking methods:

[0103] In summary, the embodiments of this application, through the design of a multimodal fusion network and a dedicated loss function, achieve accurate tracking of small drones and birds in the complex low-altitude environment of airports. Compared with traditional methods, the accuracy of small target recognition is improved by 28%, the stability of occlusion tracking is significantly enhanced, and the trajectory error is reduced by 62%, verifying the effectiveness of the flight trajectory tracking method of this application in low-altitude airspace monitoring, and effectively supporting illegal intrusion warning and airspace security.

[0104] Based on the above embodiments, another embodiment of this application provides a flight trajectory tracking device, which may include a visible light camera, an infrared thermal imaging camera, and radar.

[0105] In embodiments of this application, the flight trajectory tracking device may further include a processor and a memory storing processor-executable instructions; further, the video description device may further include a communication interface and a bus for connecting the processor, the memory and the communication interface.

[0106] In the embodiments of this application, the processor can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other types, and the embodiments of this application do not specifically limit this. A memory can be connected to the processor, wherein the memory is used to store executable program code, which includes computer operation instructions. The memory may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0107] In embodiments of this application, a bus is used to connect a communication interface, a processor, and a memory to enable communication between these devices.

[0108] In embodiments of this application, a memory is used to store instructions and data.

[0109] In practical applications, the aforementioned memory can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, providing instructions and data to the processor.

[0110] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0111] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment.

[0112] Specifically, the program instructions corresponding to a video description method in this embodiment can be stored on a storage medium such as an optical disc or a hard disk. When the program instructions corresponding to a video description method in the storage medium are read or executed by a video description device, the following steps are included: Acquire image data and point cloud data of the target being tracked; wherein, the image data includes images acquired by visible light cameras and infrared thermal imaging cameras, and the point cloud data represents point clouds acquired by radar; Preprocessing is performed on image data and point cloud data to obtain processed data; The flight trajectory of the tracked target in the processed data is analyzed using the trained tracking model to obtain the target trajectory data; The trained tracking model is obtained by training the initial tracking model based on a preset loss function. The preset loss function includes detection loss, tracking loss, and occlusion compensation loss. The detection loss is used to constrain the matching degree between the predicted bounding box and the real target, the tracking loss is used to constrain the continuity of the trajectory prediction, and the occlusion compensation loss is used to compensate for the historical trajectory when the target confidence is lower than a preset threshold.

[0113] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0114] This application is described with reference to schematic diagrams and / or block diagrams illustrating the implementation of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic diagrams and / or block diagrams, and combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the schematic diagrams and / or one or more blocks of the block diagrams.

[0115] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0117] The above embodiments are merely preferred embodiments provided to fully illustrate this application, and the scope of protection of this application is not limited thereto. Equivalent substitutions or modifications made by those skilled in the art based on this application are all within the scope of protection of this application.

Claims

1. A flight trajectory tracking method, characterized by, The method comprises: acquiring image data and point cloud data of a tracking target; wherein the image data comprises images collected by a visible light camera and an infrared thermal imaging camera, and the point cloud data represents a point cloud acquired by a radar; preprocessing the image data and the point cloud data to obtain processed data; analyzing a flight trajectory of the tracking target in the processed data by using a trained tracking model to obtain target trajectory data; wherein the trained tracking model is obtained by training an initial tracking model based on a preset loss function; the preset loss function comprises a detection loss, a tracking loss, and an occlusion compensation loss; the detection loss is used to constrain the matching degree of a predicted box and a real target, the tracking loss is used to constrain the continuity of trajectory prediction, and the occlusion compensation loss is used to compensate the historical trajectory when the target confidence is lower than a preset threshold.

2. The flight trajectory tracking method according to claim 1, characterized in that, The method further comprises: acquiring a training data set; wherein the training data set represents a data set constructed based on data collected by a visible light camera, an infrared thermal imaging camera, and a radar; performing target detection and trajectory prediction on the training data set by using the initial tracking model to obtain a first result; determining a loss function value corresponding to the preset loss function based on the difference information between the first result and labeled data in the training data set, and determining gradient information by using the loss function value, so as to iteratively update the parameters of the initial tracking model by using the gradient information to obtain the trained tracking model.

3. The flight trajectory tracking method according to claim 2, characterized in that, The determination of the loss function value corresponding to the preset loss function based on the difference information between the first result and the labeled data in the training data set comprises: determining a detection loss value based on the target detection box in the first result and the real target box in the labeled data; determining a tracking loss value based on the predicted trajectory in the first result and the real trajectory in the labeled data; determining whether the to-be-tested target is occluded based on the confidence of the to-be-tested target in the analysis result to obtain an occlusion judgment result, and determining an occlusion compensation loss value based on the occlusion judgment result; determining the loss function value based on the detection loss value, the tracking loss value, the occlusion compensation loss value, and weight information.

4. The flight trajectory tracking method according to claim 3, characterized in that, The determination of the detection loss value based on the target detection box in the first result and the real target box in the labeled data comprises: determining the area of the to-be-tested target based on the target detection box; determining the detection loss value based on an improved intersection over union, a target weight, the area of the to-be-tested target, and the image area corresponding to the to-be-tested target; wherein the value of the target weight is a first numerical value when the area of the to-be-tested target is less than a preset area, otherwise, the value of the target weight is a second numerical value.

5. The flight trajectory tracking method according to claim 3, wherein, The determination of the tracking loss value based on the predicted trajectory in the first result and the real trajectory in the labeled data comprises: determining the speed difference between adjacent time instants of the to-be-tested target based on the first result, and determining the norm of the speed difference; The tracking loss value is determined based on a norm of a position error between the predicted trajectory and the real trajectory and a norm of the speed difference.

6. The flight trajectory tracking method according to claim 3, characterized in that, The occlusion compensation loss value is determined based on the occlusion judgment result. In a case where the confidence corresponding to the target to be detected in the analysis result is less than a preset confidence, the occlusion judgment result is determined to be that the target to be detected is occluded, and the value of the occlusion compensation loss value is determined to be a third value; otherwise, the occlusion judgment result is determined to be that the target to be detected is not occluded, and the value of the occlusion compensation loss value is determined to be 0. The third value is determined based on a first position obtained by fitting a historical trajectory and a current predicted position.

7. The flight trajectory tracking method according to any one of claims 1 to 6, characterized in that, The trained tracking model includes a first preset network and a second preset network; the processed data includes processed image data and processed point cloud data; and the flight trajectory of the tracking target in the processed data is analyzed by using the trained tracking model to obtain target trajectory data, including: An image feature of the processed image data is determined by using the first preset network; the image feature includes a texture feature and a bounding box of the tracking target. A point cloud feature of the processed point cloud data is determined by using the second preset network; the point cloud feature includes a three-dimensional position feature and a speed feature of the tracking target. The image feature and the point cloud feature are subjected to spatial alignment processing to obtain an aligned image feature and an aligned point cloud feature. The aligned image feature and the aligned point cloud feature are fused based on a preset weight to obtain fused feature data; the preset weight includes a first preset weight corresponding to a visible light feature in the aligned image feature, a second preset weight corresponding to an infrared feature in the aligned image feature, and a third preset weight corresponding to the aligned point cloud feature. The target trajectory data is determined based on the fused feature data.

8. The flight trajectory tracking method according to claim 7, characterized in that, The target trajectory data is determined based on the fused feature data, including: Target detection is performed based on the fused feature data to obtain a target detection result; the target detection result includes a target confidence. In a case where the target confidence is less than a first confidence and greater than or equal to a second confidence, the third preset weight is adjusted, and the process of feature fusion is performed again based on the adjusted third preset weight to determine the target trajectory data based on updated fused features. In a case where the target confidence is less than the second confidence, the target trajectory data is determined by performing trajectory prediction on the tracking target. In a case where the target confidence is greater than a third confidence, the target trajectory data is determined based on the target detection result; the first confidence is greater than the second confidence and less than the third confidence.

9. A flight trajectory tracking apparatus characterized by comprising: A computer program product comprising a computer readable medium having stored thereon instructions that, when executed by a processor, implement the method of any of claims 1-8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program, when executed by a processor, implements the method of any of claims 1-8.

Images