Unmanned aerial vehicle body tracking method and system

Abstract

The invention discloses an unmanned aerial vehicle body tracking method and system, and the method comprises the steps: marking a to-be-tracked target from an image shot by an unmanned aerial vehicle, taking the to-be-tracked target as a template image, and taking the image shot by the unmanned aerial vehicle as a search image, the template image and a search image in the currently shot video frame are input into a passive tracking module for feature extraction, interaction between the features of the search image and the features of the template image is realized, and passive tracking is performed frame by frame; the tracking result of the passive tracking module is sent to the unmanned aerial vehicle control module, the unmanned aerial vehicle control module carries out feature extraction on the historical tracking result, the search image of the current frame and the search image of the historical frame, then carries out feature interaction and target state prediction, and determines the target state according to the predicted target state and the current control parameters of the unmanned aerial vehicle. The control parameters of the unmanned aerial vehicle at the next moment are calculated, and updating of the control parameters of the unmanned aerial vehicle is achieved; the method has the advantages that a network framework with body tracking is provided, and continuous robust tracking of the unmanned aerial vehicle to the target can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent unmanned systems and machine vision technologies, and in particular to a method and system for tracking an unmanned aerial vehicle (UAV). Background Art

[0002] Visual tracking technology captures the image information of a target object through a camera or image sensor and uses computer vision and image processing techniques to track and locate the target in real time. Its core is to analyze the image information between consecutive frames and predict and locate the target's position in the next frame.

[0003] Traditional visual tracking methods rely on matching the target's explicit features (such as color and texture) in consecutive frames. When the target is temporarily occluded or drastically deformed, the tracking success rate drops significantly. Manual recalibration of the target is required to relocate the target.

[0004] Most studies, such as the multimodal fusion-based long-term drone tracking method and system disclosed in Chinese Patent Publication No. CN116168322A, focus solely on target positioning and fail to incorporate environmental parameters (such as obstacle distribution and illumination gradients) into the decision-making model. Field data shows that in complex environments, ignoring obstacles significantly increases the risk of collision and tracking loss due to sudden changes in illumination.

[0005] Furthermore, existing tracking tasks are all performed using controlled video footage captured by controlled cameras (such as surveillance equipment, handheld devices, and human-controlled drones). This is called passive tracking, and existing technologies can achieve passive tracking. Embodied tracking, on the other hand, combines passive tracking with the ability to automatically adjust the drone's position and posture based on the target's state and intent, achieving embodied and continuous tracking. Existing tracking tasks do not address this aspect.

[0006] In summary, the traditional method has the following defects:

[0007] (1) Poor passive tracking performance

[0008] Traditional methods extract templates and search features independently, ignoring the relationship between them and failing to extract template-specific search features in a targeted manner. Under the challenges of small targets and strong interference, tracking performance is greatly reduced.

[0009] (2) Embodied tracking is not possible

[0010] (3) Tracking continuity is difficult to maintain when the target disappears

[0011] Traditional methods rely on continuous visual feedback, and the target's temporary disappearance (such as entering a blind spot) will lead to tracking interruption.

[0012] (4) Poor adaptability to dynamic environments

[0013] The tracking failure rate of existing solutions increases significantly in scenarios with sudden changes in lighting (such as entering and exiting tunnels) and obstacle interference (such as vehicles passing through). Summary of the Invention

[0014] The technical problems to be solved by the present invention are the two major problems of the UAV visual tracking method: poor passive tracking performance and inability to achieve embodied tracking.

[0015] The present invention solves the above technical problems through the following technical means: a drone embodied tracking method, comprising: marking a target to be tracked from an image taken by a drone, using the target to be tracked as a template image, and using the image taken by the drone as a search image; inputting the template image and the search image in the currently taken video frame into a passive tracking module for feature extraction and realizing the interaction between the features of the search image and the features of the template image, and performing passive tracking frame by frame; the tracking results of the passive tracking module are given to the drone control module, and the drone control module extracts features from the historical tracking results, the search image of the current frame, and the search image of the historical frame respectively, and then performs feature interaction and target state prediction; through the predicted target state and the current control parameters of the drone, the control parameters of the drone at the next moment are calculated to realize the update of the drone control parameters.

[0016] The present invention inputs the template image and the search image in the currently captured video frame into the passive tracking module for feature extraction and realizes the interaction between the features of the search image and the features of the template image, pays attention to the relationship between the template and the search features, and extracts the template-specific search features in a targeted manner. Under the challenges of small targets and strong interference, the tracking performance is guaranteed, thereby achieving good passive tracking performance. In addition, while realizing passive tracking, the present invention also automatically adjusts the control parameters of the drone, that is, the position of the drone, according to the state of the target in real time to achieve continuous tracking, thereby enabling embodied tracking.

[0017] Furthermore, the passive tracking module includes a convolutional token embedding module and a multi-layer multi-interaction Transformer module. The template image and the search image in the currently captured video frame are respectively extracted with the convolutional token embedding module with fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens and search tokens. These tokens are integrated into a feature map and input into the current layer multi-interaction Transformer module for feature extraction and interaction between the features of the search image and the features of the template image. The result of the current layer multi-interaction Transformer module is continued to be sent to the next layer multi-interaction Transformer module to enhance and interact the features layer by layer. The output result of the multi-interaction Transformer module of the last layer is input into the target box prediction head to obtain the predicted target box, and the predicted target box is given to the drone control module as the tracking result.

[0018] Furthermore, the template image and the search image in the currently captured video frame are respectively extracted using a convolutional token embedding module to extract fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens and search tokens, including:

[0019] The template image in the first frame of the currently shot video frame, the template image with a reliability exceeding a threshold in the currently shot video frame, and the search image in the currently shot video frame are convolved and flattened using a convolutional token embedding module to obtain a fixed template token, a dynamic template token, and a search token. The fixed template token is used for hybrid template token initialization, and the hybrid template token of the previous frame is used for temporal template token.

[0020] Furthermore, the processing of the multi-interacting Transformer module is as follows:

[0021] The multi-interaction Transformer module performs layer normalization on the input feature map and then splits it into different tokens according to the data dimension. A convolution operation is performed on each token to obtain the query vector, key vector and value vector corresponding to each token. The query vector, key vector and value vector of the fixed template token and the dynamic template token are spliced to obtain a new query vector, key vector and value vector, and attention calculation is performed to obtain the attention calculation result of the template image; the query vector of the mixed template token and the search token is spliced to obtain a new query vector, the key vectors of all tokens are spliced to obtain a new key vector, the value vectors of all tokens are spliced to obtain a new value vector, and then attention calculation is performed to obtain the attention calculation result of the search image; the attention calculation result of the template image and the attention calculation result of the search image are spliced and then layer normalization is performed again and the first multi-layer perceptron outputs the final result.

[0022] Furthermore, the UAV control module performs feature extraction on the historical tracking results, the search image of the current frame, and the search image of the historical frame, including:

[0023] The historical tracking results are constructed into a trajectory graph, which is input into the multi-layer GCN and pooled to obtain the historical features f T ; Construct the search image of the current frame into an environment relationship graph, input the environment relationship graph into another multi-layer GCN and perform a pooling operation to obtain the environment relationship feature f G ; The search image of the historical frame is extracted through ResNet-50 convolution features and then subjected to global average pooling and linear transformation to obtain the visual feature f I ; f G and f I The fusion feature z0 is obtained by splicing.

[0024] Furthermore, the drone control module performs feature interaction and target state prediction, including:

[0025] Construct a target state prediction network, which includes sequentially connected multi-layer attention modules and a second multi-layer perceptron. Each layer of attention module includes a first self-attention unit, a second self-attention unit, a first cross-attention unit, and a second cross-attention unit. The first self-attention unit and the second self-attention unit of the first layer of attention module receive the fusion feature z0 and the historical feature f respectively. T The first self-attention unit and the second self-attention unit multiply the input value by different weight coefficients to obtain the corresponding query vector, key vector and value vector, and perform self-attention calculation. The first cross-attention unit uses the query vector of the first self-attention unit as its query vector, and the key vector and value vector of the second self-attention unit as its key vector and value vector, and performs cross-attention calculation. The second cross-attention unit uses the query vector of the second self-attention unit as its query vector, and the key vector and value vector of the first self-attention unit as its key vector and value vector, and performs cross-attention calculation. The calculation results of the first self-attention unit and the calculation results of the first cross-attention unit are added as the image stream output of the current attention module and transmitted to the next layer of attention module. The calculation results of the second self-attention unit and the calculation results of the second cross-attention unit are added as the trajectory stream output of the current attention module and transmitted to the next layer of attention module. The image stream output and the trajectory stream output of the last layer of attention module are spliced to obtain the target stream output. The target stream output is predicted by the second multi-layer perceptron at the next moment. The present invention establishes a target state prediction network and maintains the tracking chain through the active movement of the drone. Compared with the traditional method, it does not rely on continuous visual feedback, and the temporary disappearance of the target will not cause tracking interruption.

[0026] Furthermore, the calculating of the control parameters of the UAV includes:

[0027]

[0028] p t+1 =p t +Δp

[0029] Among them, CA represents the cross attention calculation corresponding to the target state, Q p ,K s ,V s Calculate the corresponding query vector, key vector and value vector for the cross attention of the target state respectively, f p is the current UAV control parameter p t The linear projection of f p =W p p t +b p , W p and b p are the first weight and the first bias respectively; is the predicted target state at the next moment; Q p ,K ′ ,V s The weight coefficient, W Δ and b Δ are the second weight and the second bias respectively, Δp is the control increment, p t+1 is the control parameter of the UAV at the next moment.

[0030] Furthermore, the drone embodied tracking method also includes a training process:

[0031] Adjust the parameters of the passive tracking module and the drone control module, train the entire model, and calculate the value of the loss function. Stop training when the value of the loss function is minimum, and obtain the optimized model. Use the optimized model to perform drone tracking. The formula of the loss function is

[0032]

[0033] Among them, λ pred ,λ ctrl ,λ reg are all hyperparameters, Ω(Θ) is the parameter regularization term, and are the true target state and the ideal UAV pose, respectively.

[0034] Furthermore, the drone embodied tracking method also includes an environment adaptive adjustment process:

[0035] When the brightness change rate of adjacent frames exceeds 20% / second, adjust the drone's camera parameters and reduce or increase exposure compensation. If the brightness change rate of adjacent frames exceeds 20% / second continuously, use the average of multiple historical template images as the updated template image. When the light is too low or too high, use the automatic light enhancement strategy.

[0036] A safe obstacle distance is set. When an obstacle intrudes, the drone will circumvent it. If there is insufficient space, it will trigger an emergency hover and initiate a 360-degree sweep to detect any moving obstacles. This adaptive environmental adjustment process enables tracking in scenarios with sudden changes in illumination (such as entering or exiting a tunnel) or interference from obstacles (such as vehicles), preventing tracking failures.

[0037] The present invention also provides a system for executing the above-mentioned drone embodied tracking method, comprising a passive tracking module and a drone control module, wherein the passive tracking module is connected to the drone control module.

[0038] The advantages of the present invention are:

[0039] (1) The present invention inputs the template image and the search image in the currently captured video frame into the passive tracking module for feature extraction and realizes the interaction between the features of the search image and the features of the template image, pays attention to the relationship between the template and the search features, and extracts the template-specific search features in a targeted manner. Under the challenges of small targets and strong interference, the tracking performance is guaranteed, thereby achieving good passive tracking performance. In addition, while realizing passive tracking, the present invention also automatically adjusts the control parameters of the drone, that is, the position of the drone, according to the state of the target in real time to realize continuous tracking, thereby enabling embodied tracking.

[0040] (2) The present invention establishes a target state prediction network and maintains the tracking chain through the active movement of the UAV. Compared with traditional methods, it does not rely on continuous visual feedback, and the temporary disappearance of the target will not cause tracking interruption.

[0041] (3) The present invention can track objects in scenarios with sudden changes in illumination (such as entering or exiting a tunnel) or interference from obstacles (such as vehicles weaving in), thus avoiding tracking failures through the above-mentioned environmental adaptive adjustment process.

[0042] (4) The multi-interactive Transformer module of the present invention combines convolution operations with Transformer operations. The convolutional token embedding module implemented by the convolution operation is responsible for extracting local information, such as edges, textures and other important information; the Transformer operation simultaneously implements global feature extraction and template search information interaction tasks. The multi-interactive Transformer module performs a layer normalization on the input feature map and then splits it according to the data dimension into different tokens. The convolution operation is performed on each token to obtain the query vector, key vector and value vector corresponding to each token, thereby replacing the traditional linear projection operation. This achieves the goal of retaining the global modeling advantages of the Transformer while enhancing the local feature extraction capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flow chart of a drone embodied tracking method disclosed in an embodiment of the present invention;

[0044] Figure 2 This is an architectural diagram of a drone embodied tracking system in a drone embodied tracking method disclosed in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of a multi-interactive Transformer module in a drone embodied tracking method disclosed in an embodiment of the present invention;

[0046] Figure 4 This is a schematic diagram of a drone control module in a drone embodied tracking method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0048] Example 1

[0049] The tracking tasks in the existing technology are all controlled by shooting equipment (such as monitoring equipment, handheld devices, and human-controlled drones), which are manually shot to capture videos and implement tracking on these videos. They are called passive tracking. The method designed by the present invention is to track the target in the video frame (passive tracking method) and automatically adjust the position of the drone (drone control module) according to the state of the target in real time to achieve continuous tracking. It is called embodied tracking. Figure 1As shown, embodiment 1 of the present invention provides a method for embodied tracking of a drone, comprising: using ground station software to mark a target to be tracked from an image captured by a drone, the drone continuously tracking the target, the target to be tracked serving as a template image, the image captured by the drone serving as a search image, the template image and the search image in the currently captured video frame being input into a passive tracking module for feature extraction and interaction between the features of the search image and the features of the template image, performing passive tracking frame by frame, marking the position and size of the target to be tracked with a target box as the tracking result of the frame, and saving the result. The tracking result of the passive tracking module is given to the drone control module, which performs feature extraction on the historical tracking results, the search image of the current frame, and the search image of the historical frame, respectively, and then performs feature interaction and target state prediction (e.g., the target is blocked, turning, far away from the drone, etc.). Based on the predicted target state and the current control parameters of the drone, the control parameters of the drone at the next moment are calculated to update the drone control parameters. In terms of hardware, the drone platform of the present invention adopts a customized version of DJI Mavic 3 Enterprise, including a visual sensor, an edge computing unit, and an auxiliary sensor. Visual Sensor: Hasselblad L2D-20c camera (4 / 3-inch CMOS, 20 million effective pixels), supports RAW shooting, and has a 7x optical zoom. It also features a three-axis mechanically stabilized gimbal (pitch range ±90°, roll ±45°, accuracy ±0.02°) and an active image stabilization algorithm with 95% shake suppression. Edge Computing Unit: NVIDIA Jetson Xavier NX (6-core ARM v8.2 CPU, 384-core Volta GPU), built-in 32GB eMMC storage, running Ubuntu 20.04 LTS. Expansion Interfaces: 2 x USB 3.1 (for IMU and ToF sensor), 1 x MIPI CSI-2 (camera data input). Auxiliary Sensor: 6-axis IMU (BMI088, gyroscope range ±2000° / s, noise density 0.015° / s / √Hz). ToF altimeter (VL53L5CX, maximum range 4m, 4×4 resolution detection area). In terms of software, the entire method process is detailed as follows:

[0050] S1. Target initialization

[0051] The user selects the target area (at least 50×50 pixels) through the ground station software. If the light level is less than 30, the low-light enhancement mode (image gamma correction parameter γ = 2.0) is automatically enabled.

[0052] S2. Passive tracking

[0053] like Figure 2 and Figure 3First, the template image and the search image in the currently captured video frame are input into the passive tracking module, which contains a search image frame and several template images; then, the input image first passes through a multi-layer multi-interactive Transformer module to uniformly implement feature extraction and template search information interaction, so that the final extracted features effectively integrate the target information; finally, the search features are sent to the target box prediction head to realize the prediction of the target box.

[0054] In order to achieve the coordinated optimization of local feature extraction and global modeling, the multi-interaction Transformer module of this method combines convolution operations and Transformer operations. The convolution token embedding module implemented by the convolution operation is responsible for extracting local information, such as edges, textures and other important information; the Transformer operation simultaneously realizes the global feature extraction and template search information interaction tasks. Figure 1 Combine Figure 2 and Figure 3 The passive tracking module includes a convolutional token embedding module and a multi-layer multi-interaction Transformer module. The template image and the search image in the currently shot video frame use the convolutional token embedding module to extract fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens and search tokens respectively. The main purpose is to use the convolutional token embedding module to convolve and flatten the template image in the first frame image of the currently shot video frame, the template image with a reliability exceeding the threshold in the currently shot video frame, and the search image in the currently shot video frame to obtain fixed template tokens, dynamic template tokens and search tokens. The fixed template token is used for the hybrid template token initialization, and the temporal template token uses the hybrid template token of the previous frame. Among them, the convolutional token embedding module includes convolution and flattening operations. Specifically, the input is a two-dimensional feature map. First, the feature map is convolved to extract local features and reduce the dimension. Then, the two-dimensional feature map is converted into the tokens required for the Transformer operation through the flattening operation. The extraction method of each token is as follows:

[0055] Fixed Template Tokens: For each video sequence, the template of the first frame is manually annotated, and the quality of this template is often high. Therefore, we use the template image in the first frame of the video sequence and perform the convolutional token embedding operation on it to obtain the fixed template token.

[0056] Dynamic Template Tokens: During the tracking process, the model updates a highly reliable template image based on the strategy described in the paper "Cui Y, Jiang C, Wang L, et al. Mixformer: End-to-end tracking with iterative mixed attention[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022:13608-13618." This image is then incorporated into the tracking model as a dynamic template to adapt to changes in the target's appearance and improve tracking robustness. Dynamic template tokens are generated by performing a convolutional token embedding operation on the dynamic template. It is important to note that in the early stages of model tracking, before dynamic templates are generated, fixed template tokens are used instead of dynamic template tokens.

[0057] Hybrid template tokens: Hybrid template tokens incorporate search features. Although the search image contains a large amount of background information, some information exists that can contribute to template learning. For example, similar objects can enhance the model's ability to discriminate against targets. Through the interaction of template information and search information, as detailed in the multi-layer attention module below, queries for fixed template tokens and search tokens are concatenated, and hybrid template tokens are obtained through attention operations. It should be noted that for the input of the first-layer multi-interaction Transformer module, that is, before the first multi-interaction operation is performed, the template and search features have not yet interacted, and the hybrid template token is replaced by the fixed template token.

[0058] Temporal template token: Use the hybrid template token feature of the previous frame as the temporal template token. In this way, during the model tracking process, it can not only provide the template information of the previous frame, but also utilize the search information of the previous frame to enhance the features of the template. Set the current time to t, and the hybrid template token of the previous frame to mz t-1 , then the timing template token of the current frame is sz t =mz t-1 .

[0059] After the above tokens are determined, the fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens, and search tokens are integrated into feature maps and input into the current layer's multi-interactive Transformer module for feature extraction and interaction between the search image's features and the template image's features. The results of the current layer's multi-interactive Transformer module are then fed into the next layer's multi-interactive Transformer module, where the features are enhanced and interacted layer by layer. The output of the last layer's multi-interactive Transformer module is fed into the target box prediction head to obtain a predicted target box, which is then given to the drone control module as a tracking result. After the above operations, the multi-layer multi-interactive Transformer module will effectively extract target information. The target box prediction head adopts the corner prediction head that has been most commonly used in recent years. Refer to the paper "Yan B, Peng H, Fu J, et al. Learning spatio-temporal transformer for visual tracking [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 10448-10457.". By using only a few convolution-normalization-ReLU layers, it is possible to predict the top left and bottom right corners of the target box position. The predicted target box is obtained by calculating the expected probability distribution of these corner points. The processing process of the multi-interactive Transformer module is as follows:

[0060] To better adapt the representation process to the target being tracked, this paper implements unified modeling of the interaction between feature extraction and template search information. Leveraging the Transformer's powerful global modeling capabilities and the flexibility of attention operations, a multi-interaction Transformer module is constructed as shown in the figure. For simplicity, z and x will be used to represent the template and search area, and q, k, and v will represent the query, key, and value required for the attention operation, respectively.

[0061] The multi-interactive Transformer module performs a layer-normalization on the input feature map, then splits it into different tokens according to the data dimension. It then performs a convolution operation on each token to obtain the corresponding query vector, key vector, and value vector, replacing the traditional linear projection operation. This achieves the goal of retaining the Transformer's global modeling advantages while enhancing local feature extraction capabilities. The convolution operation first reshapes the token into a 2D feature map and finally flattens the mapped features into a one-dimensional vector for subsequent attention operations.

[0062] For the template branch, considering that the quality of the first frame template is usually higher, the first frame template is retained during the tracking process of the entire sequence, which is recorded as a fixed template. In addition, the fixed template alone cannot effectively cope with the appearance changes of the target. During the tracking process, a template is periodically selected from the search frame for update, which is recorded as a dynamic template. Considering that the search feature contains a lot of background information, this module only applies this multi-interactive attention mechanism between the template features of the two modalities, and will not be disturbed by irrelevant information in the search feature. Specifically, to facilitate the subsequent calculation of mixed attention, the generated query, key and value are first spliced as shown in formula (1); then the attention map of the template branch is calculated using formula (2), and finally, after a layer normalization and a multi-layer perceptron, the enhanced features are obtained through residual connection. That is, the query vector, key vector and value vector of the fixed template token and the dynamic template token are spliced to obtain a new query vector, key vector and value vector for attention calculation to obtain the attention calculation result of the template image. The relevant formula is as follows:

[0063] q z =Concat(q iz ,q dz ),k z =Concat(k iz ,k dz ),v z =Concat(v iz ,v dz ) (1)

[0064]

[0065] Where q, k, and v represent the query vector, key vector, and value vector, respectively; iz represents a fixed template token; dz represents a dynamic template token; Attention represents the attention calculation; d represents the dimension of the feature vector; and Softmax represents the Softmax function.

[0066] For the search branch, this module applies a multi-interaction attention mechanism between the search features and various template features. Through this interaction mode, extensive information interaction between the template and the search can be achieved. Specifically, similar to the template branch, the query, key, and value are first generated and spliced. It should be noted that the key and value of the search branch need to splice four template features and one search feature, as shown in formula (3); then use formula (4) to calculate the attention map of the search branch, and finally, after a layer normalization and a multi-layer perceptron, the enhanced features are obtained through residual connections. That is, the query vectors of the mixed template token and the search token are spliced to obtain a new query vector, the key vectors of all tokens are spliced to obtain a new key vector, and the value vectors of all tokens are spliced to obtain a new value vector. Then, attention calculation is performed to obtain the attention calculation result of the search image; the relevant formula is as follows:

[0067]

[0068] Among them, mz represents the hybrid template token, x represents the search token, and sz represents the sequential template token.

[0069] After the attention calculation results of the template image and the attention calculation results of the search image are spliced together, layer normalization is performed again, and the first multi-layer perceptron outputs the final result to the drone control module. Through the above operations, enhanced features of the template and search frame can be obtained, and these features are sent to the multi-interactive Transformer module of the next layer for further enhancement. In order to enhance the expressive power of the model, the present invention stacks multiple multi-interactive Transformer modules to achieve progressive interactive feature enhancement, forming a novel tracking network model.

[0070] S3. UAV control process

[0071] like Figure 4 As shown in Figure 1, the working process of the UAV control module is divided into three parts: (1) feature extraction (responsible for extracting high-level representations from the current image frame, environmental relationship map and historical trajectory map), (2) feature interaction and target state prediction (the target state prediction network adopts a dual-stream Transformer cross-inference mechanism to achieve feature fusion, and predicts the target state information through the fused features), and (3) updating of the UAV's control parameters (the UAV control network uses the predicted target state to generate the UAV's control instructions, adjust the UAV posture in real time, and thus affect the observation of the next frame).

[0072] (1) Feature extraction is to extract features from the historical tracking results, the search image of the current frame, and the search image of the historical frame, including:

[0073] The search image of the historical frame is passed through ResNet-50 to extract convolution features and then undergoes global average pooling and linear transformation to obtain the visual feature f I The process is formulated as follows: The UAV collects RGB frames at time t Extract convolutional features F through ResNet-50 I =ResNet50(S t )(F I ∈R h×w×c ) is then subjected to global average pooling and linear transformation f I =W I GAP(F I )+b I (f I ∈R d ), obtain the image-level vector, that is, the historical feature f I .W I and b I are the weight term and bias term of historical features respectively.

[0074] The search image of the current frame is constructed into an environment relationship graph, which is input into another multi-layer GCN and pooled to obtain the environment relationship feature f G ; The process is formulated as follows: Using the HGR-Net network described in the literature "Bi H, Xu C, Shi C, et al. HGR-Net: Hierarchical graph reasoning network for arbitrary shape scenetext detection[J]. IEEE Transactions on Image Processing, 2023, 32: 4142-4155.", the current frame image information S is realized. t Convert to a directed graph, where nodes are objects on the image and edges are relationships between objects, and get a node set And construct a directed graph G based on geometric adjacency or prior semantics env , use L-layer graph convolutional network (GCN) to train node features Iterative updates:

[0075] Finally, the environmental relationship characteristics are obtained by reading the function The result obtained after inputting GCN into the directed graph. Pool is the pooling operation. G and b G are the weight and bias in the environmental relationship features, respectively.

[0076] The historical tracking results are constructed into a trajectory graph, which is input into the multi-layer GCN and pooled to obtain the historical features f T ; The process is formulated as follows: Construct the tracking output of the first K frames into a trajectory graph G traj , and then use the isomorphic GCN to get f T =W T pool(G traj )+b T (f T ∈R d ).

[0077] Finally, f G and f I The fusion feature z0 is obtained by splicing.

[0078] (2) The process of feature interaction and target state prediction of the UAV control module is as follows:

[0079] Continue reading Figure 3 , construct a target state prediction network, which uses a two-stream cross Transformer structure to achieve feature enhancement and information interaction. Assume that the number of layers is N, and each layer contains self-attention (SA) to capture long-range dependencies within the stream; and cross attention (CA) to achieve Query / Key Value interaction between the two streams. Specifically, the target state prediction network includes sequentially connected multi-layer attention modules and a second multi-layer perceptron. Each layer of attention module includes a first self-attention unit, a second self-attention unit, a first cross-attention unit, and a second cross-attention unit. The first self-attention unit and the second self-attention unit of the first layer of attention module receive the fusion feature z0 and the historical feature f respectively. T , the first self-attention unit and the second self-attention unit multiply the input value by different weight coefficients to obtain the corresponding query vector, key vector and value vector, and perform self-attention calculation. The calculation principle of the first self-attention unit and the second self-attention unit is the same. For any input x∈R n×d , whose self-attention is written as Q x =xW Q ,K x =xW K ,V x =xW V .

[0080] The first cross-attention unit uses the query vector of the first self-attention unit as its query vector, and the key vector and value vector of the second self-attention unit as its key vector and value vector to perform cross-attention calculation. The second cross-attention unit uses the query vector of the second self-attention unit as its query vector, and the key vector and value vector of the first self-attention unit as its key vector and value vector to perform cross-attention calculation. The calculation results of the first self-attention unit and the calculation results of the first cross-attention unit are added as the image stream output of the current attention module and passed to the next layer of attention module. The calculation results of the second self-attention unit and the calculation results of the second cross-attention unit are added as the trajectory stream output of the current attention module and passed to the next layer of attention module. The cross-attention formula is expressed as follows: The cross attention and self-attention results are added and passed to the next layer, which is expressed as

[0081]

[0082] o (l) Represents the image stream output of layer l, u (l) Represents the trajectory flow output of layer l. CA represents cross attention calculation and SA represents self attention calculation. They represent the query vector calculated based on the output of the l-th layer image stream, the key vector calculated based on the output of the l-th layer trajectory stream, and the value vector calculated based on the output of the l-th layer trajectory stream.

[0083] The image stream output and trajectory stream output of the last layer of attention module are spliced together to obtain the target stream output o (N) The target stream output is predicted by the second multi-layer perceptron to predict the target state at the next moment in Can include target center of mass (x, y), size (w, h) and speed Etc. σ represents the second multilayer perceptron, W2, b1, b2 represent the weight of the target state at the next moment and two biases respectively.

[0084] (3) Calculate the control parameters of the UAV, including:

[0085] Assume that the current pose parameter of UAV is p t ∈R r (such as three-dimensional translation + Euler angle), first linearly project f p =W p p t +b p , and then embed the target state into the UAV control semantic space with cross attention

[0086]

[0087] Final Control Increment p t+1 =p t +Δp

[0088] New pose p t+1 The information is immediately fed back to the flight control system and injected into the next frame of information stream to form a closed loop.

[0089] Among them, CA represents the cross attention calculation corresponding to the target state, Q p ,K s ,V s Calculate the corresponding query vector, key vector and value vector for the cross attention of the target state respectively, f p is the current UAV control parameter p t The linear projection of f p =W p p t +b p , W p and b p are the first weight and the first bias respectively; is the predicted target state at the next moment; Q p ,K s ,V s The weight coefficient, W Δ and b Δ are the second weight and the second bias respectively, Δp is the control increment, p t+1 is the control parameter of the UAV at the next moment.

[0090] As a further improved technical solution, the drone embodied tracking method of the present invention further includes a training process:

[0091] Adjust the parameters of the passive tracking module and the drone control module, train the entire model, and calculate the value of the loss function. Stop training when the value of the loss function is minimum, and obtain the optimized model. Use the optimized model to perform drone tracking. Specifically, the framework is optimized with the supervised learning paradigm. If and are the true target state and the ideal UAV pose respectively, then the loss function

[0092]

[0093] Among them, Ω(Θ) is the parameter regularization term; λ pred ,λ ctrl ,λ reg is a hyperparameter. Actual training can be performed on simulators such as Gazebo / AirSim or datasets collected from real flights, and all network parameters are optimized end-to-end through backpropagation.

[0094] As a further improved technical solution, the drone embodied tracking method of the present invention further includes an environment adaptive adjustment process:

[0095] When the brightness change rate of adjacent frames is greater than 20% / second, the camera parameters of the drone are adjusted to reduce or increase the exposure compensation. If the brightness change rate of adjacent frames is greater than 20% / second continuously, the average of multiple historical template images is used as the updated template image. When the light is too low or too high, an automatic light enhancement strategy is adopted. The automatic light enhancement strategy belongs to the existing technology, that is, when the light is too low or too high, the light is enhanced according to a preset strategy. For example, when the light is lower than the first preset value, the light intensity is increased by the second preset value. Similarly, when the light is higher than the third preset value, resulting in overexposure and inability to see clearly, the light intensity is increased by the third preset value.

[0096] Set the obstacle safety distance. The safety distance threshold is 5m horizontally and 3m vertically. When an obstacle intrudes into the safety distance, the drone will bypass it. If there is insufficient space to bypass, it will trigger an emergency hover and initiate a 360° scan to detect any moving obstacles in the surrounding area.

[0097] This paper ensures the effective implementation of embodied tracking by introducing multimodal fusion features and a two-stream cross-Transformer. The cross-attention mechanism ensures information complementarity between target semantics and the UAV's self-state, and enables synchronous optimization of the control network under end-to-end training. Experimental results demonstrate that this method achieves stable and accurate target tracking performance in both simulation and real-world flight tests, providing a feasible solution for the integrated perception and control of autonomous aerial robots.

[0098] Through the above technical solutions, the present invention utilizes the powerful representation capabilities of the Transformer based on a multi-interactive passive tracking module to design a multi-interactive Transformer module, effectively realizing information interaction between feature extraction and template search, and improving the robustness of the tracking model. The drone control module is designed with a multimodal context encoder (also known as the feature extraction part), a target state prediction network, and a drone control network. It uses video sequences and historical tracking results to predict the target state, generate drone control instructions, and adjust the drone's position in real time, thereby achieving embodied tracking.

[0099] Example 2

[0100] Based on Example 1, Example 2 of the present invention further provides a system for executing the drone embodied tracking method described in Example 1, comprising a passive tracking module and a drone control module, wherein the passive tracking module is connected to the drone control module.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A drone embodied tracking method, characterized in that: include: The target to be tracked is marked in the image taken by the drone. The target to be tracked is used as the template image, and the image taken by the drone is used as the search image. The template image and the search image in the current video frame are input into the passive tracking module for feature extraction and interaction between the features of the search image and the features of the template image, and passive tracking is performed frame by frame. The tracking results of the passive tracking module are given to the UAV control module. The UAV control module extracts features from the historical tracking results, the search image of the current frame, and the search image of the historical frame respectively, and then performs feature interaction and target state prediction. Based on the predicted target state and the current control parameters of the UAV, the control parameters of the UAV at the next moment are calculated to update the UAV control parameters.

2. The method for tracking a drone according to claim 1, wherein: The passive tracking module includes a convolutional token embedding module and a multi-layer multi-interaction Transformer module. The template image and the search image in the currently captured video frame are respectively extracted with the convolutional token embedding module with fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens and search tokens. These tokens are integrated into a feature map and input into the current layer multi-interaction Transformer module for feature extraction and interaction between the features of the search image and the features of the template image. The result of the current layer multi-interaction Transformer module is continued to be sent to the next layer multi-interaction Transformer module to enhance and interact with the features layer by layer. The output result of the multi-interaction Transformer module of the last layer is input into the target box prediction head to obtain the predicted target box, and the predicted target box is given to the drone control module as the tracking result.

3. The method for tracking a drone according to claim 2, wherein: The template image and the search image in the currently captured video frame are respectively extracted using a convolutional token embedding module with fixed template tokens, dynamic template tokens, hybrid template tokens, temporal template tokens and search tokens, including: The template image in the first frame of the currently shot video frame, the template image with a reliability exceeding a threshold in the currently shot video frame, and the search image in the currently shot video frame are convolved and flattened using a convolutional token embedding module to obtain a fixed template token, a dynamic template token, and a search token. The fixed template token is used for hybrid template token initialization, and the hybrid template token of the previous frame is used for temporal template token.

4. The method for tracking a drone according to claim 3, wherein: The processing process of the multi-interaction Transformer module is: The multi-interaction Transformer module performs layer normalization on the input feature map and then splits it into different tokens according to the data dimension. A convolution operation is performed on each token to obtain the query vector, key vector and value vector corresponding to each token. The query vector, key vector and value vector of the fixed template token and the dynamic template token are spliced to obtain a new query vector, key vector and value vector, and attention calculation is performed to obtain the attention calculation result of the template image; the query vector of the mixed template token and the search token is spliced to obtain a new query vector, the key vectors of all tokens are spliced to obtain a new key vector, the value vectors of all tokens are spliced to obtain a new value vector, and then attention calculation is performed to obtain the attention calculation result of the search image; the attention calculation result of the template image and the attention calculation result of the search image are spliced and then layer normalization is performed again and the first multi-layer perceptron outputs the final result.

5. The method for tracking a drone according to claim 1, wherein: The UAV control module extracts features from the historical tracking results, the search image of the current frame, and the search image of the historical frame, including: The historical tracking results are constructed into a trajectory graph, which is input into the multi-layer GCN and pooled to obtain the historical features f T ; Construct the search image of the current frame into an environment relationship graph, input the environment relationship graph into another multi-layer GCN and perform a pooling operation to obtain the environment relationship feature f G ; The search image of the historical frame is extracted through ResNet-50 convolution features and then subjected to global average pooling and linear transformation to obtain the visual feature f I ; f G and f I The fusion feature z0 is obtained by splicing.

6. The method for tracking a drone according to claim 5, wherein: The drone control module performs feature interaction and target state prediction, including: Construct a target state prediction network, which includes sequentially connected multi-layer attention modules and a second multi-layer perceptron. Each layer of attention module includes a first self-attention unit, a second self-attention unit, a first cross-attention unit, and a second cross-attention unit. The first self-attention unit and the second self-attention unit of the first layer of attention module receive the fusion feature z0 and the historical feature f respectively. T The first self-attention unit and the second self-attention unit multiply the input value by different weight coefficients to obtain the corresponding query vector, key vector and value vector, and perform self-attention calculation. The first cross-attention unit uses the query vector of the first self-attention unit as its query vector, and the key vector and value vector of the second self-attention unit as its key vector and value vector, and performs cross-attention calculation. The second cross-attention unit uses the query vector of the second self-attention unit as its query vector, and the key vector and value vector of the first self-attention unit as its key vector and value vector, and performs cross-attention calculation. The calculation result of the first self-attention unit and the calculation result of the first cross-attention unit are added as the image stream output of the current attention module and passed to the next layer of attention module. The calculation result of the second self-attention unit and the calculation result of the second cross-attention unit are added as the trajectory stream output of the current attention module and passed to the next layer of attention module. The image stream output and the trajectory stream output of the last layer of attention module are spliced to obtain the target stream output, and the target stream output is predicted by the second multi-layer perceptron to predict the target state at the next moment.

7. The method for tracking a drone according to claim 6, wherein: The calculating of the control parameters of the UAV includes: p t+1 =p t +Δp Among them, CA represents the cross attention calculation corresponding to the target state, Q p ,K s ,V s Calculate the corresponding query vector, key vector and value vector for the cross attention of the target state respectively, f p is the current UAV control parameter p t The linear projection of f p =W p p t +b p , W p and b p are the first weight and the first bias respectively; is the predicted target state at the next moment; Q p ,K s ,V s The weight coefficient, W Δ and b Δ are the second weight and the second bias respectively, Δp is the control increment, p t+1 is the control parameter of the UAV at the next moment.

8. The method for tracking a drone according to claim 7, wherein: Also includes the training process: Adjust the parameters of the passive tracking module and the drone control module, train the entire model, and calculate the value of the loss function. Stop training when the value of the loss function is minimum to obtain an optimized model, and use the optimized model for drone embodied tracking. The formula of the loss function is Among them, λ pred ,λ ctrl ,λ reg are all hyperparameters, Ω(Θ) is the parameter regularization term, and are the true target state and the ideal UAV pose, respectively.

9. The method for tracking a drone according to claim 1, wherein: It also includes the environmental adaptive adjustment process: When the brightness change rate of adjacent frames exceeds 20% / second, adjust the drone's camera parameters and reduce or increase exposure compensation. If the brightness change rate of adjacent frames exceeds 20% / second continuously, use the average of multiple historical template images as the updated template image. When the light is too low or too high, use the automatic light enhancement strategy. Set an obstacle safety distance. When an obstacle invades the safety distance, the drone will bypass it. If there is insufficient space to bypass, it will trigger an emergency hover and start a 360° scan to detect whether there are any moving obstacles in the surrounding area that invade the safety distance.

10. A system for executing the drone embodied tracking method according to any one of claims 1 to 9, characterized in that: It includes a passive tracking module and a UAV control module, and the passive tracking module is connected to the UAV control module.

Images