An unmanned implement body tracking method and system

Abstract

The application discloses a kind of unmanned machine body tracking method and system, method includes: from the image that unmanned aerial vehicle is shot, mark target to be tracked, target to be tracked as template image, the image that unmanned aerial vehicle is shot as search image, template image and current shot video frame search image are input passive tracking module to carry out feature extraction and realize the interaction of the feature of search image and the feature of template image, passive tracking is carried out frame by frame;The tracking result of passive tracking module is given unmanned aerial vehicle control module, and the search image of current frame and the search image of historical frame are respectively carried out feature extraction by unmanned aerial vehicle control module to historical tracking result, then carry out feature interaction and target state prediction, calculate the control parameter of unmanned aerial vehicle in next time by the target state of prediction and the current control parameter of unmanned aerial vehicle, realize the update of unmanned aerial vehicle control parameter;The application has the advantages that: the network framework of body tracking is proposed, and the sustained robust tracking of unmanned aerial vehicle to target can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent unmanned systems and machine vision technology, and in particular to a method and system for unmanned machine body tracking. BACKGROUND

[0002] Visual tracking technology captures image information of a target object through a camera or image sensor, and uses computer vision and image processing technology to track and locate the target in real time. The core lies in analyzing the image information between consecutive frames, predicting and locating the position of the target in the next frame.

[0003] Traditional visual tracking methods rely on matching the explicit features (such as color, texture) of the target in consecutive frames, and when the target is short-term occluded or undergoes severe deformation, the tracking success rate significantly decreases. Artificial re-labeling of the target is required to reposition the target.

[0004] Most research, such as the unmanned aerial vehicle long-time tracking method and system based on multi-modal fusion disclosed in Chinese Patent Publication No. CN116168322A, only focuses on target positioning and does not include environmental parameters (such as obstacle distribution, illumination gradient) in the decision model. Experimental data shows that in complex environments, the collision risk caused by ignoring obstacles and the tracking loss rate caused by sudden changes in illumination significantly increase.

[0005] In addition, the tracking tasks of existing technologies are controlled by shooting devices (such as monitoring devices, handheld devices, and human-controlled unmanned aerial vehicles), and videos are shot by artificial shooting, and tracking is achieved on these videos. They are called passive tracking, i.e., existing technologies can achieve passive tracking. However, body tracking is the automatic adjustment of the position and attitude of the unmanned aerial vehicle according to the state and intention of the target while achieving passive tracking, which realizes continuous tracking of the body. The tracking tasks of existing technologies do not involve this aspect.

[0006] In summary, the traditional method has the following defects:

[0007] (1) Poor passive tracking performance

[0008] Traditional methods independently extract templates and search features, ignoring their mutual relationship, and cannot extract search features specific to the template. In the case of a small target and strong interference, the tracking performance is significantly reduced.

[0009] (2) Cannot achieve body tracking

[0010] (3) Difficulty in maintaining tracking continuity when the target disappears

[0011] Traditional methods rely on continuous visual feedback, and the temporary disappearance of the target (such as entering a blind area) will cause the tracking to be interrupted.

[0012] (4) poor dynamic environment adaptability

[0013] The existing scheme has a significantly increased tracking failure rate in the light mutation (such as entering and exiting a tunnel) and obstacle interference (such as vehicle penetration) scenarios. SUMMARY

[0014] The present application aims to solve the two problems of poor passive tracking performance and the inability to achieve body tracking of the unmanned aerial vehicle vision tracking method.

[0015] The present application solves the above technical problems by the following technical means: an unmanned aerial vehicle body tracking method, comprising: labeling a to-be-tracked target from an image shot by an unmanned aerial vehicle, the to-be-tracked target serving as a template image, and the image shot by the unmanned aerial vehicle serving as a search image; inputting the template image and the search image in a current shot video frame into a passive tracking module to perform feature extraction and realize the interaction of the features of the search image and the features of the template image, and performing passive tracking frame by frame; the tracking result of the passive tracking module is given to an unmanned aerial vehicle control module, the unmanned aerial vehicle control module respectively performs feature extraction on the historical tracking result, the search image of the current frame and the search image of the historical frame, and then performs feature interaction and target state prediction, and through the predicted target state and the current control parameter of the unmanned aerial vehicle, the control parameter of the unmanned aerial vehicle at the next moment is calculated to realize the update of the control parameter of the unmanned aerial vehicle.

[0016] The present application inputs the template image and the search image in the current shot video frame into the passive tracking module to perform feature extraction and realize the interaction of the features of the search image and the features of the template image, focuses on the mutual relationship between the template and the search features, extracts the search features specific to the template, guarantees the tracking performance under the challenge of small target and strong interference, thereby having good passive tracking performance, in addition, the present application realizes passive tracking while automatically adjusting the control parameter of the unmanned aerial vehicle, i.e., the position of the unmanned aerial vehicle, according to the state of the target in real time, realizes continuous tracking, and thereby can realize body tracking.

[0017] Further, the passive tracking module comprises a convolution token embedding module and a multi-interaction Transformer module, and the template image and the search image in the current video frame are used to extract fixed template tokens, dynamic template tokens, mixed template tokens, time sequence template tokens and search tokens by using the convolution token embedding module, and the tokens are integrated into a feature map to input the current layer multi-interaction Transformer module to extract features and realize the interaction between the features of the search image and the template image. The result of the current layer multi-interaction Transformer module is continuously sent to the next layer multi-interaction Transformer module, and the features are enhanced and interacted layer by layer. The output result of the last layer multi-interaction Transformer module is input into the target box prediction head to obtain a predicted target box, which is used as a tracking result for the unmanned aerial vehicle control module.

[0018] Further, the template image and the search image in the current video frame are used to extract fixed template tokens, dynamic template tokens, mixed template tokens, time sequence template tokens and search tokens by using the convolution token embedding module, and the tokens are integrated into a feature map to input the current layer multi-interaction Transformer module to extract features and realize the interaction between the features of the search image and the template image. The result of the current layer multi-interaction Transformer module is continuously sent to the next layer multi-interaction Transformer module, and the features are enhanced and interacted layer by layer. The output result of the last layer multi-interaction Transformer module is input into the target box prediction head to obtain a predicted target box, which is used as a tracking result for the unmanned aerial vehicle control module.

[0019] The template image in the first image of the current video frame, the template image with a reliability exceeding a threshold in the current video frame and the search image in the current video frame are respectively subjected to convolution and flattening operations by using the convolution token embedding module to obtain fixed template tokens, dynamic template tokens and search tokens. In the case of mixed template token initialization, the fixed template tokens are used, and the time sequence template tokens use the mixed template tokens of the previous frame.

[0020] Further, the processing process of the multi-interaction Transformer module is as follows:

[0021] After the multi-interaction Transformer module performs layer normalization on the input feature map, the data dimension is split, and the feature map is split into different tokens. Each token is subjected to convolution operation to obtain a query vector, a key vector and a value vector corresponding to each token. The query vector, the key vector and the value vector of the fixed template token and the dynamic template token are spliced to obtain new query vector, key vector and value vector for attention calculation to obtain the attention calculation result of the template image. The query vectors of the mixed template token and the search token are spliced to obtain a new query vector, the key vectors of all tokens are spliced to obtain a new key vector, and the value vectors of all tokens are spliced to obtain a new value vector. Then, attention calculation is performed to obtain the attention calculation result of the search image. After splicing the attention calculation result of the template image and the attention calculation result of the search image, layer normalization is performed again, and the first multi-layer perceptron outputs the final result.

[0022] Further, the unmanned aerial vehicle control module respectively performs feature extraction on the historical tracking result, the search image of the current frame and the search image of the historical frame, including:

[0023] The historical tracking result is constructed into a trajectory graph, and the trajectory graph is input into a multi-layer GCN and subjected to a pooling operation to obtain a historical feature f T The search image of the current frame is constructed into an environmental relation graph, and the environmental relation graph is input into another multi-layer GCN and subjected to a pooling operation to obtain an environmental relation feature f G The search image of the historical frame is subjected to ResNet-50 convolution feature extraction and then subjected to global average pooling and linear transformation to obtain a visual feature f I The f G and f I are spliced to obtain a fusion feature z0.

[0024] Further, the unmanned aerial vehicle control module performs feature interaction and target state prediction, including:

[0025] A target state prediction network is constructed, and the target state prediction network includes sequentially connected multi-layer attention modules and a second multi-layer perceptron, each layer of the attention modules includes a first self-attention unit, a second self-attention unit, a first cross-attention unit and a second cross-attention unit, the first self-attention unit and the second self-attention unit of the first layer of the attention modules respectively receive the fusion feature z0 and the historical feature f T , the first self-attention unit and the second self-attention unit respectively multiply different weight coefficients on input values to obtain corresponding query vectors, key vectors and value vectors, perform self-attention calculation, the first cross-attention unit takes the query vector of the first self-attention unit as its query vector, takes the key vector and the value vector of the second self-attention unit as its key vector and value vector, and performs cross-attention calculation, the second cross-attention unit takes the query vector of the second self-attention unit as its query vector, takes the key vector and the value vector of the first self-attention unit as its key vector and value vector, and performs cross-attention calculation; the calculation result of the first self-attention unit and the calculation result of the first cross-attention unit are added as the image stream output of the current attention module and transmitted to the next layer of the attention module, the calculation result of the second self-attention unit and the calculation result of the second cross-attention unit are added as the trajectory stream output of the current attention module and transmitted to the next layer of the attention module; the image stream output and the trajectory stream output of the last layer of the attention module are spliced to obtain a target stream output, and the target stream output is subjected to the second multi-layer perceptron to predict the target state at the next moment. The present application establishes a target state prediction network, maintains a tracking chain through active movement of the unmanned aerial vehicle, and compared with the traditional method, does not rely on continuous visual feedback, and the target short-term disappearance will not cause tracking interruption.

[0026] Further, the calculating the control parameter of the UAV comprises:

[0027]

[0028] p t+1 = p t + Δp

[0029] wherein CA represents cross-attention calculation corresponding to the target state, Q p ,K s ,V s are respectively query vector, key vector and value vector corresponding to the cross-attention calculation of the target state, f p is linear projection of the UAV control parameter p t at the current moment and f p = W p p t + b p , W p and b p are respectively first weight and first bias; is the predicted next moment target state; are respectively weight coefficients of Q p ,K ′ ,V s , W Δ and b Δ are respectively second weight and second bias, Δp is control increment, and p t+1 is the control parameter of the UAV at the next moment.

[0030] Further, the UAV body tracking method further comprises a training process:

[0031] parameters of the passive tracking module and the UAV control module are adjusted, the whole model is trained, and a value of a loss function is calculated, the training is stopped when the value of the loss function is minimum, an optimized model is obtained, and the optimized model is used for UAV body tracking; a formula of the loss function is

[0032]

[0033] wherein λ pred , λ ctrl , λ reg are all hyperparameters, Ω(Θ) is a parameter regular term, and are respectively true value target state and ideal UAV pose.

[0034] Further, the UAV body tracking method further comprises an environment self-adapting adjustment process:

[0035] When the adjacent frame brightness change rate > 20% / second, the camera parameters of the unmanned aerial vehicle are adjusted, the exposure compensation is reduced or increased, if the adjacent frame brightness change rate > 20% / second appears continuously, the mean of multiple historical template images is used as an updated template image; when the light is too low or too high, an automatic light enhancement strategy is adopted;

[0036] The obstacle safety distance is set, when the obstacle intrudes into the safety distance, the unmanned aerial vehicle realizes detouring, if the detouring space is insufficient, emergency hovering is triggered and 360° ring scanning is started to detect whether there is a moving obstacle intruding into the safety distance. The present application can track in the light mutation (such as entering and exiting a tunnel) and obstacle interference (such as vehicle penetration) scenes through the above-mentioned environment adaptive adjustment process, and avoids tracking failure.

[0037] The present application also provides a system for executing the above-mentioned unmanned aerial vehicle body tracking method, which comprises a passive tracking module and an unmanned aerial vehicle control module.

[0038] The present application has the following advantages:

[0039] (1) The present application inputs the template image and the search image in the current video frame into the passive tracking module for feature extraction and realizes the interaction of the features of the search image and the template image, pays attention to the mutual relationship between the template and the search feature, extracts the search feature specific to the template, guarantees the tracking performance under the challenge of small target and strong interference, so that the passive tracking performance is good. In addition, the present application realizes passive tracking while automatically adjusting the control parameters of the unmanned aerial vehicle, i.e. the position of the unmanned aerial vehicle, according to the state of the target in real time, realizes continuous tracking, and thus can realize body tracking.

[0040] (2) The present application establishes a target state prediction network, and through the active movement of the unmanned aerial vehicle, the tracking chain is maintained, compared with the traditional method, the tracking is not interrupted by the temporary disappearance of the target.

[0041] (3) The present application can track in the light mutation (such as entering and exiting a tunnel) and obstacle interference (such as vehicle penetration) scenes through the above-mentioned environment adaptive adjustment process, and avoids tracking failure.

[0042] (4) The multi-interaction Transformer module of the application combines convolution operation and Transformer operation, a convolution token embedding module implemented by convolution operation is responsible for extracting local information such as edges, textures and important information, and the Transformer operation is responsible for global feature extraction and template search information interaction tasks. The multi-interaction Transformer module splits the input feature map according to the data dimension after one layer normalization, splits it into different tokens, and performs convolution operation on each token to obtain the query vector, key vector and value vector corresponding to each token, instead of the traditional linear projection operation, which realizes the advantages of preserving the global modeling of the Transformer while enhancing the local feature extraction capability. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A flowchart of the unmanned aerial vehicle body tracking method disclosed in the embodiment of the application;

[0044] Figure 2 An architecture diagram of the unmanned aerial vehicle body tracking system in the unmanned aerial vehicle body tracking method disclosed in the embodiment of the application;

[0045] Figure 3 A multi-interaction Transformer module schematic diagram in the unmanned aerial vehicle body tracking method disclosed in the embodiment of the application;

[0046] Figure 4 An unmanned aerial vehicle control module schematic diagram in the unmanned aerial vehicle body tracking method disclosed in the embodiment of the application. DETAILED DESCRIPTION

[0047] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below in conjunction with the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0048] Embodiment 1

[0049] The tracking tasks of the prior art are all controlled shooting devices (such as monitoring devices, handheld devices and human-controlled unmanned aerial vehicles), which shoot videos by artificial shooting, and realize tracking on these videos, which are called passive tracking. The method designed in the application can track the target in the video frame (passive tracking method), and automatically adjust the position of the unmanned aerial vehicle (unmanned aerial vehicle control module) according to the state of the target in real time, so as to realize continuous tracking, which is called body tracking. Figure 1As shown, the unmanned implement body tracking method provided by the embodiment 1 of the present application comprises: through a ground station software, a target to be tracked is labeled from an image shot by an unmanned aerial vehicle, the unmanned aerial vehicle continuously tracks the target, the target to be tracked is taken as a template image, an image shot by the unmanned aerial vehicle is taken as a search image, the template image and the search image in a current shot video frame are input into a passive tracking module to extract features and realize interaction of features of the search image and the template image, passive tracking is performed frame by frame, a target box is used to mark a position and a size of the target to be tracked as a tracking result of the frame, and the tracking result is saved. The tracking result of the passive tracking module is given to an unmanned aerial vehicle control module, the unmanned aerial vehicle control module respectively extracts features of a historical tracking result, a search image of a current frame and search images of historical frames, then performs feature interaction and target state (for example, a state that the target is blocked, turns, is far away from the unmanned aerial vehicle and the like) prediction, through the predicted target state and a current control parameter of the unmanned aerial vehicle, a control parameter of the unmanned aerial vehicle at a next time is calculated to realize updating of the control parameter of the unmanned aerial vehicle. In terms of hardware, the unmanned aerial vehicle platform of the present application adopts a DJI Mavic 3 Enterprise customized version, including a visual sensor, an edge computing unit and auxiliary sensors. The visual sensor: Hasselblad L2D-20c camera (4 / 3 inch CMOS, effective pixels 20 million), supports RAW format shooting, optical zoom 7 times. Three-axis mechanical stabilization gimbal (pitch axis rotation range ±90°, roll axis ±45°, accuracy ±0.02°), equipped with an active anti-shake algorithm (shaking suppression rate 95%). The edge computing unit: NVIDIA Jetson Xavier NX (6-core ARM v8.2 CPU, 384-core Volta GPU), built-in 32 GB eMMC storage, running Ubuntu 20.04 LTS. Expansion interface: 2x USB 3.1 (connecting IMU and ToF sensors), 1x MIPICSI-2 (camera data input). Auxiliary sensors: six-axis IMU (BMI088, gyroscope range ±2000° / s, noise density 0.015° / s / √Hz). ToF altimeter (VL53L5CX, maximum ranging 4 m, resolution 4x4 area detection). In terms of software, the whole method process is as follows:

[0050] S1, target initialization

[0051] A user frames a target area (at least 50x50 pixels) through a ground station software. If an illumination level <30, a low-illumination enhancement mode is automatically enabled (image gamma correction parameter γ=2.0).

[0052] S2, passive tracking

[0053] As Figure 2 and Figure 3Firstly, the template image and the search image in the current video frame are input into the passive tracking module, which includes a search image and several template images; then, the input image is first passed through the multi-interaction Transformer module to uniformly implement feature extraction and template search information interaction, so that the finally extracted feature effectively integrates the target information; finally, the search feature is input into the target box prediction head to realize the prediction of the target box.

[0054] In order to realize the collaborative optimization of local feature extraction and global modeling, the multi-interaction Transformer module of the method combines the use of convolution operation and Transformer operation, and the convolution token embedding module realized by convolution operation is responsible for extracting local information such as edge, texture and other important information; the Transformer operation is used to realize the global feature extraction and template search information interaction task at the same time. Figure 1 In combination with Figure 2 and Figure 3 , the passive tracking module includes a convolution token embedding module and a multi-layer multi-interaction Transformer module, and the template image and the search image in the current video frame are used to extract fixed template tokens, dynamic template tokens, mixed template tokens, time sequence template tokens and search tokens by using the convolution token embedding module, mainly using the convolution token embedding module to perform convolution and flattening operation on the template image in the first frame of the current video frame, the template image with reliability exceeding the threshold in the current video frame and the search image in the current video frame to obtain fixed template tokens, dynamic template tokens and search tokens, the mixed template tokens use the fixed template tokens in the initialization state, and the time sequence template tokens use the mixed template tokens of the previous frame. The convolution token embedding module includes convolution and flattening operation, specifically, the input is a two-dimensional feature map, first the feature map is convolved to extract local features and reduce dimension, and then the two-dimensional feature map is converted into the token required by the Transformer operation through the flattening operation. The extraction method of each token is as follows:

[0055] Fixed template token: for each video sequence, the template of the first frame is manually labeled, and the quality of the template is usually high. Therefore, the template image in the first frame of the video sequence is used to perform convolution token embedding operation of the convolution token embedding module to obtain the fixed template token.

[0056] Dynamic template token: During the tracking process, the model updates the template image with high reliability according to the strategy in the literature Cui Y, Jiang C, Wang L, et al. Mixformer: End-to-end tracking with iterative mixed attention [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 13608-13618. As a dynamic template, it is integrated into the tracking model to adapt to the appearance changes of the target and improve the robustness of tracking. By performing convolution token embedding operation on the dynamic template, dynamic template token is obtained. It should be noted that in the initial stage of model tracking, there is no stage to generate dynamic template, and fixed template token is used instead of dynamic template token.

[0057] Mixed template token: The mixed template token fuses the search feature. Although the search image contains a large amount of background information, there is some information that can play a role in learning the template, such as similar objects that can enhance the model's ability to distinguish targets. Through the interaction of template information and search information, see the following multi-layer attention module for specific operations, the query of fixed template token and search token is spliced, and through attention operation, mixed template token can be obtained. It should be noted that for the input of the first multi-interaction Transformer module, i.e. before the first multi-interaction operation, the features of the template and the search have not been interacted, and the mixed template token uses the fixed template token instead.

[0058] Timing template token: The mixed template token feature of the previous frame is used as the timing template token, so that in the process of model tracking, not only the template information of the previous frame can be provided, but also the search information of the previous frame can be utilized to enhance the features of the template. Let the current time be t, and the mixed template token of the previous frame be mz t-1 , then the timing template token of the current frame is sz t = mz t-1 .

[0059] After the above tokens are determined, the fixed template token, the dynamic template token, the mixed template token, the timing template token, and the search token are integrated into a feature map input current layer multi-interaction Transformer module for feature extraction and implementation of the interaction between the features of the search image and the features of the template image. The result of the current layer multi-interaction Transformer module is continuously sent to the next layer multi-interaction Transformer module, and the features are enhanced and interacted layer by layer. The output result of the last layer multi-interaction Transformer module is input into a target frame prediction head to obtain a predicted target frame, which is used as a tracking result for a UAV control module. Through the above operation, the multi-layer multi-interaction Transformer module will complete the operation of effectively extracting target information. The target frame prediction head adopts the most commonly used corner prediction head in recent years, as described in the document “Yan B, Peng H, Fu J, et al. Learning spatio-temporal transformer for visual tracking [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 10448-1057.”, which can predict the left top corner and right bottom corner vertices of the target frame position by using only several convolution-normalization- ReLU layers. The expected value can be obtained by calculating the probability distribution of these corner points. The processing process of the multi-interaction Transformer module is as follows:

[0060] In order to make the representation process better adapt to the target to be tracked, the present application realizes the unified modeling of the interaction between the feature extraction and the template search information. With the powerful global modeling capability of Transformer and the flexibility of attention operation, a multi-interaction Transformer module is constructed as shown in the figure. In order to represent concisely, z and x will be used to represent the template and the search area in the following text, and q, k, and v will represent the query (Query), key (Key), and value (Value) required in the attention operation.

[0061] The multi-interaction Transformer module splits the input feature map according to the data dimension after one layer normalization, and splits it into different tokens. Convolution operation is performed on each token to obtain the query vector, key vector, and value vector corresponding to each token, instead of traditional linear projection operation, which preserves the global modeling advantage of Transformer while enhancing the local feature extraction capability. Among them, the convolution operation first reshapes the token into a 2D feature map, and finally maps the mapped features into a one-dimensional vector for subsequent attention operation.

[0062] For the template branch, considering that the quality of the first frame template is usually high, the first frame template is retained throughout the tracking process of the entire sequence, denoted as the fixed template. In addition, only the fixed template cannot effectively cope with the appearance changes of the target, and the template will also be selected from the search frame for updating during the tracking process, denoted as the dynamic template. Considering that the search feature contains a large amount of background information, this module will only act on the template features between the two modalities, and will not be disturbed by irrelevant information in the search feature. Specifically, in order to facilitate subsequent calculation of mixed attention, first, the generated query, key and value are spliced as shown in formula (1); then the attention map of the template branch is calculated using formula (2), and finally a layer normalization and a multi-layer perception are used to obtain enhanced features through a residual connection. That is, the query vector, key vector and value vector of the fixed template token and the dynamic template token are spliced to obtain new query vector, key vector and value vector for attention calculation to obtain the attention calculation result of the template image. The related formulas are as follows:

[0063] q z =Concat(q iz ,q dz ),k z =Concat(k iz ,k dz ),v z =Concat(v iz ,v dz ) (1)

[0064]

[0065] Wherein, q, k, v respectively represent the query vector, the key vector and the value vector, iz represents the fixed template token, dz represents the dynamic template token, Attention represents the attention calculation, d represents the dimension of the feature vector. Softmax represents the Softmax function.

[0066] For the search branch, the module applies the multi-interaction attention mechanism between the search features and various template features. Through this interaction mode, extensive information interaction between the template and the search can be achieved. Specifically, similar to the template branch, the generated query, key, and value are first spliced, and it should be noted that the key and value of the search branch need to be spliced with four template features and one search feature, as shown in formula (3); then the attention map of the search branch is calculated using formula (4), and finally the enhanced features are obtained through a layer normalization and a multi-layer perception through a residual connection. That is, the query vectors of the mixed template tokens and the search tokens are spliced to obtain new query vectors, the key vectors of all tokens are spliced to obtain new key vectors, and the value vectors of all tokens are spliced to obtain new value vectors, and then the attention calculation of the search image is performed to obtain the attention calculation result of the search image; the relevant formulas are as follows:

[0067]

[0068] Wherein, mz represents a mixed template token, x represents a search token, and sz represents a time sequence template token.

[0069] The attention calculation results of the template image and the search image are spliced, and then a layer normalization and a first multi-layer perception are performed to output the final result to the UAV control module. Through the above operations, the enhanced features of the template and the search frame can be obtained, and these features are sent to the next multi-interaction Transformer module for continuous enhancement. In order to enhance the expression ability of the model, the present application stacks multiple multi-interaction Transformer modules to realize progressive interactive feature enhancement and form a novel tracking network model.

[0070] S3, UAV control process

[0071] As shown in Figure 4 , the working process of the UAV control module includes three parts: (1) feature extraction (responsible for extracting high-level representations from the current image frame, environmental relationship graph, and historical trajectory graph), (2) feature interaction and target state prediction (the target state prediction network adopts a double-flow Transformer cross-reasoning mechanism to realize feature fusion and predict the target state information through the fused features), and (3) update of the control parameters of the UAV (the UAV control network generates control instructions for the UAV based on the predicted target state, adjusts the UAV pose in real time, and thus affects the observation of the next frame).

[0072] (1) Feature extraction is to extract features from the historical tracking results, the search image of the current frame, and the search image of the historical frame, specifically including:

[0073] The search image of the historical frame is extracted by ResNet-50 to obtain the convolutional feature F, and then global average pooling and linear transformation are performed to obtain the visual feature f I The formula of the process is: the UAV collects the RGB frame at time t The convolutional feature F is extracted by ResNet-50 I = ResNet50(S t )(F I ∈R h×w×c ), and then global average pooling and linear transformation f I = W I GAP(F I )+b I (f I ∈R d ) are performed to obtain the image-level vector, i.e., the historical feature f I W I and b I are the weight term and the bias term of the historical feature, respectively.

[0074] The search image of the current frame is constructed into an environmental relationship graph, and the environmental relationship graph is input into another multi-layer GCN and is subjected to a pooling operation to obtain the environmental relationship feature f G The formula of the process is: the HGR-Net network recorded in the literature Bi H, Xu C, Shi C, et al. HGR-Net: Hierarchical graph reasoning network for arbitrary shape scene text detection[J]. IEEE Transactions on Image Processing, 2023, 32: 4142-4155. is used to convert the current frame image information S t into a directed graph, the nodes are some objects on the picture, and the edges are the relationships between the objects, to obtain a node set and a directed graph G env is constructed based on geometric adjacency or prior semantics, and the node feature is iteratively updated using an L-layer graph convolutional network (GCN):

[0075] Finally, the environmental relationship feature f is obtained through the readout function. Pool is the pooling operation. W G and b G are the weight and the bias in the environmental relationship feature, respectively.

[0076] The historical tracking results are constructed into a trajectory graph, the trajectory graph is input into a multi-layer GCN and a pooling operation is performed to obtain historical features f T ; the process is formulated as: the tracking outputs of the previous K frames are constructed into a trajectory graph G traj , and a homogeneous GCN is used again to obtain f T = W T pool(G traj )+b T (f T ∈R d ).

[0077] Finally, f G and f I are spliced to obtain a fusion feature z0.

[0078] (2) The process of the UAV control module for feature interaction and target state prediction is as follows:

[0079] Referring back to Figure 3 , a target state prediction network is constructed, which uses a double-flow cross Transformer structure to realize feature enhancement and information interaction. Let the number of layers be N, and each layer includes self-attention (SA) to capture long-range dependencies within the flow, and cross-attention (CA) to realize the interaction of Query / Key Value between two flows. Specifically, the target state prediction network includes a plurality of sequentially connected attention modules and a second multi-layer perceptron, each attention module includes a first self-attention unit, a second self-attention unit, a first cross-attention unit and a second cross-attention unit, and the first self-attention unit and the second self-attention unit of the first attention module respectively receive the fusion feature z0and the historical feature f T , the first self-attention unit and the second self-attention unit multiply the input values by different weight coefficients to obtain corresponding query vectors, key vectors and value vectors, and perform self-attention calculation. The calculation principles of the first self-attention unit and the second self-attention unit are the same, and for any input x∈R n×d , the self-attention is written as Q x = xW Q ,K x = xW K ,V x = xW V .

[0080] The first cross-attention unit takes the query vector of the first self-attention unit as its query vector, takes the key vector and the value vector of the second self-attention unit as its key vector and value vector, and performs cross-attention calculation, and the second cross-attention unit takes the query vector of the second self-attention unit as its query vector, takes the key vector and the value vector of the first self-attention unit as its key vector and value vector, and performs cross-attention calculation; the calculation result of the first self-attention unit and the calculation result of the first cross-attention unit are added as the image stream output of the current attention module to pass into the next layer attention module, and the calculation result of the second self-attention unit and the calculation result of the second cross-attention unit are added as the track stream output of the current attention module to pass into the next layer attention module; the cross-attention is expressed by the formula as The cross-attention and the self-attention result are added to pass into the next layer, which is expressed by the formula as

[0081]

[0082] o (l) represents the image stream output of the lth layer, u (l) represents the track stream output of the lth layer. CA represents cross-attention calculation, and SA represents self-attention calculation. respectively represent the query vector calculated according to the image stream output of the lth layer, the key vector calculated according to the track stream output of the lth layer, and the value vector calculated according to the track stream output of the lth layer.

[0083] The image stream output and the track stream output of the last layer attention module are spliced to obtain the target stream output o (N) , and the target stream output is predicted by the second multi-layer perceptron to obtain the target state of the next time wherein may include target centroid (x, y), size (w, h), and velocity , etc. σ represents the second multi-layer perceptron, W2, b1 and b2 respectively represent the weight and two biases of the target state of the next time.

[0084] (3) The control parameters of the UAV are calculated, including:

[0085] Suppose the current pose parameter of the UAV is p t ∈R r (such as three-dimensional translation + Euler angle), first linear projection f p = W p p t +b p , and then the target state is embedded into the UAV control semantic space by cross-attention

[0086]

[0087] Final control increment p t+1 = p t + Δp

[0088] New pose p t+1 Instant feedback to the flight control system, while injecting the next frame of information flow, forming a closed loop.

[0089] Wherein, CA represents the cross attention calculation corresponding to the target state, Q p ,K s ,V s Respectively, the query vector, the key vector and the value vector corresponding to the cross attention calculation of the target state, f p Is the linear projection of the unmanned aerial vehicle control parameter p t At the current moment and f p =W p p t +b p , W p And b p Respectively, the first weight and the first bias; Is the predicted next moment target state; Respectively, the weight coefficient of Q p ,K s ,V s , W Δ And b Δ Respectively, the second weight and the second bias, Δp is the control increment, p t+1 Is the control parameter of the unmanned aerial vehicle at the next moment.

[0090] As a further improved technical solution, the unmanned aerial vehicle body tracking method further comprises a training process:

[0091] Adjust the parameters of the passive tracking module and the unmanned aerial vehicle control module, train the whole model, calculate the value of the loss function, stop training when the value of the loss function is minimum, obtain the optimized model, and track the unmanned aerial vehicle body using the optimized model;Specifically, the framework is optimized in a supervised learning paradigm, and if And Respectively, the true value target state and the ideal UAV pose, then the loss function

[0092]

[0093] Wherein, Ω(Θ) is a parameter regularization term;λ pred ,λ ctrl ,λ reg Are hyperparameters. Actual training can be carried out on the data set collected by the simulator such as Gazebo / AirSim or real flight, and all network parameters are optimized through back propagation end to end.

[0094] As a further improved technical solution, the unmanned aerial vehicle body tracking method further comprises an environment self-adaptive adjustment process.

[0095] When the adjacent frame brightness change rate is greater than 20% / second, the camera parameters of the unmanned aerial vehicle are adjusted to reduce or increase the exposure compensation, and if the adjacent frame brightness change rate is greater than 20% / second continuously, the mean of a plurality of historical template images is used as an updated template image; when the light is too low or too high, an automatic light enhancement strategy is adopted; the automatic light enhancement strategy belongs to the prior art, that is, when the light is too low or too high, the light is enhanced according to a preset strategy, for example, when the light is lower than a first preset value, the light intensity corresponding to a second preset value is increased, and similarly, when the light is higher than a third preset value, the light intensity corresponding to a third preset value is increased to prevent overexposure and make it impossible to see.

[0096] A safety distance of an obstacle is set, and the safety distance threshold is 5m in the horizontal direction and 3m in the vertical direction. When the obstacle intrudes into the safety distance, the unmanned aerial vehicle realizes detouring, and if the detouring space is insufficient, emergency hovering is triggered and 360° ring scanning is started to detect whether a moving obstacle intrudes into the safety distance.

[0097] The present application provides a guarantee for the effective implementation of body tracking by introducing multi-modal fusion features and a double-flow cross Transformer. The cross attention mechanism ensures the information complementarity between target semantics and UAV self-state, and enables the control network to be optimized synchronously under end-to-end training. Experimental results show that this method realizes stable and accurate target tracking performance in simulation and real flight tests, providing a feasible solution for autonomous aerial robot perception-control integration.

[0098] Through the above technical solutions, the present application is based on a passive tracking module with multiple interactions, uses the powerful representation capability of the Transformer, designs a multi-interaction Transformer module, effectively realizes the information interaction of feature extraction and template search, and improves the robustness of the tracking model. The unmanned aerial vehicle control module designs a multi-modal context encoder (i.e., a feature extraction part), a target state prediction network and an unmanned aerial vehicle control network, uses video sequences and historical tracking result information to predict the target state, generates control instructions for the unmanned aerial vehicle, and adjusts the pose of the unmanned aerial vehicle in real time, thereby realizing body tracking.

[0099] Embodiment 2

[0100] Based on embodiment 1, embodiment 2 of the present application further provides a system for executing the unmanned aerial vehicle body tracking method of embodiment 1, comprising a passive tracking module and an unmanned aerial vehicle control module, wherein the passive tracking module is connected to the unmanned aerial vehicle control module.

[0101] The above examples are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that the technical solutions recorded in the foregoing examples can be modified, or some technical features can be replaced by equivalent features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An unmanned implement body tracking method characterized by, The application relates to a passive tracking method and device based on a UAV (unmanned aerial vehicle) and a UAV control method. The passive tracking module includes a convolution token embedding module and a multi-layer multi-interaction Transformer module, the template image and the search image in the current video frame are respectively extracted by the convolution token embedding module to obtain fixed template tokens, dynamic template tokens, mixed template tokens, time sequence template tokens and search tokens, the tokens are integrated into a feature map, the feature map is input into a current layer multi-interaction Transformer module to extract features and realize interaction between features of the search image and the template image, a result of the current layer multi-interaction Transformer module is continuously input into a next layer multi-interaction Transformer module, features are enhanced and interacted layer by layer, and an output result of the last layer multi-interaction Transformer module is input into a target frame prediction head to obtain a predicted target frame, the predicted target frame is taken as a tracking result and input into the UAV control module. The template image and the search image in the current video frame are respectively extracted by the convolution token embedding module to obtain fixed template tokens, dynamic template tokens, mixed template tokens, time sequence template tokens and search tokens, including:

2. The unmanned vehicle body tracking method of claim 1, wherein, The template image in the first frame of the current video frame, the template image with a reliability higher than a threshold in the current video frame and the search image in the current video frame are respectively subjected to convolution and flattening operations by the convolution token embedding module to obtain fixed template tokens, dynamic template tokens and search tokens, the mixed template tokens use the fixed template tokens in an initialization condition, and the time sequence template tokens use the mixed template tokens of the last frame. The processing process of the multi-interaction Transformer module is:

3. The unmanned vehicle body tracking method of claim 2, wherein, ​ The multi-interaction Transformer module splits the input feature map according to the data dimension after one layer normalization, splits into different tokens, and performs convolution operation on each token to obtain the query vector, key vector and value vector corresponding to each token. The query vector, key vector and value vector of the fixed template token and the dynamic template token are spliced to obtain new query vector, key vector and value vector, and attention calculation is performed to obtain the attention calculation result of the template image; the query vectors of the mixed template token and the search token are spliced to obtain a new query vector, the key vectors of all tokens are spliced to obtain a new key vector, and the value vectors of all tokens are spliced to obtain a new value vector, and then attention calculation is performed to obtain the attention calculation result of the search image; the attention calculation result of the template image and the attention calculation result of the search image are spliced, and then one layer normalization and the first multi-layer perception are performed to output the final result.

4. The unmanned vehicle body tracking method of claim 1, wherein, The UAV control module extracts features from the historical tracking results, the search image of the current frame, and the search image of the historical frame, including: Historical tracking results are constructed into a trajectory map, which is then input into a multi-layer GCN and pooled to obtain historical features. The search image of the current frame is used to construct an environment relation graph. This environment relation graph is then input into another multi-layer GCN and pooled to obtain the environment relation features. The search images of historical frames are processed by ResNet-50 to extract convolutional features, which are then subjected to global average pooling and linear transformation to obtain visual features. ;Will and The splicing results in fusion features .

5. The unmanned vehicle body tracking method of claim 4, wherein, The UAV control module performs feature interaction and target state prediction, including: The target state prediction network comprises sequentially connected multiple attention modules and a second multilayer perceptron, each attention module comprises a first self-attention unit, a second self-attention unit, a first cross-attention unit and a second cross-attention unit, the first self-attention unit and the second self-attention unit of the first attention module respectively receive the fusion features and the historical features , the first self-attention unit and the second self-attention unit respectively multiply different weight coefficients with input values to obtain corresponding query vectors, key vectors and value vectors, perform self-attention calculation, the first cross-attention unit takes the query vector of the first self-attention unit as its query vector, takes the key vector and the value vector of the second self-attention unit as its key vector and value vector, and performs cross-attention calculation, the second cross-attention unit takes the query vector of the second self-attention unit as its query vector, takes the key vector and the value vector of the first self-attention unit as its key vector and value vector, and performs cross-attention calculation; the calculation result of the first self-attention unit and the calculation result of the first cross-attention unit are added as the image stream output of the current attention module and are transmitted to the next layer of attention module, the calculation result of the second self-attention unit and the calculation result of the second cross-attention unit are added as the track stream output of the current attention module and are transmitted to the next layer of attention module; the image stream output and the track stream output of the last layer of attention module are spliced to obtain a target stream output, and the target stream output is predicted by the second multilayer perceptron to obtain a next time target state.

6. The unmanned vehicle body tracking method of claim 5, wherein, The UAV control module calculates the control parameters of the UAV, including: ， wherein, denotes a cross-attention computation corresponding to the target state, are respectively a query vector, a key vector and a value vector corresponding to the cross-attention computation of the target state, is a linear projection of the current time UAV control parameter and , and are respectively a first weight and a first bias; is a predicted next time target state; , , are respectively weight coefficients of and are respectively a second weight and a second bias, is a control increment, is a next time UAV control parameter.

7. The unmanned vehicle body tracking method of claim 6, wherein, It also includes a training process: Adjust the parameters of the passive tracking module and the UAV control module, train the entire model, calculate the value of the loss function, stop training when the value of the loss function is the smallest, obtain the optimized model, and use the optimized model for UAV body tracking; The formula of the loss function is wherein, , , are hyperparameters, is a parameter regularizer, are the true target state and ideal UAV pose, respectively.

8. The unmanned vehicle body tracking method of claim 1, wherein, It also includes an environment adaptive adjustment process: When the adjacent frame brightness change rate > 20% / second, adjust the camera parameters of the UAV, reduce or increase the exposure compensation, if the adjacent frame brightness change rate > 20% / second appears continuously, use the mean value of multiple historical template images as the updated template image; when the light is too low or too high, use the automatic light enhancement strategy; Set the obstacle safety distance, when the obstacle invades the safety distance, the UAV realizes detour, if the detour space is insufficient, trigger emergency hovering and start 360° ring scanning to detect whether there is a moving obstacle invading the safety distance.

9. A system for performing the unmanned vehicle body tracking method of any one of claims 1-8, wherein, It includes a passive tracking module and a UAV control module, and the passive tracking module is connected with the UAV control module.

Images