Aerial target tracking method and device, medium and product

Abstract

The invention discloses an aerial target tracking method and device, a medium and a product, and relates to the technical field of target tracking, and the method comprises the steps: obtaining a video sequence of a tracking target from a preset historical moment to a current moment; extracting initial feature information; acquiring position information of a tracking target; cutting the tracking frame image by taking the position information of the tracking target as a center to obtain a search area; extracting visual feature information of the search area; determining the historical motion information and the visual feature information as first input information; determining the initial feature information and the visual feature information as second input information; inputting the first input information and the second input information into a self-attention mechanism neural network model to obtain tracking frame position information of the tracking target; and traversing all tracking frame images to obtain the position information of the tracking target at the next moment. The method can improve the tracking accuracy of the aerial target, and solves the tracking problem under the condition that the aerial target is shielded.

Description

Technical Field

[0001] The present invention relates to the technical field of target tracking, and in particular to an aerial target tracking method, device, medium and product. Background Art

[0002] In this digital age, drones have become a powerful tool for urban management and services, and drone technology provides new opportunities and possibilities for the construction and operation of smart cities. However, the large-scale commercial and civilian use of drones will inevitably bring about problems such as illegal and private drone flights. It is necessary to efficiently supervise and effectively manage drones, and adopt innovative technical and regulatory measures to prevent the abuse of drones.

[0003] Therefore, conducting research on the precise tracking of aerial targets represented by drones provides support for the formulation of comprehensive and effective drone regulatory policies, while promoting the innovation of related technologies to balance the rational application of drone technology and the needs of social security.

[0004] Most of the current mainstream trackers use a standard detection and tracking framework to independently detect each captured frame to track the target. Among these trackers, correlation filter-based trackers (DCF) are widely used in aerial platforms due to their high processing efficiency and low demand for computing resources. However, when the target moves quickly, changes in appearance, or is occluded, the performance of these correlation filter-based trackers will be greatly reduced.

[0005] In recent years, trackers based on twin neural networks have demonstrated excellent performance in many competitions, and their efficiency is amazing, and they have been widely used. Trackers based on twin neural networks believe that the target will not move dramatically in adjacent frames, and the position of the target in the current frame is located in the local neighborhood of the center point of the previous frame. Therefore, objects are detected and tracked based on the principle of locality. Only when the target is lost due to occlusion, violent movement, camera shake, etc., will it switch to a global search method to detect and re-associate the target. This alternating use of local and global search strategies is widely adopted because it can effectively save computing resources and improve tracking efficiency, meeting the needs of high-performance tracking in complex scenarios.

[0006] However, when the tracking context is transferred to the air, the tracking difficulty is greatly increased because the aerial targets have more complex motion trajectories and faster moving speeds, and are not restricted by driving rules such as "lanes". In addition, possible obstructions such as clouds, trees, and buildings, as well as unfavorable factors such as lighting changes and camera shake, make tracking of aerial targets particularly difficult, especially when it comes to relocating the target after it is lost. The global search method that performs well in ground tracking tasks may fail in the air due to sudden changes in the target's motion direction and speed. The target is likely to escape the global search range, resulting in tracking failure. Summary of the invention

[0007] The purpose of the present invention is to provide an aerial target tracking method, device, medium and product, which can improve the accuracy of aerial target tracking.

[0008] To achieve the above object, the present invention provides the following solutions: An aerial target tracking method, the method comprising: Acquire a video sequence of the tracking target from a preset historical moment to a current moment; the video sequence includes a plurality of frames arranged in chronological order; the plurality of frames include a first frame and a plurality of tracking frame images starting from a second frame; Extracting target feature information of the first frame of image as initial feature information; Acquiring the position information of the tracking target in the first frame image as the position information of the tracking target at the 0th iteration; Let tracking frame number i=1; Taking the position information of the tracking target at the i-1th iteration as the center, cropping the i-th tracking frame image to obtain the i-th search area; Extracting visual feature information of the i-th search area; Determining historical motion information of the tracking target according to the historical frame image before the i-th tracking frame image; the historical motion information includes historical running speed, historical motion acceleration and historical motion direction; Determining the historical motion information and the visual feature information as first input information at the i-th iteration; Determining the initial feature information and the visual feature information as second input information at the i-th iteration; Inputting the first input information at the i-th iteration and the second input information at the i-th iteration into the self-attention mechanism neural network model, obtaining the i+1-th tracking frame position information of the tracking target, and taking the i+1-th tracking frame position information of the tracking target as the center of the position information of the tracking target at the i-th iteration; The tracking frame number is increased by 1 and the process returns to step "taking the position information of the tracking target at the i-1th iteration as the center, cropping the i-th tracking frame image to obtain the i-th search area" until all tracking frame images are traversed to obtain the position information of the tracking target at the next moment.

[0009] Optionally, a Transformer backbone network is applied to extract target feature information of the first frame image and visual feature information of the i-th search area.

[0010] Optionally, the self-attention mechanism neural network model includes a first interaction module, a second interaction module and a prediction module; In the i-th iteration, the first input information is used as the input of the first interaction module; the output of the first interaction module is connected to the input of the prediction module; the output of the first interaction module is the interaction result of the motion information and the local visual feature; In the i-th iteration, the second input information is used as the input of the second interaction module; the output of the second interaction module is connected to the input of the prediction module; the output of the second interaction module is the interaction result between the local visual feature and the template visual feature; The output of the prediction module is the position information of the tracking target at the next moment.

[0011] Optionally, the first interaction module includes a splicing module, a first two-dimensional convolution module, a multi-head self-attention module and a multi-head cross self-attention module; The motion information of the tracked target is used as the input of the splicing module; the output of the splicing module is connected to the input of the multi-head self-attention module; the output of the multi-head self-attention module is connected to the input of the multi-head cross-self-attention module; the visual feature information of the i-th search area is used as the input of the first two-dimensional convolution module; the output of the first two-dimensional convolution module is connected to the input of the multi-head cross-self-attention module; the output of the multi-head cross-self-attention module is the interaction result of motion information and local visual features.

[0012] Optionally, the second interaction module includes a second two-dimensional convolution module, a third two-dimensional convolution module, a first masked multi-head attention module, a first residual and normalization module, a second masked multi-head attention module, a second residual and normalization module, a first feedforward neural network module, and a third residual and normalization module; The target feature information of the first frame is used as the input of the third two-dimensional convolution module; the output of the third two-dimensional convolution module is respectively connected to the input of the first masked multi-head attention module and the input of the first residual and standardization module; the output of the first masked multi-head attention module is connected to the input of the first residual and standardization module; the output of the first residual and standardization module is respectively connected to the input of the second masked multi-head attention module and the input of the second residual and standardization module; the output of the second masked multi-head attention module is connected to the input of the second residual and standardization module; the output of the second residual and standardization module is respectively connected to the input of the first feedforward neural network module and the input of the third residual and standardization module; the output of the first feedforward neural network module is connected to the input of the third residual and standardization module; the output of the third residual and standardization module is the interaction result of local visual features and template visual features; the visual feature information of the i-th search area is used as the input of the second two-dimensional convolution module; the output of the second two-dimensional convolution module is connected to the input of the second masked multi-head attention module.

[0013] Optionally, the prediction module includes a third masked multi-head attention module, a fourth residual and normalization module, a fourth masked multi-head attention module, a fifth residual and normalization module, a second feedforward neural network module, a sixth residual and normalization module, a fully connected layer and a multi-layer perceptron; The interaction result of the motion information and the local visual features is used as the input of the fourth masked multi-head attention module; The interaction results of the local visual features and the template visual features are respectively used as the input of the third masked multi-head attention module and the fourth residual and standardization module; the output of the third masked multi-head attention module is used as the input of the fourth residual and standardization module; the output of the fourth residual and standardization module is respectively used as the input of the fourth masked multi-head attention module and the input of the fifth residual and standardization module; the output of the fourth masked multi-head attention module is connected to the input of the fifth residual and standardization module; the output of the fifth residual and standardization module is connected to the input of the second feedforward neural network module; the output of the second feedforward neural network module is connected to the input of the sixth residual and standardization module; the output of the sixth residual and standardization module is connected to the input of the fully connected layer; the output of the fully connected layer is connected to the input of the multi-layer perceptron; the output of the multi-layer perceptron is the position information of the tracked target at the next moment.

[0014] Optionally, determining the historical motion information of the tracking target according to the historical frame image before the i-th tracking frame image specifically includes: Acquire a historical frame image before the i-th tracking frame image; when i is greater than a preset frame number of the historical frame images, the historical frame images are arranged in chronological order and the i-1-th tracking frame image is the last frame image; the number of the historical frame images is the preset frame number; when i is less than or equal to the preset frame number, the i-1-th tracking frame image is the historical frame image; When i is greater than a preset number of frames of the historical frame images, determining historical motion information of the tracked target according to position information of the tracked target in each frame of the historical frame images; When i is less than or equal to the preset frame number, the historical motion information of the tracking target is determined according to the position information of the i-1th tracking frame image.

[0015] A computer device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any one of the above-mentioned aerial target tracking methods.

[0016] A computer-readable storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of any of the above-mentioned aerial target tracking methods are implemented.

[0017] A computer program product includes a computer program, characterized in that when the computer program is executed by a processor, the steps of any of the above-mentioned aerial target tracking methods are implemented.

[0018] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects: The present invention provides an aerial target tracking method, device, medium and product, which designs a complete aerial target tracking framework, and predicts the future motion trajectory of the object by making full use of the visual feature information of the target, the historical motion information of the target and the search background information. Because aerial targets represented by drones are usually highly maneuverable, the concept of motion factor is introduced so that the prediction results can better fit objects with different motion speeds and have better robustness. In addition, the prediction results are used as auxiliary factors in the search process, and the prediction results are used to further narrow the search range. Compared with a single locality principle, the search area cut out according to the prediction results is not only smaller in scope, but also can overcome the problem of drastic changes in the motion of aerial targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A diagram showing the interaction structure of motion information and visual information provided in Example 1 of the present invention; Figure 1 Part (a) is a schematic diagram of the structure of the first interactive module; Figure 1 Part (b) is a schematic diagram of the structure of the second interaction module; Figure 2 It is a schematic diagram of the structure of the prediction module; Figure 3 It is the overall framework diagram of the system of the present invention; Figure 4 This is a flow chart of the aerial target tracking method of the present invention; Figure 5 This is a diagram of the internal structure of a computer device. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The purpose of the present invention is to provide an aerial target tracking method, device, medium and product, which can improve the accuracy of aerial target tracking.

[0023] The present invention improves the accuracy of aerial target tracking by adding a trajectory prediction method during the target tracking process. The idea of ​​adding prediction to the target tracking task has been around for a long time, but most prediction methods are to add a predictor based on a Kalman filter after the tracker, and predict the next possible motion trajectory of the object by mathematically modeling the motion information of the object. This method is effective, but the rich and easily accessible visual feature information of the tracker is ignored, including the appearance information of the object and the environmental information that plays a key role in affecting the motion trajectory of the object. The present invention designs a complete aerial target tracking framework, which predicts the future motion trajectory of the object by making full use of the visual feature information of the target, the historical motion information of the target, and the search background information. At the same time, considering the problem of computational efficiency and the timeliness of motion information, the observation history is reduced from a long time series to three frames, and the target trajectory is predicted only based on the effective information captured in the past three frames.

[0024] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Example 1 like Figures 1 to 4 As shown, the aerial target tracking method in this embodiment includes: Step S1: Acquire a video sequence of the tracking target from a preset historical moment to the current moment; the video sequence includes multiple frame images arranged in chronological order; the multiple frame images include a first frame image and multiple tracking frame images starting from the second frame image.

[0026] Step S2: extracting target feature information of the first frame image as initial feature information.

[0027] Specifically, the backbone network of Transformer is used to extract target feature information of the first frame image.

[0028] Step S3: acquiring the position information of the tracking target in the first frame image as the position information of the tracking target at the 0th iteration.

[0029] Step S4: Set the tracking frame number i=1.

[0030] Step S5: taking the position information of the tracking target at the i-1th iteration as the center, cropping the i-th tracking frame image to obtain the i-th search area.

[0031] Step S6: extracting visual feature information of the i-th search area.

[0032] Specifically, the backbone network of Transformer is applied to extract the visual feature information of the i-th search area.

[0033] Step S7: determining the historical motion information of the tracking target according to the historical frame image before the i-th tracking frame image; the historical motion information includes the historical running speed, the historical motion acceleration and the historical motion direction.

[0034] S7 specifically includes: Step S71: Acquire the historical frame image before the i-th tracking frame image; when i is greater than the preset frame number of the historical frame images, the historical frame images are arranged in chronological order and the i-1th tracking frame image is the last frame image; the number of the historical frame images is the preset frame number; when i is less than or equal to the preset frame number, the i-1th tracking frame image is the historical frame image.

[0035] Step S72: when i is greater than a preset number of frames of the historical frame images, determining historical motion information of the tracking target according to position information of the tracking target in each frame of the historical frame images.

[0036] Step S73: when i is less than or equal to the preset frame number, determining the historical motion information of the tracking target according to the position information of the i-1th tracking frame image.

[0037] Step S8: Determine the historical motion information and the visual feature information as the first input information at the i-th iteration.

[0038] Step S9: Determine the initial feature information and the visual feature information as the second input information in the i-th iteration.

[0039] Step S10: Input the first input information at the i-th iteration and the second input information at the i-th iteration into the self-attention mechanism neural network model to obtain the i+1-th tracking frame position information of the tracking target, and use the i+1-th tracking frame position information of the tracking target as the center of the position information of the tracking target at the i-th iteration.

[0040] Specifically, the self-attention mechanism neural network model includes a first interaction module, a second interaction module and a prediction module.

[0041] In the i-th iteration, the first input information is used as the input of the first interaction module; the output of the first interaction module is connected to the input of the prediction module; and the output of the first interaction module is the interaction result of motion information and local visual features.

[0042] In the i-th iteration, the second input information is used as the input of the second interaction module; the output of the second interaction module is connected to the input of the prediction module; and the output of the second interaction module is the interaction result between the local visual features and the template visual features.

[0043] The output of the prediction module is the position information of the tracking target at the next moment.

[0044] Among them, the first interaction module includes a splicing module, a first two-dimensional convolution module, a multi-head self-attention module and a multi-head cross-self-attention module.

[0045] The motion information of the tracked target is used as the input of the splicing module; the output of the splicing module is connected to the input of the multi-head self-attention module; the output of the multi-head self-attention module is connected to the input of the multi-head cross-self-attention module; the visual feature information of the i-th search area is used as the input of the first two-dimensional convolution module; the output of the first two-dimensional convolution module is connected to the input of the multi-head cross-self-attention module; the output of the multi-head cross-self-attention module is the interaction result of motion information and local visual features.

[0046] Among them, the second interaction module includes a second two-dimensional convolution module, a third two-dimensional convolution module, a first masked multi-head attention module, a first residual and normalization module, a second masked multi-head attention module, a second residual and normalization module, a first feedforward neural network module and a third residual and normalization module.

[0047] The target feature information of the first frame is used as the input of the third two-dimensional convolution module; the output of the third two-dimensional convolution module is respectively connected to the input of the first masked multi-head attention module and the input of the first residual and standardization module; the output of the first masked multi-head attention module is connected to the input of the first residual and standardization module; the output of the first residual and standardization module is respectively connected to the input of the second masked multi-head attention module and the input of the second residual and standardization module; the output of the second masked multi-head attention module is connected to the input of the second residual and standardization module; the output of the second residual and standardization module is respectively connected to the input of the first feedforward neural network module and the input of the third residual and standardization module; the output of the first feedforward neural network module is connected to the input of the third residual and standardization module; the output of the third residual and standardization module is the interaction result of local visual features and template visual features; the visual feature information of the i-th search area is used as the input of the second two-dimensional convolution module; the output of the second two-dimensional convolution module is connected to the input of the second masked multi-head attention module.

[0048] Among them, the prediction module includes a third masked multi-head attention module, a fourth residual and normalization module, a fourth masked multi-head attention module, a fifth residual and normalization module, a second feedforward neural network module, a sixth residual and normalization module, a fully connected layer and a multi-layer perceptron.

[0049] The interaction result of the motion information and the local visual features is used as the input of the fourth masked multi-head attention module.

[0050] The interaction results of the local visual features and the template visual features are respectively used as the input of the third masked multi-head attention module and the fourth residual and standardization module; the output of the third masked multi-head attention module is used as the input of the fourth residual and standardization module; the output of the fourth residual and standardization module is respectively used as the input of the fourth masked multi-head attention module and the input of the fifth residual and standardization module; the output of the fourth masked multi-head attention module is connected to the input of the fifth residual and standardization module; the output of the fifth residual and standardization module is connected to the input of the second feedforward neural network module; the output of the second feedforward neural network module is connected to the input of the sixth residual and standardization module; the output of the sixth residual and standardization module is connected to the input of the fully connected layer; the output of the fully connected layer is connected to the input of the multi-layer perceptron; the output of the multi-layer perceptron is the position information of the tracked target at the next moment.

[0051] Step S11: increase the tracking frame number by 1 and return to step S5 until all tracking frame images are traversed to obtain the position information of the tracking target at the next moment.

[0052] As a specific implementation, N is 3. The first tracking frame is cropped with the previous moment position information as the center to obtain the first search area. The target feature information of the first frame and the visual feature information of the first search area are extracted. The first input information and the second input information are input into the self-attention mechanism neural network model to obtain the first tracking frame position information of the tracking target; the first input information is the motion information of the tracking target and the visual feature information of the first search area; the second input information is the target feature information of the first frame and the visual feature information of the first search area.

[0053] The second tracking frame is cropped with the first tracking frame position information as the center to obtain the second search area. The target feature information of the first frame and the visual feature information of the second search area are extracted. The third input information and the fourth input information are input into the self-attention mechanism neural network model to obtain the second tracking frame position information of the tracking target; the third input information is the motion information of the tracking target and the visual feature information of the second search area; the fourth input information is the target feature information of the first frame and the visual feature information of the second search area.

[0054] The third tracking frame is cropped with the second tracking frame position information as the center to obtain the third search area; the target feature information of the first frame and the visual feature information of the third search area are extracted. The fifth input information and the sixth input information are input into the self-attention mechanism neural network model to obtain the third tracking frame position information of the tracking target; the fifth input information is the motion information of the tracking target and the visual feature information of the third search area; the sixth input information is the target feature information of the first frame and the visual feature information of the third search area.

[0055] The third tracking frame position information of the tracking target is used as the next moment position information of the tracking target.

[0056] In the present invention, the first two-dimensional convolution, the second two-dimensional convolution and the third two-dimensional convolution are two-dimensional convolutions of the same structure. The first masked multi-head attention module, the second masked multi-head attention module, the third masked multi-head attention module and the fourth masked multi-head attention module are all masked multi-head attention modules of the same structure. The first feedforward neural network and the second feedforward neural network are both feedforward neural networks with consistent structures. The first residual and normalization module, the second residual and normalization module, the third residual and normalization module, the fourth residual and normalization module, the fifth residual and normalization module and the sixth residual and normalization module are all residual and normalization modules of the same structure.

[0057] The visual feature information of the i-th search area is the predicted search area feature of the corresponding tracking frame. The target feature information of the first frame is the template feature.

[0058] In practical applications, based on the aerial target tracking method disclosed in the present invention, in order to achieve the same function, an aerial target tracking system is provided, including: a Transformer backbone, a tracking module, a motion module and a prediction module.

[0059] In the prior art, the predictor based on filter operation cannot be learned, so it neither utilizes the existing large-scale data sets nor the large amount of visual feature information obtained during the detection process. Different from the prior art, the tracking framework MPAT provided by the present invention integrates historical motion information and visual feature information, and realizes more powerful and accurate tracking and prediction in an end-to-end manner.

[0060] Specifically, the video sequence captured by the camera is sent to the backbone network based on Transformer. By extracting features from the images in the video sequence frame by frame, the visual feature information of the video sequence is obtained, including the target feature information Z (Template Feature) in the first frame and the visual feature information X (Search Feature) of the subsequent tracking frame. Each frame contains the object to be tracked and other background information. The first frame of the image will specify which object to track (by giving the coordinates of the object), so the visual feature information in the first frame is called the target feature information. The other frames after the first frame will not give the coordinates of the object to be tracked, and it is necessary to find the object specified in the first frame, so the frames after the first frame are collectively called tracking frames. The target feature information is the feature information of the object to be tracked extracted from the first frame. It is very specific and only contains the feature information of the object to be tracked. As for the visual feature information of the tracking frame, because we do not know the specific position of the object to be tracked in the tracking frame, we cannot extract only the feature information of the target like the first frame, but extract a range of visual feature information.

[0061] The Tracker Model uses the predicted location information output by the Prediction Module , crop the tracking frame, and track the target based on the target feature information Z (Template Feature) obtained from the first frame and the prediction search feature PSF (Prediction Search Feature) obtained with the predicted coordinates as the center point, that is, classify and regress the target based on the prediction search feature. Classification refers to identifying the target, such as identifying a drone, and regression refers to locating the target.

[0062] Specifically, assuming that the size of a frame image is 512×512, the prediction module will output the position coordinates (x, y) of the target to be tracked at the predicted moment. The tracking module will crop the image (512×512) based on the predicted position coordinates (x, y) with (x, y) as the center point to obtain a slice (equivalent to cutting out a part of an image). The cut part (which may be 224×224 in size, which is smaller than the original 512×512) will be used for feature extraction, and the tracking target will be found in this slice. The PSF obtained after feature extraction of this slice is called Prediction Search Feature because it is cropped according to the predicted position coordinates.

[0063] The motion model extracts the historical motion information of the target based on the past K frames (in this invention, K is set to 3 by default), including the target's motion speed, motion acceleration, motion direction, etc. Considering the high maneuverability of aerial targets, the timeliness of their motion information in a continuous time period is very short, so the historical observation time is shortened from a long time series to 3 frames.

[0064] The prediction module (Prediction Model) interacts the motion information obtained from the motion module (Motion Model) with the visual feature information of the current frame to learn the impact of the local environment on the target's motion situation; it interacts the template features obtained from Backbone with the visual information of the current frame to deal with possible posture changes of aerial targets. Both are used as input to predict the motion trajectory of the target at future moments. Among them, the first frame is special because the first frame will define the tracking target. All other frames can be called tracking frames. The current frame is the frame at the current moment in the tracking frame. Because the first frame will define the tracking target, only the features of the tracking target are extracted in the first frame and this feature is used as a template.

[0065] Current mainstream trackers generally believe that the tracked target will not move dramatically in adjacent frames of the video. Therefore, during the tracking process, the tracking frame is usually cropped using the localization principle, and only the local neighborhood of the target center point in the previous frame is searched and tracked. Therefore, during the tracking process, the tracking frame is usually cropped using the locality principle, and only the local neighborhood of the target center point in the previous frame is searched and tracked. However, the high maneuverability and flexibility of aerial targets, as well as the complexity of the aerial environment, make these targets prone to occlusion, loss of vision, or direct escape from the local neighborhood. In such cases, the local search strategy may cause the tracker to fail completely.

[0066] Based on this, in order to reduce the impact of target loss, especially when the target is severely obstructed or out of the field of view. The present invention readjusts the cropping of the search area according to the output results of the predictor (Prediction Module). In the present invention, the observed visual feature information and the historical motion information of the target are comprehensively utilized to jointly predict the position coordinates of the target at time point t. In view of the "observation deformation" that may be caused by changes in the flight angle of the aerial target, the prediction results in this invention are based on the center point of the target's location.

[0067] Formally, the output position center point coordinates are formulated as: R=(X1,Y1),(X2,Y2),…,(X T , Y T ).

[0068] Where T is the length of the prediction time range, (X i , Y i ) represent the x- and y-coordinates of the target at time i, respectively.

[0069] According to the target center point coordinates (X i , Y i ), and the initialization template size (INSTANCE_SIZE), re-crop the search area, and perform local detection and tracking. The initialization template size is a hyperparameter that can be specified according to the size of the tracked target.

[0070] (1) Local detection centered on prediction results successfully overcomes the challenges brought by the complex motion trajectory of aerial targets, thereby significantly improving the accuracy of target tracking under occlusion.

[0071] The historical motion state of the target is an important factor in predicting the target's future trajectory. However, most aerial targets, represented by drones, are highly maneuverable and flexible, and can quickly change their motion state, such as flight speed, flight angle, and flight altitude, in a short period of time. Therefore, the present invention believes that the timeliness of the motion information of aerial targets is very short. Based on this, the present invention attempts to reduce the observed historical video frames from a long time series to three frames. The motion state of the target, including speed, is observed from these three frames. , acceleration , direction of movement Etc. Figure 1 As shown, these motion information can be easily calculated by the following formula.

[0072] The calculation formula for the running speed of the tracked target is as follows: (2) where, is the running speed of the tracked target; is the displacement of the tracked target on the X-axis within the time interval of k frames; is the displacement of the tracked target on the Y-axis within the time interval of k frames; is the frame rate.

[0073] The calculation formula for the motion acceleration of the tracked target is as follows: (3) where, is the motion acceleration of the tracked target; is the frame rate; represents the average motion speed of the tracked target from i - k + 1 to the intermediate moment j; represents

[0074] The calculation formula for the motion direction is as follows:

[0075] where, atan2() represents the function for calculating the azimuth angle.

[0076] If i < k, that is, there are not enough historical frames to obtain historical motion information, the position information of the target in the last frame can be used for filling.

[0077] Specifically, k represents the kth historical frame. i < k means that there are not enough k historical frames in the front, so the ith frame is the last frame. For example, in the present invention, it is set to predict the position of the next frame based on the motion information and visual information of the previous three frames. Then, when there are only two historical frames and not enough three frames, the second historical frame is the last frame, and the second frame is used for filling to make up three frames.

[0078] Furthermore, the last frame filled with the position information of the target in the last frame is the last frame among the obtained k historical frames. For example, when k = 1, that is, there is only one historical frame, but the present invention requires three historical frames, then the position information of this one historical frame is repeated three times.

[0079] The present invention proposes a prediction module (PredictionModule) based on end-to-end design and integrating multimodal data. In addition to the historical motion state of the target, the motion trajectory of the aerial target will also be affected by aerial environmental factors, just as the lanes, obstacles, etc. on the ground affect the motion trajectory of the vehicle. In particular, the local environment around the target directly affects the target's action trajectory and motion state. Therefore, in the present invention, after the motion information of the target is spliced ​​and encoded, it interacts with the visual feature information (PSF) obtained after Backbone feature extraction of the prediction search area to learn the impact of local environmental factors on the target's motion trajectory. The interaction between the two can be formulated as: (4) (5) Among them, splicing is cat (m i ,…,m i-k ), encoding is Encoder (cat (m i ,…,m i-k )), MHCA and MHSA represent Multi-head Cross-Attention module and Multi-Head Self-Attention module respectively. In the present invention, the historical motion information extracted from the Motion Module is first subjected to Cat and Encoder operations, and then cascaded to map the historical motion information into a high-dimensional space to obtain data-richer features and reduce data sparsity. Multi-Head Self-Attention (MHSA) helps the prediction model capture long-distance dependencies when processing sequential motion information, considers the correlation between the motion information of objects at different times, and allows the prediction model to simultaneously focus on the motion information of different parts and learn the weights between them. Multi-Head Cross-Attention (MHCA) establishes a global connection between motion information and visual feature information of the search area, improving the interactive fusion effect and the generalization ability of the model. At the same time, taking into account the possible posture changes of the tracking target during the motion process, as well as the influence of factors such as lighting and camera shake, the local search features of the current frame are used in the study. , interacts with the template feature Z given from the initial frame to help predict. In order to improve efficiency and avoid network complexity, the visual feature information extracted by the backbone network is directly used: Template Feature (TF) and Prediction Search Feature (PSF). The implementation details are as follows Figure 1 As shown in part (b) of , the specific interaction formula can be expressed as: (6) Among them, TS is the interaction result of visual features; TF is the template feature.

[0080] The interaction results of historical motion information and local search features , the interaction result of the same visual feature And the prediction step length t are used as the input of the Prediction Module. The prediction module structure is as follows Figure 2 As shown in the figure, the Prediction Module consists of N layers, each of which obtains two types of interactions: Motion-PSF and PSF-TF. The prediction module consists of N stacked Motion-PSF and TF-PSF interactions. The overall structure is built using a standard Transformer Decoder. The final output is processed by MLP to obtain the prediction result.

[0081] It is difficult to directly predict the trajectory coordinates of the target at the future moment, and the generalization ability in different test sets and real environments is difficult to achieve the expected goal. In order to solve this problem, the concept of motion factor is introduced. Different from directly predicting the future trajectory coordinates of the aerial target, the output of the predictor P is defined as the relative offset value of the position coordinates based on the average moving speed of the target in the past K frames. The specific formula is expressed as: (7) (8) in, Indicates the average motion speed of the target obtained based on the historical motion information of K frames; It represents the trajectory coordinates obtained when the target moves at a constant speed at the predicted time (i+t); Represents element-by-element addition. The output of the prediction module is Do element-by-element addition to get the final prediction result.

[0082] By introducing the concept of motion factor, the present invention enables the predictor to have better generalization ability and perform well in tracking objects with different moving speeds. In addition, the prediction module predicts the multimodal future motion of the tracked target, that is, the motion trajectory of the target in the next X frames. This prediction method generates a single forward-transmitted target trajectory coordinate in the frame, which greatly saves the computational cost of aligning the entire scene with the target coordinate.

[0083] The present invention has the following advantages: (1) By deeply modeling the interaction between visual features and target motion information, a simple, concise and efficient aerial target tracker is proposed.

[0084] (2) The concept of motion factor is introduced in the present invention, so that the tracker of the present invention can better fit objects with different moving speeds.

[0085] (3) The present invention further modifies the local search capability of the tracking phase according to the prediction results, thereby reducing the reasoning time and improving the accuracy.

[0086] (4) The prediction module proposed in the present invention can be easily applied to different trackers in a plug-and-play manner, with significantly improved results.

[0087] Example 2 A computer device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aerial target tracking method in embodiment 1.

[0088] Example 3 A computer-readable storage medium stores a computer program, which implements the steps of the aerial target tracking method in embodiment 1 when executed by a processor.

[0089] Example 4 A computer program product includes a computer program, which implements the steps of the aerial target tracking method in embodiment 1 when executed by a processor.

[0090] Example 5 A computer device, which may be a database, may have an internal structure as shown in Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pending transactions. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the aerial target tracking method in Example 1 is implemented.

[0091] It should be noted that the object information (including but not limited to object device information, object personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the present invention are all information and data authorized by the object or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0092] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided by the present invention may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided by the present invention may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0093] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0094] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for tracking an aerial target, characterized in that: The method comprises: Acquire a video sequence of the tracking target from a preset historical moment to a current moment; the video sequence includes a plurality of frames arranged in chronological order; the plurality of frames include a first frame and a plurality of tracking frame images starting from a second frame; Extracting target feature information of the first frame of image as initial feature information; Acquiring the position information of the tracking target in the first frame image as the position information of the tracking target at the 0th iteration; Let tracking frame number i=1; Taking the position information of the tracking target at the i-1th iteration as the center, cropping the i-th tracking frame image to obtain the i-th search area; Extracting visual feature information of the i-th search area; Determining historical motion information of the tracking target according to the historical frame image before the i-th tracking frame image; the historical motion information includes historical running speed, historical motion acceleration and historical motion direction; Determining the historical motion information and the visual feature information as first input information at the i-th iteration; Determining the initial feature information and the visual feature information as second input information at the i-th iteration; Inputting the first input information at the i-th iteration and the second input information at the i-th iteration into the self-attention mechanism neural network model, obtaining the i+1-th tracking frame position information of the tracking target, and taking the i+1-th tracking frame position information of the tracking target as the center of the position information of the tracking target at the i-th iteration; The tracking frame number is increased by 1 and the process returns to step "taking the position information of the tracking target at the i-1th iteration as the center, cropping the i-th tracking frame image to obtain the i-th search area" until all tracking frame images are traversed to obtain the position information of the tracking target at the next moment.

2. The aerial target tracking method according to claim 1, characterized in that: The Transformer backbone network is applied to extract target feature information of the first frame image and visual feature information of the i-th search area.

3. The aerial target tracking method according to claim 1, characterized in that: The self-attention mechanism neural network model includes a first interaction module, a second interaction module and a prediction module; In the i-th iteration, the first input information is used as the input of the first interaction module; the output of the first interaction module is connected to the input of the prediction module; the output of the first interaction module is the interaction result of the motion information and the local visual feature; In the i-th iteration, the second input information is used as the input of the second interaction module; the output of the second interaction module is connected to the input of the prediction module; the output of the second interaction module is the interaction result between the local visual feature and the template visual feature; The output of the prediction module is the position information of the tracking target at the next moment.

4. The aerial target tracking method according to claim 3, characterized in that: The first interaction module includes a splicing module, a first two-dimensional convolution module, a multi-head self-attention module and a multi-head cross self-attention module; The motion information of the tracked target is used as the input of the splicing module; the output of the splicing module is connected to the input of the multi-head self-attention module; the output of the multi-head self-attention module is connected to the input of the multi-head cross-self-attention module; the visual feature information of the i-th search area is used as the input of the first two-dimensional convolution module; the output of the first two-dimensional convolution module is connected to the input of the multi-head cross-self-attention module; the output of the multi-head cross-self-attention module is the interaction result of motion information and local visual features.

5. The aerial target tracking method according to claim 3, characterized in that: The second interaction module includes a second two-dimensional convolution module, a third two-dimensional convolution module, a first masked multi-head attention module, a first residual and normalization module, a second masked multi-head attention module, a second residual and normalization module, a first feedforward neural network module, and a third residual and normalization module; The target feature information of the first frame is used as the input of the third two-dimensional convolution module; the output of the third two-dimensional convolution module is respectively connected to the input of the first masked multi-head attention module and the input of the first residual and standardization module; the output of the first masked multi-head attention module is connected to the input of the first residual and standardization module; the output of the first residual and standardization module is respectively connected to the input of the second masked multi-head attention module and the input of the second residual and standardization module; the output of the second masked multi-head attention module is connected to the input of the second residual and standardization module; the output of the second residual and standardization module is respectively connected to the input of the first feedforward neural network module and the input of the third residual and standardization module; the output of the first feedforward neural network module is connected to the input of the third residual and standardization module; the output of the third residual and standardization module is the interaction result of local visual features and template visual features; the visual feature information of the i-th search area is used as the input of the second two-dimensional convolution module; the output of the second two-dimensional convolution module is connected to the input of the second masked multi-head attention module.

6. The aerial target tracking method according to claim 3, characterized in that: The prediction module includes a third masked multi-head attention module, a fourth residual and normalization module, a fourth masked multi-head attention module, a fifth residual and normalization module, a second feedforward neural network module, a sixth residual and normalization module, a fully connected layer, and a multi-layer perceptron; The interaction result of the motion information and the local visual features is used as the input of the fourth masked multi-head attention module; The interaction results of the local visual features and the template visual features are respectively used as the input of the third masked multi-head attention module and the fourth residual and standardization module; the output of the third masked multi-head attention module is used as the input of the fourth residual and standardization module; the output of the fourth residual and standardization module is respectively used as the input of the fourth masked multi-head attention module and the input of the fifth residual and standardization module; the output of the fourth masked multi-head attention module is connected to the input of the fifth residual and standardization module; the output of the fifth residual and standardization module is connected to the input of the second feedforward neural network module; the output of the second feedforward neural network module is connected to the input of the sixth residual and standardization module; the output of the sixth residual and standardization module is connected to the input of the fully connected layer; the output of the fully connected layer is connected to the input of the multi-layer perceptron; the output of the multi-layer perceptron is the position information of the tracked target at the next moment.

7. The aerial target tracking method according to claim 1, characterized in that: Determining the historical motion information of the tracking target according to the historical frame image before the i-th tracking frame image specifically includes: Acquire a historical frame image before the i-th tracking frame image; when i is greater than a preset frame number of the historical frame images, the historical frame images are arranged in chronological order and the i-1-th tracking frame image is the last frame image; the number of the historical frame images is the preset frame number; when i is less than or equal to the preset frame number, the i-1-th tracking frame image is the historical frame image; When i is greater than a preset number of frames of the historical frame images, determining historical motion information of the tracked target according to position information of the tracked target in each frame of the historical frame images; When i is less than or equal to the preset frame number, the historical motion information of the tracking target is determined according to the position information of the i-1th tracking frame image.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the aerial target tracking method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the aerial target tracking method described in any one of claims 1 to 7 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the aerial target tracking method described in any one of claims 1 to 7 are implemented.

Images