A multi-view single-target tracking method and device

Abstract

The application provides a multi-view single target tracking method, comprising the following steps: collecting video data of multiple scenes to obtain a UAV video sequence and a ground video sequence; inputting the two video sequences, and returning tracking results of two views of the past k frames before tracking processing of each frame; performing difference between the tracking results of the two views to obtain a displacement sequence; inputting the displacement sequence into a displacement prediction module to predict a target displacement of a current frame, determining an original search area according to a tracking result of a previous frame to obtain a response score; judging qualities of a UAV search area and a ground search area, and the higher the response score is, the better the quality of the search area is, and vice versa; using the good quality search area and the predicted displacement to calculate an adjusted position of a search area of another view; and inputting the search areas of the two views into a multi-view tracker; and the application also provides a multi-view single target tracking device, which introduces a displacement to ensure that a search area contains a tracking target as much as possible.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-view single-target tracking method and apparatus. Background Technology

[0002] Vision is one of the most important ways for humans to understand the world; 80% of the external information humans acquire comes from the visual system. Computational vision, based on an understanding of human vision, uses an imaging system to replace the human visual organs, and a computer to replace the human brain in processing and understanding input images. Simultaneously, with the development of information technology and intelligent science, computer vision is one of the most popular disciplines in artificial intelligence and a crucial technology in the sensing layer of the Internet of Things (IoT). The research and application of visual target tracking (single-target tracking), as an important branch of computer vision, is receiving increasing attention from academia and industry. Its task is to predict the size and position of a target in subsequent frames given its initial position and size in a video sequence. It is widely used in various fields such as science and technology, national defense, aerospace, medicine and health, and the national economy. Research on target tracking technology has significant practical value and broad development prospects. However, with the development of the visual target field, information from a single perspective is often insufficient for many challenging tracking scenarios, making multi-view collaborative tracking particularly important. Multi-view collaborative tracking uses two different perspectives to track the same target. Compared to single-view tracking, multi-view tracking offers significantly more available information, and the different viewpoints are complementary. Effective fusion of information from different viewpoints can lead to more robust tracking. In the prior art, Chinese invention patent application CN104331901A, entitled "A Multi-View Target Tracking Device and Method Based on TLD," implements multi-view target tracking. Addressing the common problems of target occlusion, ambient lighting changes, and shadow interference in monocular vision methods for moving target detection and tracking, this invention utilizes a multi-view information fusion unit to fuse multi-view information and employs a learning mechanism to enhance the robustness of the detection module's tracking capabilities, thereby improving tracking performance.

[0003] In practical applications, single-view target tracking often encounters difficulties such as low resolution and occlusion, which significantly impact the accuracy of tracking algorithms. Adding another perspective provides more information, offering supplementary information to the current target's perspective and mitigating these challenges. Furthermore, the inherent advantages of each perspective contribute to the feasibility and effectiveness of multi-view tracking. The ground-based view, being closer than the UAV view, provides detailed local information about the tracked target. Conversely, the UAV view, with its wider field of view, provides global motion information. However, associating and fusing complementary information from different perspectives of the same target is a key challenge in multi-view single-target tracking. Previous work utilized the cross-attention mechanism in the transformer architecture to design a multi-view collaborative learning module (MvCL). This module correlates the same target within the search areas from both the UAV and ground perspectives by calculating the appearance similarity between targets in two search regions, fusing complementary visual information. While it achieves significant performance improvements over previous single-view trackers, some limitations remain. For example, when the designed multi-view tracker encounters severe occlusion or light interference, the quality of the search region may be poor, or it may even not contain the tracked target. In this case, when using the collaborative learning module MvCL for target association and fusion, fusing features from a search region that does not contain the tracked target will cause the method to fail or even introduce noise, affecting the subsequent tracking process. Summary of the Invention

[0004] The technical problem to be solved by this invention is how to improve the tracking performance of the same target from multiple perspectives.

[0005] This invention solves the above-mentioned technical problems through the following technical solution: a multi-view single-target tracking method, comprising the following steps:

[0006] S10. The drone and ground-based imaging equipment collect video data from multiple scenes to obtain a drone video sequence U = {U1, U2, ..., U...} t} and ground video sequence G = {G1, G2, ..., G} t};

[0007] S20. Generate the drone video sequence U = {U1, U2, ..., U...} t} and ground video sequence G = {G1, G2, ..., G} t As input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process, and the difference between the tracking results of the two views is used to obtain the offset sequence.

[0008] S30. Input the offset sequence into the offset prediction module to predict the target offset of the current frame, determine the original search area of ​​the two views based on the tracking result of the previous frame, and obtain the response score of the tracking result.

[0009] S40. Determine the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Use the high-quality search area and the predicted offset to calculate the adjusted position of the search area from another perspective.

[0010] S50. Input the search areas from the two perspectives into the multi-view tracker, fuse the information of the target being tracked from the two perspectives, and output the tracking results.

[0011] The tracking method of this invention introduces the geometric relationship of offset between two viewpoints, i.e., the coordinate difference of the tracking results, to ensure that the search area contains the tracked target as much as possible. This method is more effective for locating search areas with better quality and a higher probability of containing the tracked target. Through a search area adjustment strategy, the original search area generated by the model is adjusted, using the position of the high-quality search area and the offset to determine the position of the modified original low-quality search area. Since the high-quality search area is more likely to contain the target, the other search area modified by the offset is also more likely to contain the tracked target, thereby effectively fusing information about the same target from different viewpoints to improve the tracking performance of both viewpoints.

[0012] Preferably, in step S20, when tracking the t-th frame of a sequence, the model returns the tracking results from two perspectives of the k frames preceding the current frame, and records the tracking result set from the UAV's perspective as Xu = [u t-k ;...;u t-2 ;u t-1 The tracking result set of the ground view is recorded as Xg = [g t-k ;...;g t-2 g t-1 ], u t-i and g t-i Represents the tracking state, Xu and Xg ∈ R k×4 k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker will output a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the upper left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The coordinate difference between the center points of the tracking results from the two views is calculated and recorded as the offset sequence.

[0013] Preferably, the coordinates (x, y) of the center point of the two viewpoint tracking results c y c The formula for calculating ) is:

[0014] x c =x + 0.5 × w

[0015] y c = y + 0.5 × h;

[0016] Tracking results center point sequence X′ from two perspectives u and X′ g for:

[0017] X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ]

[0018] X′ g =[g′ t-k ;...;g′ t-2 g′ t-1 ];

[0019] Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i ;

[0020] Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2 .

[0021] Preferably, the method for predicting the target offset of the current frame in step S30 includes:

[0022] S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, and map the features to a higher-dimensional space through the input embedding. Then add it to the positional encoding and feed it into the N-layer stacked encoder.

[0023] The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder.

[0024] S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X dec The value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

[0025] Preferably, after obtaining the two view search regions, the response scores of the two view search regions in the current frame are compared. The search region with the higher response score is of better quality, and its center point is used as the position of the target in the current frame. Based on the center point coordinates of the good quality search region, plus the predicted offset, the adjusted position of the poor quality search region is obtained.

[0026] Preferably, in step S40, when the quality of the ground view search area is poor, the adjustment method is as follows:

[0027] x′ g =x u +Δx

[0028] y′ g =y u +Δy;

[0029] When the quality of the search area from the drone's perspective is poor, the adjustment method is as follows:

[0030] x′ u =x g -Δx

[0031] y′ u =y g -Δy.

[0032] The present invention also provides a multi-view single-target tracking device, comprising:

[0033] The data acquisition module uses the drone and ground-based imaging equipment to collect video data from multiple scenes, obtaining a drone video sequence U = {U1, U2, ..., U...}. t} and ground video sequence G = {G1, G2, ..., G} t};

[0034] The model building module is used to generate a UAV video sequence U = {U1, U2, ..., U...} t} and ground video sequence G = {G1, G2, ..., G} tAs input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process, and the difference between the tracking results of the two views is used to obtain the offset sequence.

[0035] The offset prediction module is used to input the offset sequence into the offset prediction module to predict the target offset of the current frame, determine the original search area of ​​the two viewpoints based on the tracking results of the previous frame, and obtain the response score of the tracking result.

[0036] The search area adjustment module judges the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Using the better quality search area and the predicted offset, the adjusted position of the search area from another perspective is calculated.

[0037] The output module sends the search areas from both perspectives into the multi-view tracker, fuses the information of the tracked target from both perspectives, and outputs the tracking results.

[0038] Preferably, when the model building module is tracking the t-th frame of a sequence, the model will return the tracking results from two perspectives of the k frames preceding the current frame, and record the tracking result set from the UAV's perspective as Xu = [u t-k ;…;u t-2 ;u t-1 The tracking result set of the ground view is recorded as Xg = [g t-k ;...;g t-2 g t-1 ], u t-i and g t-i Represents the tracking state, Xu and Xg ∈ R k×4 k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker will output a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the upper left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The coordinate difference between the center points of the tracking results from the two views is calculated and recorded as the offset sequence.

[0039] Preferably, the coordinates (x, y) of the center point of the two viewpoint tracking results c y c The formula for calculating ) is:

[0040] x c =x + 0.5 × w

[0041] y c = y + 0.5 × h;

[0042] Tracking results center point sequence X′ from two perspectives u and X′g for:

[0043] X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ]

[0044] X′ g =[g′ t-k ;...;g′ t-2 g′ t-1 ];

[0045] Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i ;

[0046] Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2 .

[0047] Preferably, the method by which the offset prediction module predicts the target offset of the current frame includes:

[0048] S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, and map the features to a higher-dimensional space through the input embedding. Then add it to the positional encoding and feed it into the N-layer stacked encoder.

[0049] The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder.

[0050] S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X decThe value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

[0051] The advantages provided by this invention are as follows:

[0052] 1. The tracking method of this invention introduces the geometric relationship of offset between two viewpoints, i.e., the coordinate difference of the tracking results, to ensure that the search area contains the tracking target as much as possible. This method is more effective for locating search areas with better quality and a higher probability of containing the tracking target. The position of the original low-quality search area is determined by the position of the high-quality search area and the offset. Since the high-quality search area is more likely to contain the target, the other search area modified by the offset is also more likely to contain the tracking target, thereby effectively fusing information about the same target from different viewpoints to improve the tracking performance of both viewpoints.

[0053] 2. Compared with the original strategy, the search area obtained by the search area adjustment strategy of the present invention is such that the target being tracked is basically located in the center of the search area, and the target is included. The tracking result and the actual tracking target position basically coincide, and the target is always included. The tracking effect is good and the loss of target is reduced. Attached Figure Description

[0054] Figure 1 A schematic diagram of a model provided for an embodiment of the present invention;

[0055] Figure 2 A schematic diagram of the overall structure of the offset prediction module provided in an embodiment of the present invention;

[0056] Figure 3 A schematic diagram illustrating the visualization experimental results of the multi-view single-target tracking method provided in the embodiments of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0058] This embodiment provides a multi-view single-target tracking method. The multi-view approach in this embodiment is illustrated using the collaborative tracking of the UAV's view and the ground's view as an example, and includes the following steps:

[0059] Step S10: Data Acquisition. Using drones and ground-based imaging equipment, video data is collected from various scenes such as streets, parks, schools, and playgrounds, obtaining a series of paired drone video sequences U = {U1, U2, ..., U...}. t} and ground video sequence G = {G1, G2, ..., G} t}, as input to the model.

[0060] Each input consists of a pair of drone video sequences and ground video sequences, such as U1 and G1, which are fed into the model together. Different pairs of sequences may contain different numbers of frames, but sequences within the same pair contain the same number of frames.

[0061] Step S20: Model building, combined with Figure 1 In the diagram, solid lines represent the execution order of the current frame, while dashed lines represent the tracking results and response scores of the previous frame returned by the model. The upper and lower dashed lines represent the tracking results and response scores of two different viewpoints, respectively. The Offset Prediction Module is used to predict the offset of the current frame, and Compare is used to compare the response scores of the tracking results of the previous frame to determine which search region has high quality and which has low quality in order to adjust the position.

[0062] The drone video sequence U = {U1, U2, ..., U...} is used to represent the drone video sequence. t} and ground video sequence G = {G1, G2, ..., G} t As input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process. The difference between the tracking results of the two views is used to obtain the offset sequence, which reflects the positional deviation of the tracked target in the two views.

[0063] Specifically, when tracking the t-th frame of a sequence, the model returns the tracking results from two perspectives of the k frames preceding the current frame, recording the tracking sequence from the UAV's perspective as Xu = [u t-k ;...;u t-2 ;u t-1 The tracking sequence record for the ground view is Xg = [g t-k ;...;g t-2 g t-1 ], where u t-i and .g t-i Both are 1×4 vectors used to represent the tracking states [x, y, w, h], where Xu and Xg ∈ R. k×4In this context, k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker outputs a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the top-left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The difference in coordinates between the center points of the tracked target from the two views is recorded as the offset. Since the output of the tracked target is a bounding box b = [x, y, w, h], the coordinates (x, y) of the center point of the tracked target are obtained according to the following formula. c y c ):

[0064] x c =x + 0.5 × w

[0065] y c =y + 0.5 × h

[0066] Tracking results center point sequence X′ from two perspectives u and X′ g for:

[0067] X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ]

[0068] X′ g =[g′ t-k ;...;g′ t-2 g′ t-1 ];

[0069] Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i .

[0070] Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2The offset sequence refers to the set of offsets from the previous k frames, which contains k offsets.

[0071] Step S30: Input the offset sequence into the offset prediction module to predict the target offset of the current frame, determine the original search area of ​​the two viewpoints based on the tracking result of the previous frame, and obtain the response score of the tracking result.

[0072] Specifically, the tracking results returned by the model are processed into an offset sequence X. off Then, it is passed as input to the offset prediction module with a transformer architecture, which calculates the predicted current frame offset (Δx, Δy), and combines it with... Figure 2 Specifically, it includes the following steps:

[0073] S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, map the features to a higher-dimensional space through the input embedding, and then add it to the positional encoding and feed it into the N-layer stacked encoder.

[0074] The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder.

[0075] S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X dec The value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

[0076] In order for the transformer to process the input data, it is embedded into a higher D-dimensional space through the linear projection of the weight matrix Wx. For prediction tasks, the sequence order information is very important, as it represents the trend of sequence changes. The position encoding used in this invention is the same as the position encoding provided in the original transformer. In this embodiment, both the N-layer stacked encoder and the N-layer stacked decoder are six-layer stacked.

[0077] Step S40: Determine the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Use the search area with better quality and the predicted offset to calculate the adjusted position of the search area from another perspective.

[0078] Specifically, in UAV-Ground single-target tracking, each multi-view tracker generates two search regions: the UAV view search region and the ground view search region. After obtaining the two view search regions, the response scores of the two view search regions in the current frame are compared. The search region with the higher response score is of better quality, and its center point is used as the approximate position of the target in the current frame. At the same time, the other search region has poorer quality and needs adjustment. Based on the center point coordinates of the better-quality search region, plus the predicted offset, the adjusted position of the poor-quality search region is obtained. Figure 1 The light green search area in the upper branch changes to a light red search area.

[0079] When the quality of the ground-view search area is poor, adjustments should be made according to the following formula:

[0080] x′ g =x u +Δx

[0081] y′ g =y u +Δy

[0082] When the quality of the search area from the drone's perspective is poor, adjust according to the following formula:

[0083] x′ u =x g -Δx

[0084] y′ u =y g -Δy

[0085] Step S50: Input the search areas from both perspectives into the multi-view tracker, fuse the information of the tracked target from both perspectives, and output the tracking result. Specifically, this includes the following steps:

[0086] S51. The feature extraction module extracts features from the search area and the template image.

[0087] S52. Input the template features and search region features from each perspective into the feature enhancement module to perform feature enhancement.

[0088] S53. Input the search region features from the two perspectives into the MvCL module, and use the cross attention module to adaptively fuse the information of the target tracking from the two perspectives.

[0089] S54, Output the tracking results.

[0090] In UAV-to-ground single-target tracking, past methods focused solely on associating useful information from two viewpoints. They employed cross-attention mechanisms to fuse the same target from both perspectives before tracking, adaptively fusing information without explicitly utilizing geometric information between the viewpoints. Effective tracking relied on ensuring the target was present in both search regions for feature fusion to be effective. However, this method suffers from performance degradation, even failing to track the target, when faced with challenging conditions such as severe occlusion or light interference. Furthermore, the original search region generation strategy, which defined a rectangular area centered on the previous frame's tracking result, may still not include the target. Fusing features from a search region lacking the target introduces noise, rendering the method ineffective.

[0091] It should be noted that the search area adjustment of the present invention can also be used in similar dual-lens target tracking combinations such as UAV-UAV or ground-to-ground, as long as the target position offset between lenses is utilized and the offset is predicted by time series.

[0092] The tracking method of this invention introduces the geometric relationship of offset between two viewpoints, i.e., the coordinate difference of the tracking results, to ensure that the search area contains the tracking target as much as possible. This method is more effective for locating search areas with better quality and a higher likelihood of containing the tracking target. Using the coordinate difference of past frames, this invention employs a time-series prediction method to predict the offset of the current frame. The higher the response scores of the two viewpoint tracking results in the previous frame, the better the tracking effect and the easier it is to contain the tracking target. The original search area of ​​the current frame is determined by the tracking results of the previous frame, resulting in two original search areas from different viewpoints: one of high quality and one of low quality. The position of the high-quality search area and its offset are used to determine the position of the modified original low-quality search area. Because the high-quality search area is more likely to contain the target, the other search area modified by the offset is also more likely to contain the tracking target, thus effectively fusing information about the same target from different viewpoints to improve the tracking performance of both viewpoints.

[0093] The following experimental results are listed to verify the effectiveness of the proposed method. Table 1 shows the experimental performance of the tracking method of this invention compared to previous single-target tracking algorithms and multi-view single-target tracking algorithms. It can be seen that the tracking method of this invention outperforms previous algorithms in all six metrics across two viewpoints.

[0094] Table 1

[0095]

[0096] Combination Figure 3Visual experiments were used to verify that the method of this invention is more effective in finding higher-quality search regions. The visualization results show that, compared to the original strategy, the search region obtained by the search region adjustment strategy of this invention almost always places the tracked target in the center of the search region and includes the target. In contrast, the search region of the original strategy is either located at a corner of the search region or does not include the tracked target at all. The tracking results also clearly show that the tracking result of this invention (blue box) almost coincides with the actual tracked target position (red box), always including the tracked target, resulting in better tracking performance and reducing the chance of target loss. The tracking result of the original method (green box) differs significantly from the actual result.

[0097] The present invention also provides a multi-view single-target tracking device, comprising:

[0098] The data acquisition module uses drones and ground-based imaging equipment to collect video data from various scenes such as streets, parks, schools, and playgrounds, obtaining a series of paired drone video sequences U = {U1, U2, ..., U...}. t} and ground video sequence G = {G1, G2, ..., G} t}, as input to the model.

[0099] The model building module takes the UAV video sequence U = {U1, U2, ..., U...} and assigns it to UAVs. t} and ground video sequence G = {G1, G2, ..., G} t As input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process, and the difference between the tracking results of the two views is used to obtain the offset sequence.

[0100] When the model building module tracks the t-th frame of a sequence, the model returns the tracking results from two perspectives of the k frames preceding the current frame. The tracking result set from the UAV's perspective is recorded as Xu = [u t-k ;...;u t-2 ;u t-1 The tracking result set of the ground view is recorded as Xg = [g t-k ;...;g t-2 g t-1 ], u t-i and g t-i Represents the tracking state, Xu and Xg ∈ R k×4 k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker will output a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the upper left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The coordinate difference between the center points of the tracking results from the two views is calculated and recorded as the offset sequence.

[0101] The coordinates (x) of the center point of the tracking results from the two perspectives c y c The formula for calculating ) is:

[0102] x c =x + 0.5 × w

[0103] y c = y + 0.5 × h;

[0104] Tracking results center point sequence X′ from two perspectives u and X′ g for:

[0105] X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ]

[0106] X′ g =[g′ t-k ;...;g′ t-2 g′ t-1 ];

[0107] Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i ;

[0108] Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2 .

[0109] The offset prediction module predicts the target offset in the current frame. Based on the tracking results of the previous frame, it determines the original search areas for both viewpoints and obtains the response score of the tracking result. The methods used by the offset prediction module to predict the target offset in the current frame include:

[0110] S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, and map the features to a higher-dimensional space through the input embedding. Then add it to the positional encoding and feed it into the N-layer stacked encoder.

[0111] The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder.

[0112] S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X dec The value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

[0113] The search area adjustment module judges the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Using the better quality search area and the predicted offset, the module calculates the adjusted position of the search area from another perspective.

[0114] The output module sends the search areas from both perspectives into the multi-view tracker, fuses the information of the tracked target from both perspectives, and outputs the tracking results.

[0115] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-view single-target tracking method, characterized in that: Includes the following steps: S10. The drone and ground-based imaging equipment collect video data from multiple scenes to obtain a drone video sequence U = {U1, U2, ..., U...} t } and ground video sequence G = {G1, G2, ..., G} t }; S20. Generate the drone video sequence U = {U1, U2, ..., U...} t } and ground video sequence G = {G1, G2, ..., G} t As input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process, and the difference between the tracking results of the two views is used to obtain the offset sequence. S30. Input the offset sequence into the offset prediction module to predict the target offset of the current frame, determine the original search area of ​​the two views based on the tracking result of the previous frame, and obtain the response score of the tracking result. S40. Determine the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Use the high-quality search area and the predicted offset to calculate the adjusted position of the search area from another perspective. S50. Input the search areas from the two perspectives into the multi-view tracker, fuse the information of the target being tracked from the two perspectives, and output the tracking results.

2. The multi-view single-target tracking method according to claim 1, characterized in that: In step S20, when tracking the t-th frame of a sequence, the model returns the tracking results from two perspectives of the k frames preceding the current frame, recording the tracking sequence from the UAV's perspective as Xu = [u t-k ... u t-2 ;u t-1 The tracking sequence record for the ground view is Xg = [g t-k ... g t-2 g t-1 ], u t-i and g t-i Represents the tracking state, Xu and Xg ∈ R k×4 k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker will output a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the upper left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The coordinate difference between the center points of the tracking results from the two views is calculated and recorded as the offset sequence.

3. The multi-view single-target tracking method according to claim 2, characterized in that: The coordinates (x) of the center point of the tracking results from the two perspectives c y c The formula for calculating ) is: x c =x+0.5×w and c =y+0.5×h; Tracking results center point sequence X′ from two perspectives u and X′ g for: X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ] X′ g =[g′ t-k ;...;g′ t-2 ;g′ t-1 ]; Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i ; Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2 .

4. The multi-view single-target tracking method according to claim 1, characterized in that: The method for predicting the target offset of the current frame in step S30 includes: S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, and map the features to a higher-dimensional space through the input embedding. Then add it to the positional encoding and feed it into the N-layer stacked encoder. The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder. S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X dec The value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

5. The multi-view single-target tracking method according to claim 4, characterized in that: After obtaining the two view search regions, the response scores of the two view search regions in the current frame are compared. The search region with the higher response score is of better quality, and its center point is used as the position of the target in the current frame. Based on the center point coordinates of the good quality search region, plus the predicted offset, the adjusted position of the poor quality search region is obtained.

6. The multi-view single-target tracking method according to claim 5, characterized in that: When the quality of the search area from the ground view is poor, the adjustment method is as follows: x′ g =x u +Δx y′ g y u +Δy; When the quality of the search area from the drone's perspective is poor, the adjustment method is as follows: x′ u =x g -Δx y′ u y g -Δy.

7. A multi-view single-target tracking device, characterized in that: include: The data acquisition module uses the drone and ground-based imaging equipment to collect video data from multiple scenes, obtaining a drone video sequence U = {U1, U2, ..., U...}. t } and ground video sequence G = {G1, G2, ..., G} t }; The model building module is used to generate a UAV video sequence U = {U1, U2, ..., U...} t } and ground video sequence G = {G1, G2, ..., G} t As input to the model, the model returns the tracking results of two views from the past k frames before each frame tracking process, and the difference between the tracking results of the two views is used to obtain the offset sequence. The offset prediction module is used to input the offset sequence into the offset prediction module to predict the target offset of the current frame, determine the original search area of ​​the two viewpoints based on the tracking results of the previous frame, and obtain the response score of the tracking result. The search area adjustment module judges the quality of the UAV search area and the ground search area based on the response score of the tracking results. The higher the response score, the better the quality of the search area, and vice versa. Using the high-quality search area and the predicted offset, the adjusted position of the search area from another perspective is calculated. The output module sends the search areas from both perspectives into the multi-view tracker, fuses the information of the tracked target from both perspectives, and outputs the tracking results.

8. The multi-view single-target tracking device according to claim 7, characterized in that: When the model building module tracks the t-th frame of a sequence, the model returns the tracking results from two perspectives of the k frames preceding the current frame. The tracking result set from the UAV's perspective is recorded as Xu = [u t-k ... u t-2 ;u t-1 The tracking result set of the ground view is recorded as Xg = [g t-k ;...;g t-2 g t-1 ], u t-i and g t-i Represents the tracking state, Xu and Xg ∈ R k×4 k and t are both natural numbers greater than 1, and i is a natural number from 1 to 1-k. For each frame of tracking results from each tracking viewpoint, the tracker will output a bounding box b = [x, y, w, h] as the tracking state, where (x, y) is the coordinate of the upper left vertex of the bounding box, w is the width of the bounding box, and h is the height of the bounding box. The difference in coordinates between the center points of the tracking results from the two views is recorded as the offset sequence.

9. The multi-view single-target tracking device according to claim 8, characterized in that: The coordinates (x) of the center point of the tracking results from the two perspectives c y c The formula for calculating ) is: x c =x+0.5×w and c =y+0.5×h; Tracking results center point sequence X′ from two perspectives u and X′ g for: X′ u =[u′ t-k ;...;u′ t-2 ;u′ t-1 ] X′ g =[g′ t-k ;...;g′ t-2 ;g′ t-1 ]; Where, u′ t-i and g′ t-i All are 1×2 coordinates (x) c y c ) t-i ; Then, the center point sequence X′ of the tracking results from the two perspectives is... u and X′ g The difference is taken to obtain the offset sequence, denoted as X. off =X′ u -X′ g =[x t-k ;...;x t-2 ;x t-1 ]=[(Δx t-k Δy t-k );...;(Δx t-2 Δy t-2 ); (Δx t-1 Δy t-1 )], X off ∈R k×2 .

10. The multi-view single-target tracking device according to claim 7, characterized in that: The offset prediction module predicts the target offset of the current frame using the following methods: S31. Input the offset sequence into the transformer encoder, perform offset aggregation of the first k frames through the concat operation, and map the features to a higher-dimensional space through the input embedding. Then add it to the positional encoding and feed it into the N-layer stacked encoder. The S32, N-layer stacked encoder performs feature encoding through multi-head self-attention, residual connections, layer normalization, and feedforward networks within the encoder. S33. The encoded feature input is fed into an N-layer stacked decoder. The decoder input is a two-dimensional X-ray. dec X dec The value is set to (0, 0), and the decoder input X is... dec After self-attention calculation, cross-attention calculation is performed with the feature input encoding to obtain the predicted current frame offset (Δx, Δy).

Images