Moving target tracking method, device, electronic device and storage medium

Abstract

The present invention relates to the field of target recognition technology, and provides a moving target tracking method, device, electronic device, and storage medium. The method comprises: inputting a historical target bounding box sequence of a tracked target into a motion position representation module of a target tracking model to obtain a historical position embedding feature sequence of the tracked target; inputting a historical position embedding feature sequence of the tracked target into a target position prediction module of the target tracking model to obtain a current position embedding feature of the tracked target; inputting a regional image of a current search area into a visual recognition module of the target tracking model to obtain a current visual feature of the tracked target identified by the visual recognition module based on a target template; and inputting the current position embedding feature and current visual feature of the tracked target into a motion perception fusion module of the target tracking model to obtain a current target bounding box of the tracked target. The present invention improves the accuracy of tracking complex moving targets.

Description

Technical Field

[0001] The present invention relates to the field of target recognition technology, and in particular to a moving target tracking method, device, electronic equipment and storage medium. Background Art

[0002] Traditional target tracking algorithms typically involve three key steps: visual representation, target search, and model update. These methods primarily focus on addressing background interference, illumination variations, scale changes, and occlusions. While these algorithms have achieved significant success in target tracking, they are not suitable for tracking complex moving targets on mobile vehicles (e.g., drones, unmanned vehicles, and unmanned ships). Tracking accuracy can significantly decrease when the target undergoes drastic and unpredictable position changes. Summary of the Invention

[0003] The present invention provides a moving target tracking method, device, electronic device and storage medium, which are used to solve the problem in the prior art that the accuracy of target tracking is significantly reduced when the tracked target undergoes drastic and unpredictable position changes.

[0004] The present invention provides a moving target tracking method, comprising the following steps:

[0005] Inputting a historical target bounding box sequence of the tracked target into a motion position representation module of a target tracking model to obtain a historical position embedding feature sequence of the tracked target;

[0006] Inputting the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target;

[0007] Inputting the area image of the current search area into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template;

[0008] Inputting the current position embedding feature and the current visual feature of the tracked target into the motion perception fusion module of the target tracking model to obtain the current target bounding box of the tracked target;

[0009] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0010] According to a moving target tracking method provided by the present invention, the motion position representation module is used to convert the historical target bounding box sequence into the historical position embedded feature sequence based on a radial basis function, wherein the independent variable of the radial basis function is the distance from any point in the regional image of the historical search area to the two corner points of each historical target bounding box.

[0011] According to a moving target tracking method provided by the present invention, the motion position representation module is also used to convert the coordinates of each bounding box in the historical target bounding box sequence into the regional coordinate system of the current search area, and then convert the historical target bounding box sequence after the coordinate conversion into the historical position embedding feature sequence based on the radial basis function.

[0012] According to a moving target tracking method provided by the present invention, the target position prediction module includes: a convolutional encoding module, a gated spatiotemporal attention module and a convolutional decoding module, the convolutional encoding module is connected to the gated spatiotemporal attention module, and the gated spatiotemporal attention module is connected to the convolutional decoding module;

[0013] Inputting the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target, including:

[0014] Embedding the historical position of the tracked target into a feature sequence and inputting it into a convolutional coding module to obtain a historical position coding sequence of the tracked target;

[0015] Inputting the historical position code sequence into the gated spatiotemporal attention module to obtain a spatiotemporally evolved historical position code sequence;

[0016] The historical position coding sequence after temporal and spatial evolution is input into the convolution decoding module to obtain the current position embedding feature of the tracked target.

[0017] According to a moving target tracking method provided by the present invention, the target position prediction module further includes: a convolution projection module, the convolution decoding module is connected to the convolution projection module;

[0018] The convolution projection module is used to estimate a first current target prediction box of the tracked target based on the current position embedding feature of the tracked target during target tracking model training. The first current target prediction box is used to supervise the prediction of the current target bounding box during target tracking model training.

[0019] According to a moving target tracking method provided by the present invention, the motion perception fusion module includes: an addition module, a prediction head and multiple gated fusion modules, and different gated fusion modules have different sampling scales for their respective inputs;

[0020] Inputting the current position embedding feature and the current visual feature of the tracked target into the motion perception fusion module of the target tracking model to obtain the current target bounding box of the tracked target, including:

[0021] Inputting the current position embedding feature and the current visual feature of the tracked target into multiple gated fusion modules, obtaining motion enhancement features after each gated fusion module samples and fuses the current position embedding feature and the current visual feature according to the corresponding sampling scale;

[0022] Inputting motion enhancement features of different scales into the addition module to obtain a position response map of the current target bounding box after the addition module aligns the motion enhancement features of different scales to the maximum sampling scale and adds them;

[0023] The position response map of the current target bounding box is input into the prediction head to obtain the current target bounding box predicted by the prediction head.

[0024] According to a moving target tracking method provided by the present invention, the target tracking model is trained as follows:

[0025] Inputting the sample historical target bounding box sequence and the sample region image into the target tracking model, obtaining a first current target prediction box and a second current target prediction box output by the target tracking model, wherein the first current target prediction box is predicted by a target position prediction module in the target tracking model based on the sample historical target bounding box sequence, and the second current target prediction box is predicted by a motion perception fusion module in the target tracking model based on a fusion of the sample historical target bounding box sequence and the sample region image;

[0026] The first current target prediction box, the second current target prediction box and the label target bounding box are substituted into the loss function. When the loss function converges, the model training is completed.

[0027] The present invention also provides a moving target tracking device, comprising the following units:

[0028] A feature sequence representation unit is configured to input a historical target bounding box sequence of the tracked target into a motion position representation module of a target tracking model to obtain a historical position embedding feature sequence of the tracked target;

[0029] a position feature prediction unit, configured to input a historical position embedding feature sequence of the tracked target into a target position prediction module of the target tracking model to obtain a current position embedding feature of the tracked target;

[0030] a visual feature recognition unit, configured to input an area image of a current search area into a visual recognition module of the target tracking model, and obtain a current visual feature of the tracked target recognized by the visual recognition module based on a target template;

[0031] a feature fusion unit, configured to input the current position embedding feature and the current visual feature of the tracked target into a motion perception fusion module of the target tracking model to obtain a current target bounding box of the tracked target;

[0032] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0033] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the program, the moving target tracking method as described above is implemented.

[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described moving target tracking methods.

[0035] The motion target tracking method, device, electronic device and storage medium provided by the present invention extract the corresponding historical position embedded feature sequence based on the historical target bounding box sequence of the tracked target, predict the current position embedded feature of the tracked target based on the historical position embedded feature sequence, and then fuse and predict the current target bounding box of the tracked target based on the current visual feature and the current position embedded feature. Since the current position embedded feature is predicted based on the historical position embedded feature sequence, that is, the current position embedded feature contains the motion feature information of the tracked target, the current target bounding box of the tracked target can be accurately predicted even in complex motion scenarios of the tracked target, thereby improving the accuracy of tracking complex motion targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 It is a flow chart of the moving target tracking method provided by the present invention.

[0038] Figure 2It is a structural diagram of the target tracking model in the moving target tracking method provided by the present invention.

[0039] Figure 3 It is a specific structural diagram of the motion perception fusion module of the target tracking model in the moving target tracking method provided by the present invention.

[0040] Figure 4 It is a structural diagram of the moving target tracking device provided by the present invention.

[0041] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0043] The moving target tracking method of the embodiment of the present invention is as follows: Figures 1 to 3 As shown, the process includes the following steps S110 to S140.

[0044] Step S110: The historical target bounding box sequence of the tracked target is input into the motion position representation module of the target tracking model to obtain a historical position embedding feature sequence of the tracked target. Specifically, feature extraction is performed on each historical target bounding box in the historical target bounding box sequence to obtain the embedding feature corresponding to each historical target bounding box, thereby forming a historical position embedding feature sequence of the tracked target.

[0045] Specifically, the current moment t The previous n The historical target bounding box at time t is represented as , represents the coordinates of the upper left corner of the target bounding box, Represents the coordinates of the lower right corner of the target bounding box. For each corner point, i.e., the upper left corner and the lower right corner, the embedded features extracted are: and , . and Formation Moment t The previous n Complete historical location embedding features at all times Current time t All before k The historical target bounding box sequence at each moment After the motion position representation module extracts features, the historical position embedding feature sequence of the tracked target is obtained .in, Represents the dimension h × w The real matrix of Represents the dimension as h × w ×2 real matrix, h and w Represents the current time t The previous n Area image of the search area at time X t-n height and width.

[0046] Since the tracked target is in motion, its identified target bounding box will also follow the movement. Therefore, by extracting the embedded features of the historical target bounding box sequence, the motion characteristics of the tracked target can be captured.

[0047] Step S120: Input the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target, that is, predict the current position of the tracked target at the current moment based on the historical position embedding feature sequence of the tracked target. t The current position within the search area is embedded in the feature Since the historical position embedding feature sequence reflects the motion characteristics of the tracked target, this step is equivalent to using the historical motion characteristics of the tracked target to predict its current position.

[0048] Specifically, Input target position prediction module, which is based on Predict the current position embedding feature .

[0049] Step S130: inputting the area image of the current search area into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template.

[0050] The target template is the template image of the target being tracked, that is, it tells the target tracking model to track the target represented by the target template. Of course, the area image of the current search area contains the image of the target being tracked.

[0051] The visual recognition module can be constructed based on the image recognition mechanism of deep learning, and its input is the target template and the moment t An area image of the current search area X t, it should be noted that: the current search area is the time t The region of interest in the original image or video frame that contains the tracked target. X t From the moment t The image is cropped from the original captured image. Since the tracked target is basically around the center of the regional image at the previous moment, the cropping rule is to use the center point of the target bounding box at the previous moment as the center and expand the target bounding box at the previous moment by a fixed multiple (for example: 2 to 5 times) as the cropping range.

[0052] Exemplarily, the visual recognition module includes: a segmentation and embedding module and a hybrid attention module. The segmentation and embedding module is used to segment and extract features from the target template and the original input region image to obtain image embedding features, and the hybrid attention module is used to focus on the target template based on the image embedding features. Z and time t Area image X t Modeling and outputting the current visual features of the tracked target F X .

[0053] Step S140: Input the current position embedding features and current visual features of the tracked target into the motion-aware fusion module of the target tracking model to obtain the current target bounding box of the tracked target. The target tracking model is trained based on a sequence of sample historical target bounding boxes, a sample region image, a sample target template, and the corresponding labeled target bounding boxes. The sample region image is an image containing the tracked target indicated by the sample target template.

[0054] Specifically, the motion-aware fusion module fuses the current visual features and the current position embedded features to predict the current target bounding box of the tracked target. Because the current position embedded features are predicted based on a sequence of historical position embedded features, i.e., they contain information about the motion characteristics of the tracked target, they can accurately predict the current target bounding box of the tracked target even in complex motion scenarios, thereby improving the accuracy of tracking complex moving targets.

[0055] In the motion target tracking method of this embodiment, the corresponding historical position embedded feature sequence is extracted based on the historical target bounding box sequence of the tracked target, the current position embedded feature of the tracked target is predicted based on the historical position embedded feature sequence, and then the current visual feature and the current position embedded feature are fused and the current target bounding box of the tracked target is predicted. Since the current position embedded feature is predicted based on the historical position embedded feature sequence, that is, the current position embedded feature contains the motion feature information of the tracked target, the current target bounding box of the tracked target can be accurately predicted even in complex motion scenarios of the tracked target, thereby improving the accuracy of tracking complex motion targets.

[0056] In some embodiments, the motion position representation module is used to convert the historical target bounding box sequence into the historical position embedding feature sequence based on a radial basis function, wherein the independent variable of the radial basis function is the distance from any point in the regional image of the historical search area to the two corner points (for example, the upper left corner and the lower right corner) of each historical target bounding box.

[0057] Specifically, the conversion formula of the historical position embedding feature in the historical position embedding feature sequence is as follows:

[0058] (1).

[0059] in, Embedded feature representing the upper left corner of any historical target bounding box or the embedded feature in the lower right corner , An image of the historical search area at any historical moment in the coordinate system X t-n The coordinates of any point in Represents the coordinates of any corner point of the historical target bounding box in the coordinate system of the historical search area at the same historical moment. |||| represents the distance between the two points.

[0060] It is a radial basis function. Inspired by the inter-frame displacement distribution, in this embodiment, Gaussian distribution can be used as the initial weight of the radial basis function to provide powerful prior information.

[0061] In this embodiment, the independent variable of the radial basis function is the regional image of the historical search area X t-n The distance from any point to the two corner points of each historical target bounding box , radial basis function The formula is as follows:

[0062] (2).

[0063] in, represents a piecewise matrix, is the maximum range of the segment, is the total number of segments, c Indicates the c segments, Indicates time t The previous n The historical position embedding feature corresponding to any corner point of the historical target bounding box at the moment. In formula (2), ReLU is used to correct the value in each segment and exclude values ​​exceeding the maximum range. By utilizing point convolution, the response of each coordinate in the historical search area follows the radial basis function in the formula , (1×1× ) represents the point convolution weight, Represents the convolution operation, since is a one-dimensional vector, so the convolution operation in formula (2) is a point convolution.

[0064] In order to ensure that the input and output of the motion position representation module are distributed in a unified coordinate system, in some embodiments, the motion position representation module is also used to convert the coordinates of each bounding box in the historical target bounding box sequence to the regional coordinate system of the current search area, and then convert the historical target bounding box sequence after coordinate conversion into the historical position embedding feature sequence based on the radial basis function.

[0065] Specifically, the coordinate transformation formula is as follows:

[0066] (3).

[0067] and After coordinate transformation, the above formula (1) is transformed into the following formula (4):

[0068] (4).

[0069] in, Indicates Convert the coordinates to the regional coordinate system of the current search area. express Convert the coordinates to the regional coordinate system of the current search area. Indicates the coordinates of the center of the current search area in the original image coordinate system. Indicates the coordinates of the center of the current search area in the regional coordinate system. and is the scaling factor, which is determined by the scaling size of the cropped area in the original image.

[0070] In this embodiment, by converting the coordinates of each bounding box in the historical target bounding box sequence into the regional coordinate system of the current search area, the input and output of the motion position representation module are distributed in a unified coordinate system, which is more conducive to the training and reasoning of the entire model, making the final target tracking model prediction result more accurate and achieving more precise tracking.

[0071] like Figure 2 As shown, in some embodiments, the target position prediction module includes: a convolutional coding module, a gated spatiotemporal attention module ( Figure 2 The convolutional encoding module is connected to the gated spatiotemporal attention module, and the gated spatiotemporal attention module is connected to the convolutional decoding module.

[0072] Based on the above structure of the target position prediction module, step S120 specifically includes:

[0073] The historical position of the tracked target is embedded in a feature sequence and input into a convolutional coding module to obtain a historical position coding sequence of the tracked target.

[0074] The historical position coding sequence is input into the gated spatiotemporal attention module to obtain the historical position coding sequence after spatiotemporal evolution.

[0075] The historical position coding sequence after temporal and spatial evolution is input into the convolution decoding module to obtain the current position embedding feature of the tracked target.

[0076] Specifically, the historical position of the tracked target is embedded in the feature sequence Input convolutional coding module, convolutional coding module Perform spatial encoding to obtain a historical position coding sequence. After the historical position coding sequence is input into the gated spatiotemporal attention module, the historical position coding sequence is spatiotemporally evolved through channel-scale point convolution and spatial-scale dilated convolution to obtain the spatiotemporally evolved historical position coding sequence. Channel-scale point convolution can associate the features of all moments in the historical position coding sequence, and spatial-scale dilated convolution performs long-distance spatial association by expanding the receptive field, which enables the gated spatiotemporal attention module to capture the evolution laws of long time series and the entire region, thereby improving prediction accuracy. The spatiotemporally evolved historical position coding sequence is input into the convolutional decoding module for spatial decoding and reconstruction of the current position embedding features. .

[0077] In some embodiments, the target position prediction module further includes: a convolution projection module ( Figure 2 (not shown in the figure), the convolution decoding module is connected to the convolution projection module.

[0078] The convolution projection module is used to estimate the first current target prediction frame of the tracked target based on the current position embedding feature of the tracked target during target tracking model training, that is, to perform Figure 2 The feature is embedded by the current position To the first current target prediction box In the process, the first current target prediction box Used to supervise the current target bounding box during target tracking model training predictions.

[0079] Specifically, if Figure 2 As shown, in the target position prediction module, a convolutional projection module is connected after the convolutional decoding module as the prediction head ( Figure 2 Not shown), used to embed features according to the current position Predict the first current target prediction box , which can provide richer supervision signals for the autoregressive task during training and prompt the target position prediction module to learn the potential change rules of the motion trajectory of the tracked target, thereby more accurately predicting the current position embedding features of the tracked target .

[0080] like Figure 3 As shown, in some embodiments, the motion-aware fusion module includes: an addition module, a prediction head and multiple gated fusion modules, the multiple gated fusion modules are all connected to the addition module, the addition module is connected to the prediction head, and different gated fusion modules have different sampling scales for their respective inputs.

[0081] Based on the specific structure of the above motion perception fusion module, step S140 specifically includes:

[0082] The current position embedding features and current visual features of the tracked target are input into multiple gated fusion modules to obtain motion enhancement features after each gated fusion module samples and fuses the current position embedding features and current visual features according to the corresponding sampling scale.

[0083] The motion enhancement features of different scales are input into the addition module to obtain a position response map of the current target bounding box after the addition module aligns the motion enhancement features of different scales to the maximum sampling scale and adds them.

[0084] The position response map of the current target bounding box is input into the prediction head to obtain the current target bounding box predicted by the prediction head.

[0085] Specifically, the current position of the tracked target is embedded into the feature and current visual features F XMultiple gated fusion modules are input, and multiple gated fusion modules embed features of the current position in parallel. and current visual features F X Perform fusion processing to obtain multiple motion enhancement features , each motion enhancement feature The corresponding gated fusion module embeds features of the current position according to the corresponding sampling scale and current visual features F X After sampling and fusion.

[0086] For example: three gated fusion modules, the first gated fusion module directly embeds features at the current position and current visual features F X For fusion, the second gated fusion module embeds features at the current position and current visual features F X After double upsampling, the fusion is performed. The third gated fusion module embeds the features of the current position. and current visual features F X Four times upsampling is performed before fusion. In order to obtain a clear image, the current position is embedded with features during upsampling. and current visual features F X Interpolation and projection are performed separately. Due to the different scales of visual features and spatial distribution ( Figure 3 in ) are complementary, compared with the spatial feature enhancement at a single scale, the motion enhancement feature obtained in this embodiment More accurate.

[0087] In each gated fusion module, the following formula (5) uses the gating mechanism to adaptively control the information flow and select the most valuable prior probability part. The current position embeds the feature and visual features Project them separately and add them together to obtain the motion enhancement features of gated control :

[0088] (5).

[0089] in, represents the Hadamard product, Represents the gated projection layer, which is used to project the input into the gated signal space using The function acts as an activation layer to generate the gating signal. and In order to ensure that visual features dominate the template search, Added a residual branch , thus ensuring the training stability of the gated fusion module.

[0090] In the addition module, the motion enhancement features output by multiple gated fusion modules with lower sampling scales are amplified to align them with the motion enhancement features of the maximum sampling scale, and then the motion enhancement features are added together to obtain the position response map of the current target bounding box. In this embodiment, the position response map of the current target bounding box corresponding to the maximum scale is Based on this, the coordinates of the current target bounding box are predicted to achieve higher positioning accuracy.

[0091] To get the position response graph Predict the tracking result, that is, predict the current target bounding box , the prediction head is based on the position response map Perform probabilistic positioning, and the probability weights are divided into two categories: the probability weight of the upper left corner point and the probability weight of the lower right corner point , use the following formula to calculate the weighted sum of the coordinates of each corner point to obtain the coordinates of the corner points of the current target bounding box:

[0092] (6);

[0093] (7).

[0094] in Represents the upper left corner of the target bounding box. Similarly, represents the lower right corner of the target bounding box, H and W Position response plots The height and width, Represents the position response map in the regional coordinate system of the current search area The coordinates in .

[0095] In some embodiments, the target tracking model is trained as follows:

[0096] The sample historical target bounding box sequence and the sample area image are input into the target tracking model to obtain a first current target prediction box and a second current target prediction box output by the target tracking model, wherein the first current target prediction box is predicted by the target position prediction module in the target tracking model based on the sample historical target bounding box sequence, and the second current target prediction box is predicted by the motion perception fusion module in the target tracking model based on the fusion of the sample historical target bounding box sequence and the sample area image.

[0097] The first current target prediction box, the second current target prediction box and the label target bounding box are substituted into the loss function. When the loss function converges, the model training is completed.

[0098] Specifically, during the training phase, it is necessary to train the motion position representation module, target position prediction module, visual recognition module, and motion perception fusion module simultaneously to achieve end-to-end training. Therefore, its loss function is the joint learning loss function L , the joint learning loss function L It consists of the following two parts: the second current target prediction box The corresponding positioning loss function and the first current target prediction box The corresponding prediction loss function .

[0099] (8);

[0100] (9).

[0101] (10).

[0102] in, represents the label target bounding box, 、 and are all different loss weights (for example: The value is 0.1-0.3, and The value ranges from 0 to 1, such as 0.4, and is used to adjust the supervision strength of spatiotemporal prediction. Spatiotemporal prediction is the convolutional projection module in the target position prediction module that predicts the first current target prediction frame of the tracked target based on the current position embedding feature. Specifically, the current position can be embedded into the feature by probabilistic positioning. The first current target prediction box after projection to the position distribution As can be seen from formula (9), the L1 loss function is used L l1 and GIoU loss function L GIoU ; As can be seen from formula (10), the cross entropy loss function is used L ce and SIoU loss function L SIoUSIoU takes into account the overlap, distance, angle, and scale relationship between two bounding boxes, which can provide richer supervision signals for autoregressive tasks and prompt the target position prediction module to learn the underlying change rules of the trajectory. During the training process, the ADAM optimizer is used for model optimization.

[0103] During the inference process, a trajectory update mechanism is also designed, and a multi-convolution layer is designed after the prediction head ( Figure 3 Not shown), using multiple convolutional layers to calculate If the confidence score is lower than the preset score threshold, it means that the prediction result of the prediction head is rejected. , a trajectory extrapolation method is needed to estimate the target position. Experiments have shown that simply adding random offsets to historical observations works better than directly inserting special zero vectors to indicate target disappearance. Similarly, if the search area is at the beginning of the sequence, the same padding scheme is used. Since the center of mass of a highly dynamic tracked target changes significantly more than its size, a random offset is only added to the center of mass of the tracked target, as follows:

[0104] (11);

[0105] (12).

[0106] in and express t The estimated center position (i.e., center of mass) of the tracked target at moment t. 、 、 and express t - n The center position of the tracked object at any moment, the width and height of the target bounding box. Random scaling factor and All follow Gaussian distribution , and their means are and , with a variance of 1.

[0107] The hardware environment, software environment, data set, experimental settings and experimental evaluation indicators of the experiments in the above embodiments are introduced in detail below, and the experimental results are compared with the results of the previous methods.

[0108] 1. Experimental environment: The detailed information of the environment configuration is shown in Table 1.

[0109] Table 1 Experimental environment configuration

[0110]

[0111] 2. Experimental Datasets: The datasets used in the experiment include the general target tracking dataset GOT10K, MODS-T, LMD-TShip, and SeaDroneSee. The sequences of these datasets are mostly captured on mobile carriers, such as unmanned boats and drones, where targets out of view and occlusions are common.

[0112] 3. Experimental settings: There are several hyperparameters that need to be specified in the methods of the above embodiments. Specifically, in the motion position representation module, the maximum range of the segmentation process Defined as ,in is the scale of the output visual features, . is 160. Initial radial basis function is a Gaussian distribution. In the target position prediction module, the last The convolutional encoder and convolutional decoder are composed of 3 convolutional layers with a hidden dimension of 16 and a kernel size of 3. Two gated spatiotemporal attention modules are used with a hidden dimension of 256. For the motion-aware fusion module, the hidden dimension is set to 64, and the hidden dimensions of the gated fusion blocks of 1×1, 2×2, and 4×4 scales are 384, 192, and 96, respectively. In the gated fusion module, is a single convolutional layer, It is a three-layer convolutional network. Perform average pooling and linear projection before generating the signal, using activation. It is also a single convolutional layer.

[0113] During training, the motion position representation module, target position prediction module, visual recognition module, and motion perception fusion module are jointly trained on the general target tracking dataset GOT10K. The model is then supervised fine-tuned on each target tracking dataset for evaluation. For the joint learning loss, the weight of the positioning loss is The weight of the prediction loss is . Using ADAM as the optimizer, the weight decay is When training on GOT10K, the batch size is 48 and the number of training epochs is 100. The learning rate is increased from Descend to During supervised fine-tuning, the batch size is 48. To determine the optimal epoch, an early stopping scheme based on validation loss is used, and most experiments reach the optimal value within about 25 epochs. In fine-tuning, the same learning rate scheduler is used, and the learning rate weight of the visual backbone (i.e., visual recognition module) is , the spatial embedding learning rate weight is 10 to accelerate the optimization of the radial basis function.

[0114] During inference, the target template pool consists of the initial static template and the online template; the update of the online template is adjusted based on whether the confidence score is greater than 0.5. When the maximum update interval is reached, the online template is updated with the target with the highest confidence score.

[0115] 4. Experimental evaluation metrics: Two widely used tracking metrics and one specific target presence classification metric were applied. The two widely used tracking metrics are the area under the success curve (AUC) and precision (P), and the specific target presence classification metric is the F1 score. The success rate is defined as the proportion of predictions whose IoU is greater than a threshold. A curve is plotted against the success rate and threshold, and the area under the curve is used as a metric, named AUC. P measures the proportion of predictions whose center position error is within a specified threshold, indicating the accuracy of target positioning. The threshold can be set to 20 pixels. The F1 score is the harmonic mean of precision and recall, and evaluates the performance of the algorithm. Larger values ​​for all three metrics indicate more accurate target detection.

[0116] Based on the above experimental evaluation indicators, the results obtained by the methods of the above embodiments of the present invention are compared with the performance of currently popular target tracking methods on multiple data sets as shown in Table 2 below.

[0117] Table 2 Experimental results

[0118]

[0119] It can be found from Table 2 above that, no matter which data set and which indicator in the above table, the method of the embodiment of the present invention demonstrates the best target tracking performance for moving targets.

[0120] The moving target tracking device provided by the present invention is described below. The moving target tracking device described below and the moving target tracking method described above can be referenced to each other.

[0121] The moving target tracking device of the embodiment of the present invention is as follows Figure 4 As shown, the following functional units 410 to 440 are included.

[0122] The feature sequence representation unit 410 is configured to input a historical target bounding box sequence of the tracked target into a motion position representation module of the target tracking model to obtain a historical position embedding feature sequence of the tracked target.

[0123] The position feature prediction unit 420 is configured to input the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target.

[0124] The visual feature recognition unit 430 is configured to input the area image of the current search area into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template.

[0125] The feature fusion unit 440 is configured to input the current position embedding feature and the current visual feature of the tracked target into the motion perception fusion module of the target tracking model to obtain the current target bounding box of the tracked target.

[0126] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0127] The motion target tracking device provided by the present invention extracts the corresponding historical position embedded feature sequence based on the historical target bounding box sequence of the tracked target, predicts the current position embedded feature of the tracked target based on the historical position embedded feature sequence, and then fuses and predicts the current target bounding box of the tracked target based on the current visual feature and the current position embedded feature. Since the current position embedded feature is predicted based on the historical position embedded feature sequence, that is, the current position embedded feature contains the motion feature information of the tracked target, the current target bounding box of the tracked target can be accurately predicted even in complex motion scenarios of the tracked target, thereby improving the accuracy of tracking complex motion targets.

[0128] In some embodiments, the motion position representation module is used to convert the historical target bounding box sequence into the historical position embedding feature sequence based on a radial basis function, wherein the independent variable of the radial basis function is the distance from any point in the regional image of the historical search area to the two corner points of each historical target bounding box.

[0129] In some embodiments, the motion position representation module is further used to convert the coordinates of each bounding box in the historical target bounding box sequence into the regional coordinate system of the current search area, and then convert the coordinate-converted historical target bounding box sequence into the historical position embedding feature sequence based on the radial basis function.

[0130] In some embodiments, the target position prediction module includes: a convolutional encoding module, a gated spatiotemporal attention module and a convolutional decoding module, the convolutional encoding module is connected to the gated spatiotemporal attention module, and the gated spatiotemporal attention module is connected to the convolutional decoding module.

[0131] The position feature prediction unit 420 is specifically configured to:

[0132] The historical position of the tracked target is embedded in a feature sequence and input into a convolutional coding module to obtain a historical position coding sequence of the tracked target.

[0133] The historical position coding sequence is input into the gated spatiotemporal attention module to obtain the historical position coding sequence after spatiotemporal evolution.

[0134] The historical position coding sequence after temporal and spatial evolution is input into the convolution decoding module to obtain the current position embedding feature of the tracked target.

[0135] In some embodiments, the target position prediction module further includes: a convolution projection module, and the convolution decoding module is connected to the convolution projection module.

[0136] The convolution projection module is used to estimate a first current target prediction box of the tracked target based on the current position embedding feature of the tracked target during target tracking model training. The first current target prediction box is used to supervise the prediction of the current target bounding box during target tracking model training.

[0137] In some embodiments, the motion-aware fusion module includes: an addition module, a prediction head, and multiple gated fusion modules, and different gated fusion modules have different sampling scales for their respective inputs.

[0138] The feature fusion unit 440 is specifically used for:

[0139] Inputting the current position embedding feature and the current visual feature of the tracked target into multiple gated fusion modules, obtaining motion enhancement features after each gated fusion module samples and fuses the current position embedding feature and the current visual feature according to the corresponding sampling scale;

[0140] Inputting motion enhancement features of different scales into the addition module to obtain a position response map of the current target bounding box after the addition module aligns the motion enhancement features of different scales to the maximum sampling scale and adds them;

[0141] The position response map of the current target bounding box is input into the prediction head to obtain the current target bounding box predicted by the prediction head.

[0142] In some embodiments, the target tracking model is trained as follows:

[0143] The sample historical target bounding box sequence and the sample area image are input into the target tracking model to obtain a first current target prediction box and a second current target prediction box output by the target tracking model, wherein the first current target prediction box is predicted by the target position prediction module in the target tracking model based on the sample historical target bounding box sequence, and the second current target prediction box is predicted by the motion perception fusion module in the target tracking model based on the fusion of the sample historical target bounding box sequence and the sample area image.

[0144] The first current target prediction box, the second current target prediction box and the label target bounding box are substituted into the loss function. When the loss function converges, the model training is completed.

[0145] Figure 5 An example of a physical structure diagram of an electronic device is shown below. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the moving target tracking method, which includes:

[0146] The historical target bounding box sequence of the tracked target is input into the motion position representation module of the target tracking model to obtain the historical position embedding feature sequence of the tracked target.

[0147] The historical position embedding feature sequence of the tracked target is input into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target.

[0148] The area image of the current search area is input into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template.

[0149] The current position embedding feature and the current visual feature of the tracked target are input into a motion perception fusion module of the target tracking model to obtain a current target bounding box of the tracked target.

[0150] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0151] Furthermore, the logic instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0152] On the other hand, the present invention further provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the moving target tracking method provided by the above methods, which includes:

[0153] The historical target bounding box sequence of the tracked target is input into the motion position representation module of the target tracking model to obtain the historical position embedding feature sequence of the tracked target.

[0154] The historical position embedding feature sequence of the tracked target is input into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target.

[0155] The area image of the current search area is input into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template.

[0156] The current position embedding feature and the current visual feature of the tracked target are input into a motion perception fusion module of the target tracking model to obtain a current target bounding box of the tracked target.

[0157] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0158] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the moving target tracking method provided by the above methods, the method comprising:

[0159] The historical target bounding box sequence of the tracked target is input into the motion position representation module of the target tracking model to obtain the historical position embedding feature sequence of the tracked target.

[0160] The historical position embedding feature sequence of the tracked target is input into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target.

[0161] The area image of the current search area is input into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target recognized by the visual recognition module based on the target template.

[0162] The current position embedding feature and the current visual feature of the tracked target are input into a motion perception fusion module of the target tracking model to obtain a current target bounding box of the tracked target.

[0163] The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

[0164] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0165] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A moving target tracking method, characterized in that: include: Inputting a historical target bounding box sequence of a tracked target into a motion position representation module of a target tracking model to obtain a historical position embedding feature sequence of the tracked target, wherein the motion position representation module is used to convert the historical target bounding box sequence into the historical position embedding feature sequence based on a radial basis function; Inputting the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target, the target position prediction module comprising: a convolutional encoding module, a gated spatiotemporal attention module and a convolutional decoding module, the convolutional encoding module being connected to the gated spatiotemporal attention module, and the gated spatiotemporal attention module being connected to the convolutional decoding module; Inputting the regional image of the current search area into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target identified by the visual recognition module based on the target template, the visual recognition module including: a segmentation and embedding module and a hybrid attention module, the segmentation and embedding module is used to segment and extract features from the target template and the original input regional image to obtain image embedding features, and the hybrid attention module is used to model the target template and the regional image of time based on the image embedding features, and output the current visual features of the tracked target; Inputting the current position embedding feature and the current visual feature of the tracked target into a motion-aware fusion module of the target tracking model to obtain a current target bounding box of the tracked target, the motion-aware fusion module comprising: an addition module, a prediction head, and multiple gated fusion modules, wherein different gated fusion modules have different sampling scales for their respective inputs; The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

2. The moving target tracking method according to claim 1, characterized in that: The independent variable of the radial basis function is the distance from any point in the regional image of the historical search area to the two corner points of each historical target boundary box.

3. The moving target tracking method according to claim 2, characterized in that: The motion position representation module is further configured to transform the coordinates of each bounding box in the historical target bounding box sequence into the regional coordinate system of the current search area, and then transform the coordinate-transformed historical target bounding box sequence into the historical position embedding feature sequence based on a radial basis function.

4. The moving target tracking method according to claim 1, wherein: Inputting the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target, including: Embedding the historical position of the tracked target into a feature sequence and inputting it into a convolutional coding module to obtain a historical position coding sequence of the tracked target; Inputting the historical position code sequence into the gated spatiotemporal attention module to obtain a spatiotemporally evolved historical position code sequence; The historical position coding sequence after temporal and spatial evolution is input into the convolution decoding module to obtain the current position embedding feature of the tracked target.

5. The moving target tracking method according to claim 4, characterized in that: The target position prediction module further includes: a convolution projection module, and the convolution decoding module is connected to the convolution projection module; The convolution projection module is used to estimate a first current target prediction box of the tracked target based on the current position embedding feature of the tracked target during target tracking model training. The first current target prediction box is used to supervise the prediction of the current target bounding box during target tracking model training.

6. The moving target tracking method according to claim 1, wherein: Inputting the current position embedding feature and the current visual feature of the tracked target into the motion perception fusion module of the target tracking model to obtain the current target bounding box of the tracked target, including: Inputting the current position embedding feature and the current visual feature of the tracked target into multiple gated fusion modules, obtaining motion enhancement features after each gated fusion module samples and fuses the current position embedding feature and the current visual feature according to the corresponding sampling scale; Inputting motion enhancement features of different scales into the addition module to obtain a position response map of the current target bounding box after the addition module aligns the motion enhancement features of different scales to the maximum sampling scale and adds them; The position response map of the current target bounding box is input into the prediction head to obtain the current target bounding box predicted by the prediction head.

7. The moving target tracking method according to any one of claims 1 to 6, characterized in that: The target tracking model is trained as follows: Inputting the sample historical target bounding box sequence and the sample region image into the target tracking model, obtaining a first current target prediction box and a second current target prediction box output by the target tracking model, wherein the first current target prediction box is predicted by a target position prediction module in the target tracking model based on the sample historical target bounding box sequence, and the second current target prediction box is predicted by a motion perception fusion module in the target tracking model based on a fusion of the sample historical target bounding box sequence and the sample region image; The first current target prediction box, the second current target prediction box and the label target bounding box are substituted into the loss function. When the loss function converges, the model training is completed.

8. A moving target tracking device, characterized in that: include: A feature sequence representation unit is configured to input a historical target bounding box sequence of a tracked target into a motion position representation module of a target tracking model to obtain a historical position embedding feature sequence of the tracked target, wherein the motion position representation module is configured to convert the historical target bounding box sequence into the historical position embedding feature sequence based on a radial basis function; a position feature prediction unit, configured to input the historical position embedding feature sequence of the tracked target into the target position prediction module of the target tracking model to obtain the current position embedding feature of the tracked target, wherein the target position prediction module comprises: a convolutional encoding module, a gated spatiotemporal attention module, and a convolutional decoding module, wherein the convolutional encoding module is connected to the gated spatiotemporal attention module, and the gated spatiotemporal attention module is connected to the convolutional decoding module; A visual feature recognition unit is configured to input the regional image of the current search area into the visual recognition module of the target tracking model to obtain the current visual features of the tracked target identified by the visual recognition module based on the target template. The visual recognition module includes a segmentation and embedding module and a hybrid attention module. The segmentation and embedding module is configured to segment and extract features from the target template and the original input regional image to obtain image embedding features. The hybrid attention module is configured to model the target template and the regional image of time based on the image embedding features and output the current visual features of the tracked target. a feature fusion unit, configured to input the current position embedding feature and the current visual feature of the tracked target into a motion-aware fusion module of a target tracking model to obtain a current target bounding box of the tracked target, wherein the motion-aware fusion module includes an addition module, a prediction head, and multiple gated fusion modules, wherein different gated fusion modules have different sampling scales for their respective inputs; The target tracking model is trained based on a sample history target bounding box sequence, a sample region image, a sample target template and a corresponding label target bounding box.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the moving target tracking method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the moving target tracking method according to any one of claims 1 to 7 is implemented.

Images