A spatio-temporal non-registration visible-infrared target tracking method and system

Abstract

The application discloses a kind of spatio-temporal non-registration visible light-infrared target tracking method and system, belong to computer vision and embedded artificial intelligence field.To solve the over-reliance of existing RGBT tracking on spatio-temporal registration data, the problem of spatio-temporal asynchrony in the real world cannot be effectively handled, the application proposes a spatio-temporal registration free tracking network (SAFNet).The network takes the "minimum data block" as the basic processing unit, first uses the cross-modal time attention mechanism to solve the time non-registration through the spatio-temporal interaction query module (SIQ), then realizes the spatial alignment based on the response graph through the decision-level spatial guidance fusion module (DSGF), and finally performs high-quality fusion and tracking through the feature-level response awareness fusion module (FRAF).The application realizes the end-to-end integration of spatio-temporal registration, information fusion and target tracking, significantly improves the robustness and real-time performance of asynchronous and non-registration data.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a spatiotemporal unregistered visible light-infrared target tracking method and system. Background Technology

[0002] In practical engineering applications, due to the inherent differences in the physical characteristics and working mechanisms of visible light and infrared imaging sensors (such as the low photoelectric conversion efficiency, low frame rate, and the need for non-uniformity correction during operation of infrared sensors), it is difficult to achieve accurate registration of the acquired visible light and infrared video streams in time and space.

[0003] Temporal misregistration manifests as different frame rates and asynchronous acquisition times, while spatial misregistration manifests as pixel-level positional offsets. Traditional visible-infrared (RGBT) tracking algorithms typically assume that the input pairs of images are strictly spatiotemporally registered, which causes their performance to drop sharply or even fail when faced with real-world spatiotemporally misregistered data.

[0004] Existing methods typically treat spatiotemporal registration, information fusion, and target tracking as independent modules, resulting in complex systems that are difficult to train and have poor real-time performance. Furthermore, they fail to effectively utilize temporal information to enhance tracking robustness and the ability to handle dynamic changes in the target. Therefore, there is an urgent need for an end-to-end tracking framework that can uniformly handle spatiotemporal non-registration, achieve efficient cross-modal fusion, and adaptive temporal modeling. Summary of the Invention

[0005] The purpose of this invention is to address the technical challenges of existing RGBT tracking algorithms, such as over-reliance on spatiotemporal registration data, inability to effectively handle the prevalent spatiotemporal asynchrony in the real world, and difficulty in achieving efficient, real-time tracking. By designing an end-to-end architecture of "spatiotemporal interactive query - decision-guided spatial registration - feature-level response perception fusion," this invention aims to achieve the following objectives: 1) reconstructing the temporal correspondence between visible light and infrared modes at the feature level; 2) using high-level decision information to guide the lower level to achieve adaptive alignment in the spatial dimension; 3) performing high-quality information fusion and target tracking based on features aligned in time and space; and 4) effectively addressing challenges such as target deformation and occlusion by fusing target state information in the temporal dimension through a dynamic template update mechanism.

[0006] The technical solution of the present invention: According to the first aspect of the present invention, a spatiotemporal unregistered visible light-infrared target tracking method is proposed, which adopts a Spatiotemporal Alignment-free Network (SAFNet). The core of this method is to integrate the three major functions of spatiotemporal registration, information fusion and target tracking into a unified network framework. Through a distributed processing strategy of "first temporal alignment, then spatial guidance, and finally feature fusion", robust tracking of spatiotemporal unregistered data is achieved.

[0007] Overall Architecture Design: SAFNet designs a "minimum data block" as the basic processing unit, which contains X visible light images and Y infrared images acquired consecutively within a specific processing cycle. X and Y are determined by the least common multiple of the sampling intervals of the visible light and infrared cameras.

[0008] The network comprises three core modules: Spatiotemporal Interactive Query (SIQ), Decision-Level Spatial Guidance Fusion (DSGF), and Feature-Level Response Awareness Fusion (FRAF), as well as a tracking head with an integrated dynamic template update mechanism.

[0009] The method specifically includes the following steps: Step S1, construct the minimum data block: Based on the least common multiple of the sampling intervals of the visible light camera and the infrared camera, determine the X visible light images and Y infrared images continuously acquired within a processing cycle to form a minimum data block; Step S2, Spatiotemporal Interactive Query: The visible light image and infrared image in the smallest data block are respectively divided into image blocks (the backbone network of this invention uses vision). The transformer architecture divides the input image into 16x16 pixel or 32x32 pixel patches. Typically, the input image is scaled to 224x224 pixels. For example, a 16x16 patch would result in 196 patches, each flattened into a 1x256 vector (the token). Positional encoding and temporal encoding based on the absolute timestamp of image acquisition are embedded to obtain spatiotemporal embedding features. These features are then input into a cross-modal temporal attention layer to calculate the similarity between infrared image patch queries and visible light image patch keys. Weighted by a time interval penalty term, the most temporally matching visible light-infrared image pair is determined. New alignment timestamps are assigned to matching frames, and for unmatched frames, interpolation is performed based on the similarity weights with matched frames to generate supplementary timestamps. Based on the new alignment timestamps and supplementary timestamps, bilinear interpolation is used to output the temporally aligned visible light and infrared search map feature maps and template feature maps. The problem of time misregistration is solved by using the Spatiotemporal Interactive Query (SIQ) module.

[0010] Step S3, Decision-level Spatial Guided Fusion: Perform deep cross-correlation operations on the time-aligned visible light and infrared search map feature maps and template feature maps respectively to generate multi-channel preliminary target response maps for each modality; stitch and reduce the dimensionality of the preliminary target response maps for each modality to generate the final response map for each modality; using the peak point of the infrared final response map as a reference, calculate the coordinate offset between the visible light final response map and the infrared final response map, and crop and fill the visible light final response map according to the offset to generate a spatially aligned corrected response map, and record the cropping and filling parameters; The spatial non-registration problem is solved by using the Decision-Level Spatial Guidance Fusion Module (DSGF).

[0011] Step S4, Feature-level response perception fusion: Using the cropping and padding parameters, spatial correction is performed on the visible light search map feature map to obtain a spatially aligned feature map; the preliminary target response map is normalized and interpolated to the same size as the feature map, and then used as spatial weights to convolve with the spatially aligned feature map of the corresponding modality to enhance the target response region; the maximum value of each channel response map is extracted as the channel weight, and channel attention enhancement is performed on the template feature map; the enhanced visible light and infrared search feature maps and the template feature map are fused respectively to obtain fused search features and fused template features; Refined feature fusion is achieved through the Feature-Level Response Aware Fusion (FRAF) module.

[0012] Step S5, Dynamic Template Update and Tracking: This step first inputs the fused search features and fused template features into the Region Proposal Network (RPN) to regress the final accurate target bounding box. To overcome template degradation, a dynamic update strategy is designed. The network maintains a queue containing several consecutive sets (e.g., 5 sets) of fused templates. The update weights of the initial templates are fixed. For subsequent templates, a lightweight evaluation network (composed of convolutional layers, fully connected layers, and softmax layers) takes the response map of the current template and the search map as input, evaluates its reliability, and outputs update weights. The latest template is generated by weighted fusion of the initial template and subsequent templates according to their respective weights.

[0013] The tracking head outputs the final tracking result through an integrated dynamic template update mechanism.

[0014] In one possible embodiment, in step S2, the time encoding is generated using a cosine position encoding method, which converts the absolute timestamp of image acquisition into a unified time parameter and superimposes it onto the corresponding image patch features.

[0015] In one possible embodiment, in step S2, the time interval penalty term in the cross-modal temporal attention layer includes a learnable parameter for adjusting the weights of the time difference matrix to prevent mismatches in static scenes; the time difference matrix is ​​constructed based on the frame number of the visible light image and the infrared image.

[0016] In one possible embodiment, step S2, timestamp interpolation of unmatched frames includes: calculating the matching weight between unmatched visible light frames and matched infrared frames; generating supplementary timestamps based on the matching weights through linear interpolation and nonlinear correction terms; the nonlinear correction terms being generated by a multilayer perceptron based on statistics of the matching weights.

[0017] In one possible embodiment, in step S3, the deep cross-correlation operation preserves channel semantic information and generates a set of multi-channel preliminary target response maps for each modality; the stitching and dimensionality reduction are achieved through 1x1 convolution.

[0018] In one possible embodiment, step S4, the response map-guided feature enhancement step includes: normalizing and interpolating the multi-channel preliminary response map of each modality, so that its size corresponds to the features of the corresponding search map. Figure 1 The normalized response map is used as the spatial weight and then convolved element-wise with the corresponding search map feature map.

[0019] The specific implementation of this step is as follows: First, the two-dimensional spatial response map output by the previous-level cross-modal fusion module is normalized and upsampled to the same spatial resolution as the current feature map to be enhanced through bilinear interpolation to obtain a spatial attention mask; then, the spatial attention mask is multiplied element-wise with the feature map to be enhanced along the channel dimension to achieve weighted modulation of the feature map at each spatial location in the feature map, so that the feature response of the region corresponding to the response peak is selectively amplified, while the feature of the low response background region is suppressed; finally, the weighted modulated feature map is added element-wise with the original feature map to be enhanced through residual connection to output the target feature representation enhanced by spatial response guidance. This feature representation enhances the feature expression of the target discriminative region while retaining global semantic information.

[0020] In one possible embodiment, step S4, the channel attention fusion step includes: extracting the maximum value of the multi-channel preliminary response map of each modality in each channel to form a channel weight vector; and performing channel-by-channel multiplication of the channel weight vector with the corresponding template feature map to achieve channel attention enhancement.

[0021] The specific implementation of this step is as follows: First, the visible light branch feature map and the infrared branch feature map are concatenated along the channel dimension to obtain a joint feature tensor. Then, a global average pooling operation is performed on the joint feature tensor to compress it into a channel descriptor vector, which is then passed through a bottleneck structure consisting of two fully connected layers (where the first layer reduces the number of channels to 1 / r of the original number of channels, and the second layer restores the original number of channels). The attention weight vector along the channel dimension is then generated by the Sigmoid activation function. Next, the attention weight vector is applied to the visible light branch feature map and the infrared branch feature map respectively by channel-wise scalar multiplication to achieve adaptive channel recalibration of the two heterogeneous features. Finally, the two recalibrated feature maps are added element-wise along the channel dimension to output a cross-modal feature representation with adaptive channel fusion. This representation automatically adjusts the contribution ratio of each modality and each channel according to the dynamic characteristics of the input sample.

[0022] In one possible embodiment, in step S5, the lightweight evaluation network sequentially includes a convolutional layer, a fully connected layer, and a Softmax layer; the update weight of the initial template is a fixed value of 0.5; and the dynamic update strategy maintains a queue containing 5 consecutive sets of fused templates.

[0023] In one possible embodiment, a loss function step is also included: the total loss function is composed of a weighted average of classification loss and regression loss; the classification loss uses cross-entropy loss, the regression loss uses GIoU loss, and the regression loss is calculated separately for the prediction results of visible light and infrared modes.

[0024] In this step, the total loss function consists of a weighted sum of three sub-losses, each corresponding to a different optimization objective during model training: The first is the classification loss, using Focal Loss or cross-entropy loss function, which operates between the class probability distribution output by the classification prediction head and the true class label, to optimize the model's ability to distinguish between targets and background and address the imbalance between positive and negative samples; the second is the bounding box regression loss, using GIoU Loss or CIoU Loss function, which operates between the coordinates of the predicted bounding box and the coordinates of the true labeled box, to optimize target localization accuracy while also considering the consistency of overlapping area, center point distance, and aspect ratio; the third is the centrality loss or auxiliary task loss, using binary cross-entropy loss function, which operates between the output of the centrality prediction branch or a specific auxiliary prediction head and the corresponding ground truth label, to suppress low-quality predicted boxes and improve the calibration accuracy of detection confidence; finally, the three sub-losses are multiplied by preset balanced weight coefficients and linearly summed as the total optimization objective of end-to-end backpropagation, driving the synchronous update of network parameters. According to a second aspect of the present invention, a spatiotemporal unregistered visible-infrared target tracking system is provided for implementing the above-described spatiotemporal unregistered visible-infrared target tracking method, comprising: Data building unit, used to build the smallest data block based on the camera sampling interval; The spatiotemporal interactive query unit is used to segment the visible light image and infrared image in the minimum data block into image blocks, and embed position codes and time codes generated based on the acquisition timestamps to obtain spatiotemporal embedding features. The spatiotemporal embedding features are input into a cross-modal temporal attention layer to calculate the similarity between the infrared image block query and the visible light image block key, and the most temporally matching visible light-infrared image pair is determined by weighting through a time interval penalty term. A new alignment timestamp is assigned to the matching frames, and timestamp interpolation and feature map generation are performed on the unmatched frames. The output is a temporally aligned visible light and infrared search map feature map and a template feature map. The decision-level spatially guided fusion unit is used to perform deep cross-correlation operations on the time-aligned visible light features and infrared features respectively to generate multi-channel preliminary response maps of their respective modes; to stitch and reduce the dimensionality of the preliminary response maps of each mode to generate the final response map of each mode; to calculate the coordinate offset between the visible light response map and the infrared response map using the peak point of the infrared response map as a reference, to crop and fill the visible light response map according to the offset, to generate a spatially aligned corrected response map, and to record the cropping and filling parameters; The feature-level response perception fusion unit is used to perform spatial correction on the visible light search map feature map using the cropping and padding parameters; normalize and interpolate the preliminary response map, and then use it as spatial weights to convolve with the spatially aligned feature map of the corresponding modality to enhance the target response region; extract the maximum value of each channel response map as the channel weight to perform channel attention enhancement on the template features; and fuse the enhanced visible light and infrared search map features and template features respectively to obtain fused search features and fused template features. The dynamic template update and tracking unit is used to input the fused search features and fused template features into the region proposal network to regress the target bounding box; it maintains a latest queue of several sets of fused templates, evaluates the reliability of the current template based on the response map of the search graph through a lightweight evaluation network, and outputs update weights, and fused the initial template with the subsequent templates according to the weights to generate the latest template.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) "Minimum Data Block" and Spatiotemporal Decoupling Processing Paradigm: The first to use the "minimum data block" as the basic processing unit, the SIQ module first solves the temporal non-registration problem at the feature layer, and then the DSGF module solves the spatial non-registration problem based on the response graph, thus realizing effective decoupling and step-by-step processing of the spatiotemporal dimensions, greatly enhancing the adaptability to real-world asynchronous data.

[0026] 2) Decision-level guided spatial feature alignment mechanism: Breaking away from the traditional approach of spatial alignment at the pixel or shallow feature level, the DSGF module first obtains a rough target position at the decision level through deep cross-correlation, and then uses this as a guide to perform cropping / padding-style spatial alignment on the deep feature map. This method is insensitive to registration errors and is computationally efficient.

[0027] 3) Multi-level fusion and dynamic update of response perception: In the FRAF module, the target response map generated by DSGF is used as a natural quality evaluator to dynamically guide feature fusion weights and enhance high-quality modal information. Combined with the dynamic template update strategy based on lightweight network evaluation in the tracking head, dual fusion in time and feature dimensions is achieved, which significantly improves the tracking accuracy and robustness to target changes. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the SAFNet network structure; Figure 2 This is a schematic diagram illustrating the implementation process of CMTA. Detailed Implementation

[0029] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0030] A spatiotemporal unregistered visible-infrared target tracking method includes the following steps: Step S1, construct the minimum data block: Based on the least common multiple of the sampling intervals of the visible light camera and the infrared camera, determine X visible light images and Y infrared images continuously acquired within a processing cycle to form a minimum data block; where X and Y are determined by the least common multiple of the sampling intervals of the visible light camera and the infrared camera. Step S2, Spatiotemporal Interactive Query: The Spatiotemporal Interactive Query (SIQ) module solves the temporal misregistration problem. This step first segments the visible light and infrared images in the minimum data block into image patches, embedding standard positional codes and time codes generated based on the absolute timestamps of image acquisition to obtain spatiotemporal embedding features. Then, these spatiotemporal embedding features are input into a cross-modal temporal attention layer (CMTA) to calculate the similarity between each infrared image patch query and all visible light image patch keys. This similarity is then weighted using a learnable time interval penalty term to determine the temporally best-matching visible light-infrared image pair. Finally, new alignment timestamps are assigned to the matched frames, and unmatched frames (typically frames in high frame rate modalities) undergo timestamp interpolation and feature map generation based on the features of the matched frames. The output is a temporally aligned visible light and infrared search map feature map and a template feature map. Step S3, Decision-Level Spatial Guided Fusion: The spatial non-registration problem is solved through the Decision-Level Spatial Guided Fusion (DSGF) module. This step first uses deep cross-correlation operations to process the time-aligned visible light and infrared features separately, generating a set of multi-channel preliminary target response maps for each modality. Then, the preliminary response maps of each modality are stitched together and dimensionality reduced using 1x1 convolutions to generate the final response map for each modality, which characterizes the coarse spatial distribution of the target in each modality. Finally, using the peak points of the lower-resolution, more stable infrared response map as a reference, the peak point coordinate offset between the visible light response map and the infrared response map is calculated. Based on this offset, the visible light response map is cropped and padded to generate a spatially aligned corrected response map, while recording the cropping and padding parameters. Step S4, Feature-Level Response Aware Fusion: Refined feature fusion is performed through the Feature-Level Response Aware Fusion (FRAF) module. This step first uses the clipping and padding parameters recorded by the DSGF module to synchronously spatially correct the visible light search map feature map output by the SIQ module, obtaining a spatially aligned feature map. Then, the multi-channel preliminary response map generated by DSGF is normalized and interpolated to make it the same size as the feature map, and used as a spatial weight for convolution with the spatially aligned feature map of the corresponding modality to enhance the activation of the target response region in the feature map. Simultaneously, the maximum value of each channel response map is extracted as the channel weight, and channel attention enhancement is performed on the template features of the two modalities respectively. Finally, the enhanced visible light and infrared search map features and template features are element-wise added and fused and reduced in dimensionality through 1x1 convolution to obtain the final fused search features and fused template features. Step S5, Dynamic Template Update and Tracking: The tracking head, which integrates a dynamic template update mechanism, outputs the final tracking result. This step first inputs the fused search features and fused template features into the Region Proposal Network (RPN) to regress the final accurate target bounding box. To overcome template degradation, a dynamic update strategy is designed. The network maintains a queue containing several consecutive sets (e.g., 5 sets) of fused templates. The update weights of the initial templates are fixed. For subsequent templates, a lightweight evaluation network (composed of convolutional layers, fully connected layers, and a Softmax layer) takes the response map of the current template and the search map as input, evaluates its reliability, and outputs update weights. The latest template is generated by weighted fusion of the initial and subsequent templates according to their respective weights.

[0031] In one embodiment, the scenario is as follows: This embodiment is for a continuous tracking task of moving targets in a UAV or vehicle-mounted monitoring platform, where visible light and infrared cameras have different frame rates, asynchronous acquisition times, and spatiotemporal non-registration problems such as image pixel position offsets due to differences in hardware and working mechanisms.

[0032] System hardware configuration: Embedded computing platform (such as NVIDIA Jetson AGX Orin) or server (such as one equipped with NVIDIA RTX A6000) with high-performance GPU.

[0033] System software framework: Ubuntu 22.04, PyTorch 1.12.0, Python 3.8.

[0034] Step S1, construct the minimum data block The visible light and infrared cameras are used to construct a minimum data block based on the least common multiple of their acquisition intervals. This minimum data block contains X visible light images and Y infrared images. If the frame rates of the visible light and infrared cameras are the same, then X=Y=1. For example, if the visible light acquisition interval is 20ms and the infrared acquisition interval is 40ms, then the least common multiple of the acquisition intervals is 40ms, meaning the number of visible light frames is X=2 and the number of infrared frames is Y=1.

[0035] S2: Spatiotemporal Interactive Query One of the functions of the Spatiotemporal Interactive Query (SIQ) module is to solve the problem of misregistration of time series inputs from multimodal inputs, whose input is the smallest data block.

[0036] The execution steps in the SIQ module are as follows: 1) Spatiotemporal information embedding: The implementation methods for linear mapping and spatiotemporal flag embedding in the SIQ module are similar to those in the conventional ViT model.

[0037] First, process each mode in the smallest data block. All images The image is divided into multiple patches, and spatial location parameters are directly superimposed onto the corresponding image patches through a linear mapping layer. Unlike the conventional ViT model, the processing of time stamps differs. In the SIQ module, the time parameters are first obtained. The acquisition timestamp of each image is encoded into milliseconds in the "hour-minute-second-millisecond" format, and then the system startup time is subtracted to obtain the absolute time. After converting the acquisition time into a unified and easily integrated time parameter through the above time encoding, it is then superimposed onto the corresponding image patches. The above process can be described by the following formula:

[0038] in, For the first There are 1 image patch, and the size of the image patch is 1. In this embodiment, we take . It is a linear mapping matrix. It is the embedding dimension; in this embodiment, we take... , It is a position parameter matrix. It is a time parameter matrix. It is a time encoding function.

[0039] 2) Feature extraction of CMTA cross-modal temporal attention mechanism The image patches after the spatiotemporal embedding described above are fed into the CMTA layer to measure the similarity between visible and infrared images in the smallest data block. Specifically, the CMTA layer matches the query vector of each image patch belonging to the infrared mode with the key vectors of all image patches in the visible mode. Then, it multiplies the initial timestamp value vectors embedded in the linear mapping layer for each image patch to obtain the matched time-correspondence matrix. Finally, the similarity between different image patches is output through the softmax function. The implementation process of CMTA is as follows: Figure 2 As shown.

[0040] Unlike conventional attention mechanisms, the CMTA proposed in this invention adds a time interval penalty term to prevent temporal cross-alignment in static or slow-moving scenes. The above process can be described by the following formula:

[0041] in, The time difference matrix, These are learnable parameters that adjust the weights of the time difference matrix. ,in and These represent the number of visible light and infrared modal frames in the smallest data block, respectively.

[0042] In the feature extraction step of the CMTA cross-modal temporal attention mechanism, the parameters of the relevant formulas are defined as follows: Q represents the query matrix, which is obtained by linear projection of the feature sequence of the visible light or infrared branch of the current frame, and is used to represent the target query vector to be matched at the current moment; K represents the key matrix, which is obtained by linear projection of the corresponding modal feature sequence of the historical cached frames, and is used to represent the reference key vector in the historical time series; V represents the value matrix, which shares the same input source as K but is obtained by independent linear projection, and is used to carry the content information that actually participates in the aggregation in the historical time series; T represents the set of frame indexes or the temporal position encoding matrix in the time dimension, which is used to inject temporal order information into K and V, so that the attention mechanism can distinguish the feature contributions of different historical moments; the dimensions of Q, K, and V are all N×d, where N is the sequence length, d is the feature channel dimension, and the dimension of T is consistent with the time axis of K and V.

[0043] 3) Similarity assessment For unmatched frames in a certain modality (usually the visible light modality), interpolation is performed based on the matched frames and their spatiotemporal coding features to supplement the time stamps of the lower frame rate modality (usually the infrared modality). The specific steps are as follows: First, calculate the matching weights between the unmatched visible light modality frames and the matched infrared modality frames; then, calculate the linear interpolation term and the nonlinear correction term based on the matching weights of the visible light and infrared images. The calculation method for the matching weights between the unmatched visible light modality frames and the matched infrared modality frames is shown in the following formula:

[0044] in, It is the matching weight between unmatched visible light mode frames and matched infrared mode frames. These are learnable control parameters. It is a similarity function; in this embodiment, cosine similarity is selected.

[0045] : The feature vector of the i-th RGB modality, with dimension d (usually the number of channels C), is extracted by the visible light branch network and obtained through the projection layer, representing the semantic representation of the i-th spatial location or target candidate in the current query frame.

[0046] : The feature vector of the j-th IR mode, with the same dimension d, is extracted by the infrared branch network and obtained through the projection layer. It represents the semantic representation of the j-th spatial location or target candidate in the historical cache frame or search area.

[0047] : Represents the feature vector of the k-th infrared (IR) mode, with dimension d (and d). and (Same), extracted by the infrared branch network and passed through the projection layer. Here, the subscript k is the summation index variable, with a value range of k ∈ {1,2, ..., |X...}. Y|} is used to iterate through all IR feature candidate sets involved in attention calculation.

[0048] |X Y|: Represents the size of the IR modal feature set, i.e., the total number of IR feature vectors involved in the attention calculation. Here, X and Y may represent the index sets of the RGB and IR feature sequences, respectively, |X|. Y| represents the number of IR candidates to be considered in a specific context (e.g., in a missing frame completion scenario, Y is the index of observed frames, X is the full temporal index, and |X| represents the number of IR candidates to be considered in a specific context). Y| represents the number of IR feature candidates corresponding to the virtual frame to be interpolated. This value serves as the upper limit for the summation of the denominator, ensuring weight normalization.

[0049] The method for generating the supplementary timestamp is shown in the following formula:

[0050] in, is the timestamp of the generated infrared image corresponding to the i-th visible light image, and is the timestamp of the j-th frame in the original infrared image sequence. It is a learnable parameter used to control the strength of the nonlinear correction term. MLP is a multilayer perceptron, whose input is the statistics of the matching weights and whose output is the time correction. It is the maximum matching weight between the current visible light frame and all infrared frames. It is the standard deviation of the weights.

[0051] Matching weights between frames in the unmatched visible light mode and frames in the matched infrared mode. : The j-th IR eigenvector The associated real or reference timestamp, a scalar. The superscript T may represent "Target" or "Template," referring to the timestamp of the historical frame or template frame to which this IR feature belongs. In missing frame completion scenarios, The timestamps are known real frames.

[0052] 4) Missing frame completion Spatiotemporal interactive query SIQ generates feature maps of search graphs and template graphs for visible light and infrared modes through MLP layers and weight sharing.

[0053] To improve parallelism, this invention generates supplementary timestamps based on the newly allocated timestamps and their distance relationships. Then, using bilinear interpolation based on these supplementary timestamps, supplementary feature maps for the infrared modal search map and template map are generated. This ensures that the final output infrared modal feature map and visible light modal feature map have the same dimensionality.

[0054] Feature maps of visible light and infrared mode search maps and and the feature map of the template. and It will be copied, one copy for subsequent use in decision-level spatial guidance fusion, and the other copy for feature-level response perception fusion.

[0055] When new input arrives, only the new image patch is encoded; the entire minimum data block does not need to be re-encoded. The oldest image patch is popped after a newly assigned timestamp is generated. This method ensures that the minimum data block always receives the most up-to-date data.

[0056] The minimum data block update mechanism refers to the process of packaging features from multiple consecutive frames into an atomic update unit (i.e., the minimum data block) in the historical feature cache queue maintained by the spatiotemporal interaction query module to avoid memory overflow and redundant calculations caused by frame-by-frame storage. When a new frame feature arrives, the system does not append it frame by frame, but instead determines whether the current data block is full: if it is not full, the new frame feature is appended to the current block; if it is full, the current block is marked as immutable and pushed to the tail of the circular cache queue, while a new empty data block is created to receive subsequent frames. When the cache queue exceeds the preset capacity limit, batch eviction is performed from the head of the queue on a whole data block basis, rather than single-frame eviction. This mechanism ensures the memory alignment and batch processing efficiency of cache read and write operations, while aligning the computational granularity of temporal attention with the boundaries of data blocks, avoiding fragmented access across blocks.

[0057] S3: Decision-level spatial guidance and integration The Decision-level Spatial-guided FusionModule (DSGF) in SAFNe is used to achieve spatial dimension registration and pre-tracking.

[0058] In the DSGF module, each mode is processed separately using deep cross-correlation. The specific implementation is as follows: 1) Feature maps of the search maps for visible and infrared modes and and the feature map of the template. and Decompose it into channels Each feature map; A feature map is a three-dimensional tensor (H×W×C) with spatial height H, spatial width W, and number of channels C.

[0059] 2) Feature maps for each set of search graphs and Feature map of template and Perform cross-correlation operations separately to obtain the response graph. and The length and width of the response graph are respectively represented by... and express; The specific implementation of the cross-correlation operation is as follows: First, the template feature map (size H_t×W_t×C) is used as the convolution kernel, and the search region feature map (size H_s×W_s×C, where H_s>H_t and W_s>W_t) is used as the input. Depth-wise cross-correlation is performed on each of the C channels, that is, each channel is independently convolved in two dimensions, the convolution kernel stride is 1, and there is no padding. The output single-channel response map size is (H_s - H_t + 1) × (W_s - W_t + 1). Then, the single-channel response maps of the C channels are summed element-wise along the channel dimension to obtain the final two-dimensional scalar response map (size (H_s - H_t + 1) × (W_s - W_t + 1)). 1) The numerical representation of each spatial location in the response graph is the similarity intensity between the template and the corresponding local region of the search area. The peak position is the candidate coordinate of the target center. The response graph has not undergone any nonlinear activation or normalization and retains the original cross-correlation score for direct supervision of the subsequent loss function or conversion to a probability distribution via Softmax.

[0060] 3) and The connections are made along the depth direction, and then 1×1 convolutions are used to reduce the dimensionality, generating the final response map for each modality. and .

[0061] The above process can be expressed by the following formula:

[0062]

[0063] in, It is a 1×1 convolution. It's a splicing operation. This is a deep cross-correlation operation. Before performing 1×1 convolutional dimensionality reduction, it is necessary to save the response maps of each layer for both visible and infrared modes. and This will be used to assist in information fusion in the feature-level response perception module.

[0064] The final response map characterizes the location of the target. This invention uses the highest response point in the final response map of the infrared mode as the center, and crops and fills the final response map of the visible light mode to ensure that the coordinates of the highest response point in the final response map of the visible light mode are consistent with those of the infrared mode, thus obtaining a registered visible light final response map. This method allows for spatial registration at the feature level while preserving as much information as possible. The process described above is shown in the following equation:

[0065] Here, argmax represents the operation of reading the index value of the highest response point, crop represents the cropping operation, and pad represents the padding operation. The operation parameters for the above cropping and padding need to be retained for subsequent correction of other feature maps and target response maps.

[0066] S4: Feature-level response-sensory fusion The Feature-level Response-aware Fusion Module (FRAF) takes as input the feature maps of the visible and infrared modal search maps obtained after processing by the SIQ module. and and the feature map of the template. and After processing by the DSGF module, five sets of the latest consecutive visible and infrared modal target response feature maps, after spatiotemporal registration, were obtained. and .

[0067] The specific implementation steps in the FRAF module are as follows: First, extract the target response feature maps of the five visible light and infrared modes from the smallest data block. and After normalizing each pair, interpolation expansion is performed. and This makes it compatible with the visible light and infrared modal feature maps. and They have the same scale; Then, the feature response maps of each layer are stitched together to obtain... and Using it as a convolution kernel to convolve with the modal feature map enhances the high-quality feature information in the modal feature map; Finally, the enhanced feature maps of visible light and infrared modes are element-wise summed along the channel direction and then subjected to 1×1 convolution for dimensionality reduction to obtain the final fused search map feature map. The above process can be represented as:

[0068] in, It is a 1×1 convolution. It's a splicing operation.

[0069] In addition, all and The maximum response value of each layer in the model is used as the channel weight, and these values ​​are applied to the template feature maps of the spatiotemporally registered visible and infrared modes, respectively. and Channel attention enhancement is performed. This method enhances the template information that the tracker focuses on. Finally, the enhanced template feature maps of the visible and infrared modes are element-wise summed along the channel direction, and then subjected to 1×1 convolution for dimensionality reduction to obtain the template feature map. The above process can be represented as:

[0070] Finally, the feature maps of the latest five consecutive fusion templates and the search graph in the smallest data block are combined. and These feature maps have achieved spatial and temporal registration. Furthermore, high-quality information in each modality has been sufficiently enhanced, while low-quality noise has been adequately suppressed. Accurate target location regression will then be performed in the tracker.

[0071] S5: Dynamic Template Updates and Tracking In target tracking, the target position in the first frame is fixed, so the initial template has the highest reliability. Therefore, in SAFNet, the update weight of the initial template is fixed at 0.5. The weights of subsequent templates participating in template updates are based on the reliability of the subsequent templates. The reliability of subsequent templates is evaluated in SAFNet using the template and the response map of the search graph. Specifically, a lightweight convolutional network takes the template and the response map of the search graph as input, and then passes through fully connected layers and softmax layers to output the update weights of the four subsequent templates (excluding the initial template) in the smallest data block. Finally, the latest template is calculated based on the update weights. The above process can be represented as:

[0072] in It is a response diagram. It is a convolutional network that calculates and updates weights. In this way, SAFNet fuses template information over time, which helps to identify changes in the target in a timely manner.

[0073] After obtaining the final response of the target, SAFNet maps the tracking results back to the original image according to the cropping and padding parameters of the visible light and infrared modes used in the SIQ and DSGF modules, and finally outputs the target position in the visible light mode and infrared mode respectively.

[0074] The tracking head of SAFNet is built upon a Region Proposal Network (RPN) to regress the bounding box. The loss function consists of the classification loss and regression loss from the RPN network. The classification loss uses cross-entropy loss. For each anchor point... If it is foreground, it is considered a positive sample and labeled as 1; if it is background, it is considered a negative sample and labeled as 0. Assume... Anchor point The true label (0 or 1). If the model predicts that the anchor point is foreground, then the classification loss can be expressed as:

[0075] in, It is classification loss. It is the collection of all anchor points.

[0076] SAFNet uses GIoU loss to calculate the loss for the visible light and infrared modes. The regression loss function can be expressed as:

[0077] in Represents the prediction box. Represents ground truth. This represents the smallest closed bounding box. It's about area calculation. It represents the area of ​​the smallest closed box minus the area of ​​the union of the predicted box and the ground truth box.

[0078] This invention focuses on RGBT tracking and aims to adapt to unregistered datasets. Therefore, the regression loss needs to be calculated separately for the visible light mode and the infrared mode. The regression loss function can be expressed as:

[0079] Among them, and These represent the predicted bounding boxes in the visible light and infrared modes, respectively. and These represent the true bounding boxes in the visible light and infrared modes, respectively.

[0080] The total loss function can be expressed as:

[0081] in, It is the total loss. This is a hyperparameter used to balance classification and regression losses. In this invention, it is set to 0.5. Ablation experiments were conducted in the experimental section to demonstrate the rationality of this parameter selection.

[0082] During training, the SAFNet of this invention was implemented using Python 3.8 and PyTorch 1.12.0. Training was conducted on a server with four RTX A6000 GPUs, each with 48GB of VRAM. The server's operating system was Ubuntu 22.04, its CPU was an Intel Xeon W9-3495X, and its RAM was 512GB. The batch size was set to 32 during training.

[0083] During training, the template and search region sizes were set to 128×128 and 256×256, respectively. The AdamW optimizer was used to optimize the model. Momentum and weight decay were set to 0.9 and 10, respectively. -4 The initial learning rate of this invention is set to 10. -4 After training for 200 epochs, the learning rate was adjusted to 10. -5 .

[0084] like Figure 1 As shown, the input-output relationships of each module in the overall neural network are as follows: The input layer receives the visible light image I_vis ∈ ^(H×W×3) and infrared image I_ir ∈ ^(H×W×3), respectively fed into the dual-branch backbone network; The visible light backbone network outputs visible light multi-level feature maps {F_vis^L2, F_vis^L3, F_vis^L4}, and the infrared backbone network outputs infrared multi-level feature maps {F_ir^L2, F_ir^L3, F_ir^L4}. The spatial resolution of each level of feature map is 1 / 4, 1 / 8, and 1 / 16 of the original image, respectively. The spatiotemporal interactive query module receives {F_vis^L4, F_ir^L4} and the historical cache feature queue Q_hist, and outputs the spatiotemporally enhanced cross-modal fused feature map F_fused ∈ ^((H / 16)×(W / 16)×C) and the temporal alignment feature sequence T_aligned after missing frame padding; The decision-level spatial guidance fusion module receives F_fused and low-level features {F_vis^L2, F_ir^L2}, and generates a multi-scale fusion feature pyramid {P2, P3, P4} through top-down paths and lateral connections. The detection head module receives {P2, P3, P4} and outputs classification score maps S_cls^i ∈ at each scale. ^(H_i×W_i×N_cls), bounding box regression offset map S_reg^i ∈ ^(H_i×W_i×4) and centrality prediction graph S_ctr^i∈ ^(H_i×W_i×1).

[0085] The inference process is as follows: First, an initial template is extracted from the first frame of the visible light and infrared images; then, a square search region is cropped at the center of the template, with the side length of the cropped square region equal to 1.5 times the length of the template's diagonal; finally, the template and search map are scaled to the specified model size and then input into the network for inference. Template updates are performed using the dynamic template update method described in this invention.

[0086] Quantitative performance comparison SAFNet has achieved leading or highly competitive performance on mainstream RGBT tracking datasets such as GTOT, RGBT210, RGBT234, and LasHeR. Particularly on the large-scale, challenging LasHeR dataset, it achieves a tracking success rate (SR) of 59.1% and a precision (PR) of 70.9%. On artificially constructed spatiotemporal unregistered test data and its own self-built spatiotemporal unregistered dataset, SAFNet demonstrates excellent adaptability, significantly outperforming existing algorithms. On the RGBT234 dataset, SAFNet maintains high precision (PR 89.1%) while achieving an inference speed of 35.5 FPS, achieving an excellent balance between accuracy and efficiency.

[0087] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A spatiotemporal unregistered visible-infrared target tracking method, characterized in that, include: Step S1, construct the minimum data block: Based on the least common multiple of the sampling intervals of the visible light camera and the infrared camera, determine the X visible light images and Y infrared images continuously acquired within a processing cycle to form a minimum data block; Step S2, Spatiotemporal Interactive Query: The visible light image and infrared image in the smallest data block are divided into image blocks respectively, and the position code and the time code generated based on the acquisition timestamp are embedded to obtain the spatiotemporal embedding feature; The spatiotemporal embedding features are input into a cross-modal temporal attention layer to calculate the similarity between infrared image patch queries and visible light image patch keys. The most temporally matching visible light-infrared image pairs are determined by weighting the data through a time interval penalty term. New alignment timestamps are assigned to the matching frames, and timestamp interpolation and feature map generation are performed on the unmatched frames. The output is a temporally aligned visible light and infrared search map feature map and a template feature map. Step S3, decision-level spatial guided fusion: Perform deep cross-correlation operation on the time-aligned visible light features and infrared features respectively to generate multi-channel preliminary response maps of their respective modes; stitch and reduce the dimensionality of the preliminary response maps of each mode to generate the final response map of each mode; Using the peak point of the infrared response map as a reference, the coordinate offset between the visible light response map and the infrared response map is calculated. The visible light response map is then cropped and filled according to the offset to generate a spatially aligned corrected response map, and the cropping and filling parameters are recorded. Step S4, Feature-level response perception fusion: Spatial correction of the visible light search map feature map is performed using the cropping and padding parameters; After normalizing and interpolating the preliminary response map, it is used as a spatial weight to convolve with the spatial alignment feature map of the corresponding modality to enhance the target response region; the maximum value of each channel response map is extracted as the channel weight to enhance the channel attention of the template features. The enhanced visible light and infrared search map features and template features are fused separately to obtain fused search features and fused template features; Step S5, Dynamic Template Update and Tracking: Input the fused search features and fused template features into the region proposal network to regress the target bounding box; maintain a latest queue of several sets of fused templates, evaluate the reliability of the current template based on the response map of the search graph through a lightweight evaluation network and output update weights, and fused the initial template and subsequent templates according to the weights to generate the latest template.

2. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, The time encoding is generated using cosine position encoding, which converts the absolute timestamp of image acquisition into a unified time parameter and superimposes it onto the corresponding image block features.

3. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In the cross-modal temporal attention layer, the time interval penalty term includes a learnable parameter used to adjust the weights of the time difference matrix to prevent mismatches in static scenes; the time difference matrix is ​​constructed based on the frame number of the visible light image and the infrared image.

4. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In step S2, timestamp interpolation of unmatched frames includes: calculating the matching weight between unmatched visible light frames and matched infrared frames; generating supplementary timestamps based on the matching weights through linear interpolation and nonlinear correction terms; the nonlinear correction terms are generated by a multilayer perceptron based on statistics of the matching weights.

5. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In step S3, the deep cross-correlation operation preserves channel semantic information and generates a set of multi-channel preliminary target response maps for each modality; the splicing and dimensionality reduction are achieved through 1x1 convolution.

6. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In step S4, the response map-guided feature enhancement step includes: normalizing and interpolating the multi-channel preliminary response map of each modality to make its size consistent with the corresponding search map feature map; and using the normalized response map as spatial weights to perform element-wise convolution with the corresponding search map feature map.

7. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In step S4, the channel attention fusion step includes: extracting the maximum value of the multi-channel preliminary response map of each modality in each channel to form a channel weight vector; and performing channel-by-channel multiplication of the channel weight vector with the corresponding template feature map to achieve channel attention enhancement.

8. The spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, In step S5, the lightweight evaluation network consists of convolutional layers, fully connected layers, and softmax layers; the update weight of the initial template is a fixed value of 0.5; and the dynamic update strategy maintains a queue containing 5 consecutive sets of fused templates.

9. A spatiotemporal unregistered visible-infrared target tracking method according to claim 1, characterized in that, It also includes a loss function step: the total loss function is composed of a weighted average of classification loss and regression loss; the classification loss uses cross-entropy loss, the regression loss uses GIoU loss, and the regression loss is calculated separately for the prediction results of visible light and infrared modes.

10. A visible light-infrared target tracking system based on the method of any one of claims 1-9, characterized in that, include: Data building unit, used to build the smallest data block based on the camera sampling interval; The spatiotemporal interactive query unit is used to segment the visible light image and infrared image in the smallest data block into image blocks, and embed position codes and time codes generated based on the acquisition timestamps to obtain spatiotemporal embedding features; The spatiotemporal embedding features are input into a cross-modal temporal attention layer to calculate the similarity between infrared image patch queries and visible light image patch keys. The most temporally matching visible light-infrared image pairs are determined by weighting the data through a time interval penalty term. New alignment timestamps are assigned to the matching frames, and timestamp interpolation and feature map generation are performed on the unmatched frames. The output is a temporally aligned visible light and infrared search map feature map and a template feature map. The decision-level spatial guidance fusion unit is used to perform deep cross-correlation operations on the time-aligned visible light features and infrared features to generate multi-channel preliminary response maps for each mode; the preliminary response maps of each mode are then stitched together and dimensionality reduced to generate the final response map for each mode. Using the peak point of the infrared response map as a reference, the coordinate offset between the visible light response map and the infrared response map is calculated. The visible light response map is then cropped and filled according to the offset to generate a spatially aligned corrected response map, and the cropping and filling parameters are recorded. A feature-level response perception fusion unit is used to perform spatial correction on the visible light search map feature map using the cropping and padding parameters; After normalizing and interpolating the preliminary response map, it is used as a spatial weight to convolve with the spatial alignment feature map of the corresponding modality to enhance the target response region; the maximum value of each channel response map is extracted as the channel weight to enhance the channel attention of the template features. The enhanced visible light and infrared search map features and template features are fused separately to obtain fused search features and fused template features; The dynamic template update and tracking unit is used to input the fused search features and fused template features into the region proposal network to regress the target bounding box; it maintains a latest queue of several sets of fused templates, evaluates the reliability of the current template based on the response map of the search graph through a lightweight evaluation network, and outputs update weights, and fused the initial template with the subsequent templates according to the weights to generate the latest template.

Images