A fast-moving small target tracking method based on dual-modal fusion

Abstract

The present invention discloses a dual-modal fusion fast-moving small target tracking method, which solves the problem that fast-moving small targets are easily lost and difficult to locate, and belongs to the field of computer vision. The method comprises: based on the first frame label of each video sequence, cutting out a template area and a search area of ​​the target; performing data enhancement, feature extraction and feature fusion on the visible light image sequence in the search area and the corresponding event voxel grid data set to obtain a fused feature map; outputting a robust feature representation of target positioning through an encoder and a decoder; calculating the target prediction bounding box area and the comprehensive confidence; taking the center point of the target prediction bounding box of the previous frame as a reference and generating a prediction search area of ​​the current frame with a preset multiple thereof; when the comprehensive confidence is higher than an update threshold and the predicted search area contains the target, updating the template area to obtain a constructed fast-moving small target tracking network. The present invention realizes continuous and stable tracking of fast-moving small targets.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision of artificial intelligence and relates to a dual-modal fusion method for tracking a small fast-moving target. Background Art

[0002] In the field of computer vision within artificial intelligence, object tracking is a core research topic, widely used in scenarios such as autonomous driving, intelligent security, and robotic navigation. As the complexity of these applications increases, the requirements for real-time and accurate object tracking become increasingly stringent. Traditional visible light cameras, based on a frame sampling mechanism, are inherently limited by limitations such as frame rate and dynamic range. This makes them prone to motion blur and information loss when sensing fast-moving objects, making them difficult to meet the precise tracking requirements in high-speed scenarios.

[0003] Dynamic vision sensors are a new type of visual device inspired by the biological retina. They have no concept of frames, feature independent, asynchronous pixels, and respond only to event streams of brightness changes. Compared to traditional optical sensors, event cameras offer significant advantages in challenging scenes involving high dynamic range and high-speed motion, making them more suitable for tracking fast-moving targets.

[0004] Currently, target tracking based on event cameras is mainly divided into pure event stream target tracking methods and target tracking methods that combine event stream and visible light.

[0005] Pure event stream target tracking methods can achieve effective tracking in scenes with simple backgrounds. However, when the event stream is too sparse or insufficient, this method usually cannot accurately locate the target and track it effectively.

[0006] The target tracking method that combines event stream and visible light has good tracking performance in complex background scenes through the complementarity of multimodal data.

[0007] For example: Application number: 202510193101.3, Announcement number: CN 119672071 A, Invention name: A multimodal unified target tracking method based on unified modal representation. When performing multimodal fusion, directly splicing channels may not fully exploit the uniqueness of different modal data. The feature distribution, data scale, noise characteristics, etc. of different modal data are significantly different. For example, event modal data has the characteristics of high temporal resolution and asynchronous response. If a special feature extractor is designed for these characteristics, features may be extracted more accurately, improving the fusion effect and tracking accuracy;

[0008] For example: Application number: 202111024841.2, Publication number: CN 113888607 A, Invention name: Target detection and tracking method, system and storage medium based on event camera. During the tracking process, the template image is not updated in real time, and tracking loss is prone to occur. The tracking effect for small targets and fast-moving targets is poor.

[0009] For example, application number: 202410680318.2, publication number: CN 118521617 A, invention name: A method, system and device for optical flow estimation that integrates a visible light camera and an event camera. This technology focuses on optical flow estimation, mainly calculating the motion vectors of each pixel in adjacent image frames. Although it has certain adaptability to scenes with fast-moving targets, it lacks sufficient pertinence and effectiveness in dealing with the problem that small, fast-moving targets are easily lost and difficult to locate in complex environments. Its feature extraction method also makes it difficult to accurately capture the key features of small targets.

[0010] As the demand for tracking fast-moving and small targets grows in real-world applications, existing fusion algorithms still face bottlenecks in capturing rapidly changing target trajectories and identifying small target features. In fast-moving scenes, where target positions change rapidly, existing methods can struggle to quickly and accurately capture their trajectories. Furthermore, for small targets, due to their small pixel count and less distinct features, algorithms are prone to misjudgment and target loss, necessitating more effective solutions. Summary of the Invention

[0011] To address the issues of easy loss and difficulty locating fast-moving small targets, this paper proposes a dual-modal fusion method for fast-moving small target tracking. This method leverages the high temporal resolution and high dynamic range advantages of event cameras while effectively utilizing the texture information of traditional optical cameras to improve spatial resolution. Through dual-modal data synergy, both visible light images and corresponding event voxel grid data are enhanced simultaneously to simulate complex scene changes and improve model generalization. By analyzing the characteristic features of visible light images and event voxel grid data, specialized visible light feature extractors and event feature extractors are designed. The extracted feature maps are then fused at the channel level, and new feature maps are obtained through image matching. Furthermore, a dynamic template mechanism is implemented to maintain real-time updates of the template region, thereby achieving continuous and stable tracking of fast-moving small targets. This method deeply fuses information from both modalities, improving the accuracy and stability of fast-moving small target tracking. This method effectively overcomes many challenges faced by existing technologies for small target tracking in complex scenes, bringing new breakthroughs and developments to the field of target tracking.

[0012] The purpose of the present invention is specifically achieved through the following technical solutions:

[0013] The present invention discloses a dual-modal fusion fast-moving small target tracking method, which includes:

[0014] Step 1: Preprocess the target dataset consisting of video sequences containing fast-moving small targets and divide it into a training set and a test set. Based on the first frame label of each video sequence in the training set, crop the target template area and search area.

[0015] Step 2: Data enhancement, feature extraction, and feature fusion are performed on the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset to obtain a fused feature map;

[0016] Step 3: The multi-layer Transformer encoder extracts the long-range dependencies between features in the fused feature map, and the output enhanced feature representation and the target query vector are input into the multi-layer Transformer decoder to learn and output a robust feature representation for target localization.

[0017] Step 4: The bounding box prediction head calculates the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity. The features of the search area are weighted and aggregated through the similarity matrix and then input into the fully convolutional network. The coordinate probability distribution of the target bounding box is output, and the target prediction bounding box area is calculated based on the coordinate probability distribution of the target bounding box. At the same time, the comprehensive confidence of the target prediction bounding box area is obtained through the score prediction head and the intersection-union ratio.

[0018] Step 5: Taking the center point of the target prediction bounding box of the previous frame as the reference, generate the prediction search area of ​​the current frame with a preset multiple of the target prediction bounding box area; when the comprehensive confidence is higher than the update threshold and the prediction search area contains the target, the target real bounding box area is cropped from the visible light image of the current frame and the corresponding event voxel grid data, and after dynamically updating the template area, the constructed fast-moving small target tracking network is obtained; the fast-moving small target tracking network is trained and tested through the training set and test set to achieve continuous tracking of fast-moving small targets.

[0019] In step 1, the target dataset consisting of video sequences containing fast-moving small targets is preprocessed and divided into training and test sets as follows:

[0020] Collect video sequences containing multiple fast-moving small targets. Each video sequence consists of a visible light image sequence and a corresponding event stream dataset.

[0021] The duration of events in the event stream dataset is discretized, and the event polarity of each discretized event is assigned to the two closest space-time voxels to form an event voxel grid dataset;

[0022] The visible light image sequence and the corresponding event voxel grid dataset are combined into a target dataset; the target dataset is divided into a training set and a test set according to a preset ratio; wherein the event voxel grid dataset is:

[0023] ;

[0024] Where, is the event voxel grid dataset, is the coordinate of the event voxel grid data, is the timestamp of the discrete time center, is the coordinate of the event stream data, is the event polarity, is the event timestamp, B is the total number of discrete times, is the total time span of the event stream.

[0025] In step 1, based on the first frame label of each video sequence in the training set, the method for cutting out the target template area and search area is as follows:

[0026] For each video sequence in the training set, the target ground truth bounding box is cropped from the first frame of the labeled visible light image as the visible light image template; at the same time, the same area is cropped from the corresponding event voxel grid data as the event voxel template, and the visible light image template and the corresponding event voxel template are used together as the template area;

[0027] The search area for target search is generated based on the center point of the template area and the preset multiple of the template area.

[0028] In step 2, the visible light image sequence contained in each video sequence in the search area is subjected to data enhancement, feature extraction, and feature fusion with the corresponding event voxel grid dataset to obtain a fused feature map as follows:

[0029] The visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset, and a data augmentation operation simulating complex scene changes is performed on the fused dataset to obtain a data augmented dataset;

[0030] Splitting the data augmentation dataset into a visible light image augmentation sequence and an event voxel grid augmentation dataset according to the channel dimension;

[0031] After introducing the channel attention mechanism to improve the lightweight convolutional neural network MobileNetV3-Large, feature extraction is performed on the visible light image enhancement sequence to obtain the enhanced feature map of the visible light image sequence;

[0032] After introducing the time gating mechanism to improve the 3D convolutional network, feature extraction is performed on the event voxel grid enhanced dataset to obtain the enhanced feature map of the event voxel grid dataset;

[0033] The enhanced feature map of the visible light image sequence and the enhanced feature map of the event voxel grid dataset are fused at the channel level to obtain a fused feature map.

[0034] In step 2, the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset as follows:

[0035] The visible light image of the visible light image sequence of each video sequence in the search area with a size of (H1, W1, C1) is spliced ​​with the event voxel grid data of the corresponding size of (H2, W2, C2) in the event voxel grid data along the channel dimension to form fused data of size (H, W, C1 + C2); where H1 is the height of the visible light image, W1 is the width of the visible light image, C1 is the number of channels of the visible light image, H2 is the height of the event voxel grid data, W2 is the width of the event voxel grid data, C2 is the number of channels of the event voxel grid data, H is the height of the fused data, and W is the width of the fused data; where H1, H2 and H are equal, and W1, W2 and W are equal.

[0036] Methods for performing augmentation operations on the fused dataset to simulate complex scene changes include: horizontal flipping, vertical flipping, spatial translation, random offset of bounding boxes and / or center point jittering;

[0037] The fused feature map is:

[0038] ;

[0039] Where, is the fusion feature map, It is the splicing operation of channel-level feature fusion, Enhance the feature map for the visible light image sequence, Enhanced feature maps for event voxel grid datasets.

[0040] In step 3, the long-range dependencies between features in the fused feature map are extracted through a multi-layer Transformer encoder. The output enhanced feature representation and the target query vector are input into a multi-layer Transformer decoder. The method for learning and outputting a robust feature representation for target localization is as follows:

[0041] After using the bottleneck layer to compress and flatten the number of channels in the fused feature map and add absolute position encoding, the fused feature map is input into a multi-layer Transformer encoder. The multi-head self-attention mechanism and feedforward neural network contained in each layer of the encoder in the multi-layer Transformer encoder extract the long-range dependencies between the features in the fused feature map and output an enhanced feature representation.

[0042] The target query vector is combined with the enhanced feature representation and input into the multi-layer Transformer decoder. After the self-attention mechanism, encoder and decoder attention mechanism and feedforward neural network contained in each layer of the decoder in the multi-layer Transformer decoder, the target query vector set is focused on the feature area related to the target, and the robust feature representation of target positioning is learned and output.

[0043] In step 4, the method for calculating the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity by the bounding box prediction head is:

[0044] ;

[0045] Where, is the similarity matrix, is the normalization function, is a robust feature representation, is the transpose of the features of the template region, is the feature of the template region, is the scaling factor;

[0046] The method of weighted aggregation of the features of the search area through the similarity matrix to obtain the weighted aggregation feature set is:

[0047] ;

[0048] Where, is the weighted aggregate feature set, is the characteristic of the search area;

[0049] The method for calculating the target prediction bounding box area based on the coordinate probability distribution of the target bounding box is:

[0050] ;

[0051] ;

[0052] Where, The coordinates of the lower right corner of the target prediction bounding box area, Predict the upper left corner coordinate of the bounding box area for the target; is the height of the feature map in the target bounding box, is the width of the feature map in the target bounding box, is the spatial position of the feature map in the target bounding box, is the probability distribution of the coordinates of the lower right corner of the target bounding box, is the probability distribution of the upper left corner coordinate of the target bounding box.

[0053] In step 4, the method for obtaining the comprehensive confidence of the target prediction bounding box area through the score prediction head and intersection-over-union is:

[0054] ;

[0055] ;

[0056] Where, is the comprehensive confidence level, is the weight coefficient, is the intersection and union ratio, is the prediction score of the score prediction head, The intersection area of ​​the target prediction bounding box area, The union area of ​​the bounding box regions predicted for the object.

[0057] In step 5, the fast-moving small target tracking network is trained and tested using the training set and the test set. The method for continuously tracking the fast-moving small target includes:

[0058] The total loss function is constructed from the positioning loss function and the classification loss function. The fast-moving small target tracking network is trained using the training set combined with the total loss function. The score prediction head is then trained separately using the binary cross entropy loss function to obtain a trained fast-moving small target tracking network.

[0059] The test set is input into the trained fast-moving small target tracking network, and the template area and search area are initialized based on the first visible light image of each video sequence in the test set and the corresponding event voxel grid data. In each subsequent frame, the predicted search area is extracted from the current frame and input into the trained fast-moving small target tracking network to obtain the target prediction bounding box area and comprehensive confidence. When the comprehensive confidence is higher than the update threshold and the predicted search area contains the target, the template area is dynamically updated to achieve continuous tracking of fast-moving small targets.

[0060] In step 5, the total loss function is:

[0061] ;

[0062] Where, is the total loss value, is the intersection-over-union loss value in the positioning loss function, yes The coefficient of is the center distance loss value in the positioning loss function, yes The coefficient of is the cross entropy loss value in the classification loss function, yes The coefficient of ; where

[0063] The calculation method of the intersection-over-union loss is: ;

[0064] The calculation method of the center distance loss value is:

[0065] ;

[0066] The cross entropy loss value is calculated as:

[0067] ;

[0068] Where, The center point coordinates of the target prediction bounding box area, is the center point coordinate of the target real bounding box area, is the prediction confidence in the training set, is the label in the training set, i is the category number of the training set data, and C is the total number of categories of the training set data;

[0069] The binary cross entropy loss function is:

[0070] ;

[0071] Where, is the binary cross entropy loss value.

[0072] The beneficial effects of the present invention are:

[0073] 1. The present invention fuses the event voxel grid data of the event stream with the visible light image, fully integrating the advantages of the event stream's fast response and sensitivity to dynamic changes, and the visible light image's rich color and texture. This fusion strategy significantly improves the perception and tracking accuracy of fast-moving small targets, especially in complex scenes, high-speed target motion, and environments with drastic changes in lighting, while still maintaining stable and accurate target tracking.

[0074] 2. Targeting the characteristic differences between visible light imagery and event voxel grid data, a dedicated feature extraction network was designed to deeply mine the features of both data types. Channel fusion technology was then used to maximize the integration of effective information. Furthermore, the spatiotemporal Transformer network architecture was improved, leveraging its powerful long-range dependency modeling capabilities to effectively learn target spatiotemporal feature representations. This significantly enhances the model's ability to adapt to changes in target appearance and position, laying a solid foundation for accurate tracking.

[0075] 3. Through dual-modal data collaborative enhancement, data enhancement operations are performed simultaneously on the visible light image and the corresponding event voxel grid data, including horizontal / vertical flipping, spatial translation, random offset of the bounding box, and center point jitter, to simulate complex scene changes and improve the model's generalization ability.

[0076] 4. A dynamic template region update strategy is introduced, building an intelligent update decision mechanism based on real-time tracking results and comprehensive confidence. This strategy ensures that the template region can dynamically evolve with the target state and always maintain a close match with the target's current features. This effectively suppresses tracking drift caused by factors such as target appearance changes and occlusions, ensures real-time updates of the tracking template, and further improves the stability and accuracy of the target tracking process.

[0077] 5. Dual-modal fusion of event stream data and visible light images fully leverages the high temporal resolution and high dynamic range advantages of event cameras, while effectively utilizing the texture information of traditional optical cameras to enhance spatial resolution. This effectively combines the advantages of both modal data, improving the perception and tracking accuracy of fast-moving small targets, and enabling continuous and stable tracking of targets in challenging lighting conditions such as strong exposure and low light, as well as continuous and stable tracking of small targets in fast-moving scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] The present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0079] Figure 1 Schematic diagram of a visible light image under a low-light square background provided by an embodiment of the present invention.

[0080] Figure 2 Schematic diagram of an event voxel grid under a low-light square background provided by an embodiment of the present invention.

[0081] Figure 3 3 is a schematic diagram of the tracking results of dual-modal fusion in a low-light square background provided by an embodiment of the present invention.

[0082] Figure 4 Schematic diagram of a visible light image against a complex forest background provided by an embodiment of the present invention.

[0083] Figure 5 This is a schematic diagram of an event voxel grid in a complex forest background provided by an embodiment of the present invention.

[0084] Figure 6 Schematic diagram of dual-modal fusion tracking results in a complex forest background provided by an embodiment of the present invention.

[0085] Figure 7 This is a schematic diagram of comparing the accuracy of the tracking methods provided by the embodiments of the present invention.

[0086] Figure 8 2 is a schematic diagram of comparing success rates of tracking methods provided by embodiments of the present invention. DETAILED DESCRIPTION

[0087] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0088] In the present invention, feature refers to the low-level feature vectors initially extracted from the original input data, such as visible light images and event voxel grid data;

[0089] Feature representation: refers to the high-level semantic vector generated after the original features are processed by a neural network (such as Transformer), which integrates abstract information such as context association and long-range dependency.

[0090] An embodiment of the present invention provides a dual-modal fusion fast-moving small target tracking method, the method comprising:

[0091] Step 1: Preprocess the target dataset consisting of video sequences containing fast-moving small targets and divide it into a training set and a test set. Based on the first frame label of each video sequence in the training set, crop the target template area and search area.

[0092] Step 2: Data enhancement, feature extraction, and feature fusion are performed on the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset to obtain a fused feature map;

[0093] Step 3: The multi-layer Transformer encoder extracts the long-range dependencies between features in the fused feature map, and the output enhanced feature representation and the target query vector are input into the multi-layer Transformer decoder to learn and output a robust feature representation for target localization.

[0094] Step 4: The bounding box prediction head calculates the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity. The features of the search area are weighted and aggregated through the similarity matrix and then input into the fully convolutional network. The coordinate probability distribution of the target bounding box is output, and the target prediction bounding box area is calculated based on the coordinate probability distribution of the target bounding box. At the same time, the comprehensive confidence of the target prediction bounding box area is obtained through the score prediction head and the intersection-union ratio.

[0095] Step 5: Taking the center point of the target prediction bounding box of the previous frame as the reference, generate the prediction search area of ​​the current frame with a preset multiple of the target prediction bounding box area; when the comprehensive confidence is higher than the update threshold and the prediction search area contains the target, the target real bounding box area is cropped from the visible light image of the current frame and the corresponding event voxel grid data, and after dynamically updating the template area, the constructed fast-moving small target tracking network is obtained; the fast-moving small target tracking network is trained and tested through the training set and test set to achieve continuous tracking of fast-moving small targets.

[0096] In step 1, the target dataset consisting of video sequences containing fast-moving small targets is preprocessed and divided into training and test sets as follows:

[0097] Collect video sequences containing multiple fast-moving small targets. Each video sequence consists of a visible light image sequence and a corresponding event stream dataset.

[0098] The duration of events in the event stream dataset is discretized, and the event polarity of each discretized event is assigned to the two closest space-time voxels to form an event voxel grid dataset;

[0099] The visible light image sequence and the corresponding event voxel grid dataset are combined into a target dataset; the target dataset is divided into a training set and a test set according to a preset ratio; wherein the event voxel grid dataset is:

[0100] ;

[0101] Where, is the event voxel grid dataset, is the coordinate of the event voxel grid data, is the timestamp of the discrete time center, is the coordinate of the event stream data, is the event polarity, is the event timestamp, B is the total number of discrete times, is the total time span of the event stream.

[0102] In step 1, based on the first frame label of each video sequence in the training set, the method for cutting out the target template area and search area is as follows:

[0103] For each video sequence in the training set, the target ground truth bounding box is cropped from the first frame of the labeled visible light image as the visible light image template; at the same time, the same area is cropped from the corresponding event voxel grid data as the event voxel template, and the visible light image template and the corresponding event voxel template are used together as the template area;

[0104] Using the center point of the template area as a reference, a search area for target search is generated based on a preset multiple of the template area. For example, a search area four times the area of ​​the template area is generated by doubling the length of each side of the template area. This is used for target search in subsequent frames.

[0105] In step 2, the visible light image sequence contained in each video sequence in the search area is subjected to data enhancement, feature extraction, and feature fusion with the corresponding event voxel grid dataset to obtain a fused feature map as follows:

[0106] The visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset, and a data augmentation operation simulating complex scene changes is performed on the fused dataset to obtain a data augmented dataset;

[0107] Splitting the data augmentation dataset into a visible light image augmentation sequence and an event voxel grid augmentation dataset according to the channel dimension;

[0108] After introducing the channel attention mechanism to improve the lightweight convolutional neural network MobileNetV3-Large, feature extraction is performed on the visible light image enhancement sequence to obtain the enhanced feature map of the visible light image sequence;

[0109] After introducing the time gating mechanism to improve the 3D convolutional network, feature extraction is performed on the event voxel grid enhanced dataset to obtain the enhanced feature map of the event voxel grid dataset;

[0110] The enhanced feature map of the visible light image sequence and the enhanced feature map of the event voxel grid dataset are fused at the channel level to obtain a fused feature map.

[0111] In step 2, the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset as follows:

[0112] The visible light image of the visible light image sequence of each video sequence in the search area with a size of (H1, W1, C1) is spliced ​​with the event voxel grid data of the corresponding size of (H2, W2, C2) in the event voxel grid data along the channel dimension to form fused data of size (H, W, C1 + C2); where H1 is the height of the visible light image, W1 is the width of the visible light image, C1 is the number of channels of the visible light image, H2 is the height of the event voxel grid data, W2 is the width of the event voxel grid data, C2 is the number of channels of the event voxel grid data, H is the height of the fused data, and W is the width of the fused data; where H1, H2 and H are equal, and W1, W2 and W are equal.

[0113] Methods for performing augmentation operations on the fused dataset to simulate complex scene changes include: horizontal flipping, vertical flipping, spatial translation, random offset of bounding boxes and / or center point jittering;

[0114] The fused feature map is:

[0115] ;

[0116] Where, is the fusion feature map, It is the splicing operation of channel-level feature fusion, Enhance the feature map for the visible light image sequence, Enhanced feature maps for event voxel grid datasets.

[0117] In step 3, the long-range dependencies between features in the fused feature map are extracted through a multi-layer Transformer encoder. The output enhanced feature representation and the target query vector are input into a multi-layer Transformer decoder. The method for learning and outputting a robust feature representation for target localization is as follows:

[0118] The bottleneck layer compresses and flattens the number of channels in the fused feature map, adds absolute position encoding, and then inputs it into a multi-layer Transformer encoder. The multi-head self-attention mechanism and feedforward neural network contained in each layer of the encoder in the multi-layer Transformer encoder extract the long-range dependencies between features in the fused feature map, and outputs an enhanced feature representation.

[0119] The target query vector is combined with the enhanced feature representation and input into the multi-layer Transformer decoder. After the self-attention mechanism, encoder and decoder attention mechanism and feedforward neural network contained in each layer of the decoder in the multi-layer Transformer decoder, the target query vector set is focused on the feature area related to the target, and the robust feature representation of target positioning is learned and output.

[0120] In step 4, the method for calculating the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity by the bounding box prediction head is:

[0121] ;

[0122] Where, is the similarity matrix, is the normalization function, is a robust feature representation, is the transpose of the features of the template region, is the feature of the template region, is the scaling factor;

[0123] The method of weighted aggregation of the features of the search area through the similarity matrix to obtain the weighted aggregation feature set is:

[0124] ;

[0125] Where, is the weighted aggregate feature set, is the characteristic of the search area;

[0126] The method for calculating the target prediction bounding box area based on the coordinate probability distribution of the target bounding box is:

[0127] ;

[0128] ;

[0129] Where, The coordinates of the lower right corner of the target prediction bounding box area, Predict the upper left corner coordinate of the bounding box area for the target; is the height of the feature map in the target bounding box, is the width of the feature map in the target bounding box, is the spatial position of the feature map in the target bounding box, is the probability distribution of the coordinates of the lower right corner of the target bounding box, is the probability distribution of the upper left corner coordinate of the target bounding box.

[0130] In step 4, the method for obtaining the comprehensive confidence of the target prediction bounding box area through the score prediction head and intersection-over-union is:

[0131] ;

[0132] ;

[0133] Where, is the comprehensive confidence level, is the weight coefficient, is the intersection and union ratio, is the prediction score of the score prediction head, The intersection area of ​​the target prediction bounding box area, The union area of ​​the target prediction bounding box area. The specific derivation process of the formula is:

[0134] The formula for calculating the coordinates of the intersection rectangle is:

[0135] ;

[0136] ;

[0137] Where, and is the coordinate of the upper left corner of the intersection rectangle, and is the coordinate of the lower right corner of the intersection rectangle, and are the true values ​​of the upper left corner and lower right corner coordinates of the target box, is the maximum value, is the minimum value.

[0138] The formula for calculating the intersection area is:

[0139] ;

[0140] ;

[0141] ;

[0142] Where, is the width of the intersection rectangle, is the height of the intersection rectangle, is the intersection area.

[0143] The formula for calculating the union area is:

[0144] ;

[0145] ;

[0146] ;

[0147] Where, is the predicted tracking box area, is the real box area, is the area of ​​the union.

[0148] .

[0149] In step 5, the fast-moving small target tracking network is trained and tested using the training set and the test set. The method for continuously tracking the fast-moving small target includes:

[0150] The total loss function is constructed by the positioning loss function and the classification loss function. The fast-moving small target tracking network is end-to-end pre-trained through the training set combined with the total loss function, so that the network can initially learn the characteristics and tracking mode of the target; then the binary cross entropy loss function is used to train the score prediction head separately to obtain a trained fast-moving small target tracking network; among them, the positioning loss function is the intersection-over-union loss function and the center distance loss function; the classification loss function is the cross entropy loss function.

[0151] The test set is input into the trained fast-moving small target tracking network, and the template area and search area are initialized based on the first visible light image of each video sequence in the test set and the corresponding event voxel grid data. In each subsequent frame, the predicted search area is extracted from the current frame and input into the trained fast-moving small target tracking network to obtain the target prediction bounding box area and comprehensive confidence. When the comprehensive confidence is higher than the update threshold and the predicted search area contains the target, the template area is dynamically updated to achieve continuous tracking of fast-moving small targets.

[0152] In step 5, the total loss function is:

[0153] ;

[0154] Where, is the total loss value, is the intersection-over-union loss value in the positioning loss function, yes The coefficient of is the center distance loss value in the positioning loss function, yes The coefficient of is the cross entropy loss value in the classification loss function, yes The coefficient of ; where

[0155] The calculation method of the intersection-over-union loss is: ;

[0156] The calculation method of the center distance loss value is:

[0157] ;

[0158] The cross entropy loss value is calculated as:

[0159] ;

[0160] Where, The center point coordinates of the target prediction bounding box area, is the center point coordinate of the target real bounding box area, is the prediction confidence in the training set, ; is the label in the training set, i is the category number of the training set data, and C is the total number of categories of the training set data;

[0161] The binary cross entropy loss function is:

[0162] ;

[0163] Where, is the binary cross entropy loss value.

[0164] In order to illustrate the technical solution of the present invention, a specific example is now provided:

[0165] 1. Data acquisition and preprocessing: Collect video sequences containing multiple fast-moving small targets. Each video sequence consists of a visible light image sequence. and the corresponding event stream dataset , build training set and test set in the ratio of 7:3. The duration of the event is discretized, and the event polarity of each event is assigned to the two closest space-time voxels to form an event voxel grid dataset, which can be fused with the visible light image. The event voxel grid dataset is:

[0166] ;

[0167] Where, is the event voxel grid dataset, is the coordinate of the event voxel grid data, is the timestamp of the discrete time center, is the coordinate of the event stream data, is the event polarity, is the event timestamp, B is the total number of discrete times, is the total time span of the event stream. Based on repeated experiments, the best result is found when B=5.

[0168] Based on each video sequence in the training set, the target's true bounding box is cropped from the visible light image of the first frame label as a visible light image template; at the same time, the same area is cropped from the corresponding event voxel grid data as the event voxel template, and the visible light image template and the corresponding event voxel template are used together as the template area; with the center point of the template area as the reference, a search area for target search is generated according to a preset multiple of the template area.

[0169] The visible light image sequence contained in each video sequence in the template area The corresponding event voxel grid dataset A fused dataset is formed by splicing along the channel dimension, and enhancement operations simulating complex scene changes are performed on the fused dataset, including horizontal / vertical flipping, spatial translation, random offset of the bounding box, and center point jitter, so as to simulate complex scene changes and improve the generalization ability of the model to obtain an enhanced dataset; the enhanced dataset is split into a visible light image enhancement sequence and an event voxel grid enhancement dataset according to the channel dimension, providing diverse and robust input data for subsequent feature extraction.

[0170] 2. Network training:

[0171] (1) In the pre-training stage, set the training parameters. In this example, the initial learning rate is 0.001, the number of iterations is 500, and the batch size is 8. Use the intersection-over-union loss function and the center distance loss function for positioning loss functions, and the cross entropy loss function for classification loss functions. According to the total loss function The network is trained end-to-end, and the network parameters are updated through the back-propagation algorithm, so that the network gradually learns the characteristics and tracking mode of the target.

[0172] (2) In the score prediction head stage, the parameters of other parts of the network are fixed and only the score prediction head is trained. The binary cross entropy loss function is used. ,According to the labels in the training data, the parameters of the score prediction head are adjusted so that it can accurately predict the reliability score of the tracking state.

[0173] 3. Test tracking process:

[0174] The test set is input into the trained fast-moving small target tracking network, and the template area and search area are initialized based on the first visible light image of each video sequence in the test set and the corresponding event voxel grid data. In the subsequent frame, the predicted search area is extracted from the current frame and input into the trained fast-moving small target tracking network to obtain the target prediction bounding box area and comprehensive confidence. When the comprehensive confidence is higher than the update threshold and the predicted search area contains the target, the template area is updated, and then tracking is continued in the next frame to achieve continuous tracking of fast-moving small targets.

[0175] 4. Experimental Results

[0176] The proposed dual-modal fusion fast-moving small target tracking method was experimentally verified using an event drone dataset. The dataset has challenging scenes such as exposure, low light, fast motion, and complex background. After testing and evaluation, the method can achieve stable and continuous tracking of small targets and clearly lock the target position. The tracking results are as follows: Figures 1 to 6 As shown. Among them, Figure 1 This is a diagram of a visible light image against a low-light square background. Due to low light conditions, the contrast between the target and the dark background in the visible light image is extremely low. Furthermore, the pixel ratio of the small drone target is small, making it difficult to detect the target based solely on texture information, resulting in tracking loss. Figure 2 This is a schematic diagram of an event voxel grid against a low-light square background. Event stream data can effectively capture the target's motion trajectory by recording changes in pixel brightness, which is presented as dense spatiotemporal voxel clusters in the figure. Figure 3 This is the target tracking result after bimodal fusion in a low-light square background. By fusing visible light texture and event flow motion features, the network accurately outputs the target bounding box, verifying the effectiveness of bimodal data complementarity in low-light scenes. Figure 4 This is a diagram of a visible light image against a complex forest background. The forest background has high texture complexity, which makes the visual features of the drone target highly similar to those of leaves and shadows, making it difficult to clearly detect the target in the visible light image. Figure 5 This is a schematic diagram of the event voxel grid against a complex forest background. The event stream focuses on the relative motion of the target, forming a clear outline of the drone target in the figure. Figure 6 This is a schematic diagram of the target tracking results of dual-modal fusion in a complex forest background. By fusing the motion information of the event stream and the texture information of the visible light image, stable target tracking is achieved, proving the robustness of dual-modal fusion in complex backgrounds.

[0177] In addition, this example is compared with other mainstream methods. When evaluating the performance of the algorithm, the overlap success rate is calculated by calculating the intersection of the target bounding box in the tracking result and the true target bounding box. To measure. When the value is greater than the update threshold, the tracking is considered successful, and the success rate is calculated by counting the proportion of successful frames to the total number of frames. The tracking accuracy is measured by the Euclidean distance between the target center position estimated by the tracking algorithm and the target's true center position. Assuming that the target center coordinates estimated by the tracking algorithm are , the true target center coordinates are , then by the formula Calculate the distance error. The smaller the error, the higher the accuracy.

[0178] from Figure 7 The Overall Success Plot shown and Figure 8The "OverallPrecision Plot" shows that the technical solution disclosed in the present invention has significant advantages in the task of tracking small targets with fast movement. In terms of accuracy, the average accuracy of the method of the present invention is 0.757, which is significantly higher than mainstream methods such as dimp18 (0.624), dimp50 (0.663), prdimp18 (0.492), prdimp50 (0.599), super_dimp (0.645), dimp_simple (0.651), KCF (0.296), rts50 (0.692), siamfc (0.496), tamos (0.588), tomp50 (0.585), and tomp101 (0.575). In terms of success rate, the average success rate of the proposed method is 0.454, which is superior to methods such as dimp18 (0.376), dimp50 (0.425), prdimp18 (0.287), prdimp50 (0.356), super_dimp (0.394), dimp_simple (0.393), KCF (0.158), rts50 (0.410), siamfc (0.267), tamos (0.367), tomp50 (0.350), and tomp101 (0.343). Overall, the data in Figures 7 and 8 show that the technical solution disclosed in the present invention surpasses existing methods in both core indicators of accuracy and success rate, fully verifying its excellent performance and reliability in small target tracking tasks.

[0179] The proposed fast-moving small target tracking network was compared with different single-modal target tracking networks using the same network framework. The key evaluation metrics were the average tracking success rate and average tracking accuracy of the target tracking results. Table 1 shows the comparative results of the tracking experiments using different modalities in the examples of the present invention:

[0180] Table 1

[0181] Average precision Average success rate Visible light 0.498 0.317 event 0.573 0.396 Events and visible light 0.757 0.454

[0182] As can be seen from Table 1, in the fast-moving small target scene, the fast-moving small target tracking method based on the dual-modal fusion of events and visible light outperforms the single-modal tracking method of visible light or events in terms of average tracking success rate and average tracking accuracy, confirming the synergistic gain of visible light texture and event flow motion features.

[0183] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A dual-modal fusion fast-moving small target tracking method, characterized in that: The method includes: Step 1: Preprocess the target dataset consisting of video sequences containing fast-moving small targets and divide it into a training set and a test set. Based on the first frame label of each video sequence in the training set, crop the target template area and search area. Step 2: Data enhancement, feature extraction, and feature fusion are performed on the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset to obtain a fused feature map; Step 3: The multi-layer Transformer encoder extracts the long-range dependencies between features in the fused feature map, and the output enhanced feature representation and the target query vector are input into the multi-layer Transformer decoder to learn and output a robust feature representation for target localization. Step 4: The bounding box prediction head calculates the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity. The features of the search area are weighted and aggregated through the similarity matrix and then input into the fully convolutional network. The coordinate probability distribution of the target bounding box is output, and the target prediction bounding box area is calculated based on the coordinate probability distribution of the target bounding box. At the same time, the comprehensive confidence of the target prediction bounding box area is obtained through the score prediction head and the intersection-union ratio. Step 5: Taking the center point of the target prediction bounding box of the previous frame as the reference, generate the prediction search area of ​​the current frame with a preset multiple of the target prediction bounding box area; when the comprehensive confidence is higher than the update threshold and the prediction search area contains the target, the target real bounding box area is cropped from the visible light image of the current frame and the corresponding event voxel grid data, and after dynamically updating the template area, the constructed fast-moving small target tracking network is obtained; the fast-moving small target tracking network is trained and tested through the training set and test set to achieve continuous tracking of fast-moving small targets.

2. The method according to claim 1, wherein In step 1, the target dataset consisting of video sequences containing fast-moving small targets is preprocessed and divided into training and test sets as follows: Collect video sequences containing multiple fast-moving small targets. Each video sequence consists of a visible light image sequence and a corresponding event stream dataset. The duration of events in the event stream dataset is discretized, and the event polarity of each discretized event is assigned to the two closest space-time voxels to form an event voxel grid dataset; The visible light image sequence and the corresponding event voxel grid dataset are combined into a target dataset; The target dataset is divided into a training set and a test set according to a preset ratio; wherein the event voxel grid dataset is: ； Where, is the event voxel grid dataset, is the coordinate of the event voxel grid data, is the timestamp of the discrete time center, is the coordinate of the event stream data, is the event polarity, is the event timestamp, B is the total number of discrete times, is the total time span of the event stream.

3. The method according to claim 2, wherein In step 1, based on the first frame label of each video sequence in the training set, the method for cutting out the target template area and search area is as follows: For each video sequence in the training set, the target ground truth bounding box is cropped from the first frame of the labeled visible light image as the visible light image template; at the same time, the same area is cropped from the corresponding event voxel grid data as the event voxel template, and the visible light image template and the corresponding event voxel template are used together as the template area; The search area for target search is generated based on the center point of the template area and the preset multiple of the template area.

4. The method according to claim 3, wherein In step 2, the visible light image sequence contained in each video sequence in the search area is subjected to data enhancement, feature extraction, and feature fusion with the corresponding event voxel grid dataset to obtain a fused feature map as follows: The visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset, and a data augmentation operation simulating complex scene changes is performed on the fused dataset to obtain a data augmented dataset; Splitting the data augmentation dataset into a visible light image augmentation sequence and an event voxel grid augmentation dataset according to the channel dimension; After introducing the channel attention mechanism to improve the lightweight convolutional neural network MobileNetV3-Large, feature extraction is performed on the visible light image enhancement sequence to obtain the enhanced feature map of the visible light image sequence; After introducing the time gating mechanism to improve the 3D convolutional network, feature extraction is performed on the event voxel grid enhanced dataset to obtain the enhanced feature map of the event voxel grid dataset; The enhanced feature map of the visible light image sequence and the enhanced feature map of the event voxel grid dataset are fused at the channel level to obtain a fused feature map.

5. The method according to claim 4, wherein In step 2, the visible light image sequence contained in each video sequence in the search area and the corresponding event voxel grid dataset are spliced ​​along the channel dimension to form a fused dataset as follows: The visible light image of the visible light image sequence of each video sequence in the search area with a size of (H1, W1, C1) is spliced ​​with the event voxel grid data of the corresponding size of (H2, W2, C2) in the event voxel grid data along the channel dimension to form fused data of size (H, W, C1 + C2); where H1 is the height of the visible light image, W1 is the width of the visible light image, C1 is the number of channels of the visible light image, H2 is the height of the event voxel grid data, W2 is the width of the event voxel grid data, C2 is the number of channels of the event voxel grid data, H is the height of the fused data, and W is the width of the fused data; where H1, H2 and H are equal, and W1, W2 and W are equal. Methods for performing augmentation operations on the fused dataset to simulate complex scene changes include: horizontal flipping, vertical flipping, spatial translation, random offset of bounding boxes and / or center point jittering; The fused feature map is: ； Where, is the fusion feature map, It is the splicing operation of channel-level feature fusion, Enhance the feature map for the visible light image sequence, Enhanced feature maps for event voxel grid datasets.

6. The method according to claim 5, wherein In step 3, the long-range dependencies between features in the fused feature map are extracted through a multi-layer Transformer encoder. The output enhanced feature representation and the target query vector are input into a multi-layer Transformer decoder. The method for learning and outputting a robust feature representation for target localization is as follows: After using the bottleneck layer to compress and flatten the number of channels in the fused feature map and add absolute position encoding, the fused feature map is input into a multi-layer Transformer encoder. The multi-head self-attention mechanism and feedforward neural network contained in each layer of the encoder in the multi-layer Transformer encoder extract the long-range dependencies between the features in the fused feature map and output an enhanced feature representation. The target query vector is combined with the enhanced feature representation and input into the multi-layer Transformer decoder. After the self-attention mechanism, encoder and decoder attention mechanism and feedforward neural network contained in each layer of the decoder in the multi-layer Transformer decoder, the target query vector set is focused on the feature area related to the target, and the robust feature representation of target positioning is learned and output.

7. The method according to claim 6, wherein In step 4, the method for calculating the similarity matrix between the robust feature representation and the features of the template area based on the dot product similarity by the bounding box prediction head is: ； Where, is the similarity matrix, is the normalization function, is a robust feature representation, is the transpose of the features of the template region, is the feature of the template region, is the scaling factor; The method of weighted aggregation of the features of the search area through the similarity matrix to obtain the weighted aggregation feature set is: ； Where, is the weighted aggregate feature set, is the characteristic of the search area; The method for calculating the target prediction bounding box area based on the coordinate probability distribution of the target bounding box is: ； ； Where, The coordinates of the lower right corner of the target prediction bounding box area, Predict the upper left corner coordinate of the bounding box area for the target; is the height of the feature map in the target bounding box, is the width of the feature map in the target bounding box, is the spatial position of the feature map in the target bounding box, is the probability distribution of the coordinates of the lower right corner of the target bounding box, is the probability distribution of the upper left corner coordinate of the target bounding box.

8. The method according to claim 7, wherein In step 4, the method for obtaining the comprehensive confidence of the target prediction bounding box area through the score prediction head and intersection-over-union is: ； ； Where, is the comprehensive confidence level, is the weight coefficient, is the intersection and union ratio, is the prediction score of the score prediction head, The intersection area of ​​the target prediction bounding box area, The union area of ​​the bounding box regions predicted for the object.

9. The method according to claim 8, wherein In step 5, the fast-moving small target tracking network is trained and tested using the training set and the test set. The method for continuously tracking the fast-moving small target includes: The total loss function is constructed from the positioning loss function and the classification loss function. The fast-moving small target tracking network is trained using the training set combined with the total loss function. The score prediction head is then trained separately using the binary cross entropy loss function to obtain a trained fast-moving small target tracking network. The test set is input into the trained fast-moving small target tracking network, and the template area and search area are initialized based on the first visible light image of each video sequence in the test set and the corresponding event voxel grid data. In each subsequent frame, the predicted search area is extracted from the current frame and input into the trained fast-moving small target tracking network to obtain the target prediction bounding box area and comprehensive confidence. When the comprehensive confidence is higher than the update threshold and the predicted search area contains the target, the template area is dynamically updated to achieve continuous tracking of fast-moving small targets.

10. The method according to claim 9, wherein In step 5, the total loss function is: ； Where, is the total loss value, is the intersection-over-union loss value in the positioning loss function, yes The coefficient of is the center distance loss value in the positioning loss function, yes The coefficient of is the cross entropy loss value in the classification loss function, yes The coefficient of ; where The calculation method of the intersection-over-union loss is: ; The calculation method of the center distance loss value is: ； The cross entropy loss value is calculated as: ； Where, The center point coordinates of the target prediction bounding box area, is the center point coordinate of the target real bounding box area, is the prediction confidence in the training set, is the label in the training set, i is the category number of the training set data, and C is the total number of categories of the training set data; The binary cross entropy loss function is: ； Where, is the binary cross entropy loss value.

Images