Dual-mode target tracking method, system and equipment based on optical camera and event camera and medium

Abstract

A bimodal target tracking method, system, device and medium based on an optical camera and an event camera, the method comprising: collecting optical data and event data, constructing a multi-modal data set RE, dividing the multi-modal data set RE into a training set REtrain and a test set REtest according to a certain proportion; constructing a bimodal target tracking network based on an optical camera and an event camera; training a bimodal target tracking network based on an optical camera and an event camera by using a training set REtrain in the multi-modal data set RE to obtain an optimal training weight file FEtrack.pt; performing target tracking test on the bimodal target tracking network based on the optical camera and the event camera by using a test set REtest in the multi-modal data set RE and the optimal training weight file FEtrack.pt to obtain a target tracking result; the system, the equipment and the medium are used for implementing the method. The method is used for solving the problem of low target tracking precision of the existing target tracking algorithm under the conditions of low illumination, high dynamic range and rapid target movement.

Description

Technical Field

[0001] The present invention belongs to the technical field of target tracking, and in particular relates to a dual-modal target tracking method, system, device and medium based on an optical camera and an event camera. Background Art

[0002] Target tracking is one of the core tasks in the field of computer vision. It has extensive application value and important significance in many practical applications such as intelligent monitoring, autonomous driving, drone cruising, robot navigation, and human-computer interaction. With the rapid development of artificial intelligence technology, the research on target tracking not only meets the basic requirements of real-time and high precision, but also provides new possibilities for target recognition and tracking in complex scenes. In traditional methods, technologies based on correlation filtering or target detection are often used. Although they can achieve target tracking to a certain extent, they are often difficult to ensure the robustness and stability of tracking when faced with problems such as occlusion, background complexity, and illumination changes. In recent years, the rise of deep learning has injected new vitality into target tracking. Target tracking methods based on models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and Transformers have significantly improved the feature extraction ability and adaptability of the models. At the same time, the fusion of multimodal data has also become a research hotspot. With the help of data collaborative processing of multiple sensors such as vision, infrared, and lidar, target information can be captured more accurately in complex environments.

[0003] Although deep learning technology has made many breakthroughs in the field of target tracking, it still faces some challenges. For example, in unstructured environments, targets may move quickly, deform non-rigidly, or be occluded for a long time, making it difficult for the tracker to maintain accuracy. In addition, the trade-off between real-time performance and computational efficiency has always been a difficult problem in the design of target tracking algorithms. High-precision models are usually accompanied by high computational overhead and are difficult to deploy on resource-constrained devices. To this end, researchers have proposed methods such as lightweight network structure, attention mechanism optimization, and feature fusion module to improve tracking performance while reducing computational complexity. It is worth noting that Transformer-based target tracking methods have attracted much attention in recent years. With their powerful global modeling capabilities and flexible feature interaction mechanisms, these methods have gradually become a research hotspot in the field of target tracking. In addition, multimodal target tracking technology has gradually emerged, using the deep fusion of multiple modal information such as frame images and event data, optical and thermal imaging, etc., to provide new ideas for solving target tracking problems in complex scenes.

[0004] The patent application document with publication number CN114170269A discloses a multi-target tracking method, device and storage medium based on spatiotemporal correlation. The implementation steps of the method are as follows: for any given frame of pedestrian image, a pair of motion images is formed and sent to a trained deep neural network tracking model, and the position of the pedestrian target in the current frame image and the previous frame image is directly output, and the continuous tracking of multiple pedestrian targets is realized in this way; the deep neural network tracking model includes a feature extraction module backbone module, a detection and tracking module detect-track module, and an update module update module, wherein the update module update module does not participate in training and only works during testing; the invention uses a deep neural network model with adaptive spatiotemporal correlation to directly complete the multi-target tracking process end-to-end, with strong versatility, high real-time performance, fewer error sources, long-term tracking, and strong tracking effect robustness. However, the invention still has the disadvantage that when the target moves quickly, the target outline is blurred, and in low light conditions, the target tracking accuracy of the invention is still low.

[0005] The patent application document with publication number CN116363163A discloses a spatial target detection and tracking method based on an event camera, and the implementation steps of the method are as follows: step S1, characterizing the collected event signal sequence as a three-dimensional voxel; step S2, inputting the three-dimensional voxel into the trained target detection network to obtain the position sequence of the target within the current t moments; step S3, fitting the position sequence within the current t moments, and obtaining the target motion trajectory function within the current t moments; step S4, using the target trajectory function to calculate the target position at the next moment, and at the same time judging whether the target detection network has a detection result at the next moment, and correcting the position of the target at the next moment according to the detection result and the calculation result of the trajectory prediction; and updating the target position. The invention can use event data for target detection and tracking, and improve the detection and tracking accuracy of space long-distance weak targets. However, the invention still has the disadvantage that, due to the lack of texture features of the target in the event data, the target tracking accuracy of the invention is still low when the object moves slowly or is stationary.

[0006] Defects and shortcomings of the existing technology:

[0007] 1. Existing methods are very sensitive to changes in lighting, and their performance degrades significantly in low-light, strong backlight, or complex shadow environments. At the same time, bad weather conditions (such as rain, fog, snow) and interference from transparent or reflective surfaces can also seriously affect the tracking effect.

[0008] 2. Since event data only captures pixel-level brightness changes, the ability to track stationary targets or slowly changing scenes is limited, resulting in degraded performance when target motion is not significant.

[0009] 3. Existing methods are highly dependent on hardware characteristics (such as event resolution and latency), lack robustness and applicability, and are difficult to achieve a comprehensive balance of performance under different environments and task requirements. Summary of the invention

[0010] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a dual-modal target tracking method, system, device and medium based on an optical camera and an event camera, and to construct an efficient dual-stream feature interaction mechanism through a dual-stream attention fusion (DSAF) module to enhance the correlation and interaction ability between features in the same modality; to introduce a multi-scale feature fusion mechanism through a deep cross-feature interaction and enhancement (DCFIE) module to comprehensively integrate local information and global information, thereby enriching the detail level of feature expression; to efficiently capture the fine-grained correlation between multi-modal data through a cross-modal Transformer fusion (CMTF) module, and to achieve deep fusion; the present invention is used to solve the problem of low target tracking accuracy of existing target tracking algorithms under low light, high dynamic range and fast target movement.

[0011] In order to achieve the above object, the technical solution adopted by the present invention is:

[0012] A dual-modal target tracking method based on an optical camera and an event camera comprises the following steps:

[0013] Step 1: Collect optical data and event data, construct a multimodal dataset RE, and divide the multimodal dataset RE into training sets RE according to a certain ratio train and the test set RE test ;

[0014] Step 2: Build a dual-modal target tracking network based on optical camera and event camera;

[0015] Step 3: Use the training set RE in the multimodal dataset RE constructed in step 1 train The dual-modal target tracking network based on optical camera and event camera constructed in step 2 is trained to obtain the optimal training weight file FEtrack.pt;

[0016] Step 4: Use the test set RE in the multimodal dataset RE constructed in step 1 testThe dual-modal target tracking network based on optical camera and event camera constructed in step 2 is tested with the optimal training weight file FEtrack.pt obtained in step 3 to obtain the target tracking result.

[0017] The dual-modal target tracking network based on an optical camera and an event camera in step 2 includes a feature extraction network (Feature Extraction), a feature fusion network (Feature Fusion) and a classification and regression network (Classification & Regression); the feature extraction network (Feature Extraction) is responsible for extracting optical feature maps and event feature maps from the input optical data and event data; the feature fusion network fuses the optical feature maps and event feature maps extracted by the feature extraction network to obtain fused features; the fused features are input into the classification and regression network to complete the target tracking and positioning tasks.

[0018] The feature extraction network (Feature Extraction) includes a backbone network and two dual stream attention fusion (dual stream attention fusion module, DSAF) modules; the backbone network is composed of two groups of Swin Transformer networks, which are used to extract initial feature maps T and feature maps S from a template (Frame Template) and a search (Frame Search) of optical data or event data, wherein the Swin Transformer includes Swin Transformer stage1, Swin Transformer stage2 and Swin Transformer stage3; the dual stream attention fusion (dual stream attention fusion module, DSAF) module is used to fuse and enhance the feature maps T and S; the dual stream attention fusion (dual stream attention fusion module, DSAF) module includes a convolution module (Conv), a self-attention mechanism module (Self-Attention), a multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA), a fusion feature enhancement module (Fusion Feature Enhancement Module, FFEM) and a fully connected convolution layer (Fully Connected Layer, FC); dual stream attention fusion (dual stream attention fusion module, DSAF) module is expressed as follows:

[0019] F1=Conv(T),F2=Self-Attention(S)

[0020]

[0021] F4=ReLU(Concat(F1,F2,F3),Conv(F1,F2,F3))

[0022] F5 = σ(W2·ReLU(W1·F4))

[0023] F out =FC(F5)+F5

[0024] Among them, ReLU is the activation function, Concat is the summation operation, σ represents the Sigmoid activation function, and W2 and W1 represent weight factors;

[0025] First, use the convolution module (Conv) from the template image Extract the local feature map F1 from the search image and use the self-attention mechanism module (Self-Attention) The global feature map F2 is obtained from the local feature map F1; next, the local feature map F1 and the global feature map F2 interact with each other through the multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA) to generate a fused feature map F3; then, the fused feature map F3 is further enhanced using the fusion feature enhancement module (FusionFeature Enhancement Module, FFEM) to generate an enhanced feature map F4; then, the enhanced feature map F4 is selectively weighted using the W1 weight factor, ReLU activation function, W2 weight factor and Sigmoid activation function in sequence to generate a feature map F5; finally, the feature map F5 is optimized through the fully connected convolutional layer (Fully ConnectedLayer, FC), and the optimized result is fused with the feature map F5 to generate the final output feature map F out ;

[0026] The initial feature map T and feature map S of the optical data are processed using the dual stream attention fusion module (DSAF) to obtain the enhanced feature map X f ; The initial feature map T and feature map S of the event data are processed using the dual stream attention fusion module (DSAF) to obtain the enhanced feature map X e .

[0027] The feature fusion network (Feature Fusion) includes two deep cross feature interaction and enhancement modules (deep cross feature interaction and enhancement, DCFIE) and a cross-modal Transformer fusion module (cross-modal Transformer fusion, CMTF); the deep cross feature interaction and enhancement module (deepcross feature interaction and enhancement, DCFIE) includes two inputs, which are the feature map X output by the dual stream attention fusion module (dual stream attention fusion module, DSAF module) f and feature map X e , four convolution modules (Conv), 2L multi-head self-attention mechanism modules (Multi-Head Self-Attention, MHSA), L ≥ 3, an upsampling module (Upsample), a downsampling module (Downsample), a global average pooling (GlobalAverage Pooling, GAP) and a layer normalization module (LayerNorm, LN); the deep cross feature interaction and enhancement module (deep cross feature interaction and enhancement, DCFIE) is expressed as follows:

[0028] X1=Conv(X f ),X2=Conv(X e )

[0029] X 12 =X 12 (L) =MHSA((X1 Q ,X1 K ,X2 V ) (L-1) )

[0030] X 22 =X 22 (L) =MHSA((X2 Q ,X2 K ,X1 V ) (L-1) )

[0031] X3=Upsample(X 12 )+Downsample(X 22 )

[0032] X out =Conv(LN(GAP(X3)+Conv(X3)));

[0033] Using the input feature map X f and feature map X e , when two features can interact in the same spatial dimension, each input feature map is initialized and modified using the convolution module (Conv) to obtain feature maps X1 and X2; next, the L-layer multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA) is used to gradually enhance the information enhancement effect and enhance the interaction between features to obtain feature map X 12 and feature map X 22 ; Then use the convolution module (Conv) to perform the feature map X 12 and feature map X 22 Upsampling and downsampling are performed, and the processed feature maps are concat-operated to generate feature map X3. The global information enhancement module obtains feature map X4 by performing a global average pooling (GAP) operation on feature map X3. The local information enhancement module obtains feature map X5 by performing a convolution module (Conv) on X3. Feature maps X4 and X5 are normalized by a layer normalization module (LayerNorm, LN) to obtain feature map X6. Finally, the output X is generated by using a convolution module (Conv) on feature map X6. out ;

[0034] For the input feature map X f and feature map X e Use the deep cross feature interaction and enhancement module (DCFIE) to perform two processings to obtain feature maps M1 and M2 respectively;

[0035] The cross-modal Transformer fusion module (CMTF) includes two inputs, namely, the feature maps M1 and M2 output by the deep cross feature interaction and enhancement module (DCFIE), two convolutional neural network modules (CNN), two multi-head self-attention mechanism modules (MHSA), a convolution module (Conv), a convolution module with a convolution kernel of 3 (Conv3×3), a convolution module with a convolution kernel of 5 (Conv5×5), a convolution module with a convolution kernel of 7 (Conv7×7) and a global average pooling (GAP); the cross-modal Transformer fusion module (CMTF) is expressed as follows:

[0036] M 11 =CNN(M1),M 21 =CNN(M2)

[0037]

[0038]

[0039]

[0040] M out =Conv(M3,GAP(M3));

[0041] The cross-modal Transformer fusion module (CMTF) encodes the feature maps M1 and M2 through the convolutional neural network module (CNN) and converts the feature maps of each modality into a fixed-size feature map M 11 and M 21 ; Next, the feature map M 11 and M 21 The correlation between them will be captured by the Multi-Head Self-Attention mechanism (MHSA), which uses the multi-head self-attention mechanism to capture the feature map M 11 and M 21 Processing is performed to obtain the feature map M12 and M 22 , and then the feature map M 12 and M 22 Through summation processing, fusion is performed to obtain the feature map M Fu ; Then, the multi-scale fusion (MSF) module is used to fusion the feature map M Fu Multi-scale feature extraction is performed, where the specific operation of the multi-scale fusion (MSF) module is as follows: a convolution module with a convolution kernel of 3 (Conv3×3), a convolution module with a convolution kernel of 5 (Conv5×5), and a convolution module with a convolution kernel of 7 (Conv7×7) are used to extract the feature map M. Fu The three feature maps are processed in sequence, and the processed three feature maps are fused by summing to obtain feature map M3; finally, a global average pooling (GAP) operation is performed on feature map M3, and the result and feature map M3 are input into the convolution module (Conv) to generate the final output feature map M out ; The feature map M out The data is further input into the classification and regression network to classify and predict the target location.

[0042] In step 3, the number of training rounds is set to be greater than or equal to 100, and the batch size is set to be greater than or equal to 16.

[0043] In step 4, the test batch size is set to be greater than or equal to 4.

[0044] The present invention also provides a dual-modal target tracking system based on an optical camera and an event camera, comprising:

[0045] Data acquisition module: used to collect optical data and event data, construct a multimodal data set RE, and divide the multimodal data set RE into training sets RE according to a certain ratio train and the test set RE test ;

[0046] Network building module: used to build a dual-modal target tracking network based on optical camera and event camera;

[0047] Network training module: used to use the training set RE in the multimodal dataset RE train Train the dual-modal target tracking network based on optical camera and event camera to obtain the optimal training weight file FEtrack.pt;

[0048] Target tracking module: used to use the test set RE in the multimodal dataset RE testThe target tracking test is performed on the dual-modal target tracking network based on optical camera and event camera using the optimal training weight file FEtrack.pt to obtain the target tracking results.

[0049] The present invention also provides a dual-modal target tracking device based on an optical camera and an event camera, comprising:

[0050] Memory: a computer program storing the above-mentioned dual-modal target tracking method based on an optical camera and an event camera, which is a computer-readable device;

[0051] Processor: used to implement the dual-modal target tracking method based on an optical camera and an event camera when executing the computer program.

[0052] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the dual-modal target tracking method based on an optical camera and an event camera.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. This invention proposes a dual-stream attention fusion (DSAF) module based on the Transformer architecture, which aims to enhance the correlation and interaction between features in the same modality by constructing an efficient dual-stream feature interaction mechanism. Specifically, this module uses the global attention mechanism of Transformer to effectively capture the fine-grained feature relationships within the modality and further mine potential contextual information. By introducing the dual-stream structure, the DSAF module can perform deep modeling on different feature channels respectively and fuse multi-level feature information, thereby significantly improving the feature representation capability and providing more robust support for target recognition and detection tasks.

[0055] 2. The present invention proposes a Deep Cross-Feature Interaction and Enhancement (DCFIE) module, which introduces a multi-scale feature fusion mechanism to comprehensively integrate local information and global information, thereby enriching the detail level of feature expression. Specifically, the DCFIE module adopts a deep cross-interaction strategy to fully capture the relationship between features of different scales and enhance the synergy between features. At the same time, the module design also focuses on strengthening the recognition ability of target features and effectively suppressing background interference. In target tracking tasks, the DCFIE module can significantly improve feature representation capabilities and matching accuracy in complex scenarios, thereby greatly improving the accuracy, speed and robustness of target tracking.

[0056] 3. The present invention proposes a Cross-Modal Transformer Fusion (CMTF) module, which aims to efficiently capture the fine-grained associations between multimodal data and achieve deep fusion. Specifically, the CMTF module uses the powerful modeling capabilities of Transformer to accurately capture the semantic relationships between different modal features through a global attention mechanism, while designing a special interaction strategy to enhance the synergy between modalities. This module can not only fully retain the unique characteristics of each modality, but also effectively eliminate the information gap caused by modal differences, thereby improving the overall performance and robustness of multimodal tasks.

[0057] In summary, the present invention designs a brand-new network structure, which effectively fuses the global information and local features of the image by introducing the transformer's dual-stream attention fusion DSAF module, deep cross-feature interaction and enhancement DCFIE module and cross-modal transformer fusion CMTF module into the network, and realizes complementarity. It can effectively improve the target tracking accuracy in low light, high dynamic range and rapid target motion, and has the advantages of strong adaptability and high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a schematic diagram of a method flow of an embodiment of the present invention.

[0059] Figure 2 This is a diagram of a dual-modal target tracking network based on an optical camera and an event camera according to an embodiment of the present invention.

[0060] Figure 3 It is a dual stream attention fusion module (DSAF) module structure diagram of an embodiment of the present invention.

[0061] Figure 4 It is a structural diagram of a deep cross feature interaction and enhancement (DCFIE) module of an embodiment of the present invention.

[0062] Figure 5 It is a structural diagram of a cross-modal Transformer fusion module (CMTF) according to an embodiment of the present invention.

[0063] Figure 6 This is a target tracking accuracy diagram of the present invention and the existing method. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings.

[0065] like Figure 1 As shown, a dual-modal target tracking method based on an optical camera and an event camera includes the following steps:

[0066] like Figure 2 As shown, step 1: construct a dual-modal target tracking network based on optical camera and event camera, which consists of feature extraction network, feature fusion network and classification and regression network;

[0067] The feature extraction network is responsible for extracting rich optical feature maps and event feature maps from the input optical data and event data. The core of this step is to use the ability of deep neural networks to capture the key information of different modal data and construct their independent feature representations. Secondly, the feature fusion network plays a bridging role, fusing the optical feature maps and event feature maps extracted by the feature extraction network to obtain fused features. Since optical and event data are complementary in characteristics, the design of the feature fusion network is particularly critical. The network can effectively learn the relationship between the two modalities and enhance the robustness and accuracy of target tracking. At the same time, it also ensures that the fused features retain the core information of the two modalities to support the processing of downstream tasks. Finally, the fused features are input into the classification and regression network. The classification uses the Focal loss function to determine the category of the target, and the regression uses the GIoU loss function to predict the bounding box or position information of the target. Classification and regression work together to complete the tracking and positioning tasks of the target. Specifically, the network identifies the target in the input feature through the classification branch and predicts the bounding box or position information of the target through the regression branch. The performance of this module directly determines the accuracy and efficiency of the entire system.

[0068] A feature extraction network (Feature Extraction) includes a backbone network and two dual stream attention fusion (DSAF) modules; wherein the backbone network is composed of two groups of SwinTransformer networks, which are used to extract initial feature maps T and feature maps S from the template (Frame Template) and search (Frame Search) of optical data or event data, wherein Swin Transformer includes SwinTransformer stage1, Swin Transformer stage2 and Swin Transformer stage3; the dual stream attention fusion (DSAF) module is used to fuse and enhance the feature maps T and S; Figure 3 As shown in the figure, the dual stream attention fusion module (DSAF) module includes a convolution module (Conv), a self-attention mechanism module (Self-Attention), a multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA), a fusion feature enhancement module (FusionFeature Enhancement Module, FFEM) and a fully connected convolution layer (Fully Connected Layer, FC); The dual stream attention fusion module (DSAF) module is expressed as follows:

[0069] F1=Conv(T),F2=Self-Attention(S)

[0070]

[0071] F4=ReLU(Concat(F1,F2,F3),Conv(F1,F2,F3))

[0072] F5 = σ(W2·ReLU(W1·F4))

[0073] F out =FC(F5)+F5

[0074] Among them, ReLU is the activation function, Concat is the summation operation, σ represents the Sigmoid activation function, and W2 and W1 represent weight factors. Specifically, we first use the convolution module (Conv) to extract the Extract the local feature map F1 from the search image and use the self-attention mechanism module (Self-Attention) The global feature map F2 is obtained from the local feature map F1; next, the local feature map F1 and the global feature map F2 interact with each other through the multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA) to generate a fused feature map F3; then, the fused feature map F3 is further enhanced using the fusion feature enhancement module (FusionFeature Enhancement Module, FFEM) to generate an enhanced feature map F4; then, the enhanced feature map F4 is selectively weighted using the W1 weight factor, ReLU activation function, W2 weight factor and Sigmoid activation function in sequence to generate a feature map F5; finally, the feature map F5 is optimized through the fully connected convolutional layer (Fully ConnectedLayer, FC), and the optimized result is fused with the feature map F5 to generate the final output feature map F out , thus significantly improving the target tracking performance;

[0075] The initial feature map T and feature map S of the optical data are processed using the dual stream attention fusion module (DSAF) to obtain the enhanced feature map X f ; The initial feature map T and feature map S of the event data are processed using the dual stream attention fusion module (DSAF) to obtain the enhanced feature map X e .

[0076] The feature fusion network (Feature Fusion) includes two deep cross feature interaction and enhancement modules (DCFIE) and a cross-modal Transformer fusion module (CMTF). Figure 4As shown in Figure 2, the deep cross feature interaction and enhancement module (DCFIE) includes two inputs, namely the feature map X output by the dual stream attention fusion module (DSAF) module. f and feature map X e , four convolution modules (Conv), 2L multi-head self-attention mechanism modules (Multi-Head Self-Attention, MHSA), L ≥ 3, an upsampling module (Upsample), a downsampling module (Downsample), a global average pooling (Global Average Pooling, GAP) and a layer normalization module (LayerNorm, LN); the deep cross feature interaction and enhancement module (deep cross feature interaction and enhancement, DCFIE) is expressed as follows:

[0077] X1=Conv(X f ),X2=Conv(X e )

[0078] X 12 =X 12 (L) =MHSA((X1 Q ,X1 K ,X2 V ) (L-1) )

[0079] X 22 =X 22 (L) =MHSA((X2 Q ,X2 K ,X1 V ) (L-1) )

[0080] X3=Upsample(X 12 )+Downsample(X 22 )

[0081] X out =Conv(LN(GAP(X3)+Conv(X3)))

[0082] Specifically, using the input feature map X f and feature map X e, when two features can interact in the same spatial dimension, each input feature is initialized and modified using the convolution module (Conv) to obtain the frame feature map X1 and the event feature map X2; next, the L-layer multi-head self-attention mechanism (Multi-Head Self-Attention, MHSA) is used to gradually enhance the information enhancement effect and the interaction between features to obtain the feature map X 12 and feature map X 22 ; Then use the convolution module (Conv) to perform the feature map X 12 and feature map X 22 Upsampling and downsampling are performed, and the processed feature maps are concat-operated to generate feature map X3. This operation enables the network to fuse information of different scales and generate richer and more detailed features. On the basis of multi-scale fusion, local information enhancement module and global information enhancement module are added to process local detail information and global context information respectively. The global information enhancement module obtains feature map X4 by performing global average pooling (GAP) operation on feature map X3, and the local information enhancement module obtains feature map X5 by performing convolution module (Conv) on X3; feature map X4 and feature map X5 are normalized by layer normalization module (LayerNorm, LN) operation to obtain feature map X6; finally, output X is generated by using convolution module (Conv) on feature map X6 out ;

[0083] For the input feature map X f and feature map X e Use the deep cross feature interaction and enhancement module (DCFIE) to perform two processings to obtain feature maps M1 and M2 respectively;

[0084] like Figure 5As shown in the figure, the cross-modal Transformer fusion module (CMTF) includes two inputs, namely, the feature map M1 and feature map M2 output by the deep cross feature interaction and enhancement module (DCFIE), two convolutional neural network modules (CNN), two multi-head self-attention mechanism modules (MHSA), a convolution module (Conv), a convolution module with a convolution kernel of 3 (Conv3×3), a convolution module with a convolution kernel of 5 (Conv5×5), a convolution module with a convolution kernel of 7 (Conv7×7) and a global average pooling (GAP); the cross-modal Transformer fusion module (CMTF) is expressed as follows:

[0085] M 11 =CNN(M1),M 21 =CNN(M2)

[0086]

[0087]

[0088] M Fu =M 12 +M 22

[0089]

[0090] M out =Conv(M3,GAP(M3))

[0091] Specifically, the cross-modal Transformer fusion module (CMTF) encodes the feature map M1 and the feature map M2 through the Convolutional Neural Networks (CNN) module, and converts the feature map of each modality into a fixed-size feature map M 11 and M 21 ; Next, the feature map M 11 and M 21The correlation between them will be captured by the Multi-Head Self-Attention mechanism (MHSA), which uses the multi-head self-attention mechanism to capture the feature map M 11 and M 21 Processing is performed to obtain the feature map M 12 and M 22 , and then the feature map M 12 and M 22 Through summation processing, fusion is performed to obtain the feature map M Fu The advantage of this is that both feature maps can perceive the other feature map, thereby enhancing the complementarity between features. Next, the multi-scale fusion (MSF) module is used to fusion the feature map M Fu Multi-scale feature extraction is performed to improve the fusion of information at different scales. The specific operation of the multi-scale fusion (MSF) module is as follows: a convolution module with a convolution kernel of 3 (Conv3×3), a convolution module with a convolution kernel of 5 (Conv5×5), and a convolution module with a convolution kernel of 7 (Conv7×7) are used to extract the feature map M. Fu The three feature maps are processed in sequence, and the processed three feature maps are fused by summing to obtain feature map M3; finally, a global average pooling (GAP) operation is performed on feature map M3, and the result and feature map M3 are input into the convolution module (Conv) to generate the final output feature map M out ;

[0092] The feature map M out The data is further input into the classification and regression network to classify and predict the target location.

[0093] Multi-scale convolution captures detailed information at different scales. After multi-scale feature fusion, feature representation can be further enhanced by extracting global context information. Global average pooling is used to obtain global feature information. Finally, global features are combined with multi-scale features to generate the final fused feature output M. out .

[0094] Step 2: Use the training set RE in the multimodal dataset RE constructed in step 1 trainThe dual-modal target tracking network based on optical camera and event camera constructed in the second training step sets the training rounds to be greater than or equal to 100 and the batch size to be greater than or equal to 16, and obtains the optimal training weight file FEtrack.pt; compared with the setting of using fewer training rounds in the existing method, setting the training rounds to be greater than or equal to 100 can reduce the risk of underfitting, fully explore the data features, and enhance the adaptability of the model to the task. In addition, a batch size greater than or equal to 16 can also make more efficient use of hardware and increase training speed.

[0095] Step 3: Use the test set RE in the multimodal dataset RE constructed in step 1 test The optimal training weight file FEtrack.pt obtained in step 3 is used to perform target tracking tests on the dual-modal target tracking network based on optical cameras and event cameras constructed in step 2, and the test batch size is set to be greater than or equal to 4 to obtain the target tracking results. Compared with the existing technology, a test batch size greater than or equal to 4 can make full use of hardware resources (such as GPUs and multi-core CPUs) to calculate the prediction results of multiple samples in one forward propagation, thereby significantly improving the test speed. At the same time, it can alleviate the noise impact of the model on a single sample and improve the credibility of test indicators (such as precision and recall).

[0096] The present invention also provides a dual-modal target tracking system based on an optical camera and an event camera, comprising:

[0097] Data acquisition module: used to collect optical data and event data in step 1, construct a multimodal data set RE, and divide the multimodal data set RE into training sets RE according to a certain ratio train and the test set RE test ;

[0098] Network construction module: used to implement the dual-modal target tracking network based on optical camera and event camera in step 2;

[0099] Network training module: used to implement the training set RE in the multimodal dataset RE constructed in step 1 in step 3 train The dual-modal target tracking network based on optical camera and event camera constructed in step 2 is trained to obtain the optimal training weight file FEtrack.pt;

[0100] Target tracking module: used to implement the test set RE in the multimodal dataset RE constructed in step 1 in step 4 test The dual-modal target tracking network based on optical camera and event camera constructed in step 2 is tested with the optimal training weight file FEtrack.pt obtained in step 3 to obtain the target tracking result.

[0101] The present invention also provides a dual-modal target tracking device based on an optical camera and an event camera, comprising:

[0102] Memory: a computer program storing the above-mentioned dual-modal target tracking method based on an optical camera and an event camera, which is a computer-readable device;

[0103] Processor: used to implement the dual-modal target tracking method based on an optical camera and an event camera when executing the computer program.

[0104] The present invention also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it can implement the dual-modal target tracking method based on an optical camera and an event camera.

[0105] The effect of the present invention is further described below in conjunction with simulation experiments:

[0106] 1. Simulation Experiment Conditions

[0107] The hardware platform of the simulation experiment of the present invention is: the processor is Intel i9-13900K, the main frequency is 3.0GHz, and the running memory is 64G.

[0108] The software platform of the simulation experiment platform of the present invention is: Windows 11 operating system and PyCharm, PyTorch2.1.0, and CUDA 12.1.

[0109] 2. Simulation steps

[0110] The training set RE in the multimodal dataset RE constructed in step 1 train The image is input into a dual-modal target tracking network based on an optical camera and an event camera for optimization training. The image input size is 346×260 pixels, and the number of epochs is set to 100.

[0111] 3. Simulation content and results analysis

[0112] The simulation experiment of the present invention performs target tracking experiments on 20 pairs of optical images and event data, and the results are averaged. Figure 6 shown.

[0113] Combine the following Figure 6 The simulation effect of the present invention is further described.

[0114] Figure 6This is a target tracking accuracy diagram of the present invention and the existing method. The comparison algorithm is AMTTrack (Wang.X., Huang.J., Wang.S., Tang.C., Jiang.B., Tian.Y., & Luo.B.[J](2024). Long-term Frame-Event Visual Tracking: Benchmark Dataset and Baseline.CoRR.), and the accuracy evaluation indicators are precision (Precision, P) and success rate (Success Rate, SR). The precision rate measures whether the error between the predicted position and the actual position of the target is within an acceptable range. The success rate evaluates the performance of the tracker by the degree of overlap (IOU) of the bounding box. The larger the value of all indicators, the better the result.

[0115] like Figure 6 As shown, it can be seen that the P and SR of the present invention are higher than AMTTrack. Experimental results show that the dual stream attention fusion module (DSAF) module can dynamically capture the correlation between features at different levels by introducing the multi-head self-attention mechanism module (MHSA), and effectively integrate the position information into the feature map layer by layer. This layer-by-layer interactive mechanism not only improves the accuracy of target tracking, but also enhances the model's adaptability to changes in target position and shape. At the same time, the deep cross feature interaction and enhancement module (DCFIE) fully integrates local and global information, and uses the multi-head self-attention mechanism module (MHSA) to deeply explore the potential correlation between features, which not only effectively improves the tracking performance of the model in complex scenes, but also enhances the robustness to problems such as occlusion and rapid target movement. In addition, the cross-modal Transformer fusion module (CMTF) significantly reduces information loss by deeply fusing features between different modalities when processing multimodal data, ensuring the integrity and consistency of information transmission. This design improves the efficiency of collaborative utilization of multimodal data, thereby further enhancing the robustness and reliability of the model in diverse tasks. Overall, the synergy of these modules enables the model to fully optimize target tracking performance in complex scenarios. In general, the dual-modal target tracking method based on optical camera and event camera proposed in the present invention significantly improves the accuracy of target tracking and meets the requirements of high-precision tracking.

Claims

1. A dual-modal target tracking method based on an optical camera and an event camera, characterized in that: The following steps are involved: Step 1: Collect optical data and event data, construct a multimodal dataset RE, and divide the multimodal dataset RE into training sets RE according to a certain ratio train and the test set RE test ; Step 2: Build a dual-modal target tracking network based on optical camera and event camera; Step 3: Use the training set RE in the multimodal dataset RE constructed in step 1 train The dual-modal target tracking network based on optical camera and event camera constructed in step 2 is trained to obtain the optimal training weight file FEtrack.pt; Step 4: Use the test set RE in the multimodal dataset RE constructed in step 1 test The dual-modal target tracking network based on optical camera and event camera constructed in step 2 is tested with the optimal training weight file FEtrack.pt obtained in step 3 to obtain the target tracking result.

2. The dual-modal target tracking method based on an optical camera and an event camera according to claim 1, characterized in that: The dual-modal target tracking network based on the optical camera and the event camera in step 2 includes a feature extraction network (Feature Extraction), a feature fusion network (Feature Fusion) and a classification and regression network (Classification & Regression); the feature extraction network (Feature Extraction) is responsible for extracting optical feature maps and event feature maps from the input optical data and event data; the feature fusion network fuses the optical feature maps and event feature maps extracted by the feature extraction network to obtain fusion features; The fused features are input into the classification and regression network to complete the target tracking and positioning tasks.

3. The dual-modal target tracking method based on an optical camera and an event camera according to claim 2, characterized in that: The feature extraction network (Feature Extraction) includes a backbone network and two dual stream attention fusion (dual stream attention fusion module, DSAF) modules; the backbone network is composed of two groups of Swin Transformer networks, which are used to extract initial feature maps T and feature maps S from a template (Frame Template) and a search (Frame Search) of optical data or event data, wherein the Swin Transformer includes Swin Transformer stage1, Swin Transformer stage2 and Swin Transformer stage3; the dual stream attention fusion (dual stream attention fusion module, DSAF) module is used to fuse and enhance the feature maps T and S; the dual stream attention fusion (dual stream attention fusion module, DSAF) module includes a convolution module (Conv), a self-attention mechanism module (Self-Attention), a multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA), a fusion feature enhancement module (Fusion Feature Enhancement Module, FFEM) and a fully connected convolution layer (Fully Connected Layer, FC); dual stream attention fusion (dual stream attention fusion module, DSAF) module is expressed as follows: F1=Conv(T),F2=Self-Attention(S) F4=ReLU(Concat(F1,F2,F3),Conv(F1,F2,F3)) F5 = σ(W2·ReLU(W1·F4)) F out =FC(F5)+F5 Among them, ReLU is the activation function, Concat is the summation operation, σ represents the Sigmoid activation function, and W2 and W1 represent weight factors; First, use the convolution module (Conv) from the template image Extract the local feature map F1 from the search image and use the self-attention mechanism module (Self-Attention) The global feature map F2 is obtained from the local feature map F1; then, the local feature map F1 and the global feature map F2 interact with each other through the multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA) to generate a fused feature map F3; then, the fused feature map F3 is further enhanced using the fusion feature enhancement module (Fusion Feature Enhancement Module, FFEM) to generate an enhanced feature map F4; then, the enhanced feature map F4 is selectively weighted using the W1 weight factor, ReLU activation function, W2 weight factor and Sigmoid activation function in sequence to generate a feature map F5; finally, the feature map F5 is optimized through the fully connected convolutional layer (Fully Connected Layer, FC), and the optimized result is fused with the feature map F5 to generate the final output feature map F out ; The initial feature map T and feature map S of the optical data are processed using the dual stream attention fusion module (DSAF) module to obtain the enhanced feature map X f ; The initial feature map T and feature map S of the event data are processed using the dual stream attention fusion module (DSAF) to obtain the enhanced feature map X e .

4. The dual-modal target tracking method based on an optical camera and an event camera according to claim 2, characterized in that: The feature fusion network (Feature Fusion) includes two deep cross feature interaction and enhancement modules (deep cross feature interaction and enhancement, DCFIE) and a cross-modal Transformer fusion module (cross-modal Transformer fusion, CMTF); the deep cross feature interaction and enhancement module (deepcross feature interaction and enhancement, DCFIE) includes two inputs, which are the feature map X output by the dual stream attention fusion module (dual stream attention fusion module, DSAF module) f and feature map X e , four convolution modules (Conv), 2L multi-head self-attention mechanism modules (Multi-Head Self-Attention, MHSA), L ≥ 3, an upsampling module (Upsample), a downsampling module (Downsample), a global average pooling (GlobalAverage Pooling, GAP) and a layer normalization module (LayerNorm, LN); the deep cross feature interaction and enhancement module (deep cross feature interaction and enhancement, DCFIE) is expressed as follows: X1=Conv(X f ),X2=Conv(X e ) X 12 =X 12 (L) =MHSA((X1 Q ,X1 K ,X2 V ) (L-1) ) X 22 =X 22 (L) =MHSA((X2 Q ,X2 K ,X1 V ) (L-1) ) X3=Upsample(X 12 )+Downsample(X 22 ) X out =Conv(LN(GAP(X3)+Conv(X3))); Using the input feature map X f and feature map X e , when two features can interact in the same spatial dimension, each input feature map is initialized and modified using the convolution module (Conv) to obtain feature maps X1 and X2; next, the L-layer multi-head self-attention mechanism module (Multi-Head Self-Attention, MHSA) is used to gradually enhance the information enhancement effect and enhance the interaction between features to obtain feature map X 12 and feature map X 22 ; Then use the convolution module (Conv) to perform the feature map X 12 and feature map X 22 Upsampling and downsampling are performed, and the processed feature maps are concat-operated to generate feature map X3. The global information enhancement module obtains feature map X4 by performing a global average pooling (GAP) operation on feature map X3. The local information enhancement module obtains feature map X5 by performing a convolution module (Conv) on X3. Feature maps X4 and X5 are normalized by a layer normalization module (LayerNorm, LN) to obtain feature map X6. Finally, the output X is generated by using a convolution module (Conv) on feature map X6. out ; For the input feature map X f and feature map X e Use the deep cross feature interaction and enhancement module (DCFIE) to perform two processings to obtain feature maps M1 and M2 respectively; The cross-modal Transformer fusion module (CMTF) includes two inputs, namely, the feature maps M1 and M2 output by the deep cross feature interaction and enhancement module (DCFIE), two convolutional neural network modules (CNN), two multi-head self-attention mechanism modules (MHSA), a convolution module (Conv), and a convolution module with a convolution kernel of 3 (Conv 3×3 ), a convolution module with a convolution kernel of 5 (Conv 5×5 ), a convolution module with a convolution kernel of 7 (Conv 7×7 ) and a global average pooling (GlobalAveragePooling, GAP); the cross-modal Transformer fusion module (cross-modal Transformer fusion, CMTF) is expressed as follows: M 11 =CNN(M1),M 21 =CNN(M2) M out =Conv(M3,GAP(M3)); The cross-modal Transformer fusion module (CMTF) encodes the feature maps M1 and M2 through the convolutional neural network module (CNN) and converts the feature maps of each modality into a fixed-size feature map M 11 and M 21 ; Next, the feature map M 11 and M 21 The correlation between them will be captured by the Multi-Head Self-Attention mechanism (MHSA), which uses the multi-head self-attention mechanism to capture the feature map M 11 and M 21 Processing is performed to obtain the feature map M 12 and M 22 , and then the feature map M 12 and M 22 Through summation processing, fusion is performed to obtain the feature map M Fu ; Then, the multi-scale fusion (MSF) module is used to fusion the feature map M Fu Multi-scale feature extraction is performed, where the specific operation of the multi-scale fusion (MSF) module is as follows: a convolution module (Conv 3×3 ), the convolution module with a convolution kernel of 5 (Conv 5×5 ) and a convolution module with a convolution kernel of 7 (Conv 7×7 ) for the feature map M Fu The three feature maps are processed in sequence, and the processed three feature maps are fused by summing to obtain feature map M3; finally, a global average pooling (GAP) operation is performed on feature map M3, and the result and feature map M3 are input into the convolution module (Conv) to generate the final output feature map M out ; The feature map M out The data is further input into the classification and regression network to classify and predict the target location.

5. The dual-modal target tracking method based on an optical camera and an event camera according to claim 1, characterized in that: In step 3, the number of training rounds is set to be greater than or equal to 100, and the batch size is set to be greater than or equal to 16.

6. The dual-modal target tracking method based on an optical camera and an event camera according to claim 1, characterized in that: In step 4, the test batch size is set to be greater than or equal to 4.

7. A dual-modal target tracking system based on an optical camera and an event camera based on the method according to any one of claims 1 to 6, characterized in that: include: Data acquisition module: used to collect optical data and event data, construct a multimodal data set RE, and divide the multimodal data set RE into training sets RE according to a certain ratio train and the test set RE test ; Network building module: used to build a dual-modal target tracking network based on optical camera and event camera; Network training module: used to use the training set RE in the multimodal dataset RE train Train the dual-modal target tracking network based on optical camera and event camera to obtain the optimal training weight file FEtrack.pt; Target tracking module: used to use the test set RE in the multimodal dataset RE test The target tracking test is performed on the dual-modal target tracking network based on optical camera and event camera using the optimal training weight file FEtrack.pt to obtain the target tracking results.

8. A dual-modal target tracking device based on an optical camera and an event camera, characterized in that: include: Memory: a computer program storing a dual-modal target tracking method based on an optical camera and an event camera as described in any one of claims 1 to 6, which is a computer-readable device; Processor: used to implement the dual-modal target tracking method based on an optical camera and an event camera as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it can implement a dual-modal target tracking method based on an optical camera and an event camera as described in any one of claims 1 to 6.

Images