RGB-E multi-mode target tracking method and system

Abstract

The invention relates to the technical field of target tracking, in particular to an RGB-E multi-mode target tracking method and system. According to the method, a dynamic event subframe division mechanism based on event information entropy estimation is designed, the complexity of a current event flow is quantified through event information entropy, an event frame aggregation or event subframe division strategy is adaptively selected, an event feature extraction network is provided, and spatial offset parameters of pooling kernels are dynamically learned based on input event features, so that the complexity of the current event flow is quantified. Through adaptive adjustment of a local receptive field, the feature extraction process more pays more attention to effective events generated by real motion of a target, each frame of event image is subjected to 1 * 1 convolution, shunt pooling splicing and convolution to obtain first-stage features, each-stage features are equally divided into multiple sub-features, each sub-feature is divided according to different proportions along channel dimensions, and each sub-feature is divided into a first-stage feature and a second-stage feature; different pooling methods are adopted to process each part, and event features are obtained through fusion, so that the accuracy of the event features is effectively improved. According to the method, the RGB-E multi-mode target tracking precision is effectively improved.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and in particular to an RGB-E multimodal target tracking method and system. Background Technology

[0002] Target tracking is a core computer vision technology that predicts the position of a target in real time using a continuous sequence of images. It is widely used in dynamic scenarios such as autonomous driving and video surveillance. Traditional target tracking methods mainly rely on RGB images, tracking targets by extracting static features such as color and texture. However, RGB images are easily affected by physical imaging conditions such as motion blur, low light, and high exposure, leading to a decrease in target representation capabilities and making them unsuitable for accurate tracking tasks under fast-moving or extreme lighting conditions.

[0003] In recent years, the emergence of event cameras has brought new directions to target tracking. Event cameras employ asynchronous imaging mechanisms, enabling them to capture light intensity change events with microsecond-level temporal resolution, and possess advantages such as high dynamic range, low latency, and low power consumption. Event data can effectively record the motion edge information of the target, complementing the static semantic information of RGB images. Therefore, multimodal fusion (RGB-E) based on RGB and event data is considered an effective means to improve the robustness of target tracking in complex scenes.

[0004] However, while existing RGB-E multimodal target tracking methods have made some progress, significant shortcomings remain in feature extraction. Current event feature extraction networks often employ pooling operations to aggregate features, reducing feature dimensionality while retaining some key information. In event modality data, different channels naturally carry differentiated information: some channels may more closely reflect the target's motion edge features, some may contain more information about the target's overall motion trend, and others may primarily record background or noise events. Existing event feature extraction networks ignore these inter-channel information differences, leading to insufficient feature utilization and consequently affecting the target prediction performance after fusion with RGB image features. Furthermore, the pooling operations in existing event feature extraction networks often use fixed geometric pooling kernels, making it difficult to effectively distinguish between target motion features and noise events when processing event modality data, easily aggregating both and reducing the accuracy of event feature extraction. Summary of the Invention

[0005] Therefore, the technical problem to be solved by the present invention is to overcome the shortcomings of existing RGB-E multimodal target tracking methods, which ignore the information differences between channels in the event feature extraction network and use fixed pooling kernels to extract event features, lacking adaptive perception capabilities, resulting in poor target tracking accuracy.

[0006] To address the aforementioned technical problems, this invention provides an RGB-E multimodal target tracking method, comprising: Obtain the RGB image frame sequence to be predicted and its corresponding event image frame sequence; Each event image frame is sequentially processed through 1×1 convolution, max pooling, and 1×1 convolution, followed by max pooling and average pooling. The pooled features are then concatenated to obtain the first-stage features of each event image frame. The first event image of each frame The stage features are sequentially processed through 2×2 convolutions and 1×1 convolutions, and then equally divided into multiple sub-features along the channel dimension; among them, , To set the number of stages; The first event image of each frame The first sub-feature of each stage is processed by deformable average pooling and then added to the bias term of the first sub-feature to obtain the first event image of each frame. The fusion feature of the first sub-feature of the stage; The first event image of each frame The second sub-feature of each stage is divided into two parts along the channel dimension. One part is processed by deformable average pooling, and the other part is processed by average pooling. The two pooled features are concatenated along the channel dimension and then added to the bias term of the second sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the second sub-feature of the stage; For each frame of the event image, the first The other sub-features of the stage are divided into two parts along the channel dimension at different ratios less than 1:1. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of this sub-feature in the stage; The first event image of each frame The fused features of all sub-features of each stage are concatenated along the channel dimension and then convolved using a 1×1 convolution to obtain the first event image of each frame. Stage characteristics; The first event image of each frame Stage features, serving as event features for each frame of the event image; The features of each RGB image frame and the event features of its corresponding event image are fused to obtain the predicted target location of each RGB image frame. The input features are convolved using a 3×3 method to generate the offset field for each pixel in the input features. For each predefined initial sampling point within each pooling window, the position of the initial sampling point after offset is obtained through the offset field of the pixel corresponding to the initial sampling point; The feature value of each initial sampling point after offset is calculated using bilinear interpolation. The feature value of each initial sampling point after offset is used as the target feature value of that initial sampling point; The maximum value of the target feature value of all initial sampling points within each pooling window is taken as the maximum output feature of that pooling window. Based on the spatial position of each pooling window in the input features, the maximum output features of each pooling window are concatenated to obtain the output features of deformable max pooling. The average value of the target feature values ​​of all initial sampling points within each pooling window is taken as the average output feature of that pooling window. The average output features of each pooling window are concatenated according to the spatial position of each pooling window in the input features to obtain the output features of deformable average pooling.

[0007] Preferably, the feature value of each initial sampling point after offset is calculated using bilinear interpolation, and the formula is as follows: , in, Initial sampling points after offset eigenvalues, These are the initial sampling points after the offset. Initial sampling points after offset The neighboring pixels, It is a bilinear interpolation kernel function. for eigenvalues, It is a function with maximum value. for x-coordinate for The ordinate, for x-coordinate for The ordinate, It is an absolute value.

[0008] Preferably, it further includes: Channel separation is used to obtain the features of each channel in each frame of the event image; With a step size of 1, a histogram of 256 bins is constructed. Using the constructed histogram, the density distribution of the normalized features of each channel in each event image is statistically analyzed to obtain the frequency of occurrence of the normalized features of each channel in each event image in the 256 bins. Based on the frequency of occurrence of the normalized features of each channel in each event image in 256 bins, the information entropy of the normalized features of each channel in each event image is calculated using Shannon entropy theory. The mean of the information entropy of the normalized features of all channels in each frame of the event image is taken as the event information entropy of each frame of the event image. Determine whether the event information entropy of each event image frame exceeds a set threshold. If it does, divide the event stream segment corresponding to the event image frame into multiple sub-event stream segments and compress each sub-event stream segment into a sub-event image frame. The event features of each sub-event image corresponding to the event image of the given frame are summed element by element to obtain the event features of the event image of that frame.

[0009] Preferably, it is determined whether the event information entropy of each frame event image exceeds a set threshold. If it does, the event features of each sub-event image corresponding to the frame event image are sequentially processed through two-dimensional adaptive average pooling, 1×1 convolution, and Sigmoid activation function to obtain the adaptive weights of each sub-event image corresponding to the frame event image. The event features and their adaptive weights of each sub-event image corresponding to the event image of the given frame are multiplied element by element to obtain the target event features of each sub-event image corresponding to the event image of the given frame. The target event features of each sub-event image corresponding to the event image of the given frame are summed element by element to obtain the event features of the event image of that frame.

[0010] Preferably, based on the frequency of occurrence of the normalized features of each channel in each event image across 256 bins, the information entropy of the normalized features of each channel in each event image is calculated using Shannon entropy theory, with the following formula: , , in, For the current event image, the first Information entropy of features after channel normalization For channel indexing, Indicates the first one bin, Indicates the first one bin, For the current event image, the first The normalized features of the first channel belong to the... The probability of each bin For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin.

[0011] Preferably, it is determined whether the event information entropy of each frame of event image exceeds a set threshold. If it does, the event stream segment corresponding to the frame of event image is divided into three sub-event stream segments, and each sub-event stream segment is compressed into a sub-event image.

[0012] Preferably, the first event image of each frame... The stage features are divided into four sub-features along the channel dimension; The first event image of each frame The third sub-feature of the stage is divided into two parts along the channel dimension in a 1:3 ratio. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the third sub-feature of the stage; The first event image of each frame The fourth sub-feature of the stage is divided into two parts along the channel dimension at a ratio of 1:7. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After being concatenated along the channel dimension, it is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the fourth sub-feature of the stage.

[0013] Preferably, the step of fusing the features of each frame of RGB image and the event features of its corresponding event image to obtain the predicted target location of each frame of RGB image includes: After fusing the features of each RGB image frame with the event features of its corresponding event image, the predicted target location of each RGB image frame is obtained through a fully convolutional neural network.

[0014] The present invention also provides an RGB-E multimodal target tracking system, comprising: The memory is used to store computer programs; the processor is used to implement the steps of the above-described RGB-E multimodal target tracking method when executing the computer program.

[0015] Compared with the prior art, the above-described technical solution of the present invention has the following advantages: The RGB-E multimodal target tracking method and system described in this invention designs an event feature extraction network with a multi-stage feature extraction mechanism. In the first stage, a combination of 1×1 convolution and average pooling and max pooling operations is used to perform preliminary feature compression and key information filtering on the event image by leveraging the complementary relationship between average pooling and max pooling operations, retaining the most basic global and local features in the event data. In other stages, the features from the previous stage are divided into multiple sub-features, and different pooling strategies are applied to each sub-feature to accurately adapt to the information characteristics of different channels in the event modality, thereby gradually extracting more refined event features through multiple stages. For the first sub-feature after segmentation, deformable average pooling is used in conjunction with a bias term to capture detailed features such as the edges of the target's motion. The second sub-feature is divided into two parts: one part is processed by deformable average pooling, and the other by ordinary average pooling. The two pooled parts are then combined and fused with the corresponding bias term to enhance the extraction of the overall motion trend of the target. Other sub-features are divided according to the different emphases of channel information and are processed by deformable max pooling and ordinary max pooling respectively. This selectively preserves key dynamic information after background suppression or noise filtering, enhancing the representation ability of key motion modes and accurately matching the differentiated information characteristics such as edges, trends, and noise carried by different channels in the event modality. Through multi-stage feature extraction, the obtained event features can more accurately characterize the spatiotemporal dynamics of the target's motion, thereby improving the target tracking accuracy.

[0016] This invention proposes deformable max pooling and deformable average pooling. For each predefined initial sampling point within a pooling window, the offset position is obtained based on the offset field of its corresponding pixel. This allows the initial sampling point to accurately approach the target motion feature region while moving away from noise event regions, achieving directional capture of key features and avoiding the problem of fixed sampling points including noise in feature calculations due to positional rigidity. Since the offset position is often not an integer coordinate, bilinear interpolation is used to calculate the feature value of the offset sampling point. Bilinear interpolation can perform weighted calculation using the feature values ​​of the surrounding four integer coordinate pixels, thus obtaining a smoother and more accurate feature value. This effectively avoids the loss or distortion of feature information that may be caused by simple rounding or neighbor interpolation, and can retain the effective feature information contained in the offset position to the greatest extent, ensuring that the basic data on which subsequent pooling operations are based is more reliable. This further improves the accuracy of characterizing target motion features, reduces the interference of noise events on feature extraction, and further improves target tracking accuracy.

[0017] Existing RGB-E multimodal target tracking methods attempt to improve event temporal modeling capabilities by dividing the event into multiple event subframes. However, they generally employ a fixed number of subframes, lacking the ability to adaptively adjust to different event stream densities. When the event stream is sparse, fixed division leads to excessive dispersion of subframe data, thereby weakening the effective modeling of the continuous features of target motion. To address these technical problems, this invention considers that different channels of event images often carry information of different dimensions, such as timestamps, polarity, and intensity. The distribution characteristics of this information in the event stream vary. This invention obtains the features of each channel of each event image frame through channel separation, uses histograms to statistically analyze the frequency of features in 256 bins, and combines Shannon entropy theory to calculate event information entropy to quantify the complexity of the current event stream. This allows for the adaptive selection of event frame aggregation or event subframe division strategies, effectively alleviating the feature dispersion problem caused by fixed-number division strategies in sparse scenarios and improving target tracking accuracy. Attached Figure Description

[0018] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating an RGB-E multimodal target tracking method according to the present invention. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments described are not intended to limit the present invention.

[0020] Reference Figure 1 As shown, this embodiment provides an RGB-E multimodal target tracking method, including: Step S1: Obtain the RGB image frame sequence to be predicted and its corresponding event image frame sequence; In this embodiment, specifically, the RGB video to be predicted and the event data are synchronized to obtain the RGB image frame sequence to be predicted and its corresponding event image frame sequence, including: Step S11: For RGB modal video data Based on the video frame rate, RGB image frames are extracted using a frame-by-frame decoding method. Assuming the video frame rate is Frames per second, total duration is In seconds, an RGB image frame sequence can be obtained. ,in, , Given the total number of RGB image frames, the formula for acquiring each RGB image frame can be expressed as: , in, For the first Frame RGB image, This represents the function for decoding frames. , Indicates the first frame, The height of the RGB image. The width of the RGB image. This represents the number of channels in an RGB image.

[0021] Step S12: Event stream data for the event modality Using the RGB frame timestamps extracted in step S11 as a reference, the start and end times of each RGB image frame are determined and used as the start event time of each event stream segment. With the end of the event time ; Based on the start event time of each frame's event stream segment With the end of the event time , event stream data The event stream segments are aggregated according to time windows to obtain a set of event substreams that correspond one-to-one with the RGB frame sequence. These event stream segments will be used to generate event image frames or for feature modeling. The formula for obtaining each event stream segment is as follows: , in, Indicates the first The event stream segment within the time period corresponding to the frame RGB image. The event pixel position, For event stamps, For the polarity of the event, For the first The start event time of the event stream segment within the corresponding time period of the frame RGB image. For the first The end event time of the event stream segment within the corresponding time period of the frame RGB image.

[0022] Step S13: Based on the event stream fragments obtained in step S12, convert them into two-dimensional event image frames by polarity accumulation. Specifically, within each time window, the spatial location of the event. Statistical analysis is conducted based on event polarity. Weighted summation is performed to form a sparse image representation, the construction method of which is defined as follows: , in, For the first Frame event image In pixels The value of the position, For x-axis index, Indexed by the y-axis. It is a unit impulse function used to project events onto a two-dimensional plane coordinate system.

[0023] The final event frame image A two-dimensional event graph, with dimensions consistent with the original sensor resolution, can be represented as follows: Event images.

[0024] Existing event feature extraction networks are mostly designed based on pooling kernels with fixed geometric structures. This static pooling method has two technical limitations: First, the fixed-structure pooling kernel cannot be flexibly adjusted according to the target's motion state, lacks the ability to adaptively perceive changes in the target's shape, and is prone to mixing noisy events with real motion events in the same receptive field, reducing the accuracy of feature extraction. Second, in scenarios with sparse event data, fixed pooling and fixed subframe partitioning strategies will further exacerbate the discretization of event features, limiting the model's ability to effectively model the continuous features of target motion.

[0025] Therefore, in order to address the shortcomings of existing event feature extraction networks that use fixed pooling kernels to extract event features, lack adaptive perception capabilities, and ignore the information differences between channels, resulting in poor target tracking accuracy, this invention designs an event feature extraction network based on deformable pooling and introduces deformable pooling into this network.

[0026] Step S2: After passing each event image frame through 1×1 convolution, max pooling, and 1×1 convolution, perform max pooling and average pooling respectively. Concatenate the pooled features and then pass them through 2×2 convolution and 1×1 convolution to obtain the first-stage features of each event image frame. Specifically, the first Frame event image By sequentially applying 1×1 convolution and max pooling, we obtain the... The first feature of the first stage of the frame event image The formula is: ; The first The first feature of the first stage of the frame event image After a 1×1 convolution, max pooling and average pooling are performed respectively. Average pooling extracts high-dimensional information from the event data, while average pooling extracts low-dimensional information, thus obtaining the [previous data]. Max pooling features in the first stage of frame event image , No. Average pooling features in the first stage of frame event image The formula is: , , in, For the first Max pooling features in the first stage of frame event images For the first The average pooling features of the first stage of the frame event image, This represents a convolution operation that represents a 1×1 convolution. This indicates a max pooling operation. This indicates the average pooling operation.

[0027] By fusing the converged low-dimensional and high-dimensional event information, a more comprehensive event representation is generated, yielding the [missing information]. First-stage features of frame event images The formula is: .

[0028] For all stages except the first stage, a hierarchical hybrid pooling strategy is adopted, as shown in steps S3-S6: Step S3: For each frame of the event image, the first... The stage features are sequentially passed through 2×2 convolution and 1×1 convolution to obtain the first stage of each event image frame. Stage output characteristics The formula is: , in, For the first The first frame of the event image The output characteristics of the stage This represents a convolution operation that performs a 2×2 convolution. For the first The first frame of the event image Stage characteristics.

[0029] The first event image of each frame The output features of the stage are divided into multiple sub-features along the channel dimension; among them, , To set the number of stages; Step S4: For each frame of the event image, the first... The first sub-feature of each stage is processed by deformable average pooling and then added to the bias term of the first sub-feature to obtain the first event image of each frame. The fusion feature of the first sub-feature of the stage is expressed by the following formula: , in, For the first The first frame of the event image The fusion feature of the first sub-feature of the stage, This is a deformable average pooling operation using a 3×3 pooling core. For the first sub-feature bias term, For the first The first frame of the event image The first sub-feature of the stage.

[0030] The first event image of each frame The first sub-feature of the stage is able to autonomously learn the 3×3 sampling grid offset through deformable average pooling to capture its spatial dynamic features.

[0031] Step S5: The first event image of each frame... The second sub-feature of each stage is divided into two parts along the channel dimension. One part is processed by deformable average pooling, and the other part is processed by average pooling. The two pooled features are concatenated along the channel dimension and then added to the bias term of the second sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the second sub-feature of the stage is expressed by the following formula: , in, For the first The first frame of the event image The fusion feature of the second sub-feature of the stage, To splice along the channel dimension, To make the first The first frame of the event image The first part of the second sub-feature of the stage is the feature after it has been divided into two parts along the channel dimension. To make the first The first frame of the event image The second feature of the stage is the second feature after being divided into two parts along the channel dimension. For average pooling operations using 3×3 pooling kernels, This is the second sub-feature bias term.

[0032] The first event image of each frame The second sub-feature of the stage is divided into two parts along the channel dimension. The two features are then processed by deformable average pooling and average pooling, respectively, to balance feature stability and spatial adaptability.

[0033] Step S6: For each frame of the event image, the first... The other sub-features of the stage are divided into two parts along the channel dimension at different ratios less than 1:1. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of this sub-feature in the stage; , in, For the first The first frame of the event image Phase 1 Fusion features of sub-features To make the first The first frame of the event image Phase 1 After dividing the sub-features into two parts along the channel dimension, the proportions are equal to the corresponding features of the preceding term. To make the first The first frame of the event image Phase 1 The proportion of the sub-features after dividing them into two parts along the channel dimension corresponds to the feature. This is a max pooling operation using a 3×3 pooling kernel. This is a deformable max pooling operation using a 3×3 pooling kernel. For the first Sub-feature bias term.

[0034] In this embodiment, preferably, the first frame of each event image is... The stage features are divided into four sub-features along the channel dimension; The first event image of each frame The third sub-feature of the stage is divided into two parts along the channel dimension in a 1:3 ratio. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the third sub-feature of the stage is expressed by the following formula: , in, For the first The first frame of the event image The fusion feature of the third sub-feature of the stage, To make the first The first frame of the event image The third sub-feature of the stage is divided into two parts along the channel dimension, and then the corresponding features of the preceding term are proportionally divided. To make the first The first frame of the event image The third sub-feature of the stage, after being divided into two parts along the channel dimension, corresponds to the feature of the proportional term. This is the third sub-feature bias term.

[0035] The first event image of each frame The fourth sub-feature of the stage is divided into two parts along the channel dimension at a ratio of 1:7. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After being concatenated along the channel dimension, it is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the fourth sub-feature of the stage.

[0036] , in, For the first The first frame of the event image The fusion feature of the fourth sub-feature of the stage, To make the first The first frame of the event image The fourth sub-feature of the stage is divided into two parts along the channel dimension, and then the corresponding features of the preceding term are proportionally divided. To make the first The first frame of the event image The fourth sub-feature of the stage, after being divided into two parts along the channel dimension, corresponds to the proportional subsequent feature. This is the fourth sub-feature bias term.

[0037] From a proportional design perspective, the asymmetric partitions of 1:3 and 1:7 are the optimal choices based on experimental verification. These partitions accurately adapt to the proportions of key information and redundant / noise information in different sub-features. In the third sub-feature, the 1:1 ratio channel focuses on capturing the dynamic details that require fine detail, enhancing the characterization of complex motion patterns through adaptive sampling using deformable max pooling. The 3:3 ratio channel efficiently preserves stable features through conventional max pooling. In the fourth sub-feature, the larger 1:7 ratio further strengthens the focus on minute amounts of key dynamic information and the robust extraction of a large number of basic features, preventing key signals from being overwhelmed. After concatenation and bias term adjustment, the pooling results under both ratios fully exploit the unique information value of different sub-features. The experimentally verified proportional parameters balance the targeting of feature extraction with computational efficiency, making the third sub-feature... The fusion features of the stages retain the core dynamic characteristics of the event modality while possessing better robustness and discriminative power, laying a high-quality foundation for subsequent feature processing.

[0038] The core mechanism of deformable pooling modeling lies in achieving adaptive adjustment of the feature space through dynamic deformation sampling. The deformable pooling process described above is as follows: In this embodiment, preferably, the input features are convolved using a 3×3 method to generate an offset field for each pixel in the input features; For each predefined initial sampling point within each pooling window, the position of the initial sampling point after offset is obtained through the offset field of the pixel corresponding to the initial sampling point; Using bilinear interpolation, the feature value of each initial sampling point after offset is calculated, as shown in the formula: , in, Initial sampling points after offset eigenvalues, These are the initial sampling points after the offset. Initial sampling points after offset The neighboring pixels, It is a bilinear interpolation kernel function. for eigenvalues, It is a function with maximum value. for x-coordinate for The ordinate, for x-coordinate for The ordinate, It is an absolute value.

[0039] The feature value of each initial sampling point after offset is used as the target feature value of that initial sampling point; The maximum value of the target feature value of all initial sampling points within each pooling window is taken as the maximum output feature of that pooling window. Based on the spatial position of each pooling window in the input features, the maximum output features of each pooling window are concatenated to obtain the output features of deformable max pooling. The average value of the target feature values ​​of all initial sampling points within each pooling window is taken as the average output feature of that pooling window. The average output features of each pooling window are concatenated according to the spatial position of each pooling window in the input features to obtain the output features of deformable average pooling.

[0040] For each pixel in the input event features, the initial sampling point of the pooling kernel is predefined in the pooling operation. The offset field is generated using a 3×3 convolution module. This offset field can be dynamically learned based on input event features to adjust the position of the initial sampling points. Update to the offset position During the deformation sampling stage, bilinear interpolation is used to calculate the eigenvalues ​​of the initial sampling points after the offset. To maintain spatial continuity, offset and interpolation calculations are introduced to achieve spatial adaptive adjustment of the pooling sampling position, providing a foundation for subsequent deformable pooling modeling.

[0041] Deformable pooling dynamically learns the spatial offset parameters of the pooling kernel based on input event features. Through adaptive adjustment of the local receptive field, the feature extraction process focuses more on effective events generated by the real motion of the target, suppresses the interference of noise events, and effectively improves the accuracy of event feature extraction.

[0042] Step S7: The first event image of each frame... After concatenating the fused features of all sub-features of each stage along the channel dimension, a 1×1 convolution is applied to the concatenated features for channel compression and fusion, yielding the first feature of each frame of the event image. Stage characteristics The formula is: .

[0043] Step S8: The first event image of each frame... Stage features, serving as event features for each frame of the event image; Step S9: Fuse the features of each RGB image frame with the event features of its corresponding event image to obtain the predicted target location of each RGB image frame.

[0044] In this embodiment, specifically, each frame of RGB image is... The input is fed into a Transformer network for feature extraction, yielding the features of each frame of the RGB image. ,in, The number of channels for the features of an RGB image; In this embodiment, optionally, the features of each frame of RGB image and the event features of its corresponding event image are fused to obtain the predicted target location of each frame of RGB image, including: The features of each RGB image frame are subjected to 1×1, 3×3 and 5×5 convolution operations to extract feature information under different receptive fields. The features after different convolution operations are then concatenated to obtain the target features of each RGB image frame. The target features are richer RGB feature representations. The features of each RGB image frame are processed through the Sigmoid activation function to obtain the normalized RGB feature weights of each RGB image frame. The normalized target RGB features of each frame of RGB image are multiplied element-wise with the normalized RGB feature weights corresponding to that RGB image to obtain the weighted RGB features of each frame of RGB image. Perform a 3×3 convolution on the event features of the event image corresponding to each frame of RGB image to extract the local spatiotemporal features of the event image; The event features of the event image corresponding to each RGB image frame are processed through the Sigmoid activation function to obtain the normalized event feature weights of the event image corresponding to each RGB image frame. The event features of the event image corresponding to each RGB image frame and its corresponding normalized event feature weights are multiplied element by element to obtain the weighted event features of the event image corresponding to each RGB image frame. The weighted RGB features of each frame of RGB image and the weighted event features of the corresponding event image are added element by element to fuse the features of the two modalities to obtain the target fusion feature. The target fusion feature is then passed through a classification head to obtain a confidence map of the target fusion feature, which represents the probability that each position in the search area is a foreground target. The classification head is mostly composed of convolutional layers, attention modules, and filters (such as the target filter of DiMP). The target fusion features are passed through a regression head to obtain the center coordinates and width and height of the bounding box, thus obtaining the predicted target location for each frame of the RGB image, including: After integrating local contextual information through convolutional layers, the target fusion features are then decoded using a multilayer perceptron (MLP) to obtain a feature vector of length 1.5. The feature vector is mapped to four regression values, representing the center coordinates and width and height of the bounding box, respectively. Among them, The length of the feature vector at each location.

[0045] In this embodiment, specifically, fusing the features of each frame of RGB image and the event features of its corresponding event image to obtain the predicted target location of each frame of RGB image includes: The features of each RGB image frame and the event features of the corresponding event image are fused through a bimodal mutual guidance fusion module. The bimodal mutual guidance fusion module adopts a bidirectional cross-attention mechanism, establishes cross-modal associations between the RGB modality and the event modality through mutual querying, and guides and enhances the features of each other.

[0046] Unidirectional guidance enhancement of the RGB modality through event modality: using the normalized event features of the current event image layer as the query vector. The normalized features of the RGB image layer corresponding to the current event image are used as the key vector. Sum value vector The first attention weight is obtained by using a cross-attention mechanism, and the calculation formula is as follows: , in, For cross attention, The softmax activation function is used. for transpose, is the channel dimension of the key vector.

[0047] After weighting and fusing the normalized features of the RGB image layer corresponding to the current event image through the first attention weight, the feature token of the RGB image corresponding to the current event image is added element by element to the attention features of the RGB image corresponding to the current event image through the residual connection to obtain the fused features of the RGB image corresponding to the current event image. The above process uses event modality query vectors to perform correlation modeling on RGB modal features, extracts intermodal correlation information, and obtains enhanced RGB modal features through value vector weighting. Through cross-modal feature fusion, event features guide RGB features to focus on motion-related regions, resulting in enhanced RGB features associated with event information.

[0048] During the reverse guidance process, the normalized features of the RGB image layer corresponding to the event features of the current event image are used as the query vector. The normalized event features of the current event image layer are used as key vectors. Sum value vector The second attention weight is obtained through the cross-attention mechanism; The event features after normalization of the current event image layer are weighted and fused by the second attention weight. Through the residual connection, the feature tokens of the RGB image corresponding to the current event image are added element by element to the attention features of the current event image to obtain the fused features of the current event image. Extract the corresponding template region token from the fusion features of the current event image. Search area token ; Extract the corresponding template region token from the fusion features of the RGB image corresponding to the current event image. Search area token ; Template region tokens for two modalities , By adding elements one by one, we obtain the template features. ; Search region tokens for both modalities , By adding elements one by one, we obtain the template features. Template features Features of the search area By concatenating along the spatial dimensions, a joint feature matrix is ​​obtained. ; The joint feature matrix is ​​passed through the relation modeling module to obtain the output features of the last layer of the relation modeling module. The search region portion is extracted from the output features of the last layer of the relation modeling module and used as the target search region feature. and the target search region features Reconstructing it into a two-dimensional spatial feature map; This is achieved by independently aggregating the fusion features of the current event image and the template region tokens and search region tokens of the fusion features of the corresponding RGB image. This operation is used to fully integrate the key feature information of the target template and the search region while preserving the feature structures of the two modalities.

[0049] Subsequently, the template features are spliced ​​together using a channel splicing method. Features of the search area The features are concatenated along the spatial dimension to generate a joint feature matrix. This joint feature matrix simultaneously contains multimodal fusion features from both the template and the search region, possessing a unified spatial representation capability. Based on this, the joint feature matrix is ​​input into the relationship modeling module. The relationship modeling module employs a four-layer standard visual Transformer (ViT) structure for deep relationship modeling. Through multi-layer Transformer calculations, the target association relationship between the template token and the search region token is progressively strengthened; the calculation formula is as follows: , in, For template features, For the search region features, The template region token corresponding to the fusion features of the current event image. This is the template region token corresponding to the fusion features of the RGB image corresponding to the current event image. The token represents the search region corresponding to the fusion features of the current event image. The token for the search region corresponding to the fusion features of the RGB image corresponding to the current event image. For the joint characteristic matrix, For self-attention mechanism, For normalization, For the first The joint feature matrix output by layer -1 for After the first The joint feature matrix after initial processing by the standard visual Transformer layer. For the first The joint feature matrix output by the layer. It is a multilayer perceptron. The output features of the last layer of the relation modeling module, This indicates a decoupling operation, used to separate the features of the template and the search region. Features of the target search region Indicates the first layer, This represents the total number of layers in the standard visual Transformer for the relation modeling module. The two-dimensional spatial feature map is input into a Fully Convolutional Network (FCN) for target prediction.

[0050] The fully convolutional network shown is composed of The system consists of stacked convolutional layers (Conv), batch normalization (BN), and ReLU activation functions, outputting a target classification score map, a local offset map, and a normalized target width and height map, respectively. The total number of layers in the fully convolutional network, and the object classification score map. Used to locate the probability distribution of the target. This represents the downsampling factor, used to describe the scaling relationship between the two-dimensional feature map and the target classification score map. The height of the two-dimensional spatial feature map. The width of the two-dimensional spatial feature map, local offset map To compensate for the discretization error introduced by feature map downsampling, the target size map is normalized. Used to predict the width and height of the target. During the prediction phase, this is achieved through a target classification score map. Select the position with the highest probability value Using the target center point, the predicted target location is obtained. Position of the target in the classification score graph The probability value of the target exists, and the specific calculation method is as follows: , in, Target classification score map The x-coordinate of the position with the highest probability value. Target classification score map The y-coordinate of the position with the highest probability value. The x-coordinate of the center point of the prediction box. The ordinate of the center point of the prediction box. The width of the prediction box, This represents the height of the predicted bounding box.

[0051] in, For the characteristics of a single channel Features after normalization This represents a truncation function used to limit normalized pixel values ​​to the interval [0, 255], preventing values ​​from exceeding the bounds. and These represent the characteristics of a single channel. The minimum and maximum values ​​in the range are used to determine the dynamic density range of the channel. For event images Characteristics of a single channel.

[0052] While existing RGB-E multimodal target tracking methods have made some progress, they still have certain shortcomings in event data modeling. On the one hand, most current methods simply compress asynchronous event streams into single-frame event images through simple polarity accumulation. This process significantly loses the original high temporal dynamic information in the event stream, making it difficult to fully extract and utilize the continuous motion features of the target. Although some existing methods attempt to improve the event temporal modeling capability by dividing the event into multiple event subframes, they generally adopt a fixed number of division strategies, lacking the ability to adaptively adjust to different event stream densities. When the event stream is sparse, fixed division leads to excessively discrete subframe data, thereby weakening the effective modeling of the continuous motion features of the target.

[0053] To address the issues of insufficient utilization of event temporal information and lack of adaptability in subframe partitioning in existing technologies, leading to poor target tracking accuracy, this invention designs an information entropy estimation module. Considering the spatiotemporal distribution characteristics of event data, it proposes an information entropy quantization method based on multi-channel decoupling to calculate the event information entropy of each event image frame. A dynamic event subframe partitioning mechanism based on event information entropy estimation quantifies the complexity of the current event stream through event information entropy, adaptively selecting either event frame aggregation or event subframe partitioning strategies. This effectively alleviates the feature discrepancy problem caused by fixed partitioning methods in sparse scenes. The specific scheme is as follows: In this embodiment, preferably, channel separation is used to obtain the event image of each frame. The characteristics of each channel are analyzed; specifically, channel separation measurement is used for each frame of the event image. Perform independent decoupling processing to avoid cross-channel information interference for event images. Features of a single channel The original event density is mapped to the interval [0, 255] through dynamic linear normalization. The mapping process formula is shown below: , in, For the characteristics of a single channel Features after normalization This represents a truncation function used to limit normalized pixel values ​​to the interval [0, 255], preventing values ​​from exceeding the bounds. and These represent the characteristics of a single channel. The minimum and maximum values ​​in the range are used to determine the dynamic density range of the channel. For event images Characteristics of a single channel.

[0054] With a step size of 1, a histogram of 256 bins is constructed. Using the constructed histogram, the density distribution of the normalized features of each channel in each frame of the event image is statistically analyzed to capture the bimodal distribution characteristics of the dynamic target region and the static background in the event data, and to obtain the frequency of occurrence of the normalized features of each channel in each frame of the event image in the 256 bins. Based on the frequency of occurrence of normalized features of each channel in each event image across 256 bins, the information entropy of the normalized features of each channel in each event image is calculated using Shannon entropy theory. This entropy is used to quantify the dynamic complexity of the event data for each channel. The formula is as follows: , , in, For the current event image, the first Information entropy of features after channel normalization For channel indexing, Indicates the first one bin, Indicates the first one bin, For the current event image, the first The normalized features of the first channel belong to the... The probability of each bin reflects the distribution probability of the normalized features of that channel in each pixel density interval. For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin (pixel frequency statistics results) For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin (pixel frequency statistics).

[0055] Information entropy The information entropy value indicates the complexity of the pixel density distribution in a single-channel event image. The larger the information entropy value, the richer the dynamic region and the more complex the structure of the event image.

[0056] The information entropy of the normalized features of all channels in each event image is fused by arithmetic average to obtain the event information entropy of each event image, as shown in the formula: , in, The event information entropy of the current event image. In the current event image The information entropy of the features after axis channel normalization focuses on the horizontal spatial location. In the current event image The information entropy of the features after axis channel normalization focuses on the vertical spatial location. In the current event image The information entropy of the features after axis channel normalization focuses on time position.

[0057] Determine the event information entropy of each frame of the event image. If the threshold is exceeded, it indicates that the current event frame contains a lot of dynamic information or interference components, and event subframe segmentation is required to enhance the feature modeling effect. The event stream segment corresponding to the event image in this frame is then divided into... Each event substream is compressed into a single event image frame. If the compression time is less than one frame, the event scene is considered relatively simple. In this case, the event image frame is directly processed by the event feature extraction network to extract the event features of the event image frame.

[0058] In this embodiment, preferably, it is determined whether the event information entropy of each frame of event image exceeds a set threshold. If it does, the event stream segment corresponding to that frame of event image is divided into three sub-event stream segments, i.e. Each sub-event stream segment is compressed into a single sub-event image frame, denoted as follows: .

[0059] From an accuracy perspective, three sub-event stream segments can capture the dynamic changes of events within the original event stream segment more meticulously, avoiding the loss of motion details due to the large time span, and ensuring that the sub-event images generated by the compression of each sub-event stream segment retain richer local motion features. From a computational efficiency perspective, compared to dividing into more sub-event stream segments, three sub-event stream segments can ensure a sufficiently detailed depiction of event dynamics without excessively increasing the amount of data processing and model computational burden, effectively balancing feature extraction accuracy and computational resource consumption, and ensuring that the algorithm has good operating efficiency in practical applications.

[0060] The event features of each sub-event image corresponding to the event image of the given frame are summed element by element to obtain the event features of the event image of that frame.

[0061] The decision to re-divide event subframes by determining whether the event information entropy of each event image exceeds a set threshold reflects the adaptability and specificity of the processing strategy: when the event information entropy exceeds the threshold, it indicates that the event data within the event stream segment has high dynamic complexity and rich structure. In this case, dividing the event stream into more sub-event segments to generate sub-event images can more fully extract the motion information contained therein, avoiding the blurring caused by excessive aggregation of complex dynamic features by a single event image; when the event information entropy does not exceed the threshold, it indicates that the event dynamics are relatively simple, and key information can be effectively preserved without additional division. This on-demand division mechanism ensures the quality of feature extraction in complex scenes and avoids ineffective computation in simple scenes, significantly improving the effectiveness and economy of event feature processing.

[0062] In this embodiment, preferably, it is determined whether the event information entropy of each frame of event image exceeds a set threshold. If it does, the event features of each sub-event image corresponding to the frame of event image are sequentially processed through two-dimensional adaptive average pooling, 1×1 convolution, and Sigmoid activation function to obtain the adaptive weights of each sub-event image corresponding to the frame of event image. The formula is as follows:

[0063] in, For the first The first frame event image corresponding to Adaptive weights for sub-frame event images, It is the Sigmoid activation function. For the first The first frame event image corresponding to Event characteristics of a frame event image.

[0064] The target event features of each sub-event image corresponding to the given event image are obtained by element-wise multiplying the event features and their adaptive weights element-wise, as shown in the formula: , in, For the first The first frame event image corresponding to Target event features of a frame sub-event image This indicates element-wise multiplication.

[0065] The target event features of each sub-event image corresponding to the current event image are summed element-wise to obtain the event feature of the current event image. By adaptively fusing the target event features of multiple sub-frames, the ability to model the temporal and spatial dynamics of complex event scenarios is effectively improved. The formula is as follows: , in, For the first Event characteristics of frame event images.

[0066] To verify the effectiveness of the RGB-E multimodal target tracking method proposed in this invention, this invention compares several advanced RGB-E tracking methods, such as CEUTrack, HRCEUTrack, DS-MESA, and TENet, as shown in Table 1. Table 1 provides a brief overview of different multimodal target tracking methods.

[0067] Table 1

[0068] The proposed RGB-E multimodal target tracking method (EEDPT) uses a training batch size of 30 and employs the AdamW optimizer with default parameters. The batch size during training is set to 16, and the initial learning rate is... The dimensions of the input template and the search area are 128×128 and 256×256, respectively.

[0069] As shown in Table 2, Table 2 is a comparison of the tracking accuracy (PR), success rate (SR), and normalized accuracy (NPR) of the present invention and different multimodal target tracking methods on the COESOT dataset.

[0070] Table 2

[0071] As shown in Table 2, the EEDPT method proposed in this invention achieves tracking accuracy (PR), success rate (SR), and normalized accuracy (NPR) of 77.2%, 69.0%, and 75.5% respectively on the COESOT dataset, outperforming advanced tracking methods such as CEUTrack, HRCEUTrack, and TENet. However, compared to the DS-MESA method, the EEDPT method is 0.3%, 0.1%, and 0.4% lower in tracking accuracy, success rate, and normalized accuracy, respectively.

[0072] As shown in Table 3, Table 3 is a comparison diagram between the present invention and the DS-MESA method in terms of the number of frames to be processed, the frame processing process, the total number of frames processed, and the frame utilization rate.

[0073] Table 3

[0074] The experimental statistics in Table 3 show that when testing the EEDPT method of this invention on the COESOT dataset, only 23,468 frames were divided into 3 event subframes, processing a total of 223,525 event frames, which is only 42.2% of the number of event frames processed by the DS-MESA method. Although the performance of the EEDPT method is slightly lower than that of DS-MESA, it has a significant advantage in computational efficiency. EEDPT, through its adaptive event subframe partitioning strategy, reduces computational overhead while maintaining high tracking accuracy.

[0075] As shown in Table 4, Table 4 is a performance comparison diagram of the present invention and different multimodal target tracking methods under multiple complex scene attributes of the COESOT dataset.

[0076] Table 4

[0077] As shown in Table 4, the performance of the proposed method EEDPT was compared with various tracking methods such as CEUTrack, HRCEUTrack, TENet, and DS-MESA under multiple complex scene attributes in the COESOT dataset. The COESOT dataset covers a variety of challenging scene factors, including full occlusion (FOC), out of view (OV), scale variation (SV), motion blur (MB), fast motion (FM), illumination variation (IV), and overexposure (OE). The results show that the proposed EEDPT method achieves excellent tracking performance in most complex scenes. Although it is slightly inferior to DS-MESA in some attributes, it significantly reduces the amount of event subframes used while maintaining robustness, thus effectively improving overall computational efficiency and demonstrating a superior performance-efficiency balance.

[0078] This second embodiment provides an RGB-E multimodal target tracking system, including: The memory is used to store computer programs; the processor is used to implement the steps of the above-described RGB-E multimodal target tracking method when executing the computer program.

[0079] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0082] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0083] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An RGB-E multimodal target tracking method, characterized in that, include: Obtain the RGB image frame sequence to be predicted and its corresponding event image frame sequence; Each event image frame is sequentially processed through 1×1 convolution, max pooling, and 1×1 convolution, followed by max pooling and average pooling. The pooled features are then concatenated to obtain the first-stage features of each event image frame. The first event image of each frame The stage features are sequentially processed through 2×2 convolutions and 1×1 convolutions, and then equally divided into multiple sub-features along the channel dimension; among them, , To set the number of stages; The first event image of each frame The first sub-feature of each stage is processed by deformable average pooling and then added to the bias term of the first sub-feature to obtain the first event image of each frame. The fusion feature of the first sub-feature of the stage; The first event image of each frame The second sub-feature of each stage is divided into two parts along the channel dimension. One part is processed by deformable average pooling, and the other part is processed by average pooling. The two pooled features are concatenated along the channel dimension and then added to the bias term of the second sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the second sub-feature of the stage; For each frame of the event image, the first The other sub-features of the stage are divided into two parts along the channel dimension at different ratios less than 1:

1. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of this sub-feature in the stage; The first event image of each frame The fused features of all sub-features of each stage are concatenated along the channel dimension and then convolved using a 1×1 convolution to obtain the first event image of each frame. Stage characteristics; The first event image of each frame Stage features, serving as event features for each frame of the event image; The features of each RGB image frame and the event features of its corresponding event image are fused to obtain the predicted target location of each RGB image frame. The input features are convolved using a 3×3 method to generate the offset field for each pixel in the input features. For each predefined initial sampling point within each pooling window, the position of the initial sampling point after offset is obtained through the offset field of the pixel corresponding to the initial sampling point; The feature value of each initial sampling point after offset is calculated using bilinear interpolation. The feature value of each initial sampling point after offset is used as the target feature value of that initial sampling point; The maximum value of the target feature value of all initial sampling points within each pooling window is taken as the maximum output feature of that pooling window. Based on the spatial position of each pooling window in the input features, the maximum output features of each pooling window are concatenated to obtain the output features of deformable max pooling. The average value of the target feature values ​​of all initial sampling points within each pooling window is taken as the average output feature of that pooling window. The average output features of each pooling window are concatenated according to the spatial position of each pooling window in the input features to obtain the output features of deformable average pooling.

2. The RGB-E multimodal target tracking method according to claim 1, characterized in that, The bilinear interpolation method is used to calculate the feature value of each initial sampling point after offset, and the formula is as follows: ， in, Initial sampling points after offset eigenvalues, These are the initial sampling points after the offset. Initial sampling points after offset The neighboring pixels, It is a bilinear interpolation kernel function. for eigenvalues, It is a function with maximum value. for x-coordinate for The ordinate, for x-coordinate for The ordinate, It is an absolute value.

3. The RGB-E multimodal target tracking method according to claim 1, characterized in that, Also includes: Channel separation is used to obtain the features of each channel in each frame of the event image; Construct a histogram with a step size of 1, consisting of 256 bins. By constructing histograms, density distribution statistics are performed on the normalized features of each channel in each event image frame to obtain the frequency of occurrence of the normalized features of each channel in each event image frame in 256 bins. Based on the frequency of occurrence of the normalized features of each channel in each event image in 256 bins, the information entropy of the normalized features of each channel in each event image is calculated using Shannon entropy theory. The mean of the information entropy of the normalized features of all channels in each frame of the event image is taken as the event information entropy of each frame of the event image. Determine whether the event information entropy of each event image frame exceeds a set threshold. If it does, divide the event stream segment corresponding to the event image frame into multiple sub-event stream segments and compress each sub-event stream segment into a sub-event image frame. The event features of each sub-event image corresponding to the event image of the given frame are summed element by element to obtain the event features of the event image of that frame.

4. The RGB-E multimodal target tracking method according to claim 3, characterized in that, Determine whether the event information entropy of each frame event image exceeds a set threshold. If it does, then pass the event features of each sub-event image corresponding to the frame event image through two-dimensional adaptive average pooling, 1×1 convolution, and Sigmoid activation function in sequence to obtain the adaptive weights of each sub-event image corresponding to the frame event image. The event features and their adaptive weights of each sub-event image corresponding to the event image of the given frame are multiplied element by element to obtain the target event features of each sub-event image corresponding to the event image of the given frame. The target event features of each sub-event image corresponding to the event image of the given frame are summed element by element to obtain the event features of the event image of that frame.

5. The RGB-E multimodal target tracking method according to claim 3, characterized in that, Based on the frequency of occurrence of the normalized features of each channel in each event image across 256 bins, the information entropy of the normalized features of each channel in each event image is calculated using Shannon entropy theory. The formula is as follows: ， ， in, For the current event image, the first Information entropy of features after channel normalization For channel index, Indicates the first one bin, Indicates the first one bin, For the current event image, the first The normalized features of the first channel belong to the... The probability of each bin For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin For the current event image, the first The normalized features of the channels in the first... Frequency of occurrence in each bin.

6. The RGB-E multimodal target tracking method according to claim 3, characterized in that, Determine whether the event information entropy of each event image frame exceeds a set threshold. If it does, divide the event stream segment corresponding to that event image frame into three sub-event stream segments, and compress each sub-event stream segment into a sub-event image frame.

7. The RGB-E multimodal target tracking method according to claim 1, characterized in that, The first event image of each frame The stage features are divided into four sub-features along the channel dimension; The first event image of each frame The third sub-feature of the stage is divided into two parts along the channel dimension in a 1:3 ratio. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After concatenation along the channel dimension, the result is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the third sub-feature of the stage; The first event image of each frame The fourth sub-feature of the stage is divided into two parts along the channel dimension at a ratio of 1:

7. The feature corresponding to the first part of the ratio is processed by deformable max pooling, and the feature corresponding to the second part of the ratio is processed by max pooling. After being concatenated along the channel dimension, it is added to the bias term of the sub-feature to obtain the first sub-feature of each frame of the event image. The fusion feature of the fourth sub-feature of the stage.

8. The RGB-E multimodal target tracking method according to claim 1, characterized in that, The step of fusing the features of each RGB image frame and the event features of its corresponding event image to obtain the predicted target location of each RGB image frame includes: After fusing the features of each RGB image frame with the event features of its corresponding event image, the predicted target location of each RGB image frame is obtained through a fully convolutional neural network.

9. An RGB-E multimodal target tracking system, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the RGB-E multimodal target tracking method according to any one of claims 1 to 8 when executing the computer program.

Images