Point cloud object tracking method based on visual and motion cues

Abstract

The application discloses a point cloud target tracking method based on visual and motion cues, relates to the field of computer vision, and comprises the following steps: point cloud sequence input and preprocessing, obtaining, sampling and standardizing point clouds of a current frame and historical frames, and representing the state of a tracking target as to-be-predicted quantities; point feature extraction and cue initialization, respectively extracting point-level features from the point clouds of the current frame and the historical frames, and simultaneously initializing visual cues and motion cues; short-time domain information propagation, based on an adjacent frame deformation module enhanced by cues, adopting a three-branch decoupling plus double cue injection strategy, modeling an appearance branch, a motion branch and a mask branch respectively, and respectively injecting visual cues and motion cues in the appearance branch and the motion branch, and through long-time domain appearance cue learning and long-time domain motion cue learning, finally realizing prediction, and the application improves the accuracy and stability of point cloud target tracking without significantly increasing the calculation and memory overhead.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a point cloud target tracking method based on visual and motion cues. Background Technology

[0002] 3D point cloud perception is a crucial technology in computer vision and robotics, aiming to characterize the spatial structure of targets and their environment in the real world using sparse 3D point sets acquired by LiDAR or depth sensors. Based on this, 3D point cloud single-target tracking takes a continuous sequence of point cloud frames as input. The goal is to continuously predict the target's pose or bounding box state in subsequent frames, given an initial target position, thereby achieving stable tracking of dynamic targets. This technology has significant application value in scenarios such as autonomous driving, intelligent transportation, mobile robots, and security monitoring. For example, in autonomous driving, continuous tracking of dynamic traffic participants such as vehicles and pedestrians is required to support subsequent behavior prediction and decision-making control.

[0003] Traditional point cloud single-target tracking methods often employ a "template frame-search frame" framework, propagating target information by measuring the correlation of point features between adjacent frames to achieve target localization. Other methods utilize mechanisms such as cross-attention to promote interaction and correlation modeling between point features in adjacent frames, improving adaptability to changes in target appearance. Additionally, some methods introduce motion cues, inferring target position by estimating relative displacement between adjacent frames, or modeling target-related motion changes by aligning and fusing continuous point clouds. With the development of deep learning, researchers have begun to leverage multi-frame sequence information to enhance tracking robustness, formulating the tracking problem as a sequence prediction task to better handle complex situations such as occlusion, sparse point clouds, and changes in target appearance. However, existing multi-frame methods are often limited by fixed sliding window mechanisms, primarily focusing on information within a local time frame, easily ignoring global historical cues from the first frame to the current frame, resulting in insufficient adaptability to global changes in the target. Furthermore, some strategies relying solely on adjacent frames or local history may still experience localization drift or tracking instability when faced with significant changes in target appearance, motion speed, or nonlinear motion patterns.

[0004] Furthermore, to achieve stronger long-term modeling capabilities, State Space Models (SSMs) and their related structures, due to their linear time complexity and strong ability to model long-range dependencies, are increasingly being applied to vision and point cloud sequence tasks. However, in point cloud single-target tracking scenarios, how to fully integrate appearance and motion cues while ensuring inference efficiency, and effectively utilize historical information from the first frame to the current frame, remains a challenging problem. In particular, the target may undergo simultaneous changes in appearance and motion velocity during tracking, which places higher demands on both appearance and motion modeling; relying solely on local information often makes it difficult to obtain stable target representations in scenarios with significant appearance changes. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing point cloud target tracking technologies in adaptability to scenarios with drastic changes in target appearance, nonlinear velocity changes, occlusion, or sparse point clouds. Furthermore, most methods lack explicit cue information that can summarize the entire sequence of target states, resulting in limited tracking stability and robustness. This invention proposes a point cloud target tracking method based on visual and motion cues. This method uses cue-enhanced adjacent frames for information propagation in the short time domain and a selective scanning mechanism to compress historical appearance and trajectory information into an online-updable cue vector in the long time domain. This improves tracking accuracy and stability without significantly increasing computational and memory overhead.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a point cloud target tracking method based on visual and motion cues, comprising:

[0007] The point cloud sequence input and preprocessing steps involve acquiring, sampling, and standardizing the point clouds of the current frame and historical frames, and representing the state of the tracked target as a quantity to be predicted.

[0008] The point feature extraction and cue initialization steps extract point-level features from the point clouds of the current frame and historical frames respectively, and initialize visual cues and motion cues at the same time;

[0009] The short-term information propagation steps, based on the adjacent frame deformation module with cue enhancement, adopt a three-branch decoupling plus dual cue injection strategy, model the appearance branch, motion branch and mask branch respectively, and inject the visual cue and the motion cue into the appearance branch and the motion branch respectively;

[0010] The long-term appearance cue learning step utilizes the full-sequence compression Mamba module to compress the layered appearance features from the first frame to the current frame into a hidden state and update the visual cue online;

[0011] The long-term domain motion cue learning steps include: sequentially modeling the motion trajectory from the first frame to the current frame; constructing and updating the motion cue through motion feature compression and online update mechanisms; and...

[0012] The prediction step involves receiving appearance features and predicting the target's offset, mask, and rotation angle.

[0013] Preferably, the point cloud sequence input and preprocessing steps include:

[0014] Acquire point cloud sequence data of the target to be tracked, including the point cloud of the current frame and the historical point cloud sequence composed of multiple adjacent frames;

[0015] The point clouds of the current frame and historical frames are uniformly sampled and standardized to form point cloud blocks with a fixed number of points;

[0016] The tracking task is described as a prediction of the target state increment, wherein the predicted quantities include the target's position offset and orientation change in three-dimensional space.

[0017] Preferably, the point feature extraction and prompt initialization step includes:

[0018] Feature extraction is performed on the current frame point cloud and the historical frame point cloud respectively to obtain the current frame point feature and the historical frame point feature sequence;

[0019] And initialize two types of learnable cues, namely the visual cues and the motion cues.

[0020] Preferably, in the short time domain information propagation step, the adjacent frame deformation module for cue enhancement is composed of multiple stacked layers, each layer containing the appearance branch, the motion branch and the mask branch, and the corresponding visual cue and motion cue are introduced into each layer to participate in the calculation;

[0021] The method further includes:

[0022] Cross-attention enhanced by motion cues, with current motion features as the primary focus, aggregates information related to the target motion from historical motion features;

[0023] Enhanced cross-attention through visual cues, with current appearance features as the primary focus, aggregates information consistent with the target appearance from historical appearance features;

[0024] Historical mask-related features are propagated to the current frame mask features through mask branching;

[0025] Enhanced cross-attention through visual and motion cues enables the motion branch to be corrected using target cues provided by the appearance branch;

[0026] Self-attention enhanced by visual cues is integrated and reorganized with appearance features;

[0027] The system updates the motion features, appearance features, and mask features of the next layer through a feedforward network, and simultaneously updates the visual cues and motion cues.

[0028] Preferably, the long-term appearance cue learning step includes:

[0029] The hierarchical appearance feature sequence from the short time domain information propagation step is input into the full sequence compression Mamba module, and the corresponding hidden state and several parameters are obtained through sequence compression.

[0030] The hidden state is then fused with the current visual cue to obtain more complete global target appearance information, and the visual cue is updated based on the fusion result.

[0031] Preferably, the long-term motion cue learning step includes:

[0032] The hierarchical motion feature sequence from the short time domain information propagation step is input into the full sequence compression Mamba module, and the corresponding hidden state and several parameters are obtained through sequence compression.

[0033] The hidden state is then fused with the current motion cues to obtain more complete global target motion information, and the motion cues are updated based on the fusion result.

[0034] Preferably, the prediction step includes:

[0035] The prediction head uses appearance-related features to perform mask prediction and completes the proposal and verification of target candidate locations;

[0036] Simultaneously, motion-related features are used to regress the target position offset, thereby outputting the tracking results for the current frame.

[0037] Preferably, joint loss is used for optimization during the training phase, and the loss is composed of center regression loss, classification loss, proposal loss, bounding box loss, motion offset loss and mask loss.

[0038] Preferably, the motion-enhanced cross-attention, the visual-enhanced cross-attention, the visual and motion-enhanced cross-attention, and the visual-enhanced self-attention are all implemented by a multi-head attention mechanism.

[0039] Preferably, the full-sequence compression Mamba module is constructed based on a selective state-space model, and its discretization process is implemented through the zero-order preservation method to convert continuous parameters into discrete parameters for sequence processing.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows: The method of the present invention is dedicated to solving the robust tracking problem in single target tracking of point clouds when there are both drastic changes in target appearance and changes in motion pattern. First, visual cues and motion cues are proposed as explicit global priors. Through the cues enhancement mechanism, stable target representations and trajectory constraints are provided for the appearance branch and motion branch respectively, thereby alleviating the drift caused by relying solely on local inter-frame similarity. Then, an adjacent frame deformation module for cues enhancement is proposed. The "appearance / motion / mask" three-branch decoupled propagation and "dual cues injection" are performed to simultaneously enhance the ability of appearance association, motion consistency and foreground suppression of background interference in the range of adjacent multiple frames, thereby improving the localization stability under occlusion and sparse point cloud conditions.

[0041] Finally, we propose Incessant Visual Prompt (IPV) and Incessant Motion Prompt (IMP) learning methods. By using a selective scanning-based sequence compression mechanism, we can efficiently aggregate and update the historical appearance and trajectory information from the first frame to the current frame into a prompt stream online. This solves the problem of missing global historical information in the sliding window method without significantly increasing computational and memory overhead, and enhances the adaptability to scenarios with global state changes.

[0042] This invention can achieve higher accuracy and stronger stability in point cloud single-object tracking tasks on datasets such as KITTI, NuScenes, and Waymo, while maintaining good inference efficiency. Attached Figure Description

[0043] Figure 1 A flowchart of a point cloud target tracking method based on visual and motion cues provided in an embodiment of the present invention;

[0044] Figure 2 Pipeline diagram of the point cloud target tracking method based on visual and motion cues provided in the embodiments of the present invention;

[0045] Figure 3 This is a schematic diagram of the Prompt-Enhanced Adjacent Transformer (PAT) module, which is an example of the present invention.

[0046] Figure 4 This is a schematic diagram of the uninterrupted visual prompting module as an example of the present invention;

[0047] Figure 5 This is a schematic diagram illustrating the comparison of results between this invention and other methods on the KITTI dataset;

[0048] Figure 6This is a schematic diagram of the ablation experiment results used to verify the effects of each component in this invention.

[0049] Figure 7 This is a schematic diagram illustrating the comparison of results between this invention and other methods on the NuScenes dataset. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] Since this invention improves upon existing Mamba technology, the existing Mamba technology will be described first before describing the technical solutions of the embodiments of this invention. Mamba is a sequence modeling architecture based on a StateSpace Model (SSM). Its core innovation lies in introducing a selective state space mechanism, enabling the model to dynamically adjust the state transition path according to the input content. This balances computational efficiency and modeling capability in long sequence modeling, and Mamba has significant advantages in inference speed, memory usage, and context length scalability. Mapped to an output sequence The mapping process is as follows:

[0052]

[0053]

[0054] In the formula, It is in a hidden state. It is an evolutionary parameter. and This represents the projection parameters. To effectively apply the Mamba model to deep learning algorithms, continuous parameters need to be... and Discretization is performed. A commonly used discretization method is zero-order hold (ZOH), which is defined as follows:

[0055]

[0056]

[0057] In the formula, This is the time parameter used for discretization. Therefore, the discretized representation of the above formula is defined as:

[0058]

[0059]

[0060] Example 1

[0061] This invention provides a point cloud target tracking method based on visual and motion cues, such as... Figure 1 and Figure 2 As shown, it includes the following steps:

[0062] Point cloud sequence input and preprocessing: acquire, sample and standardize point clouds of the current frame and historical frames, and represent the state of the tracked target as the translation offset and rotation angle to be predicted;

[0063] In practice, the point cloud sequence input and preprocessing may include the following steps:

[0064] Acquire the point cloud sequence data of the target to be tracked, including the point cloud of the current frame and the point clouds of historical frames. Let the point cloud of the current frame be... Historical frame point cloud collection ,in From adjacent Frame composition, Total number of frames Indicates the frame index;

[0065] To ensure computational efficiency and stability, the point clouds of the current frame and historical frames are uniformly sampled, that is, with the target center as the origin, 1024 points around it are randomly sampled to form a standardized point cloud block for network input.

[0066] The tracking task is simplified to predicting the increment of the target's state, i.e., the target offset. With rotation angle .

[0067] Point feature extraction and cue initialization: Point-level features are extracted from the point clouds of the current frame and historical frames respectively. At the same time, visual cues and motion cues are initialized as appearance and motion priors for subsequent module injection and online updates.

[0068] In practice, the point feature extraction and prompt initialization include the following steps:

[0069] For the current frame point cloud respectively With historical frame point cloud Perform feature extraction to obtain the features of the current frame. Historical frame features ,in For point characteristic numbers, Number of channels;

[0070] Initialize two types of learnable cues: visual cues Exercise tips This is used for subsequent display guidance on appearance and motion modeling.

[0071] Based on the Prompt-Enhanced Adjacent Transformer (PAT) module, short temporal information propagation is performed. A three-branch decoupling and dual-cue injection strategy is adopted, which models the appearance branch, motion branch and mask branch respectively, and injects visual and motion cues into the appearance and motion branches respectively.

[0072] In specific implementation, such as Figure 3 As shown, the short temporal domain information propagation based on the Prompt-Enhanced Adjacent Transformer (PAT) includes the following steps:

[0073] PAT consists of multiple stacked layers (using...) Each layer contains three structurally similar branches, which respectively implement the decoupled propagation of appearance, motion, and mask features, and introduce visual cues in each layer. Exercise tips ;

[0074] For the motion branch, Motion Prompt-Enhanced Cross-Attention (MPCA) is employed, using the current motion features. For the query, the characteristics of historical movements Exercise tips By concatenating these into a key and value, historical motion information is aggregated into the current motion features.

[0075]

[0076] Where Concat represents the concatenation operation, and MPCA is implemented by Multi-Head Attention (MHA):

[0077]

[0078] In the formula , and These represent three different fully connected layers. The channel dimension representing the latent feature;

[0079] For the appearance branch, Visual Prompt-Enhanced Cross-Attention (VPCA) is used, based on the current appearance features. For a query, historical appearance features and visual cues are concatenated into a key and value, aggregating historical appearance information into the current appearance feature:

[0080]

[0081] In the formula To represent historical appearance characteristics, VPCA is still implemented by MHA;

[0082] For mask branches, MHA is used to propagate historical mask features into the current mask features:

[0083]

[0084] In the formula Obtained by processing the historical mask set through a linear layer;

[0085] To aggregate visual features into motion features, Visual and Motion Prompt-Enhanced Cross-Attention (VMCA) is employed to concatenate appearance features with visual cues. As the key and value, motion features are concatenated with motion cues to obtain... As a query:

[0086]

[0087]

[0088] In the formula, VMCA is implemented by MHA;

[0089] To further enhance the current appearance features, Visual Prompt-Enhanced Self-Attention (VPSA) is employed to fuse visual cues with appearance features:

[0090]

[0091] In the formula, VPSA is implemented by MHA.

[0092] To perform inter-layer updates and memory writes, three different feedforward networks (FFNs) are used to obtain the motion, appearance, and mask features of the next layer, as well as the updated hints:

[0093]

[0094]

[0095]

[0096] In the formula and Adding them together gives you a hint for the next level. Next, the hierarchical features are written into the memory for use in local modeling of the next frame.

[0097] Based on Incessant Visual Prompt (IVP) learning, Mamba is used to compress the hierarchical appearance features from the first frame to the current frame into hidden states and update visual cues online to form a complete appearance cue stream, thereby robustly responding to changes in the target appearance.

[0098] In specific implementation, such as Figure 4 As shown, the long-term appearance cue learning process includes the following steps:

[0099] To model the appearance features of the entire sequence while ensuring high computational efficiency, hierarchical appearance features from PAT are used. The input is fed into Complete Sequence Compression Mamba (CSCM) to obtain the hidden state. With parameter matrix :

[0100]

[0101] In the formula Indicates the number of layers in the PAT;

[0102] The hidden state is fused with the current visual cue to obtain global target information and the update value is output:

[0103]

[0104] In the formula Then updated by a linear layer Multiple iterations allow the prompt to capture more complete appearance changes.

[0105] Based on Incessant Motion Prompt (IMP) learning, the motion trajectory from the first frame to the current frame is sequentially modeled, and a complete motion prompt stream is constructed through motion feature compression and online update mechanism, thereby stably estimating the target motion;

[0106] In practice, long-term motion cue learning includes the following steps:

[0107] To model the motion features of the entire sequence while ensuring high computational efficiency, hierarchical motion features from PAT are used. Input into CSCM to get the hidden state. With parameter matrix :

[0108]

[0109] In the formula Indicates the number of layers in the PAT;

[0110] The hidden state is fused with the current motion cue to obtain global target information and the update value is output:

[0111]

[0112] In the formula Then updated by a linear layer Multiple iterations allow the prompt to capture more complete appearance changes.

[0113] Predict the appearance features received by the head, predict the target's offset, mask, and rotation angle, and train the entire network under the supervision of ground truth.

[0114] Predicting the appearance features of head reception specifically includes the following steps:

[0115] The prediction head uses appearance features for mask prediction, target candidate location proposal and verification; and uses motion features to regress position offset.

[0116] The loss function consists of center regression, classification, proposal, bounding box, motion offset, and mask loss:

[0117]

[0118] The mask loss in the formula uses cross-entropy loss. The target center loss uses the mean square error loss. The quality score and the target score are compared using cross-entropy loss. , The bounding box loss is a smooth-L1 loss. The position offset loss is the mean square error loss. , and It is a balancing factor.

[0119] Figure 6The results of ablation experiments are presented to verify the role of each component in this invention. The test scenarios were divided into two categories: scenarios where the target's appearance changes and scenarios with similar-looking interfering objects. In the first category, when the target's appearance changes significantly over time (e.g., changes in pose, partial occlusion leading to changes in point cloud morphology, changes in sparsity, or shifts in the visible area), if matching is based solely on visual information from the most recent frames, the model often only utilizes short-term local appearance cues, easily leading to mismatches or drift, thus making it difficult to provide accurate bounding boxes. By introducing global visual cues, the model can converge and retain appearance cues over a longer time span from the first frame to the current frame, forming a more stable target-level appearance memory; even if the appearance of the current frame differs significantly from recent frames, it can still relocate the target and predict more accurate bounding boxes using long-term appearance information.

[0120] In the second scenario, when there are distracting objects in the environment that closely resemble the target's appearance (e.g., nearby vehicles, similarly shaped objects, or background structures with similar point cloud distributions to the target), pure visual matching methods are prone to mistaking these distracting objects for the target, leading to incorrect feature matching directions and ultimately tracking failure. To address this issue, this invention does not rely solely on appearance feature similarity for association. Instead, it further utilizes the trajectory and dynamic information encoded by motion cues for position prediction and constraint: by modeling historical motion, predictions that better reflect the target's actual motion patterns are obtained, thus maintaining accurate target tracking and stable output even when similar distracting objects are present. In summary, the visualization results verify the complementary effects of the two types of cues: global visual cues primarily enhance the model's re-identification ability and localization stability under conditions of appearance changes and occlusion, while motion cues primarily enhance the model's resistance to false matching and robustness in scenarios with similar distracting objects; together, they significantly improve overall tracking accuracy.

[0121] The present invention will now be simulated. Figure 5 , 7This diagram illustrates the comparison between the present invention and existing technologies. Comparisons were performed on two datasets: the KITTI dataset includes targets such as cars, pedestrians, vans, and cyclists; the NuScenes dataset includes targets such as cars, pedestrians, trucks, trailers, and buses. The existing technologies used for comparison include CXTrack (mentioned in the 2023 paper "CXTrack: Optimizing 3D Point Cloud Tracking with Contextual Information" by Xu Tianxing et al.) and MBPTrack (mentioned in the 2023 paper "MBPTrack: Optimizing 3D Point Cloud Tracking Based on Memory Networks and Box Priors" by Xu Tianxing et al.). It can be seen that the tracking stability of the present invention surpasses that of existing methods in various scenarios.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for point cloud object tracking based on visual and motion cues, characterized in that, include: The point cloud sequence input and preprocessing steps involve acquiring, sampling, and standardizing the point clouds of the current frame and historical frames, and representing the state of the tracked target as a quantity to be predicted. The point feature extraction and cue initialization steps extract point-level features from the point clouds of the current frame and historical frames respectively, and initialize visual cues and motion cues at the same time; The short-term information propagation steps, based on the adjacent frame deformation module with cue enhancement, adopt a three-branch decoupling plus dual cue injection strategy, model the appearance branch, motion branch and mask branch respectively, and inject the visual cue and the motion cue into the appearance branch and the motion branch respectively; The long-term appearance cue learning step utilizes the full-sequence compression Mamba module to compress the layered appearance features from the first frame to the current frame into a hidden state and update the visual cue online; The long-term domain motion cue learning steps involve sequentially modeling the motion trajectory from the first frame to the current frame, and constructing and updating the motion cue through motion feature compression and online update mechanisms. as well as The prediction step involves receiving appearance features and predicting the target's offset, mask, and rotation angle.

2. The method according to claim 1, characterized in that, The point cloud sequence input and preprocessing steps include: Acquire point cloud sequence data of the target to be tracked, including the point cloud of the current frame and the historical point cloud sequence composed of multiple adjacent frames; The point clouds of the current frame and historical frames are uniformly sampled and standardized to form point cloud blocks with a fixed number of points; The tracking task is described as a prediction of the target state increment, wherein the predicted quantities include the target's position offset and orientation change in three-dimensional space.

3. The method according to claim 1, characterized in that, The point feature extraction and prompt initialization steps include: Feature extraction is performed on the current frame point cloud and the historical frame point cloud respectively to obtain the current frame point feature and the historical frame point feature sequence; And initialize two types of learnable cues, namely the visual cues and the motion cues.

4. The method according to claim 1, characterized in that, In the short time domain information propagation step, the adjacent frame deformation module for cue enhancement is composed of multiple stacked layers. Each layer includes the appearance branch, the motion branch, and the mask branch, and the corresponding visual cue and motion cue are introduced into each layer to participate in the calculation. The method further includes: Cross-attention enhanced by motion cues, with current motion features as the primary focus, aggregates information related to the target motion from historical motion features; Enhanced cross-attention through visual cues, with current appearance features as the primary focus, aggregates information consistent with the target appearance from historical appearance features; Historical mask-related features are propagated to the current frame mask features through mask branching; Enhanced cross-attention through visual and motion cues enables the motion branch to be corrected using target cues provided by the appearance branch; Self-attention enhanced by visual cues is integrated and reorganized with appearance features; The system updates the motion features, appearance features, and mask features of the next layer through a feedforward network, and simultaneously updates the visual cues and motion cues.

5. The method according to claim 1, characterized in that, The long-term appearance cue learning steps include: The hierarchical appearance feature sequence from the short time domain information propagation step is input into the full sequence compression Mamba module, and the corresponding hidden state and several parameters are obtained through sequence compression. The hidden state is then fused with the current visual cue to obtain more complete global target appearance information, and the visual cue is updated based on the fusion result.

6. The method according to claim 1, characterized in that, The long-term motion cue learning steps include: The hierarchical motion feature sequence from the short time domain information propagation step is input into the full sequence compression Mamba module, and the corresponding hidden state and several parameters are obtained through sequence compression. The hidden state is then fused with the current motion cues to obtain more complete global target motion information, and the motion cues are updated based on the fusion result.

7. The method according to claim 1, characterized in that, The prediction steps include: The prediction head uses appearance-related features to perform mask prediction and completes the proposal and verification of target candidate locations; Simultaneously, motion-related features are used to regress the target position offset, thereby outputting the tracking results for the current frame.

8. The method according to claim 1, characterized in that, During the training phase, a joint loss is used for optimization, which consists of center regression loss, classification loss, proposal loss, bounding box loss, motion offset loss, and mask loss.

9. The method according to claim 4, characterized in that, The motion-enhanced cross-attention, the visual-enhanced cross-attention, the visual and motion-enhanced cross-attention, and the visual-enhanced self-attention are all implemented by a multi-head attention mechanism.

10. The method according to claim 5 or 6, characterized in that, The full-sequence compression Mamba module is built on a selective state-space model, and its discretization process is achieved through the zero-order preservation method to convert continuous parameters into discrete parameters for sequence processing.

Images