Multi-modal video target tracking method and system based on contrast learning modal alignment

Abstract

The application discloses a multi-modal video target tracking method and system based on contrast learning modal alignment, and belongs to the field of target tracking. The method comprises the following steps: acquiring an RGB image and a depth image of an input video sequence, performing synchronous registration and pretreatment; dividing the pretreated image into multiple patches, inputting the patches into a feature extraction network respectively, and extracting semantic texture features of the RGB and geometric structure features of the depth; taking the extracted features as inputs, constructing a cross-modal contrast learning modal alignment module, comparing positive and negative samples of the RGB features and the depth features, realizing modal alignment, and obtaining cross-modal consistent fusion feature representation; inputting the fusion features into a Transformer backbone network, performing global modeling by using a multi-head self-attention mechanism, and outputting position prediction and tracking results of a target in a video sequence. The application can significantly enhance tracking robustness in complex scenes such as illumination change, target occlusion and similar background, and realize high-precision video target tracking under multi-modal conditions.

Description

Technical Field

[0001] This invention relates to the field of target tracking, and more specifically to a multimodal video target tracking method and system based on contrastive learning modal alignment. Background Technology

[0002] In the fields of computer vision and intelligent sensing, multimodal video target tracking is of great value in applications such as security monitoring, autonomous driving, augmented reality, and intelligent robotics. These applications require systems capable of accurately and stably tracking moving targets in complex scenarios.

[0003] In recent years, depth images have gradually gained attention as another visual modality. Depth information can provide three-dimensional geometric structure and spatial location information, which has a natural advantage in solving occlusion and background interference. However, existing multimodal tracking methods often use simple feature stitching or layer-by-layer fusion, directly superimposing RGB and depth features into the model, ignoring the distribution differences and semantic gaps between the two modalities. This coarse fusion method is prone to feature inconsistencies and even introduces noise, thereby affecting the model's accuracy and generalization ability.

[0004] Meanwhile, the information quality of RGB and depth modalities varies significantly across different scenarios. For example, RGB features degrade severely in low-light conditions, while depth sensors can still provide stable geometric information; in the presence of distant or transparent objects, depth information is easily distorted by interference. This imbalance between modalities further exacerbates the difficulty of fusion. How to effectively align RGB and depth features while ensuring computational efficiency, enabling them to complement each other within a unified semantic space, thereby improving tracking accuracy and stability in complex scenes, has become an urgent problem to be solved. Summary of the Invention

[0005] In view of the problems existing in the above-mentioned background technology, such as insufficient single-modal information of RGB images, coarse multimodal feature fusion, and difficulty in aligning modal differences, this invention proposes a multimodal video target tracking method and system based on contrastive learning modal alignment. This method introduces a cross-modal contrastive learning modal alignment module on the basis of the OSTrack framework, realizing the alignment of RGB and depth features in a unified semantic space, thereby improving tracking accuracy and robustness in complex scenes.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, embodiments of the present invention provide a multimodal video target tracking method based on contrastive learning modal alignment, comprising: S1. Obtain the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. S2. Divide the preprocessed RGB image and depth image into multiple patch units, and input them into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. S3. Using the extracted features as input, construct a cross-modal contrastive learning modality alignment module, compare positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities; S4. Input the fused features into the Transformer backbone network, use the multi-head self-attention mechanism to perform global modeling, and output the target's position prediction and tracking results in the video sequence. In one embodiment, step S1, the synchronization registration and preprocessing includes: Brightness normalization and color balance are performed on RGB images to reduce the impact of changes in lighting. Missing values ​​are filled and filtered to remove noise from depth images in order to eliminate holes and random noise generated by sensor acquisition; Based on spatial geometric constraints or extrinsic parameter calibration methods, RGB images and depth images are registered at the pixel level to ensure their consistency in spatial location and time series.

[0007] In one embodiment, step S2 specifically includes: The preprocessed RGB image and depth image are divided into fixed-size patch units according to the visual Transformer partitioning method; Convolutional neural networks are used to extract the texture and semantic information of RGB patches, and lightweight convolutional networks are used to extract the geometric and spatial structure information of depth patches. By using linear projection or mapping networks, the extracted RGB and depth features are transformed into the same semantic feature space, reducing modal differences.

[0008] In one embodiment, step S3 specifically includes: The extracted semantic texture features and depth geometric features are used as input to construct a cross-modal contrastive learning modality alignment module. A contrastive learning mechanism is used to construct positive and negative sample pairs of RGB and deep features at the patch level; By optimizing the contrastive loss function, the RGB features and deep features of the same patch are brought closer to each other in the semantic space, while features from different sources or unrelated regions are pulled apart.

[0009] In one embodiment, constructing positive and negative sample pairs of RGB and depth features includes: RGB features and depth features at the same spatial location or the same target area are defined as positive sample pairs, and RGB features and depth features at different spatial locations or different target areas are defined as negative sample pairs. Among them, positive sample pairs are aligned by minimizing the feature distance during training, while negative sample pairs are distinguished by maximizing the feature distance. Sample selection can be combined with neighborhood constraints or time consistency rules to enhance the robustness and discriminativeness of contrastive learning.

[0010] In one embodiment, in step S3, the cross-modal contrastive learning modality alignment module includes: a lightweight projection head and a similarity measurement unit; Among them, the projection head is used to map RGB and depth features to a unified low-dimensional embedding space, and the similarity measurement unit calculates the similarity between different feature pairs based on cosine similarity or dot product operation; The cross-modal contrastive learning modality alignment module adopts a shallow network design, resulting in a smaller number of additional parameters and computational overhead.

[0011] In one embodiment, step S4 specifically includes: The aligned and fused feature sequence is input into a Transformer-based backbone network, which includes a multi-layer encoder structure for global modeling of the features. A contrastive learning module is inserted into the early and mid-stage layers of the encoder in the backbone network to enhance the consistency between RGB and deep modalities before deep feature extraction. The multi-head self-attention mechanism is used to dynamically model the dependencies between patch features, and a nonlinear transformation is performed through a feedforward neural network to generate the target representation; The decoded output sequence yields the target position prediction, which is then combined with trajectory smoothing to generate a continuous tracking trajectory.

[0012] In one embodiment, the contrastive learning module is a lightweight design, with additional parameters not exceeding 10% of the backbone network, and the computation is controlled through parameter sharing.

[0013] In one embodiment, the comparison loss function expression in step S3 is as follows:

[0014] in, This is a similarity measurement function; This represents the RGB features of a specific patch. For the depth positive sample features corresponding to its spatial location, For all candidate deep features, N is the total number of samples, and j is the index pointer to the current negative sample; This is a temperature coefficient used to adjust the discriminative power of the characteristic distribution.

[0015] Secondly, embodiments of the present invention also provide a multimodal video target tracking system based on contrastive learning modal alignment, the system being configured to perform the method described in any one of the above embodiments, including: The registration and preprocessing module is used to acquire the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. The feature extraction module is used to divide the preprocessed RGB image and depth image into multiple patch units, which are then input into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. The modality alignment module takes the extracted features as input to build a cross-modal contrastive learning module, which compares positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities. The target tracking module is used to input the fused features into the Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the target's position prediction and tracking results in the video sequence.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical advantages: A lightweight contrastive learning modality alignment module is introduced into the pre-trained visual Transformer backbone network, enabling effective alignment of RGB and depth features within a unified semantic space, fully leveraging the complementary advantages of the two modalities. This invention significantly enhances tracking robustness in complex scenarios such as varying illumination, target occlusion, and similar backgrounds, while maintaining the real-time performance and generalization capabilities of the original model through lightweight design, thereby achieving high-precision video target tracking under multimodal conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a flowchart of a multimodal video target tracking method based on contrastive learning modal alignment provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the principle framework of the method described in this invention; Figure 3This is a structural diagram of the contrastive learning modality alignment module described in this invention; Figure 4 This is a comparison diagram of the tracking target response described in this invention; Figure 5 This is a block diagram of a multimodal video target tracking system based on contrastive learning modal alignment provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To effectively fuse multimodal information from video and thus increase the accuracy of tracking algorithms in responding to target locations, this invention provides a multimodal video target tracking method based on contrastive learning modal alignment. (Refer to...) Figure 1 As shown, the specific steps include S1 to S4: S1. Obtain the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. S2. Divide the preprocessed RGB image and depth image into multiple patch units, and input them into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. S3. Using the extracted features as input, construct a cross-modal contrastive learning modality alignment module, compare positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities; S4. Input the fused features into the Transformer backbone network, use the multi-head self-attention mechanism to perform global modeling, and output the target's position prediction and tracking results in the video sequence. Figure 2This diagram illustrates the underlying principle of the method. From top to bottom on the left, the layers are: depth map template, depth map search region, RGB image template, and RGB image search region. First, all input images are segmented, with each segmented patch unit represented by a square. The depth map is segmented using the same parameters as the RGB image in the base model to ensure that the semantics of the patch units remain unchanged. Next, the RGB image is processed by the base model's encoding embedding network into an initial RGB encoding, represented by a dark gray block. The depth map patch units and the initial RGB image encoding are then fed into the contrastive learning modality alignment module. After processing, the RGBD fusion encoding is obtained. This fusion encoding contains the fusion information of the depth and RGB images and has the same dimension as the RGB image encoding, allowing it to be directly processed by the base model without requiring additional modifications. Finally, the base model with frozen weights, after inputting the RGBD fusion encoding, calculates a target location prediction box based on the model's output response, represented by a red box in the diagram, thus achieving target tracking under RGBD modality fusion.

[0021] The following is a detailed explanation of each of the above steps: In step S1, RGB and depth image preprocessing and synchronous registration are performed; First, the input RGB image is normalized for brightness and balanced for color to reduce the impact of lighting changes on feature distribution, so that the image maintains a relatively consistent visual appearance under different environments.

[0022] Secondly, missing value imputation and noise suppression are performed on the depth image. For voids generated during acquisition, neighborhood interpolation can be used for filling.

[0023] in, This represents the pixel value at coordinate position (x, y) of the filled depth image; Indicates A neighborhood window centered on the center; This represents the total number of non-empty pixels in the neighborhood window. This represents the pixel value at coordinates (i, j) in the original depth image, i.e., the depth value at a specific location within the neighborhood window. Subsequently, bilateral filtering is used to further smooth the depth distribution to reduce the interference of random noise.

[0024] Then, the preprocessed depth data is transformed into a three-dimensional coordinate system. Let the pixel coordinates of the depth image be... Its three-dimensional coordinates can be represented as:

[0025] Where K is the camera intrinsic parameter matrix.

[0026] Finally, using the camera extrinsic matrix Projecting depth points onto the RGB image plane achieves pixel-level alignment between the two modalities:

[0027] Through the above processing, the RGB image and the depth image are synchronously registered in the spatial and temporal dimensions, thus providing a unified data foundation for subsequent feature extraction and modality alignment.

[0028] In step S2, this step performs patch segmentation and feature extraction: First, the preprocessed and registered RGB image and depth image are divided into fixed-size patch units in the manner of a visual Transformer. Each patch represents information of a local region of the image, thereby transforming the original image into a serialized input that can be processed by the subsequent network.

[0029] Secondly, feature extraction is performed on the RGB image patch and the depth image patch respectively. The RGB image patch extracts semantic and texture information through a convolutional neural network to capture the appearance features of the target; the depth image patch extracts geometric and spatial structure information through a lightweight convolutional network to reflect the three-dimensional shape and depth differences of the scene.

[0030] Then, to reduce the distribution differences between modalities, the RGB and depth patch features are further mapped to a unified semantic space. Let the RGB features be... The depth features are The mapping process can be represented by linear projection as follows:

[0031] in, and The learned projection matrix; This is the representation of RGB features after linear projection; This is the representation of depth features after linear projection. The projected RGB values ​​and depth features are constrained to the same dimension and distribution, making them comparable and alignable.

[0032] Finally, all mapped patch features are concatenated sequentially to form a feature sequence before fusion, providing input for subsequent contrastive learning modality alignment.

[0033] In step S3, contrastive learning modality alignment is achieved: First, the RGB features and depth features obtained in step S2 are used as input to construct a cross-modal contrastive learning modality alignment module. The goal of this module is to establish constraints in a unified semantic space to make the RGB features and depth features of the same target region closer to each other, and the features of different regions separate from each other, thereby achieving intermodal consistency.

[0034] Reference Figure 3 The diagram illustrates the structure of the contrastive learning modality alignment module. First, depth information is enhanced by extracting its gradient information as enhanced features. The original features and enhanced features are then concatenated to obtain the geometric information encoding embedding. The shading in the diagram represents the depth image encoding containing geometric information. Simultaneously, the RGB image is encoded and embedded using the base model, resulting in the initial RGB encoding represented by pure dark gray in the diagram. Next, the depth image encoding and the initial RGB encoding are fed into the contrastive learning module. This module establishes alignment at the patch unit level. The diagonal squares in the diagram represent the complementary state of the two encodings after semantic space alignment. Finally, the two encodings are fused and projected onto the same dimension as the initial RGB encoding. This residual term is added to the RGB image encoding embedding, thus integrating depth information into the RGB image features to obtain the RGBD fused encoding. Secondly, a contrastive learning mechanism is used to construct positive and negative sample pairs of RGB and depth features at the patch level. RGB and depth features from the same spatial location are defined as positive sample pairs; features from different spatial locations or unrelated target regions are defined as negative sample pairs. In this way, the correspondence between RGB and depth can be constrained at the patch level.

[0035] Then, during training, a lightweight projection head is used to map RGB and depth features into a low-dimensional embedding space, and their similarity is calculated. Let a certain RGB patch feature be... The corresponding positive sample depth feature is The negative sample depth feature is The contrastive loss function is then defined as:

[0036] in, The similarity metric function is preferably cosine similarity. , where is the temperature parameter used to adjust the discriminative power of the feature distribution; N is the total number of samples, and j is the index pointer to the current negative sample. By optimizing the contrastive loss function, the RGB features and depth features of the same patch are brought closer together in the semantic space, while features from different sources or unrelated regions are distanced, thereby improving cross-modal consistency and obtaining a more discriminative fusion representation. In other words, by minimizing this loss function, corresponding RGB and depth features are made more consistent, while non-corresponding features remain separated, thus improving the cross-modal feature alignment effect.

[0037] This loss function achieves alignment by minimizing the distance between RGB and the depth features of positive samples, while maximizing the distance between RGB and the depth features of negative samples to achieve differentiation, thus achieving effective cross-modal alignment. Sample selection can be combined with neighborhood constraints or temporal consistency rules to enhance the robustness and discriminativeness of contrastive learning.

[0038] Finally, to ensure efficiency, the cross-modal contrastive learning modality alignment module of this invention adopts a shallow structure and lightweight design, which enhances the modality alignment capability while avoiding excessive computational overhead, thereby maintaining the real-time performance of the overall network.

[0039] Specifically, the cross-modal contrastive learning modality alignment module includes a lightweight projection head and a similarity measurement unit. The projection head maps RGB and depth features to a unified low-dimensional embedding space, while the similarity measurement unit calculates the similarity between different feature pairs based on cosine similarity or dot product operations. This design ensures that the module enhances the alignment capability between RGB and depth modalities without significantly increasing computational complexity, thus preserving the original real-time performance, tracking performance, and cross-scene generalization ability of the pre-trained tracking model.

[0040] In step S4, the processing and tracking output based on the Transformer backbone network are implemented: First, the modality-aligned fused feature sequence obtained in step S3 is input into the visual Transformer-based backbone network. This backbone network adopts a multi-layer encoder structure and uses a multi-head self-attention mechanism and a feedforward neural network to globally model the input features, thereby capturing the correlation of the target in the temporal and spatial dimensions.

[0041] Secondly, a contrastive learning module is inserted into the early and mid-stage layers of the backbone network, enabling consistency enhancement between RGB and deeply fused features before entering deep modeling. In this way, the network can further improve the discriminative ability of cross-modal features while maintaining overall structural stability. The fused features are globally modeled within the Transformer's multi-head self-attention mechanism, thereby improving target tracking accuracy and robustness in complex scenes.

[0042] The contrastive learning module adopts a lightweight design. Structurally, this module uses a shallow network design, introducing only a small number of convolutional or linear transformation layers, not exceeding 10% of the original network parameters. Redundant computation is reduced through parameter sharing and weight constraint strategies.

[0043] Then, a multi-head self-attention mechanism is used to dynamically model the dependencies between patches, enabling the network to focus on the target region based on contextual information and reduce the impact of occlusion, lighting changes, and background interference. Simultaneously, the feedforward neural network layer performs a non-linear transformation on the attention output to generate a more robust target representation.

[0044] Finally, the output sequence features of the backbone network are decoded to obtain the target prediction box and trajectory. To improve tracking stability, this invention can also incorporate trajectory smoothing strategies in the output stage, such as Kalman filtering or moving average methods, to reduce prediction jitter, thereby generating continuous and stable target trajectories and achieving robust video target tracking in complex scenarios.

[0045] Reference Figure 4 The image shows a comparison of the target tracking response between the method of this invention and existing methods. The first row shows the correct target bounding boxes on the RGB and depth images, with the ground truth (GT) boxes representing the true target bounding boxes. The second row shows a comparison of the model response and detection boxes on the RGB and depth images. The left image shows the response of the basic model, which produces a large area response in an incorrect location; box 1 represents the detection result of the basic model. The right image shows the response of the method of this invention, which correctly produces a clear response at the center of the target; box 2 represents the detection result of the method of this invention. Based on the same inventive concept, this invention also provides a multimodal video target tracking system based on contrastive learning modal alignment. Since the principle of solving the problem by these systems is similar to that of the aforementioned multimodal video target tracking method based on contrastive learning modal alignment, the implementation of this system can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0046] like Figure 5 As shown, this embodiment of the invention also provides a multimodal video target tracking system based on contrastive learning modal alignment. The system is configured to perform the method described in the above embodiments, and the system includes: The registration and preprocessing module is used to acquire the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. The feature extraction module is used to divide the preprocessed RGB image and depth image into multiple patch units, which are then input into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. The modality alignment module is used to take the extracted features as input to build a cross-modal contrastive learning modality alignment module. It compares positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities. The target tracking module is used to input the fused features into the Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the target's position prediction and tracking results in the video sequence.

[0047] This system achieves multimodal complementary fusion through preprocessing of RGB and depth images, feature extraction, contrastive learning alignment, and Transformer global modeling, significantly improving tracking robustness and accuracy.

[0048] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0049] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal video target tracking method based on contrastive learning modal alignment, characterized in that, include: S1. Obtain the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. S2. Divide the preprocessed RGB image and depth image into multiple patch units, and input them into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. S3. Using the extracted features as input, construct a cross-modal contrastive learning modality alignment module, compare positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities; S4. Input the fused features into the Transformer backbone network, use the multi-head self-attention mechanism to perform global modeling, and output the target's position prediction and tracking results in the video sequence. Step S3 specifically includes: The extracted semantic texture features and depth geometric features are used as input to construct a cross-modal contrastive learning modality alignment module. The contrastive learning mechanism constructs positive and negative sample pairs of RGB and depth features at the patch level. This includes defining RGB features and depth features at the same spatial location or the same target region as positive sample pairs, and defining RGB features and depth features at different spatial locations or different target regions as negative sample pairs. During training, positive sample pairs are aligned by minimizing feature distance, while negative sample pairs are distinguished by maximizing feature distance. Sample selection can be combined with neighborhood constraints or temporal consistency rules to enhance the robustness and discriminativeness of contrastive learning. By optimizing the contrastive loss function, the RGB features and deep features of the same patch are brought closer to each other in the semantic space, while features from different sources or unrelated regions are pulled apart. Furthermore, the contrastive learning modal alignment module enhances the deep features by extracting their gradient information as enhanced features, concatenates the original deep features with the enhanced features to obtain geometric information encoding embedding, and then sends it to the contrastive learning module with the initial RGB encoding to complete the alignment at the patch unit level. Finally, the two encodings are fused and projected to the same dimension as the initial RGB encoding, and added as a residual term to the encoding embedding of the RGB image to obtain the RGBD fusion encoding.

2. The method as described in claim 1, characterized in that, In step S1, the synchronization registration and preprocessing include: Brightness normalization and color balance are performed on RGB images to reduce the impact of changes in lighting. Missing values ​​are filled and filtered to remove noise from depth images in order to eliminate holes and random noise generated by sensor acquisition; Based on spatial geometric constraints or extrinsic parameter calibration methods, RGB images and depth images are registered at the pixel level to ensure their consistency in spatial location and time series.

3. The method as described in claim 1, characterized in that, Step S2 specifically includes: The preprocessed RGB image and depth image are divided into fixed-size patch units according to the visual Transformer partitioning method; Convolutional neural networks are used to extract the texture and semantic information of RGB patches, and lightweight convolutional networks are used to extract the geometric and spatial structure information of depth patches. By using linear projection or mapping networks, the extracted RGB and depth features are transformed into the same semantic feature space, reducing modal differences.

4. The method as described in claim 1, characterized in that, In step S3, the cross-modal contrastive learning modality alignment module includes: a lightweight projection head and a similarity measurement unit; Among them, the projection head is used to map RGB and depth features to a unified low-dimensional embedding space, and the similarity measurement unit calculates the similarity between different feature pairs based on cosine similarity or dot product operation; The cross-modal contrastive learning modality alignment module adopts a shallow network design, resulting in a smaller number of additional parameters and computational overhead.

5. The method as described in claim 1, characterized in that, Step S4 specifically includes: The aligned and fused feature sequence is input into a Transformer-based backbone network, which includes a multi-layer encoder structure for global modeling of the features. A contrastive learning module is inserted into the early and mid-stage layers of the encoder in the backbone network to enhance the consistency between RGB and deep modalities before deep feature extraction. The multi-head self-attention mechanism is used to dynamically model the dependencies between patch features, and a nonlinear transformation is performed through a feedforward neural network to generate the target representation; The decoded output sequence yields the target position prediction, which is then combined with trajectory smoothing to generate a continuous tracking trajectory.

6. The method as described in claim 5, characterized in that, The comparative learning module is designed to be lightweight, with additional parameters not exceeding 10% of the backbone network, and calculations are controlled through parameter sharing.

7. The method as described in claim 1, characterized in that, In step S3, the expression for the comparison loss function is as follows: in, This is a similarity measurement function; This represents the RGB features of a specific patch. For the depth positive sample features corresponding to its spatial location, For all candidate deep features, N is the total number of samples, and j is the index pointer to the current negative sample; This is a temperature coefficient used to adjust the discriminative power of the characteristic distribution.

8. A multimodal video target tracking system based on contrastive learning modal alignment, characterized in that, The system is configured to perform the method according to any one of claims 1-7, including: The registration and preprocessing module is used to acquire the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. The feature extraction module is used to divide the preprocessed RGB image and depth image into multiple patch units, which are then input into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. The modality alignment module is used to take the extracted features as input to build a cross-modal contrastive learning modality alignment module. It compares positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities. The target tracking module is used to input the fused features into the Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the target's position prediction and tracking results in the video sequence.

Images