Multi-modal video target tracking method and system based on comparative learning modal alignment

Abstract

The invention discloses a multi-modal video target tracking method and system based on comparative learning modal alignment, and belongs to the field of target tracking. The method comprises the following steps: acquiring an RGB image and a depth image of an input video sequence, and performing synchronous registration and preprocessing; dividing the preprocessed image into a plurality of patches, respectively inputting the patches into a feature extraction network, and extracting semantic texture features of RGB and geometric structure features of depth; taking the extracted features as input, constructing a cross-modal comparison learning modal alignment module, performing positive and negative sample comparison on the RGB features and the depth features, realizing modal alignment, and obtaining cross-modal consistent fusion feature representation; and inputting the fusion features into a Transform backbone network, performing global modeling by using a multi-head self-attention mechanism, and outputting a position prediction and tracking result of the target in the video sequence. According to the method, the tracking robustness can be remarkably enhanced in complex scenes such as illumination variation, target shielding and similar backgrounds, and high-precision video target tracking under a multi-mode condition is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target tracking, and more particularly to a multi-modal video target tracking method and system based on contrast learning modal alignment. BACKGROUND

[0002] In the field of computer vision and intelligent perception, multi-modal video target tracking has important value in applications such as security monitoring, autonomous driving, augmented reality, and intelligent robots. These applications require the system to accurately and stably track moving targets in complex scenes.

[0003] In recent years, depth images have gradually attracted attention as another visual modality. Depth information can provide three-dimensional geometric structure and spatial location information, and has a natural advantage in solving occlusion and background interference. However, existing multi-modal tracking methods often use simple feature splicing or layer-by-layer fusion, directly superimposing RGB and depth features into the model, ignoring the distribution differences and semantic gaps between the two modalities. This rough fusion method can easily cause inconsistent features and even introduce noise, thereby affecting the accuracy and generalization ability of the model.

[0004] At the same time, the information quality of RGB and depth modalities differs greatly in different scenes. For example, when the light is insufficient, the RGB feature degrades severely, while the depth sensor can still provide stable geometric information; in the case of long distance or transparent objects, depth information is easily disturbed and distorted. This imbalance between modalities further exacerbates the difficulty of fusion. How to effectively align RGB and depth features while ensuring computational efficiency, so that they complement each other in a unified semantic space, thereby improving tracking accuracy and stability in complex scenes, has become a problem that needs to be solved. SUMMARY

[0005] In view of the problems of insufficient RGB image single modality information, rough multi-modal feature fusion, and difficult modal difference alignment in the above background art, the present application proposes a multi-modal video target tracking method and system based on contrast learning modal alignment. This method introduces a cross-modal contrast learning modal alignment module based on the OStrack framework, aligns RGB and depth features in a unified semantic space, and improves tracking accuracy and robustness in complex scenes.

[0006] To achieve the above purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a multi-modal video target tracking method based on contrast learning modal alignment, comprising: S1, obtaining the RGB image and depth image of the input video sequence, and performing synchronous registration and preprocessing to eliminate interference between modalities and ensure spatial and temporal consistency between modalities; S2, divide the pre-processed RGB image and the depth image into a plurality of patch units, respectively input a feature extraction network, and extract semantic texture features of the RGB and geometric structure features of the depth; S3, input the extracted features, construct a cross-modal contrast learning modal alignment module, compare the positive and negative sample pairs of the RGB features and the depth features, realize modal alignment, and obtain cross-modal consistent fusion feature representation; S4, input the fusion features into a Transformer backbone network, perform global modeling by using a multi-head self-attention mechanism, and output position prediction and tracking results of the target in the video sequence. In one embodiment, the step S1 includes: The RGB image is subjected to brightness normalization and color equalization to weaken the influence of illumination changes; The depth image is subjected to missing value filling and filtering denoising to eliminate holes and random noise generated by the sensor acquisition; Based on the spatial geometric constraint or the external parameter calibration method, the RGB image and the depth image are registered at the pixel level, so as to ensure the consistency of the two in the spatial position and the time sequence.

[0007] In one embodiment, the step S2 specifically includes: The pre-processed RGB image and the depth image are cut into patch units of a fixed size according to the division method of the visual Transformer; The texture and semantic information of the RGB patch is extracted by using a convolutional neural network, and the geometric and spatial structure information of the depth patch is extracted by using a lightweight convolutional network; The extracted RGB and depth features are converted to the same semantic feature space by using a linear projection or mapping network, so as to reduce the modal difference.

[0008] In one embodiment, the step S3 specifically includes: The extracted semantic texture features of the RGB and the geometric structure features of the depth are input, and a cross-modal contrast learning modal alignment module is constructed; The positive and negative sample pairs of the RGB and the depth features are constructed at the patch level by using a contrast learning mechanism; By optimizing the contrast loss function, the RGB features and the depth features of the homologous patch are close to each other in the semantic space, while the features of different sources or irrelevant regions are far away from each other.

[0009] In one embodiment, the positive and negative sample pairs of the RGB and the depth features are constructed, including: RGB features and depth features at the same spatial location or the same target area are defined as positive sample pairs, and RGB features and depth features at different spatial locations or different target areas are defined as negative sample pairs. Among them, positive sample pairs are aligned by minimizing the feature distance during training, while negative sample pairs are distinguished by maximizing the feature distance. Sample selection can be combined with neighborhood constraints or time consistency rules to enhance the robustness and discriminativeness of contrastive learning.

[0010] In one embodiment, in step S3, the cross-modal contrastive learning modality alignment module includes: a lightweight projection head and a similarity measurement unit; Among them, the projection head is used to map RGB and depth features to a unified low-dimensional embedding space, and the similarity measurement unit calculates the similarity between different feature pairs based on cosine similarity or dot product operation; The cross-modal contrastive learning modality alignment module adopts a shallow network design, resulting in a smaller number of additional parameters and computational overhead.

[0011] In one embodiment, step S4 specifically includes: The aligned and fused feature sequence is input into a Transformer-based backbone network, which includes a multi-layer encoder structure for global modeling of the features. A contrastive learning module is inserted into the early and mid-stage layers of the encoder in the backbone network to enhance the consistency between RGB and deep modalities before deep feature extraction. The multi-head self-attention mechanism is used to dynamically model the dependencies between patch features, and a nonlinear transformation is performed through a feedforward neural network to generate the target representation; The decoded output sequence yields the target position prediction, which is then combined with trajectory smoothing to generate a continuous tracking trajectory.

[0012] In one embodiment, the contrastive learning module is a lightweight design, with additional parameters not exceeding 10% of the backbone network, and the computation is controlled through parameter sharing.

[0013] In one embodiment, the comparison loss function expression in step S3 is as follows:

[0014] in, This is a similarity measurement function; This represents the RGB features of a specific patch. For the depth positive sample features corresponding to its spatial location, For all candidate deep features, N is the total number of samples, and j is the index pointer to the current negative sample; This is a temperature coefficient used to adjust the discriminative power of the characteristic distribution.

[0015] Secondly, embodiments of the present invention also provide a multimodal video target tracking system based on contrastive learning modal alignment, the system being configured to perform the method described in any one of the above embodiments, including: The registration and preprocessing module is used to acquire the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. The feature extraction module is used to divide the preprocessed RGB image and depth image into multiple patch units, which are then input into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. The modality alignment module takes the extracted features as input to build a cross-modal contrastive learning module, which compares positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities. The target tracking module is used to input the fused features into the Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the target's position prediction and tracking results in the video sequence.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention has the following technical advantages: A lightweight contrastive learning modality alignment module is introduced into the pre-trained visual Transformer backbone network, enabling effective alignment of RGB and depth features within a unified semantic space, fully leveraging the complementary advantages of the two modalities. This invention significantly enhances tracking robustness in complex scenarios such as varying illumination, target occlusion, and similar backgrounds, while maintaining the real-time performance and generalization capabilities of the original model through lightweight design, thereby achieving high-precision video target tracking under multimodal conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a flowchart of a multimodal video target tracking method based on contrastive learning modal alignment provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the principle framework of the method described in this invention; Figure 3This is a structural diagram of the contrastive learning modality alignment module described in this invention; Figure 4 This is a comparison diagram of the tracking target response described in this invention; Figure 5 This is a block diagram of a multimodal video target tracking system based on contrastive learning modal alignment provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] To effectively fuse multimodal information from video and thus increase the accuracy of tracking algorithms in responding to target locations, this invention provides a multimodal video target tracking method based on contrastive learning modal alignment. (Refer to...) Figure 1 As shown, the specific steps include S1 to S4: S1. Obtain the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. S2. Divide the preprocessed RGB image and depth image into multiple patch units, and input them into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. S3. Using the extracted features as input, construct a cross-modal contrastive learning modality alignment module, compare positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities; S4. Input the fused features into the Transformer backbone network, use the multi-head self-attention mechanism to perform global modeling, and output the target's position prediction and tracking results in the video sequence. Figure 2This diagram illustrates the underlying principle of the method. From top to bottom on the left, the layers are: depth map template, depth map search region, RGB image template, and RGB image search region. First, all input images are segmented, with each segmented patch unit represented by a square. The depth map is segmented using the same parameters as the RGB image in the base model to ensure that the semantics of the patch units remain unchanged. Next, the RGB image is processed by the base model's encoding embedding network into an initial RGB encoding, represented by a dark gray block. The depth map patch units and the initial RGB image encoding are then fed into the contrastive learning modality alignment module. After processing, the RGBD fusion encoding is obtained. This fusion encoding contains the fusion information of the depth and RGB images and has the same dimension as the RGB image encoding, allowing it to be directly processed by the base model without requiring additional modifications. Finally, the base model with frozen weights, after inputting the RGBD fusion encoding, calculates a target location prediction box based on the model's output response, represented by a red box in the diagram, thus achieving target tracking under RGBD modality fusion.

[0021] The following is a detailed explanation of each of the above steps: In step S1, RGB and depth image preprocessing and synchronous registration are performed; First, the input RGB image is normalized for brightness and balanced for color to reduce the impact of lighting changes on feature distribution, so that the image maintains a relatively consistent visual appearance under different environments.

[0022] Secondly, missing value imputation and noise suppression are performed on the depth image. For voids generated during acquisition, neighborhood interpolation can be used for filling.

[0023] in, This represents the pixel value at coordinate position (x, y) of the filled depth image; Indicated by A neighborhood window centered on the center; This represents the total number of non-empty pixels in the neighborhood window. This represents the pixel value at coordinates (i, j) in the original depth image, i.e., the depth value at a specific location within the neighborhood window. Subsequently, bilateral filtering is used to further smooth the depth distribution to reduce the interference of random noise.

[0024] Then, the preprocessed depth data is transformed into a three-dimensional coordinate system. Let the pixel coordinates of the depth image be... Its three-dimensional coordinates can be represented as:

[0025] Where K is the camera intrinsic parameter matrix.

[0026] Finally, using the camera extrinsic matrix Projecting depth points onto the RGB image plane achieves pixel-level alignment between the two modalities:

[0027] Through the above processing, the RGB image and the depth image are synchronously registered in the spatial and temporal dimensions, thus providing a unified data foundation for subsequent feature extraction and modality alignment.

[0028] In step S2, this step performs patch segmentation and feature extraction: First, the preprocessed and registered RGB image and depth image are divided into fixed-size patch units in the manner of a visual Transformer. Each patch represents information of a local region of the image, thereby transforming the original image into a serialized input that can be processed by the subsequent network.

[0029] Secondly, feature extraction is performed on the RGB image patch and the depth image patch respectively. The RGB image patch extracts semantic and texture information through a convolutional neural network to capture the appearance features of the target; the depth image patch extracts geometric and spatial structure information through a lightweight convolutional network to reflect the three-dimensional shape and depth differences of the scene.

[0030] Then, to reduce the distribution differences between modalities, the RGB and depth patch features are further mapped to a unified semantic space. Let the RGB features be... The depth features are The mapping process can be represented by linear projection as follows:

[0031] in, and The learned projection matrix; This is the representation of RGB features after linear projection; This is the representation of depth features after linear projection. The projected RGB values ​​and depth features are constrained to the same dimension and distribution, making them comparable and alignable.

[0032] Finally, all mapped patch features are concatenated sequentially to form a feature sequence before fusion, providing input for subsequent contrastive learning modality alignment.

[0033] In step S3, contrastive learning modality alignment is achieved: First, the RGB features and depth features obtained in step S2 are used as input to construct a cross-modal contrastive learning modality alignment module. The goal of this module is to establish constraints in a unified semantic space to make the RGB features and depth features of the same target region closer to each other, and the features of different regions separate from each other, thereby achieving intermodal consistency.

[0034] Reference Figure 3 The diagram illustrates the structure of the contrastive learning modality alignment module. First, depth information is enhanced by extracting its gradient information as enhanced features. The original features and enhanced features are then concatenated to obtain the geometric information encoding embedding. The shading in the diagram represents the depth image encoding containing geometric information. Simultaneously, the RGB image is encoded and embedded using the base model, resulting in the initial RGB encoding represented by pure dark gray in the diagram. Next, the depth image encoding and the initial RGB encoding are fed into the contrastive learning module. This module establishes alignment at the patch unit level. The diagonal squares in the diagram represent the complementary state of the two encodings after semantic space alignment. Finally, the two encodings are fused and projected onto the same dimension as the initial RGB encoding. This residual term is added to the RGB image encoding embedding, thus integrating depth information into the RGB image features to obtain the RGBD fused encoding. Secondly, a contrastive learning mechanism is used to construct positive and negative sample pairs of RGB and depth features at the patch level. RGB and depth features from the same spatial location are defined as positive sample pairs; features from different spatial locations or unrelated target regions are defined as negative sample pairs. In this way, the correspondence between RGB and depth can be constrained at the patch level.

[0035] Then, during training, a lightweight projection head is used to map RGB and depth features into a low-dimensional embedding space, and their similarity is calculated. Let a certain RGB patch feature be... The corresponding positive sample depth feature is The negative sample depth feature is The contrastive loss function is then defined as:

[0036] in, The similarity metric function is preferably cosine similarity. , where is the temperature parameter used to adjust the discriminative power of the feature distribution; N is the total number of samples, and j is the index pointer to the current negative sample. By optimizing the contrastive loss function, the RGB features and depth features of the same patch are brought closer together in the semantic space, while features from different sources or unrelated regions are distanced, thereby improving cross-modal consistency and obtaining a more discriminative fusion representation. In other words, by minimizing this loss function, corresponding RGB and depth features are made more consistent, while non-corresponding features remain separated, thus improving the cross-modal feature alignment effect.

[0037] This loss function achieves alignment by minimizing the distance between RGB and the depth features of positive samples, while maximizing the distance between RGB and the depth features of negative samples to achieve differentiation, thus achieving effective cross-modal alignment. Sample selection can be combined with neighborhood constraints or temporal consistency rules to enhance the robustness and discriminativeness of contrastive learning.

[0038] Finally, to ensure efficiency, the cross-modal contrastive learning modality alignment module of this invention adopts a shallow structure and lightweight design, which enhances the modality alignment capability while avoiding excessive computational overhead, thereby maintaining the real-time performance of the overall network.

[0039] Specifically, the cross-modal contrastive learning modality alignment module includes a lightweight projection head and a similarity measurement unit. The projection head maps RGB and depth features to a unified low-dimensional embedding space, while the similarity measurement unit calculates the similarity between different feature pairs based on cosine similarity or dot product operations. This design ensures that the module enhances the alignment capability between RGB and depth modalities without significantly increasing computational complexity, thus preserving the original real-time performance, tracking performance, and cross-scene generalization ability of the pre-trained tracking model.

[0040] In step S4, the processing and tracking output based on the Transformer backbone network are implemented: First, the modality-aligned fused feature sequence obtained in step S3 is input into the visual Transformer-based backbone network. This backbone network adopts a multi-layer encoder structure and uses a multi-head self-attention mechanism and a feedforward neural network to globally model the input features, thereby capturing the correlation of the target in the temporal and spatial dimensions.

[0041] Secondly, a contrastive learning module is inserted into the early and mid-stage layers of the backbone network, enabling consistency enhancement between RGB and deeply fused features before entering deep modeling. In this way, the network can further improve the discriminative ability of cross-modal features while maintaining overall structural stability. The fused features are globally modeled within the Transformer's multi-head self-attention mechanism, thereby improving target tracking accuracy and robustness in complex scenes.

[0042] The contrastive learning module adopts a lightweight design. Structurally, this module uses a shallow network design, introducing only a small number of convolutional or linear transformation layers, not exceeding 10% of the original network parameters. Redundant computation is reduced through parameter sharing and weight constraint strategies.

[0043] Then, a multi-head self-attention mechanism is used to dynamically model the dependencies between patches, enabling the network to focus on the target region based on contextual information and reduce the impact of occlusion, lighting changes, and background interference. Simultaneously, the feedforward neural network layer performs a non-linear transformation on the attention output to generate a more robust target representation.

[0044] Finally, the output sequence features of the backbone network are decoded to obtain the target prediction box and trajectory. To improve tracking stability, this invention can also incorporate trajectory smoothing strategies in the output stage, such as Kalman filtering or moving average methods, to reduce prediction jitter, thereby generating continuous and stable target trajectories and achieving robust video target tracking in complex scenarios.

[0045] Reference Figure 4 The image shows a comparison of the target tracking response between the method of this invention and existing methods. The first row shows the correct target bounding boxes on the RGB and depth images, with the ground truth (GT) boxes representing the true target bounding boxes. The second row shows a comparison of the model response and detection boxes on the RGB and depth images. The left image shows the response of the basic model, which produces a large area response in an incorrect location; box 1 represents the detection result of the basic model. The right image shows the response of the method of this invention, which correctly produces a clear response at the center of the target; box 2 represents the detection result of the method of this invention. Based on the same inventive concept, this invention also provides a multimodal video target tracking system based on contrastive learning modal alignment. Since the principle of solving the problem by these systems is similar to that of the aforementioned multimodal video target tracking method based on contrastive learning modal alignment, the implementation of this system can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0046] like Figure 5 As shown, this embodiment of the invention also provides a multimodal video target tracking system based on contrastive learning modal alignment. The system is configured to perform the method described in the above embodiments, and the system includes: The registration and preprocessing module is used to acquire the RGB and depth images of the input video sequence, and perform synchronous registration and preprocessing to eliminate intermodal interference and ensure spatial and temporal consistency between modalities. The feature extraction module is used to divide the preprocessed RGB image and depth image into multiple patch units, which are then input into the feature extraction network to extract the semantic texture features of RGB and the geometric structure features of depth. The modality alignment module is used to take the extracted features as input to build a cross-modal contrastive learning modality alignment module. It compares positive and negative samples of RGB features and depth features to achieve modality alignment and obtain a consistent fusion feature representation across modalities. The target tracking module is used to input the fused features into the Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the target's position prediction and tracking results in the video sequence.

[0047] This system achieves multimodal complementary fusion through preprocessing of RGB and depth images, feature extraction, contrastive learning alignment, and Transformer global modeling, significantly improving tracking robustness and accuracy.

[0048] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0049] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-modal video object tracking method based on contrastive learning modality alignment, characterized in that, The application relates to a method for cross-modal video object tracking, comprising the following steps: S1, acquiring an RGB image and a depth image of an input video sequence, and performing synchronous registration and preprocessing to eliminate interference between the two modalities and ensure spatial and temporal consistency between the two modalities; S2, dividing the preprocessed RGB image and depth image into a plurality of patch units, and inputting the RGB image and the depth image into a feature extraction network respectively to extract semantic texture features of the RGB image and geometric structure features of the depth image; S3, taking the extracted features as inputs to construct a cross-modal contrast learning modal alignment module, comparing the RGB features and the depth features as positive and negative sample pairs, realizing modal alignment, and obtaining cross-modal consistent fusion feature representation; S4, inputting the fusion features into a Transformer backbone network, performing global modeling by using a multi-head self-attention mechanism, and outputting position prediction and tracking results of a target in a video sequence.

2. The method of claim 1, wherein, In the step S1, the synchronous registration and preprocessing comprises the following steps: Performing brightness normalization and color equalization on the RGB image to weaken the influence of illumination changes; Performing missing value filling and filter denoising on the depth image to eliminate holes and random noise generated by the sensor during acquisition; Based on a spatial geometric constraint or an external parameter calibration method, the RGB image and the depth image are registered at the pixel level, so that the consistency of the two in spatial position and time sequence is ensured.

3. The method of claim 1, wherein, The step S2 specifically comprises the following steps: The preprocessed RGB image and depth image are cut into patch units of a fixed size according to the division mode of a visual Transformer; Texture and semantic information of the RGB patch is extracted by using a convolutional neural network, and geometric and spatial structure information of the depth patch is extracted by using a lightweight convolutional network; The extracted RGB and depth features are converted to the same semantic feature space by using a linear projection or mapping network to reduce the modal difference.

4. The method of claim 1, wherein, The step S3 specifically comprises the following steps: The extracted semantic texture features of the RGB image and the geometric structure features of the depth image are taken as inputs to construct a cross-modal contrast learning modal alignment module; The positive and negative sample pairs of the RGB and depth features are constructed at the patch level by using a contrast learning mechanism; By optimizing a contrast loss function, the RGB features and the depth features of the same source patch are close to each other in the semantic space, while the features of different sources or irrelevant regions are far away from each other.

5. The method of claim 4, wherein, The positive and negative sample pairs of the RGB and depth features are constructed, comprising the following steps: The RGB features and the depth features of the same spatial position or the same target region are defined as the positive sample pairs, and the RGB features and the depth features of different spatial positions or different target regions are defined as the negative sample pairs; In the training process, the positive sample pairs are aligned by minimizing the feature distance, and the negative sample pairs are distinguished by maximizing the feature distance; The sample selection can be combined with neighborhood constraints or time consistency rules to enhance the robustness and discriminability of the contrast learning.

6. The method of claim 5, wherein, In the step S3, the cross-modal contrast learning modal alignment module comprises a lightweight projection head and a similarity measurement unit. The projection head is configured to map RGB and depth features to a unified low-dimensional embedding space, and the similarity measurement unit is configured to calculate the similarity between different feature pairs based on cosine similarity or dot product operation. The cross-modal contrast learning modal alignment module is structurally designed with a shallow network, and has small additional parameter quantity and computational overhead.

7. The method of claim 1, wherein, The step S4 specifically includes: The aligned fusion feature sequence is input into a Transformer-based backbone network, which includes a multi-layer encoder structure for global modeling of the features. A contrast learning module is inserted in the early and middle layers of the encoder of the backbone network to enhance the consistency of the RGB and depth modalities before deep feature extraction. The dependency relationship between patch features is dynamically modeled using a multi-head self-attention mechanism, and a feedforward neural network is used for nonlinear transformation to generate target representations. The target position prediction is obtained by decoding the output sequence, and the continuous tracking trajectory is generated by combining trajectory smoothing processing.

8. The method of claim 7, wherein, The contrast learning module is designed to be lightweight, with an additional parameter quantity of no more than 10% of the backbone network, and the calculation is controlled through parameter sharing.

9. The method of claim 4, wherein, In step S3, the contrast loss function expression is as follows: wherein, is a similarity measure function; is a RGB feature representation of a certain patch, is a positive depth sample feature corresponding to the spatial position of the patch, is all candidate depth features, N is the total amount of samples, and j is a subscript pointer pointing to the current negative sample; is a temperature coefficient, used to adjust the discrimination degree of the feature distribution.

10. A multi-modal video object tracking system based on contrastive learning modality alignment, characterized in that, The system is configured to perform the method according to any one of claims 1-9, comprising: A registration and preprocessing module is configured to obtain RGB images and depth images of an input video sequence, and perform synchronous registration and preprocessing to eliminate interference between modalities and ensure spatial and temporal consistency between modalities. A feature extraction module is configured to divide the preprocessed RGB images and depth images into multiple patch units, and input them into a feature extraction network respectively to extract semantic texture features of the RGB and geometric structure features of the depth. A modal alignment module is configured to input the extracted features as input to construct a cross-modal contrast learning modal alignment module, compare the RGB features and depth features with positive and negative samples, realize modal alignment, and obtain cross-modal consistent fusion feature representation. A target tracking module is configured to input the fusion features into a Transformer backbone network, perform global modeling using a multi-head self-attention mechanism, and output the position prediction and tracking result of the target in the video sequence.

Images