RGBD target tracking method based on deep and shallow feature fusion

Through the integration of depth and shallow feature fusion and the use of attention mechanisms, the problem of insufficient fusion of modal information in the existing RGBD tracking technology is solved, efficient and accurate RGBD target tracking is achieved, and the goal identification ability and tracking in complex scenarios is improved. quality.

CN119027446AActive Publication Date: 2024-11-26NANJING UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202310586055.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-23
Publication Date
2024-11-26
Estimated Expiration
2043-05-23

AI Technical Summary

Technical Problem

The existing RGBD tracking technology lacks efficient and robust modal fusion methods, which makes the texture details provided by the visible light mode being ignored, and the contour information and semantic information provided by the depth mode being difficult to extract, reducing the target recognition ability.

Method used

Using the deep and shallow feature fusion method, the shallow and deep features of visual and depth images are extracted by using ResNet-50, and the shallow features are enhanced through the attention mechanism, combining the feature redistribution module for shallow depth and shallow layers Dynamic fusion of features improves the efficiency of modal information utilization.

Benefits of technology

It realizes efficient and accurate RGBD target tracking, makes full use of depth image information, improves the quality of target tracking in complex indoor scenarios, and improves the real-time and robustness of the tracker model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027446A_ABST
    Figure CN119027446A_ABST
Patent Text Reader

Abstract

An RGBD target tracking method based on deep and shallow feature fusion is characterized in that RGBD target tracking is realized through a target tracking model, the target tracking model comprises a backbone network and a deep and shallow feature fusion and target discrimination module, and the target tracking model is used for online tracking and online updating of model parameters after offline training. According to the method, a deep and shallow feature fusion algorithm is constructed to learn robust feature expression, detail texture information of shallow features and contour semantic information in deep features are extracted through an attention mechanism, full fusion is realized through a feature redistribution module, and a target position and a score are searched by using a discrimination model. The RGBD tracking method solves the problems that most RGBD tracking methods at present are insufficient in bimodal feature mining and incomplete in fusion, and a good tracking effect is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision target tracking, and in particular relates to an RGBD target tracking method based on deep and shallow layer feature fusion. Background Art

[0002] As a kind of multimodal target tracking technology, RGBD tracking technology combines visible light and depth images for tracking tasks to make full use of the rich color, texture and detail features provided by visible light and the object contour shape and distance data captured by depth images. This combination helps to distinguish the difference between the target and the background in complex indoor scenes, and to deal with the problem of poor tracking effect of a single modality caused by many similar targets. However, because visible light and depth modalities have different focuses on feature expression and different forms of expression, directly using addition or cascade methods for feature fusion may lead to loss of modal information, and the contribution of the two modalities will be different in different environments. It is also very important to adjust the modal feature allocation method according to the actual environment. How to effectively fuse the features of the two and give full play to their respective advantages is the key issue of RGBD tracking tasks.

[0003] The main problem of existing methods is the lack of efficient and robust modality fusion methods. Common modality fusion methods include shallow feature fusion and deep feature fusion, which refer to the fusion of features extracted by deep neural networks at shallow layers or deep layers, respectively. When the network layer is shallow, the receptive field of the network is small, and the network extracts a large number of texture and detail features; when the network layer is deepened, the receptive field becomes larger, and the network extracts a large number of contour features and semantic information. For RGBD tracking, the visible light modality can provide a large amount of texture detail information. The use of deep feature fusion methods is likely to ignore these features, reducing the tracking method's ability to distinguish similar objects or camouflaged targets. The use of shallow feature fusion makes it difficult to extract the contour information and semantic information provided by the deep modality, making the method's ability to identify targets in complex backgrounds low. How to fully utilize and fuse the complementary features provided by visible light and deep modalities is an urgent problem to be solved in RGBD, which will help improve the tracking quality of RGBD. Summary of the invention

[0004] The problem to be solved by the present invention is: in view of the shortcomings of the existing tracking technology, an RGBD target tracking method with deep and shallow feature fusion is proposed to improve the real-time and robustness of the tracker model.

[0005] The technical solution of the present invention is: an RGBD target tracking method based on deep and shallow feature fusion, which realizes RGBD target tracking through a target tracking model, comprising the following steps:

[0006] Step S1: construct a data set for model training and testing;

[0007] Step S2: Construct a target tracking model, including a backbone network, deep and shallow feature fusion, and tracking discrimination modules, as follows:

[0008] Step S2.1: Using the spatiotemporally registered visual image and depth image as input to the tracking model;

[0009] Step S2.2: construct a two-stream backbone network with shared weights, where the two streams refer to the visible light stream corresponding to the visual image and the depth stream corresponding to the depth image, and use the ResNet-50 network convolution as the backbone network to extract shallow features and deep features of the visual image and the depth image respectively;

[0010] Step S2.3: Fuse the shallow and deep features, and construct an attention-based shallow feature enhancement module for the shallow features. After adding the visual shallow features and deep shallow features extracted in step S2.2, they are enhanced through channel attention and spatial attention to obtain shallow fusion features; construct a feature redistribution module to dynamically redistribute the shallow fusion features, visual deep features, and deep deep features, and fuse them into aggregate features;

[0011] Step S2.4: Construct a tracking and discrimination module, including a convolutional layer, a precise pooling layer, and a fully connected layer, to determine the confidence of the target and output the target position and score;

[0012] Step S3: Offline model training, load the pre-trained model ResNet50, train the model built in step S2, use the AdamW algorithm to train the model until the loss converges, and obtain the trained model parameters;

[0013] Step S4: Online training and tracking, as follows:

[0014] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, extract the depth image and visual image features, and fuse the deep and shallow features to obtain the fused features;

[0015] Step S4.2: Use the tracking and discrimination module to determine the target position and perform online tracking. First, use the target of the first frame to build an initial set after data enhancement, which is used to optimize the discrimination model obtained through offline training. As the tracking sequence progresses, new tracking results are added to the initial set to update the parameters of the discrimination model.

[0016] Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire RGBD video sequence.

[0017] As a further technical solution of the present invention, in step S2.3, the attention-based shallow feature enhancement module includes channel attention and spatial attention. The channel attention is divided into two parts: compression and excitation. The compression part compresses the global spatial information. The compression part performs maximum pooling and average pooling on the features, and then performs feature learning in the channel dimension. The excitation part assigns weights to each channel, uses a multi-layer perceptron, and then performs element-wise summation of the two output feature vectors and performs Sigmoid activation before output. The channel attention formula is as follows:

[0018] F channel =σ(MLP(AvgPool(F ri ))+MLP(Max Pool(Fo ri )))

[0019] In the above formula, F channel represents the feature after channel attention processing, σ represents the Sigmoid activation function, MLP is a multi-layer perceptron, AvgPool and MaxPool represent full average pooling and maximum pooling respectively, Fo ri is the added visual shallow features and deep shallow features;

[0020] Spatial attention is used to strengthen the meaningful local area in the feature, so that the network focuses on the most informative part. The features output by the channel attention are sequentially subjected to maximum pooling and average pooling operations, and then concatenated and multiplied by a valid feature descriptor. After convolution and activation operations, the result is output. The formula is as follows:

[0021] F spatial =σ(f 7×7 ([AvgPool(F channel );MaxPool(F channel )]))

[0022] In the above formula, F spatial Represents the features after spatial attention processing, and f is a 7*7 convolution kernel.

[0023] As a further improvement of the present invention, the feature redistribution module of step S2.3 is specifically as follows: first, the shallow fusion features, the visual deep features and the deep deep features are added and then enter the global average pooling operation, and the global feature vector is obtained by average pooling and full connection, and then the global feature vector is divided into three parts, namely the visual branch, the deep branch and the shallow branch. Each branch is fully connected and activated once, and a weight is output for each channel of the branch. The weight is used to describe the importance of each channel of the three features. The weight is element-wise multiplied with the input three-channel feature to obtain the fused deep and shallow features, that is, the fused features. The feature redistribution is as follows:

[0024]

[0025] FC stands for fully connected layer, GP stands for global pooling, F R 、F D and F G Represent visual deep features, deep deep features and shallow fusion features respectively. represents element plus;

[0026] The global redistribution result F gl o bal Enter each branch for redistribution and combine with the original feature element F i take:

[0027]

[0028] In the above formula, represents the corresponding features after processing, Represents element-wise multiplication.

[0029] The beneficial effect of the present invention is to provide an efficient and accurate RGBD marker tracking technology, which has the following advantages over the prior art.

[0030] (1) The present invention fully combines the depth image information, makes up for the defects of the visible light modality, and can cope with target tracking tasks in complex indoor scenes.

[0031] (2) The present invention proposes an attention-based shallow feature extraction module, which extracts the detailed texture information of shallow features and the contour semantic information of deep features through the attention mechanism, solving the problem that most current RGB-D tracking methods do not fully exploit bimodal features.

[0032] (3) The present invention proposes a feature reallocation module to reallocate the deep and shallow features of RGBD to ensure effective feature fusion, thereby solving the problems of low fusion efficiency and incomplete fusion in existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is the overall flow chart of the present invention.

[0034] Figure 2 Schematic diagram of the attention-based shallow feature extraction module of the present invention.

[0035] Figure 3 It is a schematic diagram of the feature reallocation module of the present invention.

[0036] Figure 4 It is the overall tracking index of the present invention on the Depth-Track dataset. DETAILED DESCRIPTION

[0037] The technical solution of the present invention is further described in detail below in conjunction with the corresponding drawings:

[0038] This paper proposes an RGBD target tracking method based on deep and shallow feature fusion, constructs a deep and shallow feature fusion algorithm to learn robust feature expression, extracts the detailed texture information of shallow features and the contour semantic information of deep features through the attention mechanism, and achieves full fusion through the feature redistribution module, and uses the discriminant model to find the target position and score to achieve target tracking. Figure 1 As shown, the present invention includes the following steps.

[0039] Step S1: Construct a dataset and divide the corresponding data from the public RGBD object tracking dataset for model training and testing.

[0040] Step S2: Construct a target tracking model, including a feature extraction module, an attention-based shallow feature enhancement module, a feature reallocation module and a target discrimination module, as follows.

[0041] Step S2.1: Use the spatiotemporally registered visible light and depth images as input to the tracking model.

[0042] Step S2.2: Construct a dual-stream feature extraction module with shared weights, where the dual streams refer to the visible light stream corresponding to the visual image and the depth stream corresponding to the depth image. ResNet-50 network convolution is used as the backbone network to extract shallow features and deep features of the visual image and depth image respectively.

[0043] Step S2.3: Based on the deep and shallow feature fusion algorithm, the deep and shallow features are fused. The present invention constructs an attention-based shallow feature enhancement module for the shallow features, and then fuses the deep features of the visual image and the depth image through feature redistribution based on the fusion of the shallow features.

[0044] For the shallow features extracted in step S2.2, an attention-based shallow feature enhancement module is constructed, and the attention mechanism is used to enhance the shallow features of the deep image and the visual image, so as to make full use of the rich texture detail features in the shallow features. The present invention uses an attention mechanism to achieve shallow feature enhancement, which includes a channel attention and a spatial attention. The purpose of the former is to enhance the channel information that needs to be paid attention to in the feature, while the purpose of the latter is to enhance the part that is useful for tracking in the feature space distribution. Channel attention is divided into two parts: compression and excitation. Figure 2As shown in (a), the purpose of the compression part is to compress the global spatial information. The compression part performs feature learning in the channel dimension by max pooling and average pooling. In the excitation part, weights are assigned to each channel, and a multi-layer perceptron is used. The two feature vectors are then summed and output after Sigmoid activation. The channel attention formula is as follows:

[0045] F channel =σ(MLP(AvgPool(F ri ))+MLP(Max Pool(Fo ri )))(1)

[0046] In the above formula, F channel represents the features after channel attention processing, σ represents the Sigmoid activation function, MLP is a multi-layer perceptron, AvgPool and MaxPool represent full average pooling and maximum pooling respectively, Fo ri It is the added visual shallow features and deep shallow features.

[0047] The role of spatial attention is to strengthen the meaningful local regions in the features, such as Figure 2 As shown in (b), the network focuses on the most informative part, which is a supplement to channel attention. First, the features output by channel attention are sequentially subjected to maximum pooling and average pooling operations, and then concatenated and multiplied by a valid feature descriptor. After convolution and activation operations, the result is output. The formula is as follows:

[0048] F spatial =σ(f 7×7 ([AvgPool(F channel );MaxPool(F channel )]))(2)

[0049] In the above formula, F spatial represents the features after channel attention processing, and f is a 7*7 convolution kernel.

[0050] Construct a feature redistribution module, such as Figure 3As shown. Shallow features and deep features are dynamically reallocated to fuse shallow and deep features to obtain fused features. The input of this module includes deep visual features, deep depth features, and shallow fusion features. First, the three features are added together and then enter the global average pooling operation. This operation obtains a global feature vector through average pooling and full connection to comprehensively consider the three aspects of features. The feature vector is then divided into three parts, namely the visual branch, the deep branch, and the shallow branch. Each branch is fully connected and activated once, and a weight is output for each channel of the branch. The weight describes the importance of each channel of the three features. The weight is element-wise multiplied with the input three-channel feature to obtain the adjusted deep and shallow features. This feature not only comprehensively considers the required feature information globally, but also includes contour and texture features. It is of great help to subsequent tracking tasks.

[0051] The formula for global redistribution is as follows:

[0052]

[0053] In the above formula, FC represents the fully connected layer, GP represents the global pooling, and F R 、F D and F G They represent deep visual features, deep depth features, and shallow fusion features respectively. Represents element plus.

[0054] After global redistribution, the global redistribution result is sent to each branch redistribution weight and multiplied by the original feature element. The formula is as follows:

[0055]

[0056] In the above formula, represents the corresponding feature after processing, σ(FC i (F gl o bal )) is the weight of each branch, Represents element-wise multiplication, resulting in The sum of them is the fusion feature.

[0057] Step S2.4: Construct a tracking discriminant module, which is the same as the discriminant module used by DIMP, including convolutional layers, precise pooling layers, and fully connected layers to judge the confidence of the target. For target tracking DIMP, please refer to "Learning Discriminative Model Prediction for Tracking" (ICCV2019).

[0058] Step S3: Offline model training, load the pre-trained model ResNet50, repeat step S2 to train the constructed model, use the AdamW algorithm to train the model until the loss converges, and obtain the trained model parameters.

[0059] Step S4: Online training and tracking, as follows:

[0060] Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, extract the features of the depth image and the visible light image, and fuse the features using the deep and shallow feature fusion algorithm;

[0061] Step S4.2: Use the discriminant model to determine the target position and perform online tracking. First, use the target in the first frame to build an initial set after data enhancement, which is used to optimize the discriminant model trained offline. As the tracking sequence progresses, new tracking results are added to the initial set to update the parameters of the discriminant model.

[0062] Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire RGBD sequence.

[0063] In order to verify the effectiveness of the tracking model, a comparative experiment is conducted on the widely used RGBD dataset. The experimental results are as follows: Figure 4 Compared with the most advanced multimodal tracking methods, the accuracy and success rate of the present invention on the DepthTrack dataset are significantly improved compared with the tracking network SPT and the advanced OSTrack method. In addition, the running rate of the present invention is approximately 60 frames per second, which meets the real-time requirements.

Claims

1. An RGBD target tracking method based on deep and shallow feature fusion, characterized by RGBD target tracking is implemented through a target tracking model, including the following steps: Step S1: construct a data set for model training and testing; Step S2: Construct a target tracking model, including a backbone network, deep and shallow feature fusion, and tracking discrimination modules, as follows: Step S2.1: Using the spatiotemporally registered visual image and depth image as input to the tracking model; Step S2.2: construct a two-stream backbone network with shared weights, where the two streams refer to the visible light stream corresponding to the visual image and the depth stream corresponding to the depth image, and use the ResNet-50 network convolution as the backbone network to extract shallow features and deep features of the visual image and the depth image respectively; Step S2.3: Fuse the shallow and deep features, and construct an attention-based shallow feature enhancement module for the shallow features. After adding the visual shallow features and deep shallow features extracted in step S2.2, they are enhanced through channel attention and spatial attention to obtain shallow fusion features; construct a feature redistribution module to dynamically redistribute the shallow fusion features, visual deep features, and deep deep features, and fuse them into aggregate features; Step S2.4: Construct a tracking and discrimination module, including a convolutional layer, a precise pooling layer, and a fully connected layer, to determine the confidence of the target and output the target position and score; Step S3: Offline model training, load the pre-trained model ResNet50, train the model built in step S2, use the AdamW algorithm to train the model until the loss converges, and obtain the trained model parameters; Step S4: Online training and tracking, as follows: Step S4.1: Obtain the label of the first frame of the video sequence as the initial tracking target, extract the depth image and visual image features, and fuse the deep and shallow features to obtain the fused features; Step S4.2: Use the tracking and discrimination module to determine the target position and perform online tracking. First, use the target of the first frame to build an initial set after data enhancement, which is used to optimize the discrimination model obtained through offline training. As the tracking sequence progresses, new tracking results are added to the initial set to update the parameters of the discrimination model. Step S4.3: Repeat steps S4.1-S4.2 to gradually calculate the target tracking result of each frame, thereby achieving target tracking of the entire RGBD video sequence.

2. The RGBD target tracking method based on deep and shallow feature fusion according to claim 1 is characterized in that In step S2.3, the attention-based shallow feature enhancement module includes channel attention and spatial attention. The channel attention is divided into two parts: compression and excitation. The compression part compresses the global spatial information. The compression part performs maximum pooling and average pooling on the features, and then performs feature learning in the channel dimension. The excitation part assigns weights to each channel, uses a multi-layer perceptron, and then sums the two output feature vectors and performs Sigmoid activation before output. The channel attention formula is as follows: F channel =σ(MLP(AvgPool(F ori ))+MLP(Max Pool(F ori ))) In the above formula, F channel represents the features after channel attention processing, σ represents the Sigmoid activation function, MLP is a multi-layer perceptron, AvgPool and MaxPool represent full average pooling and maximum pooling respectively, F ori is the added visual shallow features and deep shallow features; Spatial attention is used to strengthen the meaningful local area in the feature, so that the network focuses on the most informative part. The features output by the channel attention are sequentially subjected to maximum pooling and average pooling operations, and then concatenated and multiplied by a valid feature descriptor. After convolution and activation operations, the result is output. The formula is as follows: F spatial =σ(f 7×7 ([AvgPool(F channel );MaxPool(F channel )])) In the above formula, F spatial Represents the features after spatial attention processing, and f is a 7*7 convolution kernel.

3. The RGBD target tracking method based on deep and shallow feature fusion according to claim 1 is characterized in that The feature redistribution module of step S2.3 is specifically as follows: first, the shallow fusion features, the visual deep features and the deep deep features are added and then enter the global average pooling operation, and the global feature vector is obtained through average pooling and full connection, that is, global redistribution, and then the global redistribution feature vector is input into three branches respectively, namely the visual redistribution branch, the deep redistribution branch and the shallow redistribution branch, and each branch is fully connected and activated once, and a weight is output for each channel of the branch. The weight is used to describe the importance of each channel of the three features, and the weight is element-wise multiplied with the input three-channel features to obtain the redistributed shallow and deep features; The feature redistribution is as follows: FC stands for fully connected layer, GP stands for global pooling, F R 、F D and F G Represent visual deep features, deep deep features and shallow fusion features respectively. represents element plus; The global redistribution result F global Enter each branch for redistribution and combine with the original feature element F i take: In the above formula, represents the corresponding features after processing, Represents element-wise multiplication, resulting in The sum of them is the fusion feature.

Citation Information

Patent Citations

  • RGB-D feature target tracking method based on twin network

    CN112785624A

  • Semantic segmentation method for RGB-D bimodal feature fusion

    CN114693929A

  • Real-time RGBT target tracking method based on multi-modal interaction and multi-stage optimization

    CN115170605A