Object perception image fusion method for multi-modal target tracking

Abstract

The invention discloses an object perception image fusion method for multi-modal target tracking, and the method comprises the following steps that: a self-adaptive fusion image is obtained, the saliency detection of the image is carried out, the outputs of different layers in a network are cascaded and connected, an RGB image and a thermal modal image are inputted into two channels, each channel comprises three aggregation modules, and reconstruct a fusion image is ed by judging the gray value, pixel intensity, similarity measurement and consistency loss of the image according to the connection deep features and adaptively guiding the network; the feature combination module combines the features extracted from the sample and the search image by using depth cross-correlation operation to generate corresponding similarity features for subsequent target positioning, and then performs classification and regression to perform target positioning; a regression network without an anchor frame is used for tracking training, all pixels in a bounding box in groundtruth are used as training samples, weak prediction can be corrected to a certain extent, and a correct position is corrected; and the sampling position is changed by adopting deformable convolution, so that the sampling position is aligned with the prediction bounding box, and the classification confidence corresponds to the target object, so that the classification confidence is more reliable.

Description

technical field [0001] The invention relates to the field of target tracking, in particular to an object-aware image fusion method for multi-modal target tracking. Background technique [0002] Target tracking is widely used in video surveillance, automatic driving and robotics. It has always been the focus of computer vision research. Its definition is as follows: given the size and position of the target in the initial test frame of a video sequence, predict the target in the subsequent frame size and position. The main challenge of tracking is that the target object may have severe occlusion, large deformation and illumination changes. [0003] In recent years, it has been found that thermal infrared can provide more stable signals, and the promotion of thermal infrared cameras has promoted the development of several fields, such as object segmentation, person Re-ID and pedestrian detection, so multimodal (RGBT) tracking has caused More research. Multimodal tracking ca...

AI Technical Summary

Problems solved by technology

This way weights the two modalities equally, however one modality may actually have a better value than the other; on the other hand, by extending the single-modal tracker to a multi-modal tracker, the designed Some benchmark RGBT trackers, done by concatenating features from RGB and thermal modalities directly into vectors, which are then fed into the tracker, these weights are used to fuse complementary features from RGB and thermal images so that they can be effectively properties of deployment-specific modalities, but ignores the potential value of modality sharing and object information, which is crucial for multimodal tracking

[0005] These methods rely on hand-crafted features or single-structure adapter deep networks for object localization, which are difficult to deal with the challenges of appearance changes due to object deformation, sudden motion, background clutter, and occlusion, etc.

Images