Heterogeneous multi-modal remote sensing target detection method based on language hub pre-training

Abstract

The application discloses a heterogeneous multi-modal remote sensing target detection method based on language pivot pre-training, constructs a pre-training framework containing a visual encoder and a large language model, uses a large language model as a semantic pivot through a concept sharing instruction alignment mechanism, maps visual features of heterogeneous sensors such as RGB, SAR and infrared to a shared language semantic space, and realizes implicit cross-modal alignment; meanwhile, a hierarchical visual-semantic annealing mechanism is introduced, multi-scale intermediate layer features of the visual encoder are gradually aggregated through time-dependent dynamic coefficients, and the granularity mismatch problem between language high-level semantics and detection task fine-grained features is solved. In the pre-training stage, the application decouples modal alignment and downstream task learning, and significantly improves the training stability, convergence speed and detection precision of the model on heterogeneous remote sensing data.

Images