Domain increment image-text retrieval method for evaluating distillation based on Banzhaf alliance

Abstract

The invention discloses a domain increment image-text retrieval method based on Banzhaf alliance assessment distillation, which is characterized in that image-text retrieval is modeled into a multi-element cooperative game process through Banzhaf alliance assessment, and more efficient cross-modal comparison is realized and the model accuracy is improved by calculating the contribution of the alliance of an image token and a text token to retrieval accuracy. The method comprises the following steps: utilizing comparative learning distillation and Banzhaf alliance assessment distillation, retaining structured semantic information, and respectively constraining cross-task structure knowledge from sample and token-level similarity to reduce disastrous forgetting. According to the method, by adding token-level dynamic semantic prompts, semantic information differences between modes are compensated in a self-adaptive mode, and task knowledge is reserved by reserving domain-independent semantic prompts. The method mainly solves the technical problems of missing of structured semantic information of different modal features and inter-task knowledge deviation during cross-modal knowledge migration.

Images