A visual language combination understanding method based on cross-modal combination concept mask modeling

Abstract

The application discloses a visual language combination understanding method based on cross-modal combination concept mask modeling, comprising the following steps: 1, using a text scene graph parser and an open set target detection model, respectively positioning combination concept word units and spatial regions in the text and the image, and generating corresponding masks; 2, constructing a symmetric cross-modal mask modeling path, using the complete context information of another mode to guide the mask mode to reconstruct in the word table space or the pixel space, and integrating the global class feature into the local mask word unit in the feature processing stage to enhance the semantic representation density; 3, through the introduction of the cross-modal alignment loss and the intra-modal regularization loss enhanced by the mask feature, combined with the mask modeling loss, joint training is carried out. The application deeply excavates the potential combination concept alignment signal in the text and the image, significantly alleviates the “bag of words effect” of the visual language model, and improves the perception and reasoning ability of the model to the object relationship and attribute binding.

Images