Human-object interaction model training method and system, medium and detection method

Abstract

The application provides a human-object interaction model training method, system, medium and detection method. The training method comprises: obtaining visual interaction features; constructing English interaction descriptions; translating into non-English interaction descriptions; obtaining English text embedding and multilingual text embedding; obtaining multilingual enhanced interaction text representation; aligning, querying a bounding box and an interaction category; and jointly optimizing parameters of the trainable projection matrix, the lightweight adapter, the visual branch and the human-object interaction decoder. The training method provided by the application fully utilizes the complementary semantic information of the same interaction in different languages, is designed for human-object interaction, a kind of fine-grained local relationship, can utilize the difference in multilingual semantics to alleviate the problem of polysemy of verbs and cross-cultural semantic shift, and significantly improves the accuracy and generalization ability of human-object interaction detection in a large-scale open vocabulary scene.

Images