A dual-modal behavior recognition method based on a base model and competitive selection

Abstract

The application relates to a dual-mode behavior recognition method based on a basic model and competitive selection, which comprises the following steps: for infrared picture sequences and visible light picture sequences of a user in the same scene, adopting a competitive mechanism to dynamically generate mask information of the infrared pictures and the visible light pictures, and recording positions of the images being masked as mask marks and positions of the images not being masked as visible marks; according to the positions of the images being masked and not being masked, inputting the infrared picture sequences and the visible light picture sequences and corresponding mask information into infrared encoders and visible light encoders respectively, and extracting corresponding modal features; feeding the modal features obtained after encoding into a hierarchical mask fusion module, combining the positions of the images being masked and not being masked to calculate fusion features, and feeding the fusion features into a classifier to calculate a behavior category of the user in the corresponding scene; and the application can improve the efficiency and accuracy of a behavior recognition task.

Images