A fatigue driving detection method and system based on multi-modal feature fusion

Abstract

The application discloses a kind of fatigue driving detection method and system based on multimodal feature fusion, the method is by DROZY dataset as example, the invention method is described.By pre-processing data set, extract time one-to-one corresponding image data and PSG time series data, image and time series data are used for subsequent system input.By extracting the feature information of shallow, medium and deep network of two kinds of modal data, the feature information is optimized by residual convolution attention module after the dimension of three feature blocks of two kinds of modal is adjusted to be consistent.Then, the weight information of different layers of single modal data is obtained by weight calculation method, and then the feature vector extraction result of image data and time series data is obtained by multi-scale weight fusion of weight information and original feature block.The feature vector data of two modes is fused, and the classification result is output by the classifier through the fusion feature, to realize fatigue driving detection.

Images