This invention provides a method and
system for detecting abnormal driver states based on multimodal
information fusion. The method includes: Step 1, acquiring real-time facial video images of the driver using an in-vehicle camera; and simultaneously acquiring the driver'
s voice signal using an in-vehicle
microphone; Step 2, detecting driver fatigue abnormal states based on the facial video images acquired in Step 1 to obtain a fatigue probability; Step 3, preprocessing the voice
signal, extracting Mel-frequency cepstral coefficients as voice features, and using a time-aware bidirectional multi-scale network for voice
emotion recognition to obtain an
anger probability; Step 4, adaptively weighting and fusing the fatigue probability and the
anger probability to calculate a fusion risk value. This invention achieves non-contact, highly robust driver abnormal state detection, with a lightweight model and fast detection speed, making it suitable for deployment on in-vehicle edge devices.