The invention discloses a video-based weather phenomenon recognition method. The method achieves classification recognition of common weather phenomena such as sun, cloud, rain, snow and fog. The method comprises the steps of training an off-line classifier, wherein an image sequence is sampled for a given training video; on the one hand, video characteristics of the image sequence are extracted; on the other hand, key frame images and image characteristics of the key frame images are extracted from the image sequence, the AdaBoost is adopted for conducting learning and training on the extracted video characteristics, the extracted image characteristics and manual annotations to obtain the classifier; recognizing the weather phenomena in an online mode, wherein a plurality of sets of image sequences are sampled for a testing video, video characteristics and image characteristics of each set of image sequences are extracted, the characteristics are sent into the classifier for classification to obtain a corresponding recognition result, then decision fusion is carried out in a voting mode, and the voting result is used as the weather phenomenon recognition result of the testing video.