The invention relates to a passenger counting system and a passenger counting method, which belong to the technical field of statistics for the number of passengers. Firstly, the passenger counting system comprises a passenger video acquisition module, a video data processing module, a video data analysis module, a passenger counting module and a boarding direction recognition module, which are connected in sequence. Secondly, the passenger counting method includes the steps: environment deployment, passenger video acquisition, video data processing, video data analysis, passenger counting and direction analysis. Under coordination of the modules, the shortage that existing computer vision and pattern recognition techniques are easy to err in case of crowdedness or poor light is overcome, and the system and the method are capable of accurately counting passengers in real time all day long under any natural conditions.