The invention provides a traffic behavior analysis system, which comprises a video acquisition module, a vehicle detection module, a vehicle tracking module, a vehicle trajectory extraction module, a vehicle behavior analysis module, a license plate recognition module, a vehicle information description module and a communication module. The traffic behavior analysis system acquires a traffic video image from a traffic monitoring scene, detects a vehicle target in the acquired traffic video image, performing movement tracking on the vehicle target to obtain information such as the trajectory and the speed of the vehicle target, analyzes behaviors of the vehicle target according to the obtained information, judges whether the vehicle target has behaviors such as illegal lane changing, speed limit exceeding, wrong-direction running and illegal turning, acquires license plate information of the regulation breaking vehicle in real time and sends regulation breaking video and vehicle information of the vehicle target to a management center in real time. The traffic behavior analysis system is high in real-time performance, high in recognition accuracy and capable of effectively performing traffic monitoring round the clock.