The invention discloses a rail fastener anomaly detection system based on binocular vision and laser speckle, which is composed of a laser speckle projector, a camera, a laser displacement sensor, a wheel encoder, an RFID detector and an industrial computer. The speckle projector and the camera constitute a binocular stereo vision speckle imaging system, which is used for three-dimensional measurement of the fastener; the laser displacement sensor is used to measure the height of the fastener installation support, which is used to control the camera imaging, wheel encoder and The RFID detector is used for mileage counting, the industrial computer completes the image acquisition and processing, and the method based on binocular vision stereo matching is used to detect whether the fastener is abnormal. The device uses a laser speckle light source to artificially increase the mottled texture on the surface of the fastener, which can improve the matching accuracy of image blocks and accurately reconstruct the three-dimensional shape of the fastener, thereby effectively detecting various abnormalities of different types of fasteners.