The invention relates to the technical field of cable braking, and discloses a magnetorheological cable drive braking system and a control method thereof. The magnetorheological cable drive braking system comprises a base, a controller, a motor and a braking mechanism; the brake mechanism comprises a case, and a movable piece, a reciprocating screw and a guide rod are arranged in an inner cavity of the case; the motor is in driving connection with the reciprocating screw, and the reciprocating screw is coaxially and fixedly connected with a winding drum; a rope is wound on the winding drum; anelectromagnetic coil is wound on the side wall of the case, and the inner cavity of the case is filled with magnetorheological fluid; and the controller is in communication connection with a speed measuring mechanism, and is electrically connected with the electromagnetic coil. According to the magnetorheological cable drive braking system, the movable piece is driven by the reciprocating screw to reciprocate in the case, and when the falling speed is too high, the electromagnetic coil is switched on through the controller to generate a magnetic field, so that the viscosity of magnetorheological fluid is increased, the resistance of the movable piece is increased, and the speeds of the reciprocating screw and the winding drum are quickly reduced until the reciprocating screw and the winding drum stop or are rotated at constant speed; therefore, the loaded cable end stops falling or slides down at constant speed, safe braking of the cable is achieved, and the motor is protected.