The invention discloses a three-dimensional roughness measurement system based on zooming technology for an object surface and a method of the system. The measurement system comprises following hardware: an L-type z-directional support rack, a stepping motor, a ball screw, a pair of lead screw nut seats, an upper limiting switch, a lower limiting switch, a sliding linkage block, a limiting switch baffle plate, an optical probe formed by an optical lens and an image sensor, a scale raster and a raster reading head. In case that the optical probe and a to-be-measured object are in different z-directional distances, image sequences of out-of-focus, focusing and out-of-focus are acquired. Through software programming, a digital image related method and a focusing evaluation function are mainly used for processing the acquired image sequences, and a roughness calculation formula is used for obtaining a three-dimensional roughness evaluation parameter of the object surface. According to the invention, the three-dimensional surface roughness of complex geometric structures with various shapes, different materials and different surface reflectivity can be measured and the measurement precision can reach micro-nano level.