The invention relates to a defect detection method of an irregular shape metal working surface based on depth learning, By preprocessing the collected surface images of metal machining, image enhancement, YOLOv3-based network architecture, Deep learning network suitable for defect detection is built. The samples are labeled manually and the deep learning network is trained with labeled samples toobtain defect detection model. Finally, the defect detection model is used to detect the surface image of metal machining, and the defect detection results are obtained. Compared with the prior art, the network structure in the invention has good adaptability to the detection of small objects and small targets, and unifies the four basic steps of the candidate region generation, the feature extraction, the classification and the position refinement of the target detection into the same depth network framework, thereby improving the running speed and making the detection more accurate.