A deep learning-based sheet metal part surface defect detection method and system

Abstract

This invention discloses a deep learning-based method and system for detecting surface defects in sheet metal parts, relating to the field of industrial defect detection technology. The method includes: acquiring and preprocessing a surface image of the component to be tested; inputting the preprocessed image into a semantic segmentation model and a target detection model for parallel detection, with the semantic segmentation model outputting a defect segmentation mask, defect area, and location information, and the target detection model outputting a defect bounding box, category, and confidence information; fusing the detection results of the two models and determining whether the sheet metal part has defects based on preset acceptance criteria; if defects exist, capturing the defective component to be tested and placing it in the defect placement area; this method employs a multi-model collaborative detection architecture to improve detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of industrial defect detection technology, and in particular to a method and system for detecting surface defects in sheet metal parts based on deep learning. Background Technology

[0002] Sheet metal parts, as crucial components in modern industrial manufacturing, are widely used in the automotive, aerospace, and electronic equipment industries. During processing, various defects inevitably arise on the surface of sheet metal parts, such as scratches, blemishes, cracks, dents, inclusions, and oxide scale. These defects not only affect the product's appearance quality but can also lead to serious problems like stress concentration and fatigue failure. Traditional defect detection methods primarily rely on manual visual inspection and traditional machine vision-based methods, which suffer from limitations such as low efficiency, poor consistency, limited generalization ability, and insufficient robustness, making it difficult to meet the high-efficiency and high-precision requirements of modern industrial production.

[0003] In recent years, deep learning technology has provided a new technical path for industrial defect detection. However, existing deep learning-based defect detection systems still have the following shortcomings: First, most systems only use a single detection model, which is prone to missed detections or false detections when faced with complex defect types; second, existing solutions focus on the detection algorithm itself and lack efficient collaborative control with execution mechanisms such as loading, positioning, and sorting, making it difficult to form a closed-loop automated detection unit, resulting in the detection results not being promptly converted into actual quality control actions; finally, the automation level of the system is insufficient, and the entire process from sheet metal loading to defective product sorting still requires manual intervention, making it impossible to achieve truly unmanned continuous detection. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, this invention proposes a method and system for detecting surface defects in sheet metal parts based on deep learning, and adopts a multi-model collaborative detection architecture to improve detection accuracy.

[0005] This invention proposes a deep learning-based method for detecting surface defects in sheet metal parts, comprising: Acquire and preprocess the surface image of the component under test; The preprocessed image is input into the semantic segmentation model and the object detection model for parallel detection. The semantic segmentation model outputs the defect segmentation mask, defect area, and location information, while the object detection model outputs the defect bounding box, category, and confidence information. By combining the detection results of the two models, the presence of defects in sheet metal parts is determined based on preset qualification indicators; If a defect exists, grab the defective component to be tested and place it in the defect placement area.

[0006] Furthermore, the preset qualification indicators include the number of defects, the defect confidence threshold, and the defect area threshold. When the number of defects is zero, the confidence of all defects is lower than the threshold, or the area of ​​all defects is less than the threshold, it is judged as qualified.

[0007] Furthermore, the fusion strategy for combining the detection results of the two models is as follows: Calculate the intersection-union ratio (IUU) between the minimum bounding rectangle of the defect segmentation mask output by the semantic segmentation model and the defect bounding box output by the object detection model; If the crossover ratio is greater than the preset threshold, it is determined to be the same defect. The weighted average of the confidence scores of the two models is taken as the final confidence score, and the defect category is based on the output of the target detection model. If the crossover ratio is less than or equal to a preset threshold, it is considered an independent defect and retained.

[0008] Furthermore, the defect types include plaques, cracks, dents, inclusions, scratches, and oxide scale.

[0009] Furthermore, it also includes a data management step: storing the detection results in a database, the detection results including detection time, detection image, defect type, defect location and defect quantity.

[0010] A deep learning-based surface defect detection system for sheet metal parts includes: Conveying and supporting components are used to carry and transport the component under test. An image acquisition component is disposed above the conveying and supporting component and is used to acquire images of the surface of the component under test; A control component, electrically connected to the image acquisition component, is used to receive image data and execute a deep learning-based defect detection algorithm. The defect detection algorithm adopts a dual-model parallel detection architecture, including a semantic segmentation model and an object detection model. The semantic segmentation model is used for pixel-level defect segmentation, and the object detection model is used for defect localization and classification. The detection results of the two models are fused for defect evaluation. The material gripping component is electrically connected to the control component and is used to grip the defective component under test according to the detection results; A material storage component is located within the working range of the material gripping component and is used to store the gripped test component.

[0011] Furthermore, the conveying and support assembly includes feeding rollers, a conveyor line bracket, and feet. Multiple feeding rollers are arranged in parallel in sequence and are rotatably connected to the conveyor line bracket. The conveyor line bracket is connected to the feet via columns.

[0012] Furthermore, the image acquisition component (3) includes a light source mounting adapter, a light source, an industrial camera, and a light source and camera bracket in the form of a gantry structure, wherein the light source and camera bracket is positioned above the conveying and support component; The light source mounting adapter is fixed to the light source and camera bracket. The light source and the light source mounting adapter are rotatably connected. The industrial camera is mounted on the light source and camera bracket, with the lens facing the surface of the component being tested.

[0013] Furthermore, four light sources are distributed around the industrial camera to provide uniform and stable illumination for the camera.

[0014] Furthermore, the material gripping assembly includes a multi-degree-of-freedom robotic arm and an end effector connected to the output end of the multi-degree-of-freedom robotic arm. The end effector is a vacuum suction cup or a mechanical gripper, used to grip the component under test.

[0015] The advantages of the deep learning-based sheet metal surface defect detection method and system provided by this invention are as follows: It employs a multi-model collaborative detection architecture to improve detection accuracy and achieves fully automated closed-loop control from material loading and detection to sorting. This effectively replaces manual inspection, significantly improving detection efficiency and product quality consistency, thus solving technical problems such as low detection efficiency, unstable accuracy, and insufficient automation in existing technologies, thereby meeting the actual needs of modern industrial production. Furthermore, the deep learning-based detection algorithm possesses strong adaptive learning and generalization capabilities, enabling it to adapt to the detection needs of different types and sizes of sheet metal parts, and exhibits strong robustness to environmental factors such as changes in lighting and surface texture. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of the present invention; Figure 2 This is a structural schematic diagram of the conveying and support components; Figure 3 This is a schematic diagram of the image acquisition component. Figure 4 This is a flowchart of a defect detection method; Figure 5 A flowchart illustrating parallel detection for semantic segmentation and object detection models. Detailed Implementation

[0017] The technical solution of the present invention will now be described in detail through specific embodiments. Many specific details are set forth in the following description to provide a thorough understanding of the invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0018] like Figures 1 to 5 As shown, the present invention proposes a deep learning-based method for detecting surface defects in sheet metal parts, comprising: Acquire and preprocess the surface image of the component under test; The preprocessed image is input into the semantic segmentation model and the object detection model for parallel detection. The semantic segmentation model outputs the defect segmentation mask, defect area, and location information, while the object detection model outputs the defect bounding box, category, and confidence information. By combining the detection results of the two models, the presence of defects in sheet metal parts is determined based on preset qualification indicators; If a defect exists, grab the defective component to be tested and place it in the defect placement area.

[0019] This embodiment develops a sheet metal surface defect detection method with high integration, high automation, high detection accuracy, and fast detection speed. The method should adopt a multi-model collaborative detection architecture to improve detection accuracy and realize full-process automated closed-loop control from material loading and detection to sorting, so as to solve the technical problems of low detection efficiency, unstable accuracy, and insufficient automation in the existing technology, thereby meeting the actual needs of modern industrial production.

[0020] This embodiment proposes a sheet metal surface defect detection system based on deep learning. The overall structure of the system includes a conveying and support component 1, a test component 2, an image acquisition component 3, a control component 4, a material gripping component 5, and a material storage component 6.

[0021] The conveying and support assembly 1 is used to carry and convey the sheet metal parts to be inspected. For example... Figure 2 As shown, the conveying and support assembly 1 includes a feeding roller 1.1, a conveyor line bracket 1.2, and feet 1.3. The feeding roller 1.1 employs a roller conveyor structure to transport sheet metal parts from the loading position to the inspection area. Multiple feeding rollers 1.1 are arranged in parallel and rotatably connected to the conveyor line bracket 1.2. The conveyor line bracket 1.2 is connected to the feet 1.3 via columns. The conveyor line bracket 1.2 uses an aluminum profile or steel frame structure to provide stable support for the conveying system and ensure the smoothness of the conveying process. The feet 1.3 support the entire conveying and support assembly 1 and fix it to the ground, ensuring the stability of the system.

[0022] The component under test 2 is the sheet metal part to be tested, which can be sheet metal products of various shapes and sizes, including but not limited to automotive sheet metal parts, electrical housings, chassis and cabinets, etc.

[0023] like Figure 3As shown, the image acquisition component 3 is positioned above the conveying and support component 1 and is used to acquire surface images of the component under test 2. The image acquisition component 3 includes a light source mounting adapter 3.1, a light source 3.2, an industrial camera 3.3, and a light source and camera bracket 3.4. The light source and camera bracket 3.4 has a gantry structure and spans above the conveying and support component 1, providing a mounting platform for the image acquisition component 3. The light source mounting adapter 3.1 is fixed to the light source and camera bracket 3.4, providing an installation interface for the light source 3.2. The light source 3.2 is mounted on the light source and camera bracket 3.4 via the light source mounting adapter 3.1, using an LED light source to provide uniform and stable illumination, eliminate shadows and reflections, and ensure image quality. The industrial camera 3.3 is mounted on the light source and camera bracket 3.4, with its lens facing the surface of the component under test 2, and is used to acquire high-resolution surface images. In this embodiment, the industrial camera 3.3 is an industrial-grade camera with at least 2 megapixels and a frame rate of no less than 30fps, which meets the requirements of real-time detection.

[0024] For example, four light sources 3.2 are distributed around the industrial camera 3.3 to provide uniform and stable illumination for the industrial camera 3.3.

[0025] Control component 4 is electrically connected to image acquisition component 3. It receives and processes image data, executes a deep learning-based defect detection algorithm, and controls the coordinated operation of all system components. Control component 4 includes an industrial computer and a vision inspection software solution. The industrial computer uses an industrial-grade computer, equipped with a high-performance processor and graphics processing unit, enabling it to quickly process image data and run deep learning models. The vision inspection software solution includes an image preprocessing module and a dual-model detection module.

[0026] The image preprocessing module preprocesses the acquired raw images, including image normalization, resizing, and color space conversion, converting the images into a format suitable for input to deep learning models.

[0027] The dual-model detection module includes a semantic segmentation model and an object detection model. The semantic segmentation model employs an encoder-decoder structure, achieving accurate segmentation of defect regions through pixel-level classification, and outputting defect segmentation masks, defect areas, and location information. The object detection model uses a single-stage detection architecture, enabling rapid defect localization and classification, outputting defect bounding boxes, categories, and confidence information. The two models work in parallel, complementing each other; the semantic segmentation model excels at accurately locating defect boundaries, while the object detection model excels at quickly identifying and classifying defect types. In the result fusion stage, the defect mask output by the semantic segmentation model is first converted into a minimum bounding box, and the intersection-over-union (IoU) ratio between this bounding box and the defect bounding box output by the object detection model is calculated. The IoU calculation formula is: ; Where A is the area of ​​the minimum bounding rectangle of the semantic segmentation model, and B is the area of ​​the defect bounding box of the object detection model.

[0028] When the maximum intersection-union ratio is greater than the preset threshold If the confidence level is 0.5, it is determined to be the same defect. The weighted average of the confidence levels of the two models is selected as the final confidence level, and the defect category is based on the output of the target detection model. If it is less than the preset threshold, it is considered as the same defect. If a defect is found to be an independent defect, it is retained. This fusion strategy significantly improves detection accuracy and robustness through the dual-model approach.

[0029] Control component 4 is connected to a programmable logic controller (PLC) via industrial Ethernet or serial communication protocol. The PLC is used to control the operating status of the conveying and support component 1, including conveying speed control, start / stop control, etc., and also controls the action of the material gripping component 5. Control component 4 sends the detection results to the PLC via the communication protocol, and the PLC executes the corresponding control logic based on the detection results.

[0030] The material gripping assembly 5 is located on one side of the conveying and support assembly 1 and is electrically connected to the control assembly 4. It is used to grip defective sheet metal parts based on inspection results. The material gripping assembly 5 includes a robotic arm and an end effector connected to the output end of the multi-degree-of-freedom robotic arm. The robotic arm is a multi-degree-of-freedom industrial robotic arm with sufficient working range and load capacity, enabling it to accurately reach designated positions. The end effector is configured as a vacuum suction cup or a mechanical gripper depending on the shape and size of the sheet metal part. Vacuum suction cups are suitable for sheet metal parts with flat surfaces, while mechanical grippers are suitable for sheet metal parts requiring edge gripping. The selection of the end effector can be flexibly configured according to actual application requirements, without limiting the specific structural form.

[0031] The material storage component 6 is located within the working range of the material gripping component 5 and is used to store the gripped defective sheet metal parts. In this embodiment, the material storage component 6 adopts a material rack structure, which can store various sheet metal parts containing defects, facilitating subsequent defect analysis and quality traceability.

[0032] Example 1 This invention provides a deep learning-based method for detecting surface defects in sheet metal parts, applicable to the aforementioned deep learning-based sheet metal part surface defect detection system. See [link to relevant documentation]. Figure 4 As shown, the method includes the following steps: S1: The sheet metal part is placed on the feeding roller 1.1 of the conveying and support assembly 1, and is smoothly conveyed to the image acquisition position by the feeding roller 1.1. The operator or the automatic feeding mechanism places the sheet metal part to be inspected on the feeding roller 1.1, the feeding roller 1.1 starts, and smoothly conveys the sheet metal part to the image acquisition position of the inspection area.

[0033] S2: After the sheet metal part reaches the designated position, the position sensor detects the sheet metal part's arrival signal, and the PLC sends a trigger signal to the control component 4. At the same time, it controls the feeding roller 1.1 to pause or reduce the conveying speed to ensure the stability of image acquisition.

[0034] S3: After receiving the trigger signal, the control component 4 controls the industrial camera 3.3 of the image acquisition component 3 to acquire images. The light source 3.2 provides stable and uniform illumination, and the industrial camera 3.3 acquires high-resolution images of the sheet metal surface and transmits the image data to the control component 4.

[0035] S4: Control component 4 preprocesses the acquired image. Image preprocessing includes: first, image normalization, normalizing pixel values ​​to the range of 0-1; then, resizing, adjusting the image to the input size required by the deep learning model; and finally, color space conversion, converting it to RGB or grayscale images according to the model's requirements.

[0036] S5: The preprocessed images are input into the semantic segmentation model and the object detection model respectively for parallel detection. See also Figure 5 As shown, the semantic segmentation model performs pixel-level classification of the image and outputs a defect segmentation mask. By analyzing the mask, the defect area and precise location information can be obtained. The object detection model locates and classifies defects in the image and outputs defect bounding boxes, class labels, and confidence scores. The two models work simultaneously, making full use of computational resources and improving detection efficiency.

[0037] S6: Control component 4 fuses and comprehensively evaluates the detection results of the two models. The fusion strategy includes: for the same defect area, if both models detect it, the weighted average of the confidence scores of the two models is selected as the final confidence score, and the defect category is based on the output of the target detection model; for defects detected by only one model, the judgment is made based on the confidence score of that model and the defect area.

[0038] The comprehensive evaluation is based on preset acceptance criteria, which include: defect quantity threshold, defect confidence threshold, and defect area threshold. A product is considered acceptable (OK) if any of the following conditions are met: zero defects; all detected defects have a confidence level below a preset threshold (e.g., 0.5); or all defect areas are smaller than a preset threshold (e.g., 100 square pixels). Otherwise, it is considered defective (NG).

[0039] S7: If it is determined to be a qualified product, the control component 4 notifies the PLC through the communication protocol. The PLC controls the feeding roller 1.1 to resume the normal conveying speed, and the sheet metal part continues to be transported by the feeding roller 1.1 to the subsequent workstations, entering the next production process. If it is determined to be a defective product, the control component 4 sends the detection results to the PLC through the communication protocol. The detection results include the OK / NG determination result, the number of defects, the defect type code, and the defect position coordinates. The defect types include six common defect types such as patches, cracks, depressions, inclusions, scratches, and mill scale, and each defect type corresponds to a unique code.

[0040] S8: The PLC controls the movement of the robotic arm of the material grasping component 5 according to the received detection results. The robotic arm moves above the defective sheet metal part, the end effector descends and grasps the sheet metal part, then moves the sheet metal part above the material storage component 6, and releases the sheet metal part to make it fall into the material rack. After the grasping is completed, the robotic arm returns to the initial position, waiting for the next grasping instruction. At the same time, the PLC controls the detection and conveying section 1.2 to continue conveying, preparing to detect the next sheet metal part.

[0041] In this embodiment, the entire detection process realizes a high degree of automation, from the feeding of sheet metal parts to the sorting of defective products, without manual intervention. The system can operate continuously and stably, achieving high-speed detection with high detection accuracy. Compared with traditional manual detection and detection methods based on traditional machine vision, the missed detection rate and misdetection rate are significantly reduced.

[0042] Embodiment 2 Based on Embodiment 1, in order to further improve the detection accuracy and adapt to more complex detection requirements, multiple industrial cameras 3.3 can be added to the image acquisition component 3 to collect the surface images of the sheet metal parts from different angles. For example, an industrial camera can be set above and on the side of the sheet metal part respectively to achieve a full-range detection of the surface and edges of the sheet metal part. The images collected by multiple cameras can be respectively input into the dual-model detection module for detection, and then the multiple detection results are fused to further improve the comprehensiveness and accuracy of the detection.

[0043] Compared with the prior art, this embodiment has the following beneficial effects: This embodiment adopts a dual-model parallel detection architecture, combines the advantages of the semantic segmentation model and the object detection model, and realizes high-precision detection and positioning of the surface defects of sheet metal parts. Compared with the detection system using a single model, the detection accuracy is significantly improved, and the missed detection rate and misdetection rate are effectively reduced. <

[0044] This embodiment realizes the full-process automation from the feeding of sheet metal parts, image acquisition, defect detection, result determination to automatic sorting, without manual intervention, greatly improving the detection efficiency, achieving high-speed detection, and being able to meet the rhythm requirements and real-time detection needs of industrial production lines.

[0045] This embodiment integrates the conveying system, image acquisition system, detection system and sorting system through integrated design, which reduces the number of times the workpiece is transferred between different workstations, avoids secondary damage to sheet metal parts during the transfer process, and improves the reliability and stability of the system.

[0046] This embodiment employs a deep learning-based detection algorithm, which has strong adaptive learning and generalization capabilities. It can adapt to the detection needs of sheet metal parts of different types and sizes, and has strong robustness to environmental factors such as changes in lighting and surface texture.

[0047] This embodiment uses a programmable logic controller to achieve coordinated control of various components. The system has a fast response speed, high control precision, and can achieve continuous and stable operation for 24 hours, which significantly reduces labor costs and improves production efficiency and product quality consistency.

[0048] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A deep learning-based sheet metal surface defect detection method, characterized by, include: Acquire and preprocess the surface image of the component under test; The preprocessed image is input into the semantic segmentation model and the object detection model for parallel detection. The semantic segmentation model outputs the defect segmentation mask, defect area, and location information, while the object detection model outputs the defect bounding box, category, and confidence information. By combining the detection results of the two models, the presence of defects in sheet metal parts is determined based on preset qualification indicators; If a defect exists, grab the defective component to be tested and place it in the defect placement area.

2. The method of claim 1, wherein, The preset qualification indicators include the number of defects, the defect confidence threshold, and the defect area threshold. When the number of defects is zero, the confidence of all defects is lower than the threshold, or the area of ​​all defects is less than the threshold, it is judged as qualified.

3. The method of claim 1, wherein, The fusion strategy for combining the detection results of the two models is as follows: Calculate the intersection-union ratio (IUU) between the minimum bounding rectangle of the defect segmentation mask output by the semantic segmentation model and the defect bounding box output by the object detection model; If the crossover ratio is greater than the preset threshold, it is determined to be the same defect. The weighted average of the confidence scores of the two models is taken as the final confidence score, and the defect category is based on the output of the target detection model. If the crossover ratio is less than or equal to a preset threshold, it is considered an independent defect and retained.

4. The method according to claim 1, characterized in that, The defect types include plaques, cracks, dents, inclusions, scratches, and oxide scale.

5. The method according to claim 1, characterized in that, It also includes a data management step: storing the detection results in a database, the detection results including detection time, detection image, defect type, defect location and defect quantity.

6. A deep learning-based surface defect detection system for sheet metal parts, characterized in that, include: A conveying and support assembly (1) is used to carry and convey the component under test (2). An image acquisition component (3) is located above the conveying and support component (1) and is used to acquire images of the surface of the component under test (2); The control component (4) is electrically connected to the image acquisition component (3) and is used to receive image data and execute a deep learning-based defect detection algorithm. The defect detection algorithm adopts a dual-model parallel detection architecture, including a semantic segmentation model and a target detection model. The semantic segmentation model is used for pixel-level defect segmentation, and the target detection model is used for defect localization and classification. The detection results of the two models are fused for defect evaluation. The material gripping component (5) is electrically connected to the control component (4) and is used to grip the defective test component (2) according to the detection result. The material storage component (6) is located within the working range of the material gripping component (5) and is used to store the gripped test component (2).

7. The system according to claim 6, characterized in that, The conveying and support assembly (1) includes a feeding roller (1.1), a conveyor support (1.2) and a foot (1.3). Multiple feeding rollers (1.1) are arranged in parallel in sequence and are rotatably connected to the conveyor support (1.2). The conveyor support (1.2) is connected to the foot (1.3) through a column.

8. The system according to claim 6, characterized in that, The image acquisition component (3) includes a light source mounting adapter (3.1), a light source (3.2), an industrial camera (3.3), and a light source and camera bracket (3.4) in the form of a gantry structure, wherein the light source and camera bracket (3.4) is mounted above the conveying and support component (1); The light source mounting adapter (3.1) is fixed on the light source and camera bracket (3.4). The light source (3.2) is rotatably connected to the light source mounting adapter (3.1). The industrial camera (3.3) is mounted on the light source and camera bracket (3.4) with its lens facing the surface of the component under test (2).

9. The system according to claim 8, characterized in that, Four light sources (3.2) are distributed around the industrial camera (3.3) to provide uniform and stable illumination for the industrial camera (3.3).

10. The system according to claim 8, characterized in that, The material gripping component (5) includes a multi-degree-of-freedom robotic arm and an end effector connected to the output end of the multi-degree-of-freedom robotic arm. The end effector is a vacuum suction cup or a mechanical gripper, used to grip the component under test (2).

Images