Vision-based stamping part defect detection method

Abstract

The invention relates to the technical field of visual inspection, in particular to a stamping part defect detection method based on vision, which comprises the following steps: S1, shooting setting; s2, synchronizing video streams; s3, acquiring a part bounding box and a hole position bounding box; s4, associating part bounding boxes belonging to the same stamping part among the frame part images, and associating hole position bounding boxes belonging to the same screw hole position; s5, correspondingly associating the bounding boxes of the target parts belonging to the same stamping part in the part image; s6, when a hole position boundary frame matched with the view angle template image of the corresponding view angle exists in any target detection image under any view angle, determining that a screw hole position exists; and when the matched hole position boundary frame is not found in the target detection image under the corresponding visual angle in the visual angle template images of all the visual angles, the screw hole position is missing. The stamping part defect detection device can be used for defect detection of stamping parts in the conveying process, the consistency of detection results is improved, and the detection efficiency is improved.

Description

Technical Field

[0001] This invention relates to the technical field of visual inspection, and more specifically, to a vision-based method for detecting defects in stamped parts. Background Technology

[0002] Traditional defect detection in automotive stamping parts largely relies on visual inspection by quality control personnel. However, this method is inefficient, especially in high-volume production lines where the inspection speed of personnel often cannot meet production demands. Furthermore, it typically requires experienced quality control personnel, and this reliance on experience can lead to missed or false positives. Additionally, inspection results can fluctuate due to differences in personnel and inspection environments, resulting in poor consistency and hindering stable quality control and product consistency requirements. To address this, existing technologies for defect detection in parts collect a large number of defect image samples for neural network training to obtain a defect model, which is then used to detect defects in stationary, actual parts. However, in actual production, the number of part images available as defect samples is limited, making it difficult to obtain a comprehensive defect model. This results in inconsistent inspection results and hinders effective improvement in inspection efficiency. Summary of the Invention

[0003] The purpose of this invention is to overcome the lack of consistency in the detection results of stamped parts defects in existing technologies, and to provide a vision-based method for detecting stamped parts defects, which can be used for defect detection of stamped parts during the transmission process, thereby improving the consistency of detection results and increasing detection efficiency.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A vision-based defect detection method for stamped parts is provided, comprising the following steps: S1. Camera setup: A multi-view camera is installed on the conveyor belt used to transport stamped parts, and the shooting range of the multi-view camera can cover the entire conveyor belt. S2. Video stream synchronization: Synchronize the video streams from various perspectives captured by the multi-view cameras in time. S3. Boundary box detection: In each frame of part image captured by the camera from its respective viewpoint, the part boundary box of the corresponding stamping part is obtained based on the object detection algorithm, and the hole boundary box of each screw hole on the stamping part is obtained. S4. Target tracking: In the part images captured by the cameras from their respective perspectives, the bounding boxes of the parts belonging to the same stamping part in each frame of part images are correlated accordingly, and the bounding boxes of the holes belonging to the same screw hole position under the same stamping part in each frame of part images are also correlated accordingly. S5. Part Recognition: In the part images captured by the cameras from their respective perspectives, the target part bounding box is obtained based on the relative position of the part bounding box on the conveyor belt; the target part bounding boxes belonging to the same stamping part in the part images from all perspectives in the same frame are associated accordingly. S6. Multi-view template matching: Obtain the template image of the stamped part according to each viewpoint; select multiple effective part images with a certain degree of similarity to the template image under each viewpoint as target detection images; match the target detection images one by one with the corresponding hole bounding boxes of the template image. When there is a matching hole bounding box in any target detection image under any viewpoint and the viewpoint template image of the corresponding viewpoint, it is considered that the screw hole corresponding to the hole bounding box exists. If no matching hole bounding box is found in the target detection image at the corresponding viewpoint in the viewpoint template images of all views, then the screw hole corresponding to the hole bounding box is considered to be missing.

[0005] This invention discloses a vision-based defect detection method for stamped parts. It simultaneously captures images of stamped parts transported on a conveyor belt using multi-view cameras, ensuring that the entire stamped part is covered by the image, providing a sufficient data foundation for subsequent defect detection. Next, a target detection algorithm extracts various features of the stamped parts from the images, and then obtains accurate part bounding boxes and hole bounding boxes based on these features. Then, from each viewpoint, the bounding boxes of the same stamped part and the bounding boxes of the same screw hole are correlated across different frames of part images, achieving target tracking and enabling defect detection in areas where defects are found. The invention maintains stable detection even in obstructed or occluded scenarios, improving detection stability. Furthermore, by associating the bounding boxes of target parts belonging to the same stamping part from all viewpoints within the same frame, multi-view part recognition is achieved, making the invention applicable to accurate recognition in complex scenarios in actual production environments. Finally, by matching the target detection image with the corresponding normal viewpoint template image, multi-view template matching is achieved, enabling defect detection without collecting a large number of defect samples. This invention can be used for defect detection of missing holes in stamping parts during transport, improving the consistency of detection results and increasing detection efficiency.

[0006] Further, in step S1, the multi-view camera includes one camera each positioned directly above, on the upper left, on the upper right, and at an oblique distance from the corresponding conveyor belt; wherein, the camera positioned directly above is located within the projection area directly above the conveyor belt; the cameras on the upper left and upper right are located on either side of the camera positioned directly above, and the cameras on the upper left, upper right, and directly above are all located in the same vertical plane, and this vertical plane is perpendicular or approximately perpendicular to the conveying direction of the conveyor belt; the camera at an oblique distance is located within the projection area directly above the conveyor belt.

[0007] Further, in step S4, the intersection-union ratio (IUU) between the bounding boxes of the parts in the current frame and the previous frame is calculated using a target tracking algorithm to obtain the part overlap. For parts with an overlap greater than or equal to a preset overlap threshold, the corresponding current frame and the bounding boxes of the parts in the previous frame are associated. The hole overlap is calculated using a target tracking algorithm to obtain the hole overlap. For holes with an overlap greater than or equal to a preset overlap threshold, the corresponding current frame and the bounding boxes of the holes in the previous frame are associated.

[0008] Furthermore, in the target tracking in step S4: if a certain hole bounding box is not associated with a corresponding hole in n consecutive frames of part images, then the hole bounding box is marked as lost, and the hole bounding box marked as lost will not be executed in subsequent steps; where n≥3; if a hole bounding box that did not appear in the previous frame is associated with a corresponding hole in an adjacent frame of part images, then the hole bounding box is given a new mark, and the newly marked hole bounding box will be executed in subsequent steps.

[0009] Further, step S5 includes the following steps: S51. Based on the center point position of the part bounding box output by the target detection algorithm, allocate state variables according to the relative position on the conveyor belt under the corresponding view camera; wherein, the state variables include the state box that is located on the conveyor belt and set along the conveying direction of the conveyor belt, the state box that has entered, and the state box that has left. S52. The bounding boxes of parts that have entered the state box or have already entered the state box are identified as the target part bounding boxes, and each frame of part image after obtaining the target part bounding box is taken as a valid part image. S53. Group the valid part images of all viewpoints in the same frame into pairs, and then calculate the overlap ratio between the bounding boxes of the target parts in each group to obtain the overlap. When the overlap ratio is greater than or equal to the corresponding set overlap threshold, the bounding boxes of the target parts are considered to belong to the same stamping part, and the bounding boxes of the target parts belonging to the same stamping part are associated.

[0010] Further, step S53 includes the following steps: S531. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the upper left and upper right views in the same frame to obtain the left and right overlap, and compare the left and right overlap with the preset left and right overlap threshold. S532. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the upper left or upper right view and the top view in the same frame to obtain the side-to-top overlap, and compare the side-to-top overlap with the preset side-to-top overlap threshold. S533. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the downsloping far side view and the direct upward view in the same frame to obtain the far-positive overlap, and compare the far-positive overlap with the preset far-positive overlap threshold. S534. When the overlap calculated in each of steps S531, S532, and S533 is greater than or equal to the corresponding preset overlap threshold, it is considered that the bounding boxes of the target parts under each view belong to the same stamping part, and then the bounding boxes of the target parts belonging to the same stamping part are associated.

[0011] Furthermore, in steps S532 and S533, the bounding box of the target part from one viewpoint is mapped to the coordinate system of another viewpoint using a homography transformation matrix. Then, the overlap ratio between the transformed bounding box of the target part from one viewpoint and the bounding box of the target part from the other viewpoint is calculated.

[0012] Further, step S6 includes the following steps: S61. Based on the top, top left, top right, and oblique far-angle views, respectively obtain the template image of the template stamping part from the top, top left, top right, and oblique far-angle views. S62. Under each viewpoint, calculate the cross-union ratio between the effective part image in each frame and the target part bounding box in the template image of that viewpoint to obtain the similarity overlap. Then, use the effective part image with a similarity overlap greater than or equal to the preset similarity overlap threshold as the target detection image. S63. Under each viewpoint, the target detection image is matched one by one with the corresponding hole position bounding box of the template image of that viewpoint: the intersection-union ratio between the current target detection image and the hole position bounding box in the template image of that viewpoint is calculated to obtain the matching overlap. If the matching overlap is greater than or equal to the preset matching overlap threshold, it means that the screw hole position corresponding to the hole position bounding box exists. Then, when processing the next target detection image, the hole position bounding boxes that have been matched are removed and the matching overlap of the remaining hole position bounding boxes is calculated, until the hole position bounding boxes of all target detection images have undergone matching overlap calculation. If no hole bounding box with a matching overlap threshold greater than or equal to the preset threshold is obtained in the target detection image under the corresponding viewpoint template image in all viewpoints, it indicates that the screw hole corresponding to the hole bounding box is missing.

[0013] Furthermore, step S3 also includes: obtaining the part category of the corresponding stamping part in each frame of part image based on the target detection algorithm.

[0014] Further, step S2 includes the following steps: S21. Combine all frame component images captured by video streams from different viewpoints at the same time into a frame packet; S22. Extract the part image of one frame from each frame packet for inspection. If the part image of the frame is normal, record the timestamp of the corresponding frame packet and store it in the corresponding buffer. S23. Compare the timestamps of the frame packets in each buffer, and obtain the frame packet with the earliest timestamp as the reference base timestamp; S24. Calculate the time difference between the timestamps of other frame packets and the reference timestamp. For lagging timestamps, the frame packets in their corresponding buffers will not be output temporarily until their timestamps catch up with the reference timestamps. S25. Once the timestamps in all buffers are aligned, output synchronized frame packets.

[0015] Compared with the prior art, the beneficial effects of the present invention are: This invention discloses a vision-based defect detection method for stamped parts. It simultaneously captures images of stamped parts conveying on a conveyor belt using multi-view cameras, ensuring that the entire stamped part is covered in the images, providing a sufficient data foundation for subsequent defect detection. Next, a target detection algorithm extracts various features of the stamped parts from the images, and then obtains accurate part bounding boxes and hole bounding boxes based on the extracted features. Finally, from each viewpoint, the bounding boxes of the same stamped part and the bounding boxes of the same screw hole are correlated across different frames of part images. This invention achieves target tracking, enabling stable detection even in scenarios with missed detections or occlusion, thus improving detection stability. Then, by associating the bounding boxes of target parts belonging to the same stamping part from all viewpoints within the same frame, multi-view part recognition is achieved, making the invention applicable to accurate recognition in complex scenarios in actual production environments. Finally, multi-view template matching is achieved by matching the target detection image with the viewpoint template image of the corresponding viewpoint. This invention can be used for defect detection of missing holes in stamping parts during the conveying process, improving the consistency of detection results and increasing detection efficiency. Attached Figure Description

[0016] Figure 1 This is a flowchart of a vision-based defect detection method for stamped parts according to the present invention; Figure 2 This is a schematic diagram of the part bounding box and hole bounding box of the part image in step S3 of the present invention; Figure 3 This is a schematic diagram of the state variables in step S51 of the present invention. The arrows in the diagram indicate the conveyor belt conveying direction. Detailed Implementation

[0017] The present invention will be further described below with reference to specific embodiments. The accompanying drawings are for illustrative purposes only, representing schematic diagrams rather than actual physical objects, and should not be construed as limiting the scope of this patent. To better illustrate the embodiments of the present invention, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0018] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0019] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0020] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] Example 1 like Figure 1 The figure shows a first embodiment of a vision-based defect detection method for stamped parts according to the present invention, which includes the following steps: S1. Camera setup: A multi-view camera is installed on the conveyor belt used to transport stamped parts, and the shooting range of the multi-view camera can cover the entire conveyor belt. S2. Video stream synchronization: Synchronize the video streams from various perspectives captured by the multi-view cameras in time. S3. Boundary box detection: In each frame of part image captured by the camera from its respective viewpoint, the part boundary box of the corresponding stamping part is obtained based on the object detection algorithm, and the hole boundary box of each screw hole on the stamping part is obtained. S4. Target tracking: In the part images captured by the cameras from their respective perspectives, the bounding boxes of the parts belonging to the same stamping part in each frame of part images are correlated accordingly, and the bounding boxes of the holes belonging to the same screw hole position under the same stamping part in each frame of part images are also correlated accordingly. S5. Part Recognition: In the part images captured by the cameras from their respective perspectives, the target part bounding box is obtained based on the relative position of the part bounding box on the conveyor belt; the target part bounding boxes belonging to the same stamping part in the part images from all perspectives in the same frame are associated accordingly. S6. Multi-view template matching: Obtain the template image of the stamped part according to each viewpoint; select multiple effective part images with a certain degree of similarity to the template image under each viewpoint as target detection images; match the target detection images one by one with the corresponding hole bounding boxes of the template image. When there is a matching hole bounding box in any target detection image under any viewpoint and the viewpoint template image of the corresponding viewpoint, it is considered that the screw hole corresponding to the hole bounding box exists. If no matching hole bounding box is found in the target detection image at the corresponding viewpoint in the viewpoint template images of all views, then the screw hole corresponding to the hole bounding box is considered to be missing.

[0023] This invention discloses a vision-based defect detection method for stamped parts. It simultaneously captures images of stamped parts conveying on a conveyor belt using multi-view cameras, ensuring that the entire stamped part is covered in the images, providing a sufficient data foundation for subsequent defect detection. Next, a target detection algorithm extracts various features of the stamped parts from the images, and then obtains accurate part bounding boxes and hole bounding boxes based on the extracted features. Finally, from each viewpoint, the bounding boxes of the same stamped part and the bounding boxes of the same screw hole are correlated across different frames of part images. This invention achieves target tracking, enabling stable detection even in scenarios with missed detections or occlusion, thus improving detection stability. Then, by associating the bounding boxes of target parts belonging to the same stamping part from all viewpoints within the same frame, multi-view part recognition is achieved, making the invention applicable to accurate recognition in complex scenarios in actual production environments. Finally, multi-view template matching is achieved by matching the target detection image with the viewpoint template image of the corresponding viewpoint. This invention can be used for defect detection of missing holes in stamping parts during the conveying process, improving the consistency of detection results and increasing detection efficiency.

[0024] Example 2 This embodiment is a second embodiment of a vision-based defect detection method for stamped parts. This embodiment is similar to the first embodiment, except that in step S1, the multi-view camera includes one camera positioned directly above, one on the upper left, one on the upper right, and one at a distant angle to the conveyor belt. The camera positioned directly above the conveyor belt is located within its projection area. The cameras on the upper left and upper right are located on either side of the camera positioned directly above the conveyor belt, and all three cameras are located in the same vertical plane, which is perpendicular or approximately perpendicular to the conveyor belt's direction. The camera at a distant angle to the conveyor belt is located within its projection area directly above the conveyor belt. Furthermore, the sum of the shooting ranges of the cameras positioned directly above, on the upper left, on the upper right, and at a distant angle to the conveyor belt covers the entire conveyor surface. In this embodiment, a high-definition camera with a video resolution of 1920*1080 and an H.264 encoding format is selected for real-time shooting.

[0025] In this embodiment, step S2 includes the following steps: S21. Combine all frame component images captured by video streams from different viewpoints at the same time into a frame packet; Specifically, the RTSP video source address is read from each camera, and then video frames are obtained from the video source; the same moment can be set within the same 1 second, and the 1 second contains multiple frames of part images; S22. Extract the part image of one frame from each frame packet for inspection. If the part image of the frame is normal, record the timestamp of the corresponding frame packet and store it in the corresponding buffer. S23. Compare the timestamps of the frame packets in each buffer, and obtain the frame packet with the earliest timestamp as the reference base timestamp; S24. Calculate the time difference between the timestamps corresponding to other frame packets and the reference timestamp. For lagging or lost timestamps, the frame packets in their corresponding buffers will not be output temporarily until their timestamps catch up with the reference timestamps. Also, if a video frame is found to be lost or in an abnormal state during a capture, a retry mechanism is triggered to try to acquire the video frame again. If the frame cannot be acquired after a certain number of retries, the previous frame is used to fill the gap to ensure that the number of output frames is consistent. S25. Once the timestamps in all buffers are aligned, output synchronized frame packets.

[0026] In this embodiment, the target detection algorithm in step S3 is the YOLOv8 algorithm. Specifically, based on the collected stamping part sample images, a sample image dataset is constructed, and the sample image dataset is divided into a training set, a validation set, and a test set according to a certain ratio. The data is then preprocessed, and the preprocessed data is input into a convolutional neural network for training. The convolutional neural network extracts features of the input sample images from the pixel level through layer-by-layer convolution and pooling: shallow layers extract low-level features such as edges, corners, and colors, while deep layers learn shape, texture, spatial relationships, and semantic information. Based on the extracted features, the bounding box coordinates, category, and position confidence of each feature map unit are calculated through fully connected layers. Finally, a non-maximum suppression algorithm is used to eliminate redundant overlapping bounding boxes. In this embodiment, in each frame of part images captured by the top-view camera, the bounding box of the corresponding stamping part and the bounding boxes of the corresponding screw holes on the stamping part are obtained based on the YOLOv8 algorithm. Similarly, the processing of each frame of part images captured by the top-left, top-right, and oblique-far-side cameras can be obtained. like Figure 2 The image shown is a part image processed by an object detection algorithm. The large colored bounding boxes represent the part bounding boxes, and the small colored bounding boxes represent the hole bounding boxes. Since a stamped part has several screw holes, there will also be several hole bounding boxes within a part bounding box.

[0027] In this embodiment, in step S4, based on the target tracking algorithm, the bounding boxes of the part images obtained from the top-down view, the top-left view, the top-right view, and the oblique far-side view are associated in their respective views. The target tracking algorithm is the SORT algorithm, which defines the state of each part bounding box and each hole bounding box as... The seven-dimensional vector, using the bounding box position information from the previous frame, is used to establish a state prediction equation using a Kalman filter to solve for the bounding box state in the current frame. Specifically: The overlap of parts is obtained by calculating the intersection-over-union ratio (IOU) between the bounding boxes of parts in the current frame and the previous frame using the SORT algorithm. For parts with an overlap greater than or equal to a preset overlap threshold, the bounding boxes of the corresponding current frame and the previous frame are associated. In this embodiment, the overlap threshold is set to 0.35. The IOU is defined as:

[0028] In the formula, A represents the bounding box of the part in the current frame, B represents the bounding box of the part in the previous frame, and IOU represents the overlap between the two bounding boxes; similarly, the calculation of each subsequent overlap can be obtained. The hole overlap is obtained by calculating the intersection-union ratio (IUU) between the hole bounding boxes in the current frame part image and the previous frame part image using the SORT algorithm. For hole overlaps greater than or equal to a preset hole overlap threshold, the corresponding hole bounding boxes in the current frame and the previous frame are associated. In this embodiment, the hole overlap threshold is set to 0.35.

[0029] In this embodiment, during target tracking in step S4: If a hole bounding box is not associated with a corresponding bounding box in n consecutive frames of part images, then the hole bounding box is marked as missing, and subsequent steps will not be executed for hole bounding boxes marked as missing; where n≥3; If a hole bounding box that did not appear in the previous frame is associated with a part image in an adjacent frame, then the hole bounding box is given a new mark, and the hole bounding box with the new mark will be subject to subsequent steps.

[0030] In this embodiment, step S5 includes the following steps: S51. Based on the center point position of the part bounding box output by the target detection algorithm, allocate state variables according to the relative position on the conveyor belt under the corresponding view camera; wherein, the state variables include the state box that is located on the conveyor belt and set along the conveying direction of the conveyor belt, the state box that has entered, and the state box that has left. like Figure 3 As shown, the blue diagram represents the ENTERING state box, the green diagram represents the ENTERED state box, the purple diagram represents the LEAVING state box, and the red diagram represents the part bounding box. Figure 3In the image, the center point of the part's bounding box is located within the "Entered" state box, so the state variable assigned to the current part image is "Entered". In this embodiment, in step S51, the state variables also include an unknown state box (UNKOWN) located outside the conveyor belt, which can be used to assign state variables to the bounding boxes of the stamping parts located outside the conveyor belt, thereby avoiding misidentification of stamping parts outside the conveyor belt. S52. The bounding boxes of parts that have entered the state box or have already entered the state box are identified as the target part bounding boxes, and each frame of part image after obtaining the target part bounding box is taken as a valid part image. S53. Group the valid part images of all viewpoints in the same frame into pairs, and then calculate the overlap ratio between the bounding boxes of the target parts in each group to obtain the overlap. When the overlap ratio is greater than or equal to the corresponding set overlap threshold, the bounding boxes of the target parts are considered to belong to the same stamping part, and the bounding boxes of the target parts belonging to the same stamping part are associated.

[0031] Specifically, step S53 includes the following steps: S531. Calculate the intersection-union ratio (IUU) between the target part bounding boxes in the effective part images of the upper left and upper right views in the same frame to obtain the left-right overlap, and compare the left-right overlap with a preset left-right overlap threshold; in this embodiment, the left-right overlap threshold is set to 0.3. S532. Calculate the intersection-union ratio (IU) between the target part bounding boxes in the effective part images of the upper left or upper right view and the top view in the same frame to obtain the side-to-top overlap, and compare the side-to-top overlap with a preset side-to-top overlap threshold; in this embodiment, the side-to-top overlap threshold is set to 0.3. S533. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the oblique far-side view and the direct upward view in the same frame to obtain the far-positive overlap, and compare the far-positive overlap with the preset far-positive overlap threshold; in this embodiment, the far-positive overlap threshold is set to 0.3. S534. When the overlap calculated in each of steps S531, S532, and S533 is greater than or equal to the corresponding preset overlap threshold, it is considered that the bounding boxes of the target parts under each viewpoint belong to the same stamping part. Then, the bounding boxes of the target parts belonging to the same stamping part are associated. Specifically, they can be associated by assigning the same ID number to the bounding boxes of the target parts belonging to the same stamping part.

[0032] In steps S532 and S533, the bounding box of the target part from one viewpoint is mapped to the coordinate system of the other viewpoint using a homography transformation matrix. Then, the overlap ratio is calculated between the transformed bounding boxes of the target part from one viewpoint and the other viewpoint. Specifically: The homography transformation matrix is ​​defined as follows: Let the coordinates of point P1 in the part image from its own perspective be (x, y), and the coordinates of the corresponding point P2 in the part image from another perspective be (x', y'). Then the relationship between the two points can be represented by the following matrix:

[0033] ,in,

[0034] By obtaining N (N≥4) pairs of corresponding points from two calibrated part images, the following system of equations can be constructed for each pair of corresponding points:

[0035]

[0036] In the above formula, H is a 3*3 homography transformation matrix. During calculation, it can be obtained by transforming h... 33 Setting the value to 1 simplifies the equations, resulting in a combined equation of the form A*h=0, where A is an 8*9 matrix and h is a column vector. Singular value decomposition (SVD) of matrix A yields a least-squares solution. This solution can be used to represent the mapping between two perspectives; that is, a point P1 in one's own perspective can be multiplied by the transformation matrix M of the least-squares solution to obtain the corresponding point P2 = M in the other perspective. P1.

[0037] In this embodiment, step S6 includes the following steps: S61. Obtain template images of the stamping part from the top, top left, top right, and oblique far-angle views respectively; wherein, each screw hole position in each template image has been assigned a corresponding ID number; it should be noted that the obtained template images of each view can be set as the corresponding view images taken when the stamping part is placed near the middle of the conveyor belt. S62. From their respective perspectives, the cross-union ratio (CUI) between each frame of valid part images and the bounding boxes of the target parts in the viewpoint template image is calculated to obtain the similarity overlap. Then, valid part images with a similarity overlap greater than or equal to a preset similarity overlap threshold are used as target detection images. In this embodiment, the similarity overlap threshold is set to 0.75. It should be noted that by setting the similarity overlap threshold, only valid part images located approximately at the same position as the stamping part in the viewpoint template image can be used as target detection images. That is, only valid part images transported to the vicinity of the middle position of the conveyor belt can be used as target detection images. S63. From their respective perspectives, the target detection images are matched one by one with the corresponding hole bounding boxes of the template image of that perspective: the intersection-union ratio (IUU) between the current target detection image and the hole bounding boxes in the template image of that perspective is calculated to obtain the matching overlap. If the matching overlap is greater than or equal to a preset matching overlap threshold, it indicates that the screw hole corresponding to the hole bounding box exists. In this embodiment, the matching overlap threshold is set to 0.75. If the calculated matching overlap is greater than or equal to 0.75, it indicates that the screw hole corresponding to the hole bounding box number exists. Then, when processing the next target detection image, the matched hole bounding boxes are removed and the matching overlap of the remaining hole bounding boxes is calculated, until the hole bounding boxes of all target detection images have undergone matching overlap calculation. When no hole bounding box with a matching overlap threshold greater than or equal to the preset threshold is obtained in the target detection image under the corresponding viewpoint in all viewpoint template images, that is, if a hole bounding box number that has not been matched in the target detection images of the four viewpoints appears in the viewpoint template images of the four viewpoints, it indicates that the screw hole corresponding to the hole bounding box is missing.

[0038] It should be noted that the final test results can be displayed on a set display device, making it convenient for quality inspectors to view and record them in real time.

[0039] Example 3 This embodiment is a third embodiment of a vision-based method for detecting defects in stamped parts. This embodiment is similar to Embodiment 1 or 2, except that step S3 further includes: obtaining the part category of the corresponding stamped part in each frame of part images based on a target detection algorithm; such as... Figure 3 As shown, the corresponding part category can be marked at the part bounding box of the part image.

[0040] Specifically, in step S4, by setting the part overlap threshold, it is possible to avoid the part bounding boxes corresponding to different types of stamping parts being associated between adjacent frames. Therefore, when associating the hole bounding boxes in the subsequent process, it is also possible to avoid the association of screw holes on different types of stamping parts.

[0041] Specifically, in step S53, by setting various overlap thresholds, it is also possible to prevent the bounding boxes of stamped parts of different categories from being associated between different viewpoints.

[0042] Specifically, in step S61, each category of stamped parts has a corresponding top-view template image, top-left view template image, top-right view template image, and oblique-far view template image; in steps S62 and S63, the similarity overlap threshold and matching overlap threshold can also prevent stamped parts of different categories from being matched.

[0043] In the specific implementation of the above embodiments, the technical features can be combined in any non-contradictory way. For the sake of brevity, not all possible combinations of the above technical features are described. However, as long as the combination of these technical features is not contradictory, it should be considered to be within the scope of this specification.

[0044] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A vision-based method for detecting defects in stamped parts, characterized in that, Includes the following steps: S1. Camera setup: A multi-view camera is installed on the conveyor belt used to transport stamped parts, and the shooting range of the multi-view camera can cover the entire conveyor belt. S2. Video stream synchronization: Synchronize the video streams from various perspectives captured by the multi-view cameras in time. S3. Boundary box detection: In each frame of part image captured by the camera from its respective viewpoint, the part boundary box of the corresponding stamping part is obtained based on the object detection algorithm, and the hole boundary box of each screw hole on the stamping part is obtained. S4. Target tracking: In the part images captured by the cameras from their respective perspectives, the bounding boxes of the parts belonging to the same stamping part in each frame of part images are correlated accordingly, and the bounding boxes of the holes belonging to the same screw hole position under the same stamping part in each frame of part images are also correlated accordingly. S5. Part Recognition: In the part images captured by the cameras from their respective perspectives, the target part bounding box is obtained based on the relative position of the part bounding box on the conveyor belt; the target part bounding boxes belonging to the same stamping part in the part images from all perspectives in the same frame are associated accordingly. S6. Multi-view template matching: Obtain the template image of the stamped part according to each viewpoint; select multiple effective part images with a certain degree of similarity to the template image under each viewpoint as target detection images; match the target detection images one by one with the corresponding hole bounding boxes of the template image. When there is a matching hole bounding box in any target detection image under any viewpoint and the viewpoint template image of the corresponding viewpoint, it is considered that the screw hole corresponding to the hole bounding box exists. If no matching hole bounding box is found in the target detection image at the corresponding viewpoint in the viewpoint template images of all views, then the screw hole corresponding to the hole bounding box is considered to be missing.

2. The vision-based defect detection method for stamped parts according to claim 1, characterized in that, In step S1, the multi-view camera includes one camera each positioned above, on the upper left, on the upper right, and at an oblique distance from the corresponding conveyor belt. The camera positioned above is located within the projection area directly above the conveyor belt. The upper left and upper right cameras are positioned on either side of the camera positioned above, and all three cameras are located in the same vertical plane, which is perpendicular or approximately perpendicular to the conveying direction of the conveyor belt. The camera at an oblique distance is located within the projection area directly above the conveyor belt.

3. The vision-based defect detection method for stamped parts according to claim 1, characterized in that, In step S4, the intersection-union ratio (IUU) between the bounding boxes of the parts in the current frame and the previous frame is calculated using a target tracking algorithm to obtain the part overlap. For parts with an overlap greater than or equal to a preset part overlap threshold, the bounding boxes of the corresponding current frame and the previous frame are associated. The cross-union ratio (CUI) between the hole bounding boxes in the current frame and the previous frame of the part image is calculated using a target tracking algorithm to obtain the hole overlap. For hole overlap values ​​greater than or equal to a preset hole overlap threshold, the corresponding hole bounding boxes in the current frame and the previous frame are associated.

4. The vision-based defect detection method for stamped parts according to claim 3, characterized in that, In target tracking in step S4: If a hole bounding box is not associated with a corresponding location in any of the n consecutive frames of part images, then the hole bounding box is marked as missing, and subsequent steps will not be executed for hole bounding boxes marked as missing; where n≥3; If a hole bounding box that did not appear in the previous frame is associated with a part image in an adjacent frame, then the hole bounding box is given a new mark, and the hole bounding box with the new mark will be subject to subsequent steps.

5. The vision-based defect detection method for stamped parts according to claim 2, characterized in that, Step S5 includes the following steps: S51. Based on the center point position of the part bounding box output by the target detection algorithm, allocate state variables according to the relative position on the conveyor belt under the corresponding view camera; wherein, the state variables include the state box that is located on the conveyor belt and set along the conveying direction of the conveyor belt, the state box that has entered, and the state box that has left. S52. The bounding boxes of parts that have entered the state box or have already entered the state box are identified as the target part bounding boxes, and each frame of part image after obtaining the target part bounding box is taken as a valid part image. S53. Group the valid part images of all viewpoints in the same frame into pairs, and then calculate the overlap ratio between the bounding boxes of the target parts in each group to obtain the overlap. When the overlap ratio is greater than or equal to the corresponding set overlap threshold, the bounding boxes of the target parts are considered to belong to the same stamping part, and the bounding boxes of the target parts belonging to the same stamping part are associated.

6. The vision-based defect detection method for stamped parts according to claim 5, characterized in that, Step S53 includes the following steps: S531. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the upper left and upper right views in the same frame to obtain the left and right overlap, and compare the left and right overlap with the preset left and right overlap threshold. S532. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the upper left or upper right view and the top view in the same frame to obtain the side-to-top overlap, and compare the side-to-top overlap with the preset side-to-top overlap threshold. S533. Calculate the cross-union ratio between the target part bounding boxes in the effective part images of the oblique far-side view and the direct upward view in the same frame to obtain the far-positive overlap, and compare the far-positive overlap with the preset far-positive overlap threshold. S534. When the overlap calculated in each of steps S531, S532, and S533 is greater than or equal to the corresponding preset overlap threshold, it is considered that the bounding boxes of the target parts under each viewpoint belong to the same stamping part, and then the bounding boxes of the target parts belonging to the same stamping part are associated.

7. The vision-based defect detection method for stamped parts according to claim 6, characterized in that, In steps S532 and S533, the bounding box of the target part from one viewpoint is mapped to the coordinate system of another viewpoint using a homography transformation matrix. Then, the overlap ratio between the transformed bounding box of the target part from one viewpoint and the bounding box of the target part from the other viewpoint is calculated.

8. The vision-based defect detection method for stamped parts according to any one of claims 2 to 7, characterized in that, Step S6 includes the following steps: S61. Based on the top, top left, top right, and oblique far-angle views, respectively obtain the template image of the template stamping part from the top, top left, top right, and oblique far-angle views. S62. Under each viewpoint, calculate the cross-union ratio between the effective part image in each frame and the target part bounding box in the template image of that viewpoint to obtain the similarity overlap. Then, use the effective part image with a similarity overlap greater than or equal to the preset similarity overlap threshold as the target detection image. S63. Under each viewpoint, the target detection image is matched one by one with the corresponding hole position bounding box of the template image of that viewpoint: the intersection-union ratio between the current target detection image and the hole position bounding box in the template image of that viewpoint is calculated to obtain the matching overlap. If the matching overlap is greater than or equal to the preset matching overlap threshold, it means that the screw hole position corresponding to the hole position bounding box exists. Then, when processing the next target detection image, the hole position bounding boxes that have been matched are removed and the matching overlap of the remaining hole position bounding boxes is calculated, until the hole position bounding boxes of all target detection images have undergone matching overlap calculation. If no hole bounding box with a matching overlap threshold greater than or equal to the preset threshold is obtained in the target detection image under the corresponding viewpoint template image in all viewpoints, it indicates that the screw hole corresponding to the hole bounding box is missing.

9. The vision-based defect detection method for stamped parts according to any one of claims 1 to 7, characterized in that, Step S3 further includes: obtaining the part category of the corresponding stamping part in each frame of part image based on the target detection algorithm.

10. The vision-based defect detection method for stamped parts according to any one of claims 1 to 7, characterized in that, Step S2 includes the following steps: S21. Combine all frame component images captured by video streams from different viewpoints at the same time into a frame packet; S22. Extract the part image of one frame from each frame packet for inspection. If the part image of the frame is normal, record the timestamp of the corresponding frame packet and store it in the corresponding buffer. S23. Compare the timestamps of the frame packets in each buffer, and obtain the frame packet with the earliest timestamp as the reference base timestamp; S24. Calculate the time difference between the timestamps of other frame packets and the reference timestamp. For lagging timestamps, the frame packets in their corresponding buffers will not be output temporarily until their timestamps catch up with the reference timestamps. S25. Once the timestamps in all buffers are aligned, output synchronized frame packets.

Images