Large sheet metal part punching defect detection method and system based on deep learning

Abstract

The invention belongs to the related technical field of surface defect detection, and discloses a large sheet metal part punching defect detection method and system based on deep learning, and the method comprises the steps: (1) collecting a multi-view image of a to-be-detected sheet metal part; (2) training a deep learning hole site target detection model by using the image, and further obtaining position data of a hole site on the to-be-detected sheet metal part based on the model and the acquired image; (3) hole site matching between different camera images is carried out through a multi-view geometric principle, and repeated hole sites in different images are detected; and (4) removing repeated hole sites from all the hole sites obtained by the deep learning hole site target detection model, counting, comparing an obtained counting result with the number of designed hole sites or comparing the positions of the hole sites with corresponding hole site design positions, and further judging whether hole site punching omission exists or not according to a comparison result. According to the invention, the real-time performance and the accuracy of the missed punching detection of the stamping part are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field related to surface defect detection, and more specifically, to a method and system for detecting punching defects of large sheet metal parts based on deep learning. Background Art

[0002] Sheet metal parts are widely used in high-end manufacturing industries such as aerospace and automobiles due to their good forming properties, low density, and high strength. Among the characteristics of sheet metal stamping, hole position is the most common and important forming feature of sheet metal parts, and is the basis for ensuring the smooth assembly and accurate appearance of sheet metal parts. Due to the lack of stamping process, wear of die punches, insufficient punching depth and other reasons, the stamping process of sheet metal parts may produce missing punching defects. Missing punching can cause its assembly to fail smoothly, thus failing to meet the design function, and also indicates that the die has reached the wear limit. Therefore, in the stamping process, accurate and timely detection of missing punching is of great significance to ensure the production quality of sheet metal parts.

[0003] At present, the detection of missing holes in sheet metal parts is mainly divided into two methods: manual detection and detection based on machine vision. The manual detection method relies on manual visual inspection and the assistance of inspection tools. This detection method is inefficient, prone to counting errors, and difficult to detect missing holes in time. The detection method based on machine vision generally detects the hole position target through the sheet metal image, counts the number of holes and compares it with the design number, so as to detect the missing hole defects. Limited by the characteristics of sheet metal parts, such as large structural dimensions, complex morphology, severe surface reflection, and many interference factors in the production environment, as well as the lack of accuracy of traditional punching hole detection algorithms, the detection accuracy of missing holes in sheet metal parts is still relatively low. And limited by the camera field of view, the current detection size range of missing holes in sheet metal parts is limited, which is not suitable for large sheet metal parts. Summary of the invention

[0004] In response to the above defects or improvement needs of the prior art, the present invention provides a method and system for detecting missing punching holes in large sheet metal parts based on deep learning, which aims to solve the problems of low efficiency and accuracy in the existing missing punching hole detection of large sheet metal parts.

[0005] To achieve the above object, according to one aspect of the present invention, a method for detecting punching defects of large sheet metal parts based on deep learning is provided, and the method comprises the following steps:

[0006] (1) Using a multi-camera array to capture multi-view images of the sheet metal to be tested, the multiple images obtained can cover all the hole positions of the sheet metal to be tested;

[0007] (2) using the obtained multiple images to train a deep learning hole position target detection model, and then obtaining the position data of the hole positions on the sheet metal part to be tested based on the trained deep learning hole position target detection model and the collected image of the sheet metal part to be tested; wherein the input of the deep learning hole position target detection model is the image of the sheet metal part to be tested, and the output is the position of the hole position on the sheet metal part to be tested;

[0008] (3) Hole positions are matched between images from different cameras using the principle of multi-view geometry to detect repeated hole positions in different images;

[0009] (4) Count all the holes obtained by the deep learning hole target detection model after removing the duplicate holes and compare the counting results with the designed number of holes or compare the positions of the holes with the corresponding designed positions of the holes, and then judge whether there are any hole leaks based on the comparison results.

[0010] Furthermore, before image acquisition, it is necessary to calibrate the internal and external parameters between the dual camera sets and calculate the essential matrix E between the dual cameras. The essential matrix E is used to calculate the epipolar lines.

[0011] Furthermore, the formula for calculating the essential matrix E is:

[0012]

[0013] Among them, t is the translation vector of the dual-camera extrinsic parameters, t× is the antisymmetric matrix calculated from t, and R is the rotation matrix of the dual-camera extrinsic parameters.

[0014] Furthermore, the deep learning hole position target detection model is a convolutional neural network for target detection, which includes three parts: a backbone network, a Neck layer and a detection head.

[0015] Furthermore, the backbone network adopts a ResNet network with a deep residual structure.

[0016] Furthermore, the Neck layer uses a feature pyramid network for feature fusion, which extracts and fuses image features at different scales and reuses feature maps at different levels.

[0017] Furthermore, the detection head adopts a decoupling head structure, which is used to adjust the backbone network according to the hole position detection effect.

[0018] Furthermore, the specific steps of hole position matching are:

[0019] S4.1, randomly select two images with overlapping fields of view, where the hole center point set of the left camera image is P left = {P l |P l ∈P left,l=1,2,…,m}, the hole center point set under the right camera image is P right = {P r |P r ∈P right ,l=1,2,…,n};

[0020] S4.2, traverse the hole center image point P under the left camera image l , calculate P from the essential matrix l The epipolar line l in the right camera image r , traverse all the hole center points in the right camera image and select the outer pole line l r The hole center image point P is further calculated for each hole center point whose distance is less than the set threshold. l The epipolar line l under the left camera system l , if P l To l If the distance is also less than the set threshold, P l With each P r As pre-registered hole pairs;

[0021] S4.3, when the same P l There are multiple P r Correspondingly, calculate the center points P of multiple right camera images r Is there a point in the image that is uniquely matched by the center point of a left camera image hole? If so, remove the P r point; if there is still ambiguity, calculate each P r The left-right relationship of each matched hole position pair along the polar line direction is comprehensively considered. l and multiple P r The pixel size ratio of the hole position P is obtained by eliminating ambiguity. l The only corresponding P r Or exclude the hole position matching relationship;

[0022] S4.4, according to the size of the overlapping hole position range between the images, the hole position matching results between all image combinations with overlapping hole position areas are calculated in descending order.

[0023] The present invention also provides a large sheet metal punching defect detection system based on deep learning, the system includes a memory and a processor, the memory stores a computer program, and the processor executes the large sheet metal punching defect detection method based on deep learning as described above when executing the computer program.

[0024] The present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the large sheet metal punching defect detection method based on deep learning as described above.

[0025] In general, compared with the prior art, the above technical solution conceived by the present invention has the following beneficial effects:

[0026] 1. Through the deep learning hole position target detection model, vision-based punching defect detection can improve the real-time and accuracy of punching defect detection of stamped parts, improve the intelligence level of related quality inspection environment, and meet the production cycle requirements and punching process optimization data needs.

[0027] 2. Through multi-view hole position matching based on multi-view geometry, the problem of incomplete coverage of a single view for large sheet metal parts and the problem of easy omission or miscounting of holes at the edge of the image can be effectively solved, thus realizing intelligent, efficient and accurate detection of punching defects of large sheet metal parts.

[0028] 3. The backbone network adopts a ResNet network with a deep residual structure. Through the jump connection of the feature map, the backbone network can learn a more robust feature representation, improve the detection performance and generalization performance of the deep learning hole target detection model, and because the residual structure allows information to jump in the network, the gradient can be propagated faster during the back propagation process, thereby accelerating the convergence of the deep learning hole target detection model.

[0029] 4. The Neck layer uses a feature pyramid network for feature fusion. By extracting and fusing image features at different scales, reusing feature maps at different levels, and generating feature representations with semantic information on low-level feature maps, the detection performance of holes of various scales of sheet metal parts is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of a large sheet metal punching defect detection method based on deep learning provided by the present invention;

[0031] Figure 2 It is a schematic diagram of a large sheet metal punching defect detection system based on deep learning provided in an embodiment of the present invention.

[0032] In all the drawings, the same reference numerals are used to represent the same elements or structures, wherein: 1 - CCD or CMOS camera, 2 - lens, 3 - large sheet metal part to be tested. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0034] See also Figure 1 The present invention provides a large sheet metal punching defect detection method based on deep learning, in order to achieve comprehensive, accurate and efficient missed punching detection of large and complex sheet metal parts, and provide data support for stamping process optimization and stamping parts quality assurance.

[0035] The detection method mainly comprises the following steps:

[0036] Step 1: calibrate the internal and external parameters between the two-camera set, and calculate the essential matrix E between the two cameras. The essential matrix E is used to calculate the epipolar line.

[0037] Camera calibration: When using a multi-camera array for image acquisition, use a checkerboard or dot array calibration plate to calibrate the internal and external parameters. When using a mechanical motion mechanism for image acquisition, the camera external parameters can be calculated based on the motion parameters.

[0038] The formula for calculating the essential matrix E is:

[0039]

[0040] Among them, t is the translation vector of the dual-camera extrinsic parameters, t× is the antisymmetric matrix calculated from t, and P is the rotation matrix of the dual-camera extrinsic parameters.

[0041] In one embodiment, dual cameras in a multi-camera array fixed by a fixture are calibrated, and internal and external parameters are calibrated using a checkerboard or dot array calibration plate; when a mechanical motion mechanism is used for image acquisition, only the internal parameters can be calibrated using the checkerboard or dot array calibration plate, and the external parameters between the dual cameras can be calculated based on the motion parameters.

[0042] Step 2: Collect multi-view images of the sheet metal part to be tested, and the obtained multiple images can cover all the hole positions of the sheet metal part to be tested.

[0043] Among them, multi-angle image acquisition is performed through a multi-camera array or a mechanical motion platform to obtain multiple images that can cover all the hole positions of the sheet metal to be tested.

[0044] In one embodiment, the multi-camera array uses a tooling to achieve stable installation of the multi-cameras to ensure that the relative posture of the cameras, that is, the external parameters, remain unchanged under the vibration environment of the factory, thereby laying the foundation for hole position matching between sheet metal images, such as using a CCD or CMOS camera 1 and a lens 2 to obtain multi-view images of a large sheet metal part 3 to be tested. In addition, a mechanical motion platform can be used to carry a single camera to collect images of sheet metal parts from multiple perspectives in multiple postures.

[0045] Step three, use the obtained multiple images to train the deep learning hole target detection model, and then obtain the position data of the holes on the sheet metal to be tested based on the trained deep learning hole target detection model and the collected image of the sheet metal to be tested; wherein the input of the deep learning hole target detection model is the image of the sheet metal to be tested, and the output is the position of the hole on the sheet metal to be tested.

[0046] The output data of the deep learning hole target detection model include the hole center coordinates, hole width and height. The deep learning hole target detection model is a convolutional neural network for target detection, which includes a backbone network, a Neck layer, and a detection head. Among them, the backbone network adopts a ResNet network with a deep residual structure. Through the jump connection of the feature map, the backbone network can learn a more robust feature representation, improve the detection performance and generalization performance of the deep learning hole target detection model, and because the residual structure allows information to jump in the network, the gradient can be propagated faster during the back propagation process, thereby accelerating the convergence of the deep learning hole target detection model.

[0047] Among them, the Neck layer uses a feature pyramid network for feature fusion. By extracting and fusing image features at different scales, reusing feature maps at different levels, and generating feature representations with semantic information on low-level feature maps, the detection performance of holes of various scales of sheet metal parts is improved.

[0048] Among them, the detection head adopts a decoupled head structure, which can adjust the backbone network according to the hole detection effect, thereby speeding up the construction speed of the deep learning hole target detection network model.

[0049] Step 4: Match the hole positions between different camera images using the principle of multi-view geometry to detect repeated hole positions in different images.

[0050] Take any two images with overlapping parts, and based on multiple constraints such as epipolar constraints, two-dimensional size constraints, uniqueness constraints, and position constraints, use the principle of multi-view geometry to match the holes between images from different cameras and check the same holes in different images.

[0051] Among them, the specific steps of hole position matching are:

[0052] S4.1, randomly select two images with overlapping fields of view, where the hole center point set of the left camera image is P left = {P l |P l ∈P left ,l=1,2,…,m}, the hole center point set under the right camera image is P right = {P r |P r ∈P right ,l=1,2,…,n}.

[0053] S4.2, traverse the hole center image point P under the left camera image l , calculate P from the essential matrix l The epipolar line l in the right camera image r , traverse all the hole center points in the right camera image and select the outer pole line l r The hole center image point P is further calculated for each hole center point whose distance is less than the set threshold. l The epipolar line l under the left camera system l , if P l To l If the distance is also less than the set threshold, P l With each P r as pre-registered hole pairs.

[0054] S4.3, when the same P l There are multiple P r Correspondingly, calculate the center points P of multiple right camera images r Is there a point in the image that is uniquely matched by the center point of a left camera image hole? If so, remove the P r point; if there is still ambiguity, calculate each P r The left-right relationship of each matched hole position pair along the polar line direction is comprehensively considered. l and multiple P r The pixel size ratio of the hole position P is obtained by eliminating ambiguity. l The only corresponding P r Or eliminate the hole matching relationship.

[0055] S4.4, according to the size of the overlapping hole position range between the images, the hole position matching results between all image combinations with overlapping hole position areas are calculated in descending order.

[0056] Step five, count all the holes obtained by the deep learning hole target detection model after removing the duplicate holes and compare the counting results with the designed number of holes, or compare the positions of the holes with the corresponding designed positions of the holes, and then judge whether there are any missed holes based on the comparison results.

[0057] In one embodiment, the hole position counting is to count all the hole positions of the sheet metal after excluding overlapping matching holes (i.e., repeated holes). After the hole position counting, it is necessary to compare with the designed hole positions to determine whether there are any sheet metal holes that are missed in the stamping process.

[0058] The embodiment of the present invention also provides a sheet metal part punching defect detection system based on deep learning, comprising: a multi-camera array, a computer-readable storage medium and a processor. The multi-camera array is used to collect images of all hole positions of the sheet metal part from multiple angles to achieve image coverage of all hole positions of the sheet metal part; the computer-readable storage medium is used to store executable instructions to record the above algorithm flow; the processor is used to read the executable instructions stored in the computer-readable storage medium to execute the large sheet metal part punching defect detection method based on deep learning as described above.

[0059] like Figure 2 As shown, a sheet metal punching defect detection system based on deep learning provided by one embodiment of the present invention includes a CCD or CMOS camera 1, a lens 2, a large sheet metal part 3 to be tested, and a tooling for fixing multiple cameras. The camera is used to collect images of local sheet metal parts at various viewing angles; the tooling is used to firmly fix multiple cameras; and the sheet metal part to be tested is located within the camera field of view and depth of field.

[0060] The present invention also provides a large sheet metal punching defect detection system based on deep learning, the system includes a memory and a processor, the memory stores a computer program, and the processor executes the large sheet metal punching defect detection method based on deep learning as described above when executing the computer program.

[0061] The present invention also provides a computer-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the large sheet metal punching defect detection method based on deep learning as described above.

[0062] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A large sheet metal punching defect detection method based on deep learning, characterized in that: The method comprises the following steps: (1) Using a multi-camera array to capture multi-view images of the sheet metal to be tested, the multiple images obtained can cover all the hole positions of the sheet metal to be tested; (2) using the obtained multiple images to train a deep learning hole position target detection model, and then obtaining the position data of the hole positions on the sheet metal part to be tested based on the trained deep learning hole position target detection model and the collected image of the sheet metal part to be tested; wherein the input of the deep learning hole position target detection model is the image of the sheet metal part to be tested, and the output is the position of the hole position on the sheet metal part to be tested; (3) Hole positions are matched between images from different cameras using the principle of multi-view geometry to detect repeated hole positions in different images; (4) Count all the holes obtained by the deep learning hole target detection model after removing the duplicate holes and compare the counting results with the designed number of holes or compare the positions of the holes with the corresponding designed positions of the holes, and then judge whether there are any hole leaks based on the comparison results.

2. The large sheet metal punching defect detection method based on deep learning as claimed in claim 1, characterized in that: Before image acquisition, the internal and external parameters between the dual-camera units need to be calibrated, and the essential matrix E between the dual cameras needs to be calculated. The essential matrix E is used to calculate the epipolar lines.

3. The large sheet metal punching defect detection method based on deep learning as claimed in claim 2, characterized in that: The formula for calculating the essential matrix E is: Among them, t is the translation vector of the dual-camera extrinsic parameters, t× is the antisymmetric matrix calculated from t, and R is the rotation matrix of the dual-camera extrinsic parameters.

4. The large sheet metal punching defect detection method based on deep learning as claimed in claim 1, characterized in that: The deep learning hole position target detection model is a convolutional neural network for target detection, which includes three parts: a backbone network, a Neck layer and a detection head.

5. The large sheet metal punching defect detection method based on deep learning as claimed in claim 4, characterized in that: The backbone network uses a ResNet network with a deep residual structure.

6. The large sheet metal punching defect detection method based on deep learning as claimed in claim 4, characterized in that: The Neck layer uses a feature pyramid network for feature fusion, extracting and fusing image features at different scales and reusing feature maps at different levels.

7. The large sheet metal punching defect detection method based on deep learning as claimed in claim 4, characterized in that: The detection head adopts a decoupling head structure, which is used to adjust the backbone network according to the hole position detection effect.

8. The large sheet metal punching defect detection method based on deep learning according to any one of claims 1 to 7, characterized in that: The specific steps for hole matching are: S4.1, randomly select two images with overlapping fields of view, where the hole center point set of the left camera image is P left = {P l |P l ∈P left ,l=1,2,…,m}, the hole center point set under the right camera image is P right = {P r |P r ∈P right ,l=1,2,…,n}; S4.2, traverse the hole center image point P under the left camera image l , calculate P from the essential matrix l The epipolar line l in the right camera image r , traverse all the hole center points in the right camera image and select the outer pole line l r The hole center image point P is further calculated for each hole center point whose distance is less than the set threshold. l The epipolar line l under the left camera system l , if P l To l If the distance is also less than the set threshold, P l With each P r As pre-registered hole pairs; S4.3, when the same P l There are multiple P r Correspondingly, calculate the center points P of multiple right camera images r Is there a point in the image that is uniquely matched by the center point of a left camera image hole? If so, remove the P r point; if there is still ambiguity, calculate each P r The left-right relationship of each matched hole position pair along the polar line direction is comprehensively considered. l and multiple P r The pixel size ratio of the hole position P is obtained by eliminating ambiguity. l The only corresponding P r Or exclude the hole position matching relationship; S4.4, according to the size of the overlapping hole position range between the images, the hole position matching results between all image combinations with overlapping hole position areas are calculated in descending order.

9. A large sheet metal punching defect detection system based on deep learning, characterized in that: The system includes a memory and a processor, the memory stores a computer program, and the processor executes the large sheet metal punching defect detection method based on deep learning as described in any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores machine-executable instructions. When the machine-executable instructions are called and executed by the processor, the machine-executable instructions prompt the processor to implement the large sheet metal punching defect detection method based on deep learning as described in any one of claims 1-8.

Images