A method and system for defect detection

Abstract

A method and system for defect detection. The method comprises: acquiring a 2D picture of an object to be detected; inputting the acquired 2D picture into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an intersection over union threshold improved layer by layer, and wherein the 2D defect mask comprises information about a defect type, a defect size and a defect position of a segmented defect region; and judging the segmented 2D defect mask based on predefined defect rules to output a defect detection result.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and system for defect detection. BACKGROUND

[0002] In the field of modern industrial manufacturing, defect detection of industrial products is a key part of quality detection of industrial products, which is very important for improving product processes and improving yield.

[0003] However, in traditional industrial manufacturing, defect detection of industrial products usually adopts manual observation, which is difficult to directly observe defects when the object to be detected is very small, and has the problems of high detection cost and low quality inspection efficiency. SUMMARY

[0004] In view of the above problems, the present application provides a method and system for defect detection, which can detect defects when the object to be detected is very small, significantly reduce the detection cost, and greatly improve the quality inspection efficiency.

[0005] In a first aspect, the present application provides a method for defect detection, comprising: collecting a two-dimensional (2D) picture of an object to be detected; inputting the collected 2D picture into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an increasing intersection over union (IoU) threshold at each level, and the 2D defect mask includes information about the defect type, defect size and defect position of the segmented defect region; and judging the segmented 2D defect mask based on a predefined defect rule to output a defect detection result.

[0006] In the technical scheme of the present application, by designing an instance segmentation network based on a multi-level feature extraction architecture, better segmentation and detection results can be achieved in a semantic segmentation model where positive samples are particularly difficult to obtain or particularly few, achieving no missed killing and greatly reducing the probability of over-killing.

[0007] In some embodiments, the method further comprises: collecting a three-dimensional (3D) picture of an object to be detected; pre-processing the collected 3D picture to obtain an image with depth information of the object to be detected; inputting the obtained image with depth information into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask comprises information about the defect depth of the segmented defect region; and fusing the segmented 2D defect mask and 3D defect mask based on a predefined defect rule to output a defect detection result. By combining the results of 2D and 3D detection, the segmentation result about the depth for the actual requirement is obtained, so that the detection result is more accurate, and when the defect is not obvious in the 2D image form, the no-miss killing is realized by fusing the 2D and 3D detection results, and the probability of over-killing is greatly reduced.

[0008] In some embodiments, fusing the segmented 2D defect mask and 3D defect mask based on a predefined defect rule further comprises: using a coordinate transformation matrix between the collected 2D picture and 3D picture to align the 2D picture and 3D picture at the pixel level to obtain an aligned picture; filling the segmented 2D defect mask and 3D defect mask into the aligned picture; and fusing the segmented 2D defect mask and 3D defect mask on the aligned picture based on a predefined defect rule to output a defect detection result. By aligning and filling the corresponding mask when fusing the 2D and 3D segmentation results, defects can be more intuitively and accurately detected.

[0009] In some embodiments, the multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, wherein the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, 0.3-0.45 in the second level, and 0.5-0.7 in the third level. By setting a lower IoU threshold in the first stage when sampling positive and negative samples, the overfitting problem caused by the imbalance of positive and negative samples is effectively avoided, and at the same time, the IoU threshold is gradually increased layer by layer to continuously refine the sampling, thereby obtaining higher feature extraction accuracy, so that the result of defect detection is more accurate.

[0010] In some embodiments, the IoU threshold is set to 0.3 in the first level, 0.4 in the second level, and 0.5 in the third level. By setting the IoU threshold of each level in the multi-level feature extraction instance segmentation network to 0.3, 0.4 and 0.5 respectively, the positive and negative samples can be sampled more finely layer by layer, so that the result of defect detection is more accurate.

[0011] In some embodiments, the defect mask output by the multi-level feature extraction instance segmentation network is obtained based on a weighted average of the defect masks output by the instance segmentation network at each level. By performing a weighted average of the defect masks output by the network at each level, the result of defect detection is more accurate, and the probability of overkill is greatly reduced.

[0012] In some embodiments, judging the segmented defect mask based on the predefined defect rule to output the defect detection result further comprises: outputting a defect class as the defect detection result when the size or depth of the segmented defect region is greater than a predefined threshold for the defect type of the defect region. By applying different defect rules for judging different types of defects, the threshold of defect specification during detection can be customized, making the detection algorithm more flexible.

[0013] In some embodiments, the defect types include pit defects and bump defects, and judging the segmented defect mask based on the predefined defect rule to output the defect detection result further comprises: in the case that the defect type of the defect region is a pit defect, if the segmented 2D defect mask and the 3D defect mask both include the defect region, or the depth of the defect region is greater than a predefined threshold for pit defects, outputting a defect class as the defect detection result; or in the case that the defect type of the defect region is a bump defect, if the segmented 2D defect mask and the 3D defect mask both include the defect region, and the size and depth of the defect region are greater than a predefined threshold for bump defects, outputting a defect class as the defect detection result. By applying different defect rules for judging different categories of pit defects and bump defects, the threshold of defect specification during detection can be customized, making the detection algorithm more flexible.

[0014] In a second aspect, the present application provides a system for defect detection, comprising: an image acquisition module configured to acquire a two-dimensional (2D) picture of an object to be detected; a defect segmentation module configured to input the acquired 2D picture into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an increasing intersection over union (IoU) threshold at each level, and wherein the 2D defect mask includes information about the defect type, defect size, and defect location of the segmented defect region; and a defect judgment module configured to judge the segmented 2D defect mask based on a predefined defect rule to output a defect detection result.

[0015] In the technical scheme of the embodiment of the present application, by designing an instance segmentation network based on a multi-level feature extraction architecture, better segmentation and detection results can be achieved in a semantic segmentation model in which positive samples are particularly difficult to obtain or particularly few, no missed killing is realized, and the probability of over-killing is greatly reduced.

[0016] In some embodiments, the image acquisition module is further configured to acquire a three-dimensional (3D) picture of the object to be detected; the defect segmentation module is further configured to: pre-process the acquired 3D picture to obtain an image with depth information of the object to be detected; input the obtained image with depth information into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask includes information about the defect depth of the segmented defect region; and the defect judgment module is further configured to fuse and judge the segmented 2D defect mask and 3D defect mask based on a predefined defect rule to output a defect detection result. By combining the results of 2D and 3D detection, the segmentation result about depth for actual needs is obtained, so that the detection result is more accurate, and when the defect is not obvious in the 2D image form, no missed killing is realized by fusing the 2D and 3D detection results, and the probability of over-killing is greatly reduced.

[0017] In some embodiments, the defect judgment module is further configured to: use a coordinate transformation matrix between the acquired 2D picture and 3D picture to align the 2D picture and 3D picture at the pixel level to obtain an aligned picture; fill the segmented 2D defect mask and 3D defect mask into the aligned picture; and fuse and judge the segmented 2D defect mask and 3D defect mask on the aligned picture based on a predefined defect rule to output a defect detection result. When fusing the 2D and 3D segmentation results, by aligning and filling the corresponding masks, defects can be more intuitively and accurately detected.

[0018] In some embodiments, the multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, wherein the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, set to 0.3-0.45 in the second level, and set to 0.5-0.7 in the third level. When performing positive and negative sample sampling, by setting a lower IoU threshold in the first stage, the overfitting problem caused by the imbalance of positive and negative samples is effectively avoided, and at the same time, the IoU threshold is gradually increased layer by layer to continuously refine the sampling, thereby obtaining higher feature extraction accuracy, so that the result of defect detection is more accurate.

[0019] In some embodiments, the IoU threshold is set to 0.3 in the first level, to 0.4 in the second level, and to 0.5 in the third level. By setting the IoU threshold of each level in the multi-level feature extraction instance segmentation network to 0.3, 0.4 and 0.5 respectively, the positive and negative samples can be sampled more finely layer by layer, so that the result of defect detection is more accurate.

[0020] In some embodiments, the defect mask output by the multi-level feature extraction instance segmentation network is obtained based on a weighted average of the defect mask output by the instance segmentation network of each level. By performing a weighted average of the defect mask output by the network of each level, the result of defect detection is more accurate, and the probability of overkill is greatly reduced.

[0021] In some embodiments, the defect judgment module is further configured to output a defect class as a defect detection result when the size or depth of the segmented defect region is greater than a predefined threshold for the defect type of the defect region. By applying different defect rules for different types of defects for judgment, the threshold of defect specification during detection can be customized, making the detection algorithm more flexible.

[0022] In some embodiments, the defect type includes pit defects and protrusion defects, and the defect judgment module is further configured to: in the case that the defect type of the defect region is a pit defect, if the segmented 2D defect mask and 3D defect mask both include the defect region, or the depth of the defect region is greater than a predefined threshold for pit defects, output a defect class as a defect detection result; or in the case that the defect type of the defect region is a protrusion defect, if the segmented 2D defect mask and 3D defect mask both include the defect region, and the size and depth of the defect region are greater than a predefined threshold for protrusion defects, output a defect class as a defect detection result. By applying different defect rules for different categories of pit defects and protrusion defects for judgment, the threshold of defect specification during detection can be customized, making the detection algorithm more flexible.

[0023] In a third aspect, the present application provides a device for defect detection, the device comprising: a memory, the memory storing computer executable instructions; and at least one processor, the computer executable instructions, when executed by the at least one processor, causing the device to: collect a two-dimensional (2D) picture of an object to be detected; input the collected 2D picture into the trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an increasing IoU threshold at each level, and wherein the 2D defect mask comprises information about a defect type, a defect size, and a defect position of a segmented defect region; and judge the segmented 2D defect mask based on a predefined defect rule to output a defect detection result.

[0024] In the technical solution of the embodiments of the present application, by designing an instance segmentation network based on a multi-level feature extraction architecture, better segmentation and detection results can be achieved in a semantic segmentation model in which positive samples are particularly difficult to obtain or particularly few, no missed killing is achieved, and the probability of over-killing is greatly reduced.

[0025] In some embodiments, the computer executable instructions, when executed, further cause the at least one processor to: collect a three-dimensional (3D) picture of an object to be detected; the defect segmentation module is further configured to: pre-process the collected 3D picture to obtain an image of the object to be detected with depth information; input the obtained image with depth information into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask comprises information about the depth of a segmented defect region; and the defect judgment module is further configured to fuse and judge the segmented 2D defect mask and the 3D defect mask based on a predefined defect rule to output a defect detection result. By combining the results of 2D and 3D detection, a segmentation result about depth for actual needs is obtained, so that the detection result is more accurate, and when the defect is not obvious in the 2D image form, no missed killing is achieved by fusing the 2D and 3D detection results, and the probability of over-killing is greatly reduced.

[0026] In some embodiments, the computer-executable instructions, when executed, further cause the at least one processor to: perform pixel-level alignment between the 2D picture and the 3D picture to obtain an aligned picture by using a coordinate transformation matrix between the 2D picture and the 3D picture; fill the segmented 2D defect mask and 3D defect mask into the aligned picture; and perform fusion judgment on the segmented 2D defect mask and 3D defect mask on the aligned picture based on a predefined defect rule to output a defect detection result. By aligning and filling the corresponding masks when fusing the 2D and 3D segmentation results, defects can be more intuitively and accurately detected.

[0027] In some embodiments, the multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, wherein the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, 0.3-0.45 in the second level, and 0.5-0.7 in the third level. By setting a lower IoU threshold in the first stage when sampling positive and negative samples, the problem of overfitting caused by imbalance of positive and negative samples is effectively avoided, and at the same time, the IoU threshold is gradually increased layer by layer to continuously refine the sampling, thereby obtaining higher feature extraction accuracy and making the defect detection result more accurate.

[0028] In some embodiments, the IoU threshold is set to 0.3 in the first level, 0.4 in the second level, and 0.5 in the third level. By setting the IoU threshold of each level in the multi-level feature extraction instance segmentation network to 0.3, 0.4, and 0.5, respectively, more refined sampling of positive and negative samples can be performed layer by layer, making the defect detection result more accurate.

[0029] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions, when the computer-executable instructions are executed by a computing device, causing the computing device to implement the method for defect detection according to any of the preceding aspects.

[0030] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the following detailed description can be implemented according to the content of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the following detailed description of the specific embodiments of the present application is given. BRIEF DESCRIPTION OF DRAWINGS

[0031] So that the manner in which the above recited features of the present application can be understood in detail, a brief description of the embodiments can be had by reference to various implementations, some of which are illustrated in the drawings. It is to be noted, however, that the appended drawings illustrate only typical aspects of this application and are therefore not to be considered limiting of its scope, for the description can admit to other equally effective aspects.

[0032] Figure 1 is an example flowchart of a method for defect detection according to an embodiment of the present application;

[0033] Figure 2 is a structural diagram of an instance segmentation network based on a multi-level feature extraction architecture according to an embodiment of the present application;

[0034] Figure 3 is a diagram of a 2D defect segmentation result according to an embodiment of the present application;

[0035] Figure 4 is an example flowchart of a method for defect detection according to another embodiment of the present application;

[0036] Figure 5 is a diagram of a pseudo-color image according to an embodiment of the present application;

[0037] Figure 6 is a diagram of a 3D defect segmentation result according to an embodiment of the present application;

[0038] Figure 7 is an example flowchart of a 2D and 3D segmentation result fusion process according to an embodiment of the present application;

[0039] Figure 8 is a schematic architecture diagram of a system for defect detection according to an embodiment of the present application;

[0040] Figure 9 is a schematic architecture diagram of an apparatus for defect detection according to another embodiment of the present application.

[0041] Reference signs in the detailed description of the embodiments are listed as follows:

[0042] Defect detection system 800, image acquisition module 801, defect segmentation module 802, defect detection module 803;

[0043] Apparatus 900, memory 901, processor 902. DETAILED DESCRIPTION

[0044] The embodiments of the technical solutions of the present application will be described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore only serve as examples, and cannot limit the protection scope of the present application.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the use of the terms "including," "comprising," or "having" and variations thereof herein is intended to be broad and encompass the terms "consisting of" and "consisting essentially of" and variations thereof. The description herein of any embodiments, including preferred embodiments, is intended to be illustrative and is not intended to be limiting.

[0046] In the description of embodiments of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specifically defined. Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, or necessarily alternatives to other embodiments. It is explicitly and implicitly understood that embodiments described herein can be combined with other embodiments.

[0047] In the description of embodiments of the present application, the term "and / or" is merely an association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the associated objects.

[0048] At present, from the development of market situation, the application of battery is more and more widely. Energy storage battery is applied to energy storage power supply system of water power, fire power, wind power and solar power station, and power battery is also widely used in electric bicycles, electric motorcycles, electric vehicles and other electric vehicles, military equipment, aerospace and other fields. With the continuous expansion of the application field of power battery, the market demand is also increasing. Sealing nail welding is an indispensable link in the production process of power battery, and whether the sealing nail welding meets the standard directly affects the safety of the battery. The sealing nail welding area is called weld, and due to the changes of temperature, environment, laser angle and other factors during welding, there are often defects such as burst line (pit) and fusion bead on the weld.

[0049] With the development of machine vision and industrial automation, there are methods for automatically detecting defects based on artificial intelligence. However, in the case of very small objects to be detected or defects that are difficult to identify, using traditional feature extraction networks cannot detect defects well, and may easily result in missed detection or overkill. For example, the existing scheme uses Res2Net as a feature extraction module. However, due to the fact that some defects of the sealing nail are not very obvious and there are very small defects, using only Res2Net for training may easily result in an imbalance between the positive and negative samples, which may lead to a substandard model convergence, thereby limiting the detection capability of the model for difficult samples. In addition, the existing samples only use two-dimensional (2D) pictures for defect detection. However, the pits and protrusions in laser welding are defects caused by depth problems, and the 2D shape of the defects may not be very obvious. Therefore, for defects with depth, using only 2D pictures for detection may easily result in misjudgment.

[0050] Based on the above considerations, in order to solve the problem of missed detection or overkill for defects that are not obvious or only have depth in defect detection, the inventors have designed a multi-level backbone network structure that more accurately extracts defect features and a defect detection algorithm that fuses the results of 2D and 3D instance segmentation algorithms. By using an instance segmentation network based on a multi-level feature extraction architecture, the present application can still achieve good segmentation and detection results in a semantic segmentation model in which positive samples are particularly difficult to obtain or there are very few positive samples. In addition, by combining the results of 2D and 3D detection, the present application can customize the fusion of the results and obtain a model result related to depth that meets actual needs, thereby making the detection of defects with depth more accurate. Compared with previous algorithms, the present application reduces the missed detection of defects such as sealing nail pits and fusion beads to 0%, and reduces overkill to within 0.02%. In addition, the present application can also customize the adjustment of the threshold of the defect specification during detection, making the detection algorithm more flexible.

[0051] It can be appreciated that the present application can be applied to the field of defect detection combined with artificial intelligence (AI). The method and system for defect detection disclosed in the embodiments of the present application can be used for defect detection of sealing nail welds, but are not limited thereto, and can also be used for defect detection of other types of products in modern industrial manufacturing.

[0052] The following embodiments are described for convenience with respect to defect detection of sealing nail welds.

[0053] Figure 1 is an example flowchart of a method 100 for defect detection according to an embodiment of the present application. According to one embodiment of the present application, reference is made to Figure 1 and further reference is made to Figure 2 and Figure 3 whereinFigure 2 is a structure diagram of an instance segmentation network based on a multi-level feature extraction architecture according to an embodiment of the present application, Figure 3 is a schematic diagram of a 2D defect segmentation result according to an embodiment of the present application. The method 100 starts from step 101 of acquiring a two-dimensional (2D) picture of an object to be detected. In step 102, the acquired 2D picture is input into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an IoU threshold value increasing layer by layer, and the segmented 2D defect mask includes information about the defect type, defect size and defect position of the segmented defect region. In step 103, the segmented 2D defect mask is judged based on a predefined defect rule to output a defect detection result.

[0054] The network architecture of the defect segmentation model in step 102 is shown in Figure 2 , which is obtained by cascading three levels of instance segmentation networks. Generally, in the training stage of the instance segmentation network, the intersection over union (IoU) between each candidate box (proposal) and the ground truth (gt) is first calculated, and the candidate boxes are divided into positive samples (foreground) and negative samples (background) by setting an IoU threshold value (for example, usually 0.5), and the positive and negative samples are sampled so that their ratio as much as possible satisfies 1:3 (the total number of the two is usually 128), and then the candidate boxes (for example, usually 128) are sent to the ROI (region of interest) pooling layer, and finally the class classification and bounding box regression are performed. However, in the current application scenario, since the pit features are not very significant, directly setting the IoU threshold value to 0.5 will cause the following two problems: 1) the candidate boxes that meet this threshold condition are very few, which can easily lead to overfitting; and 2) a serious mismatch problem, since the current semantic segmentation structure itself has this problem, this problem is more serious when the IoU threshold value is set to be higher. The above two problems will lead to the decline of the defect feature extraction performance. Therefore, in the present application, the extraction of the candidate boxes is divided into three stages, wherein a smaller IoU threshold value is set in the first stage and the IoU threshold value is increased stage by stage. Figure 2 The 2D defect mask (Mask) output by the defect segmentation model in Figure 3 is shown in Figure 3 , the right lower corner gray area is the shape after coloring the segmented mask, which indicates the detected pit defect, and the segmentation result is relatively accurate.

[0055] Thus, in the defect detection, by using the instance segmentation network based on the multi-level feature extraction architecture, better segmentation and detection results can still be achieved in the instance segmentation model in which positive samples are particularly difficult to obtain or particularly few, no missed killing is realized, and the probability of over-killing is greatly reduced.

[0056] According to an embodiment of the present application, optionally, referring to Figures 4 to 6 , Figure 4 is an example flowchart of a method 400 for defect detection according to another embodiment of the present application, Figure 5 is a schematic diagram of a rendered pseudo-color image according to an embodiment of the present application, Figure 6 is a schematic diagram of a 3D defect segmentation result according to an embodiment of the present application. Steps 401 and 402 in method 400 are the same as steps 101 and 102 in method 100, which will not be repeated here. Further, in step 403, a three-dimensional (3D) picture of the object to be detected is collected. In step 404, the collected 3D picture is preprocessed to obtain an image with depth information of the object to be detected. In step 405, the obtained image with depth information is input into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask includes information about the defect depth of the segmented defect region. In step 406, the segmented 2D defect mask and the 3D defect mask are fused based on a predefined defect rule to output a defect detection result.

[0057] The 3D image of the object to be detected is usually collected by a 3D camera (e.g., a depth camera), and the collected 3D image can be saved in the form of a depth image, wherein the gray value of the depth image represents the depth information in the Z direction. The depth image and the 2D image usually have positional consistency, in other words, the pixel points on the depth image and the pixel points in the 2D image are one-to-one corresponding. In order to make the depth information more easily recognized by humans and machines, it is usually necessary to preprocess the collected depth image to obtain an image with depth information. In some cases, the photographed depth image can be rendered to obtain a pseudo-color image, as shown in Figure 5 For example, a predefined color mapping (e.g., using applycolormap (pseudo-color function) in OpenCV) can be used to pseudo-colorize the gray image, so that the depth information can be more easily perceived by humans and machines. Figure 2 The 3D defect mask (Mask) output by the defect segmentation model in Figure 6 is shown in Figure 6For clarity, the 3D defect mask segmented when detecting the sealing bead is shown superimposed on the original 3D picture, where the gray area in the upper left corner is the outline of the segmented mask, which indicates the depth map-based segmentation information of the relevant defect. The type, size, and depth of the defect can be directly determined from the result.

[0058] By combining the results of 2D and 3D detection, the segmentation result about depth for actual needs is obtained, so that the detection result is more accurate, and when the defect is not obvious in the 2D image form (for example, the pit is not significantly different from the normal bead in the 2D image form), the 2D and 3D detection results are fused to achieve no missed killing, and the probability of over-killing is greatly reduced.

[0059] According to an embodiment of the present application, optionally, with reference to Figure 3 , 4 and 6, and further with reference to Figure 7 , Figure 7 is an example flowchart of a 2D and 3D segmentation result fusion process 700 according to an embodiment of the present application. The steps of post-processing the 2D and 3D segmentation results start from block 701, using the coordinate transformation matrix between the captured 2D picture and 3D picture to align the 2D picture and 3D picture at the pixel level to obtain an aligned picture. In block 702, the segmented 2D defect mask and 3D defect mask are filled into the aligned picture. In block 703, the segmented 2D defect mask and 3D defect mask are fused on the aligned picture based on predefined defect rules to output a defect detection result.

[0060] Since the segmented 2D and 3D defect masks need to be fused for judgment, the original captured 2D and 3D pictures need to be aligned at the pixel level to locate the defect position, so that it can be determined whether the defect is detected in both 2D and 3D pictures when fusion judgment is performed. Specifically, first, the captured 2D and 3D pictures are preprocessed respectively, wherein the 2D picture is grayed to obtain a 2D grayscale picture, and a brightness picture is separated from the 3D picture. Subsequently, in the obtained 2D grayscale picture and 3D brightness picture, three same position points of the sealing bead are selected in advance, spatial transformation (for example, using opencv) is performed, the coordinate transformation equation is solved, the coordinate transformation matrix between the 2D picture and the 3D picture is obtained, and the 2D picture and the 3D picture are transformed and aligned at the pixel level by using the coordinate transformation matrix, thereby obtaining an aligned superimposed picture. After alignment, the segmented 2D defect mask and 3D defect mask can be filled into the corresponding positions of the aligned superimposed picture. Subsequently, the segmented 2D defect mask and 3D defect mask can be fused based on predefined defect rules to output a defect detection result. As Figure 7As shown in the middle, the 2D defect mask and the 3D defect mask include the same defect area (such as the gray marked area), and the type of the defect area is a pit, which indicates that the pit in the 2D detection result is re-detected on the 3D detection, and thus the defect detection result is output as a defect class.

[0061] As can be seen, when the 2D and 3D segmentation results are fused, by aligning and filling in the corresponding masks, the defect area can be located in the 2D and 3D pictures at the same time, so that the defect can be detected more intuitively and accurately.

[0062] According to an embodiment of the present application, optionally, with reference to Figure 2 The defect segmentation model for defect segmentation in the present application is obtained by training a multi-level feature extraction instance segmentation network, wherein the multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, wherein the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, 0.3-0.45 in the second level, and 0.5-0.7 in the third level.

[0063] As described before, in the present application, the extraction of candidate boxes is divided into three stages, wherein a small IoU threshold is set in the first stage, for example, the IoU threshold can be set to 0.2-0.4, in this stage, more positive samples than directly setting the IoU threshold to 0.5 can be obtained, but at the expense of a certain accuracy, which can be understood as preliminary screening. Then in the second stage, further sampling is performed on the basis of the candidate boxes extracted in the original stage, and the IoU threshold for sampling can be set to 0.3-0.45, thereby obtaining more refined sampling on the basis of the existing sampling. Then in the third stage, further sampling is performed on the results of the second stage, and the IoU threshold for sampling can be set to 0.5-0.7. Finally, the results of the third level are directly output to obtain the final instance segmentation results.

[0064] According to an embodiment of the present application, optionally, with reference to Figure 2 In training the defect segmentation model, the IoU threshold for positive and negative sample sampling is set to 0.3 in the first level, 0.4 in the second level, and 0.5 in the third level.

[0065] By setting a lower IoU threshold in the first stage when performing positive and negative sample sampling, the problem of overfitting caused by unbalanced positive and negative samples is effectively avoided, and by gradually increasing the IoU threshold layer by layer to perform increasingly refined sampling, higher feature extraction accuracy is obtained.

[0066] According to an embodiment of the present application, optionally, with reference to Figure 2The defect mask output by the multi-level feature extraction instance segmentation network can be obtained based on a weighted average of the defect masks output by each level of the instance segmentation network.

[0067] By performing a weighted average of the defect masks output by each level of the network, the result of defect detection is more accurate, and the probability of overkill is greatly reduced.

[0068] According to an embodiment of the present application, optionally, after defect segmentation, judging the segmented defect mask based on a predefined defect rule to output a defect detection result further comprises: when the size or depth of the segmented defect region is greater than a predefined threshold for the defect type of the defect region, outputting a defect class as the defect detection result.

[0069] In some examples, the predefined defect rule can include identifying the detection object as a defect class only when the defect is detected in both 2D and 3D detection results, and the size or depth of the defect region of the defect is greater than a predefined threshold, otherwise identifying as a normal class. In other examples, the predefined defect rule can include identifying the detection object as a defect class when the defect in the 2D detection result is re-detected in the 3D detection, or the depth of the defect is greater than a predefined threshold, otherwise identifying as a normal class. The predefined defect rule can be different for different types of defects.

[0070] According to an embodiment of the present application, optionally, the defect type includes pit defects and bump defects, and judging the segmented defect mask based on a predefined defect rule to output a defect detection result further comprises: in the case that the defect type of the defect region is a pit defect, if the segmented 2D defect mask and the 3D defect mask both include the defect region, or the depth of the defect region is greater than a predefined threshold for pit defects, then outputting a defect class as the defect detection result, or in the case that the defect type of the defect region is a bump defect, if the segmented 2D defect mask and the 3D defect mask both include the defect region, and the size and depth of the defect region are greater than a predefined threshold for bump defects, then outputting a defect class as the defect detection result.

[0071] For example, in a seal pin weld defect detection, for pit type defects, once the pit in the 2D detection result is re-detected on the 3D detection, or in the 3D detection, the size or depth of the defect region exceeds the threshold requirement in the specification for pit, then it is determined as a defect class (also known as NG). For bump type defects, after being detected in the 2D detection, in the 3D detection, both the size and depth of the defect region exceed the threshold requirement in the specification for bump, then it is determined as a defect class.

[0072] By applying different defect rules based on different types of defects for judgment, the threshold of defect specification during detection can be customized, and the detection algorithm is more flexible.

[0073] Figure 8 is a schematic architecture diagram of a system 800 for defect detection according to an embodiment of the present application. According to an embodiment of the present application, referring to Figure 8 , the system 800 at least includes an image acquisition module 801, a defect segmentation module 802 and a defect judgment module 803. The image acquisition module 801 can be used to acquire a two-dimensional (2D) picture of an object to be detected. The defect segmentation module 802 can be used to input the acquired 2D picture into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with IoU threshold increasing layer by layer, and wherein the 2D defect mask includes information about the defect type, defect size and defect location of the segmented defect region. The defect judgment module 803 can be used to judge the segmented 2D defect mask based on predefined defect rules to output a defect detection result.

[0074] Corresponding to the above-mentioned method 100 for defect detection, the system for defect detection according to the present application can still achieve better segmentation and detection results in an instance segmentation model where positive samples are particularly difficult to obtain or particularly few, realizes no missed killing, and greatly reduces the probability of over-killing by using an instance segmentation network based on a multi-level feature extraction architecture.

[0075] According to an embodiment of the present application, optionally, the image acquisition module 801 can be further configured to acquire a three-dimensional (3D) picture of the object to be detected. The defect segmentation module 802 can be further configured to: pre-process the acquired 3D picture to obtain an image of the object to be detected with depth information, input the obtained image with depth information into a trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask includes information about the defect depth of the segmented defect region. The defect judgment module 803 can be further configured to perform fusion judgment on the segmented 2D defect mask and the 3D defect mask based on the predefined defect rules to output a defect detection result.

[0076] Thus, when the defect is not obvious in the 2D image form, fusion of 2D and 3D detection results is realized to achieve no missed killing, and greatly reduce the probability of over-killing. Here, for specific details of the operations performed by each module of the system for defect detection according to the present application, see the description made above for Figures 1 to 7 , and here the details are not repeated for the sake of brevity.

[0077] Those skilled in the art can understand that the system of the present disclosure and each module thereof can be implemented in the form of hardware or software, and each module can be combined or combined in any suitable manner.

[0078] Figure 9 is a schematic architecture diagram of an apparatus 900 for defect detection according to another embodiment of the present application. According to an embodiment of the present application, referring to Figure 9 , the apparatus 900 can include a memory 901 and at least one processor 902. The memory 901 can store computer executable instructions. The computer executable instructions, when executed by the at least one processor 902, cause the apparatus 900 to perform the following operations: acquiring a two-dimensional (2D) picture of an object to be detected, inputting the acquired 2D picture into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with an increasing IoU threshold at each level, wherein the 2D defect mask includes information about the defect type, defect size and defect location of the segmented defect region, and judging the segmented 2D defect mask based on a predefined defect rule to output a defect detection result.

[0079] The memory 901 can include RAM, ROM, or a combination thereof. In some cases, the memory 901 can include, among other things, a BIOS, which can control basic hardware or software operation such as the interaction with peripheral components or devices. The processor 902 can include an intelligent hardware device, (e.g., a general-purpose processor, a DSP, a CPU, a microcontroller, an ASIC, an FPGA, a programmable logic device, a discrete gate or transistor logic component, a discrete hardware component, or any combination thereof).

[0080] Thus, corresponding to the above-mentioned method 100 for defect detection, the apparatus for defect detection according to the present application can still achieve better segmentation and detection results in an instance segmentation model where positive samples are particularly difficult to obtain or particularly few, realize no missed killing, and greatly reduce the probability of over-killing by using an instance segmentation network based on a multi-level feature extraction architecture. Here, the computer executable instructions, when executed by the at least one processor 902, cause the at least one processor 902 to perform various operations described above with reference to Figures 1-7 , which will not be repeated here for brevity.

[0081] The various illustrative blocks and modules described in connection with the disclosure herein can be implemented or performed with a general-purpose processor, a DSP, an ASIC, an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration).

[0082] The functions described herein can be implemented in hardware, software executed by a processor, firmware, or any combination thereof. If implemented in software executed by a processor, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Other examples and implementations are within the scope of the disclosure and appended claims. For example, due to the nature of software, functions described herein can be implemented using software executed by a processor, hardware, firmware, hardwiring, or combinations of any of these. Features implementing functions can also be physically located at various positions, including being distributed such that portions of functions are implemented at different physical locations.

[0083] While the present application has been described with reference to the preferred embodiments, the application is not limited by the preferred embodiments and various modifications can be made without departing from the scope of the application. In particular, the technical features mentioned in each of the embodiments can be combined in any manner as long as there is no structural conflict. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for defect detection, characterized in that: The method comprises: Acquire a two-dimensional (2D) image of the object to be detected; Inputting the acquired 2D image into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with a layer-by-layer increase in the intersection-over-union (IoU) threshold, wherein the 2D defect mask includes information about the defect type, defect size, and defect location of the segmented defect area; and Judging the segmented 2D defect mask based on predefined defect rules to output defect detection results; The method further comprises: Acquire a three-dimensional (3D) image of the object to be inspected; Preprocessing the collected 3D image to obtain an image with depth information of the object to be detected; Inputting the obtained image with depth information into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask includes information about defect depths of the segmented defect areas; and Based on predefined defect rules, the segmented 2D defect mask and 3D defect mask are fused and judged to output defect detection results; Performing a fusion judgment on the segmented 2D defect mask and the 3D defect mask based on predefined defect rules further includes: aligning the 2D image with the 3D image at a pixel level using a coordinate transformation matrix between the acquired 2D image and the 3D image to obtain an aligned image; Filling the segmented 2D defect mask and 3D defect mask into the aligned pictures; and A fusion judgment is performed on the segmented 2D defect mask and the 3D defect mask on the aligned image based on a predefined defect rule to output a defect detection result.

2. The method according to claim 1, wherein The multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, where the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, 0.3-0.45 in the second level, and 0.5-0.7 in the third level.

3. The method according to claim 2, wherein The IoU threshold is set to 0.3 in the first level, 0.4 in the second level, and 0.5 in the third level.

4. The method according to any one of claims 1 to 3, wherein: The defect mask output by the multi-level feature extraction instance segmentation network is obtained based on the weighted average of the defect masks output by the instance segmentation network at each level.

5. The method according to any one of claims 1 to 3, wherein: Judging the segmented defect mask based on predefined defect rules to output defect detection results further includes: When the size or depth of the segmented defect area is greater than a predefined threshold for the defect type of the defect area, the defect class is output as the defect detection result.

6. The method according to any one of claims 1 to 3, wherein: The defect types include pit defects and protrusion defects, and judging the segmented defect mask based on predefined defect rules to output defect detection results further includes: In a case where the defect type of the defect area is a pit defect, if both the segmented 2D defect mask and the segmented 3D defect mask include the defect area, or the depth of the defect area is greater than a predefined threshold for the pit defect, then outputting the defect class as the defect detection result; or In the case where the defect type of the defect area is a convex defect, if the segmented 2D defect mask and the 3D defect mask both include the defect area, and the size and depth of the defect area are greater than the predefined threshold for the convex defect, the defect class is output as the defect detection result.

7. A system for defect detection, characterized in that: The system comprises: an image acquisition module, wherein the image acquisition module is configured to acquire a two-dimensional (2D) picture of the object to be detected; a defect segmentation module configured to input the acquired 2D image into a trained defect segmentation model to obtain a segmented 2D defect mask, wherein the defect segmentation model is trained based on a multi-level feature extraction instance segmentation network with a layer-by-layer increase in the intersection-over-union (IoU) threshold, wherein the 2D defect mask includes information about the defect type, defect size, and defect location of the segmented defect area; and a defect judgment module, the defect judgment module being configured to judge the segmented 2D defect mask based on predefined defect rules to output a defect detection result; The image acquisition module is further configured to acquire a three-dimensional (3D) picture of the object to be detected; The defect segmentation module is further configured to: Preprocessing the collected 3D image to obtain an image with depth information of the object to be detected; Inputting the obtained image with depth information into the trained defect segmentation model to obtain a segmented 3D defect mask, wherein the 3D defect mask includes information about defect depths of the segmented defect areas; and The defect judgment module is further configured to perform a fusion judgment on the segmented 2D defect mask and the 3D defect mask based on a predefined defect rule to output a defect detection result; The defect judgment module is further configured to: aligning the 2D image with the 3D image at a pixel level using a coordinate transformation matrix between the acquired 2D image and the 3D image to obtain an aligned image; Filling the segmented 2D defect mask and 3D defect mask into the aligned pictures; and A fusion judgment is performed on the segmented 2D defect mask and the 3D defect mask on the aligned image based on a predefined defect rule to output a defect detection result.

8. The system according to claim 7, wherein: The multi-level feature extraction instance segmentation network is obtained by cascading three levels of instance segmentation networks, where the IoU threshold for positive and negative sample sampling is set to 0.2-0.4 in the first level, 0.3-0.45 in the second level, and 0.5-0.7 in the third level.

9. The system according to claim 8, wherein The IoU threshold is set to 0.3 in the first level, 0.4 in the second level, and 0.5 in the third level.

10. The system according to any one of claims 7 to 9, characterized in that: The defect mask output by the multi-level feature extraction instance segmentation network is obtained based on the weighted average of the defect masks output by the instance segmentation network at each level.

11. The system according to any one of claims 7 to 9, characterized in that: The defect judgment module is further configured to: When the size or depth of the segmented defect area is greater than a predefined threshold for the defect type of the defect area, the defect class is output as the defect detection result.

12. The system according to any one of claims 7 to 9, characterized in that: The defect types include pit defects and protrusion defects, and the defect judgment module is further configured to: In a case where the defect type of the defect area is a pit defect, if both the segmented 2D defect mask and the segmented 3D defect mask include the defect area, or the depth of the defect area is greater than a predefined threshold for the pit defect, outputting the defect class as the defect detection result; or In the case where the defect type of the defect area is a convex defect, if the segmented 2D defect mask and the 3D defect mask both include the defect area, and the size and depth of the defect area are greater than the predefined threshold for the convex defect, the defect class is output as the defect detection result.

13. A device for defect detection, characterized in that: The device comprises: a memory storing computer-executable instructions; and at least one processor, and when the computer executable instructions are executed by the at least one processor, the apparatus implements the method for defect detection according to any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable instructions, which, when executed by a computing device, enable the computing device to implement the method for defect detection according to any one of claims 1 to 6.

Images