Deviation defect detection method and device, electronic equipment and storage medium

Abstract

The invention provides a deviation defect detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a pre-labeled defect image, obtaining a training set, processing the defect image, obtaining processing data, optimizing an initial loss function of a pre-constructed initial segmentation model through the processing data, and obtaining a deviation defect detection result. Obtaining a target loss function, and training the initial segmentation model to obtain a target segmentation model; inputting a to-be-detected image into the target segmentation model to obtain edge positions of a part area of the to-be-detected part and a reference area of the to-be-detected part; and determining whether the to-be-detected part has the deviation defect or not based on the part area of the to-be-detected part and the edge position of the reference area of the to-be-detected part. According to the method, the loss function of the existing segmentation model is updated to obtain the target segmentation model, so that the target segmentation model can be used for accurately extracting and segmenting the edge region of the product, and the accuracy of detecting the deviation defect is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and storage medium for detecting offset defects. Background Art

[0002] In modern industrial production, product quality requirements are getting higher and higher. Traditional industrial offset defect detection mainly focuses on the identification of surface defects, such as cracks, dents or uneven colors. These defects often directly affect the appearance and basic functions of the product, so they are easily taken seriously. However, in addition to surface defects, there is a more common defect type - positional deviation, referred to as offset.

[0003] In the related art, an edge of a part is extracted by image processing techniques, and then the edge of the reference area is located in a similar way. The distance between the two edges is calculated to determine whether the part is offset. If the two edges are very close or very far apart, it means that an offset defect has occurred. However, during image detection, there will be large differences in the incoming materials in production, the degree of reflection on the surface of the part, the degree of wrinkling of the part, the turning and milling quality of the reference area, etc., which will cause large fluctuations in the pixel values ​​of the part and the reference area. It is difficult to obtain the edge position stably with a set of parameters, and there are problems of over-inspection and missed inspection. The target detection model can also be used directly to detect offset defects. The main detection model currently used is mainly to use a general target detection model to identify the part area with offset features. However, the traditional detection method based on supervised training requires too many training samples and is difficult to have a high generalization ability when facing a highly complex and changeable production environment. It is necessary to collect a certain amount of defect samples for each offset direction in order to have a good detection accuracy. The existing offset detection results are not accurate enough. Summary of the invention

[0004] The present application provides a method, device, electronic device and storage medium for detecting offset defects, so as to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present application, a method for detecting an offset defect is provided, the method comprising:

[0006] Obtain pre-labeled defect images to obtain a training set;

[0007] Processing the pre-marked defect image to obtain processed data;

[0008] Optimizing an initial loss function of a pre-constructed initial segmentation model based on the processed data to obtain a target loss function;

[0009] Training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model;

[0010] Inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested;

[0011] Based on the edge positions of the part area of ​​the part to be measured and the reference area of ​​the part to be measured, it is determined whether the part to be measured has a misalignment defect.

[0012] In one possible implementation manner, the processing the pre-marked defect image to obtain processed data includes:

[0013] Adjusting the pre-marked defect image to a defect image of a preset size;

[0014] Performing edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0015] Performing a morphological dilation operation on the edge gradient image to obtain a dilated image;

[0016] The dilated image is binarized based on a preset binarization threshold to obtain a binarized image; wherein all the obtained binarized images are used as processing data.

[0017] In one possible implementation manner, the optimizing the initial loss function of the pre-constructed initial segmentation model based on the processed data to obtain the target loss function includes:

[0018] Acquire the initial loss function, where the initial loss function is used to characterize the difference between each pixel point of the segmentation result obtained after the initial segmentation model calculates the defect image and the pre-labeled defect image;

[0019] Determine, based on the initial loss function, the pixel position of the defect image in the processed data whose pixel value is a preset value;

[0020] Determine a pixel difference value between a pixel at the pixel position and a pixel of a pre-marked defect image, and discard pixel difference values ​​whose pixel difference values ​​are lower than a first threshold;

[0021] The initial loss function is updated based on the preset magnification coefficient and the difference value of the remaining pixel points after discarding to obtain the target loss function.

[0022] In one embodiment, the target loss function is calculated in the following manner:

[0023]

[0024] e i,j =(Seg i,j -Mask i,j ) 2 ,(i=0,...w,j=0,...,h)

[0025] Among them, Loss is the target loss function, i is the horizontal coordinate of each pixel in the defect image, j is the vertical coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image. i，j The analysis result of the initial segmentation model is the pixel at the pixel position ij, Mask i，j is the pixel of the pre-annotated defect image at the pixel position ij, G′ i，j For the training defect image at the pixel position ij, e' i，j It is the difference between the analysis result of the target segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image.

[0026] In one possible implementation manner, the step of inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested includes:

[0027] Obtaining the reference area contour of the defect-free part image;

[0028] Inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part area of ​​the part to be tested;

[0029] Determine the affine transformation relationship between the defect-free image and the image to be tested based on the reference area contour and the image to be tested using a contour matching algorithm, so as to map the reference area contour to a corresponding position in the image to be tested, and obtain a determination map;

[0030] Based on the determination diagram, it is determined whether the part to be tested has a misalignment defect.

[0031] In one possible implementation manner, determining whether the part to be tested has a misalignment defect based on the determination diagram includes:

[0032] Determine the outer contour of the part area in the determination diagram;

[0033] Determine whether each contour point in the outer contour of the part region exceeds the contour range of the reference region;

[0034] Counting all contour points beyond the contour range of the reference area, and calculating the distance between the contour points and the contour of the reference area;

[0035] Determine whether the part to be tested has a displacement defect according to the distance value.

[0036] In one possible implementation manner, determining whether the part to be tested has a misalignment defect according to the distance value includes:

[0037] The maximum value among all the distance values ​​is determined, and the maximum value is compared with a second threshold value. When the maximum value exceeds the second threshold value, it is determined that the part to be tested has a misalignment defect.

[0038] In one possible implementation manner, determining whether the part to be tested has a misalignment defect according to the distance value includes:

[0039] The number of contour points having distance values ​​is determined, and the number is compared with a number threshold. When the number exceeds the number threshold, it is determined that the part to be tested has a misalignment defect.

[0040] In one possible implementation manner, the step of obtaining the reference area contour of the defect-free part image includes:

[0041] Inputting the defect-free image into the target segmentation model to obtain a first edge of a reference area of ​​the defect-free image;

[0042] Binarization is performed on the first edge to obtain the reference area contour.

[0043] In one possible implementation manner, the step of inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part region of the part to be tested includes:

[0044] Inputting the acquired image of the part to be tested into the target segmentation model;

[0045] The target segmentation model uses a skeleton extraction algorithm to extract the contour to be measured of the part to be measured and the outer contour of the part area of ​​the part to be measured.

[0046] According to a second aspect of the present application, a device for detecting a misalignment defect is provided, the device comprising:

[0047] An acquisition module is used to acquire pre-labeled defect images to obtain a training set;

[0048] A processing module, used for processing the pre-marked defect image to obtain processing data;

[0049] An optimization module, used for optimizing an initial loss function of a pre-built initial segmentation model based on the processed data to obtain a target loss function;

[0050] A training module, used for training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model;

[0051] An input module, used for inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested;

[0052] The detection module is used to determine whether the part to be tested has a misalignment defect based on the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested.

[0053] In one possible implementation, the acquisition module includes:

[0054] An adjusting unit, used for adjusting the pre-marked defect image to a defect image of a preset size;

[0055] A processing unit, configured to perform edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0056] An operation unit, used for performing a morphological dilation operation on the edge gradient image to obtain a dilated image;

[0057] The binarization unit is used to perform binarization processing on the dilated image based on a preset binarization threshold value to obtain a binarized image; wherein all the obtained binarized images are used as processing data.

[0058] In one possible implementation, the optimization module includes:

[0059] A first acquisition unit, used to acquire the initial loss function, where the initial loss function is used to characterize the difference between each pixel point of the segmentation result obtained after the initial segmentation model calculates the defect image and the pre-labeled defect image;

[0060] A determining unit, configured to determine, based on the initial loss function, a pixel point position of a defect image in the processed data whose pixel value is a preset value;

[0061] a discarding unit, configured to determine a pixel difference value between a pixel at the pixel position and a pixel of a pre-marked defect image, and discard a pixel difference value whose pixel difference value is lower than a first threshold;

[0062] An updating unit is used to update the initial loss function based on a preset magnification coefficient and the difference value of the remaining pixel points after discarding to obtain a target loss function.

[0063] In one embodiment, the target loss function is calculated in the following manner:

[0064]

[0065] e i,j =(Seg i,j -Mask i,j 0 2 ,(i=0,...w,j=0,...,h)

[0066] Among them, Loss is the target loss function, i is the horizontal coordinate of each pixel in the defect image, j is the vertical coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image. i，j The analysis result of the initial segmentation model is the pixel at the pixel position ij, Mask i，j is the pixel of the pre-annotated defect image at the pixel position ij, G′ i，j For the training defect image at the pixel position ij, e' i，j It is the difference between the analysis result of the target segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image.

[0067] In one possible implementation, the input module includes:

[0068] A second acquisition unit is used to acquire the reference area contour of the defect-free part image;

[0069] An input unit, used for inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part area of ​​the part to be tested;

[0070] A transformation unit, configured to determine an affine transformation relationship between the defect-free image and the image to be tested based on the reference area contour and the image to be tested by using a contour matching algorithm, so as to map the reference area contour to a corresponding position in the image to be tested, and obtain a determination map;

[0071] A determination unit is used to determine whether the part to be tested has a misalignment defect based on the determination diagram.

[0072] In one possible implementation, the determination unit includes:

[0073] A first determination subunit, used to determine the outer contour of the part area in the determination diagram;

[0074] A second determination subunit is used to determine whether each contour point in the outer contour of the part area exceeds the contour range of the reference area;

[0075] A calculation unit, used for counting all contour points beyond the contour range of the reference area, and calculating the distance between the contour points and the contour of the reference area;

[0076] The third determination subunit is used to determine whether the part to be tested has a displacement defect according to the distance value.

[0077] In one possible implementation, the third determining subunit includes:

[0078] The first determination super subunit is used to determine the maximum value of all distance values, compare the maximum value with a second threshold value, and when the maximum value exceeds the second threshold value, determine that the part to be tested has a misalignment defect.

[0079] In one possible implementation, the third determining subunit includes:

[0080] The second determination super subunit is used to determine the number of contour points with distance values, compare the number with a number threshold, and when the number exceeds the number threshold, determine that the part to be tested has a misalignment defect.

[0081] In one possible implementation, the second acquisition unit includes:

[0082] A segmentation subunit, configured to input the defect-free image into the target segmentation model to obtain a first edge of a reference area of ​​the defect-free image;

[0083] The processing subunit is used to perform binarization processing on the first edge to obtain the reference area contour.

[0084] In one possible implementation, the input unit includes:

[0085] The extraction subunit is used to extract the contour to be measured of the part to be measured and the outer contour of the part area of ​​the part to be measured by using a skeleton extraction algorithm.

[0086] According to a third aspect of the present application, an electronic device is provided, including:

[0087] at least one processor; and

[0088] a memory communicatively connected to the at least one processor; wherein,

[0089] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present application.

[0090] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the method described in the present application.

[0091] In the present application, firstly, a pre-labeled defect image is obtained to obtain a training set, and the pre-labeled defect image is processed to obtain processed data, and the initial loss function of the pre-constructed initial segmentation model is optimized using the processed data to obtain a target loss function, thereby training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model; the image to be tested of the part to be tested is input into the target segmentation model to obtain the edge position of the part area to be tested and the reference area of ​​the part to be tested; based on the edge position of the part area to be tested and the reference area of ​​the part to be tested, determine whether the part to be tested has an offset defect. The present application updates the loss function of the existing segmentation model to obtain a target segmentation model, so that the edge area of ​​the product can be accurately extracted and segmented using the target segmentation model, thereby improving the accuracy of detecting offset defects.

[0092] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become easily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0094] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0095] Figure 1 The schematic diagram of the implementation process of the offset defect detection method of the present application is shown in FIG. Figure 1 ;

[0096] Figure 2 A schematic diagram of the structure of the offset defect in an embodiment of the present application is shown;

[0097] Figure 3 A schematic diagram of the process of extracting edges in a segmentation model in an embodiment of the present application is shown;

[0098] Figure 4 The schematic diagram of the implementation process of the offset defect detection method of the present application is shown in FIG. Figure 2 ;

[0099] Figure 5A schematic diagram of the structure of the misalignment defect detection device according to an embodiment of the present application is shown;

[0100] Figure 6 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0101] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0102] Misalignment defects refer to problems that occur during the assembly or installation process of a product that do not comply with production rules. Generally, a certain area of ​​a part exceeds the reference area allowed for installation. For example, a piece of tape should have been completely inside a groove, but during the installation process, a corner of the tape exceeded the groove, which is a misalignment defect. This problem may not immediately affect the appearance of the product, but it may cause processing failures or performance degradation in subsequent processing or actual use of the product. Since misalignment usually involves the relative position or combination of components, they are often difficult to detect using traditional image processing technology.

[0103] It should be noted that the segmentation model is used because segmentation does not require the collection of too many defect samples. Only about 60 samples are needed to allow the model to have a certain segmentation effect on both the part area and the reference area for offset detection, while the traditional target detection model requires about 200 labeled defect samples to achieve the same accuracy. Among them, the reference area includes the area range of grooves, positioning lines, bosses, etc.

[0104] However, traditional segmentation models generally do not perform well for edge segmentation, because the area of ​​edge locations is relatively small for the model, while the area of ​​non-edge locations is larger. The calculation process of the model loss function will be dominated by the latter, resulting in the model not paying much attention to whether the edge area can be accurately segmented. In this application, the training process of the segmentation model is optimized so that it can focus on the accurate segmentation of edge locations.

[0105] The offset defect detection method provided in the present application uses a segmentation model to segment the part position and the edge position of the reference area, and then uses an image processing algorithm to analyze whether the part exceeds the reference area and calculate specific quantitative indicators such as the width of the exceeding area, thereby determining whether the part to be tested has an offset defect.

[0106] The following is a detailed description of the offset defect detection method, device, electronic device and storage medium provided by the present application in conjunction with the accompanying drawings.

[0107] The present application embodiment provides a method for detecting offset defects. Figure 1 As shown, the method includes:

[0108] S101: Acquire pre-labeled defect images to obtain a training set;

[0109] In this step, we first need to explain the offset defects, such as Figure 2 As shown, Figure 2 In the example, the image without defects is OK, and the image with deviation is a defective product.

[0110] This application first obtains some defect images in a defect data set, and then annotates the defects on the defect images to obtain annotated defect images, and uses these annotated defect images as a training set.

[0111] S102: Processing the pre-marked defect image to obtain processed data;

[0112] It is understandable that after obtaining the labeled defect image, some preprocessing can be performed on the defect image, such as size adjustment, grayscale processing, etc. to obtain processed data. In this application, the processed data is used to improve the initial loss function.

[0113] S103: Optimizing an initial loss function of a pre-constructed initial segmentation model based on the processed data to obtain a target loss function;

[0114] The segmentation model is used in this step because segmentation does not require the collection of too many defect samples. Only about 60 samples are needed to allow the model to have a certain segmentation effect on both the part area and the reference area for offset detection, while the traditional target detection model needs about 200 labeled defect samples to achieve the same accuracy. Specifically, the reference area is an area where parts such as grooves, positioning lines, and bosses are located.

[0115] The segmentation model in this application adopts U-Net. The segmentation model calculates the difference between each pixel of the segmentation result Seg and the annotated image Mask and integrates them as the initial loss function. In this application, the initial loss function is optimized to obtain the target loss function, wherein the target loss function can focus on the edge area so as to achieve accurate segmentation.

[0116] Specifically, the initial loss function in this application is

[0117]

[0118] e i,j=(Seg i,j -Mask i,j ) 2 ,(i=0,...w,j=0,...,h)

[0119] Where i is the horizontal coordinate of each pixel in the defect image, j is the vertical coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j Mask is the difference between the analysis result of the initial segmentation model and the pixel point of the pre-annotated defect image at the pixel position ij. i，j is the pixel of the pre-annotated defect image at the pixel position ij, G′ i，j The pixel at the pixel position ij of the defect image in the training is obtained. After optimizing the initial loss function, the optimized target loss function is obtained.

[0120] S104: Training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model;

[0121] In this step, the initial segmentation model is trained through the target loss function using the pre-labeled defect image until the initial segmentation model meets the convergence condition. It can be understood that the convergence condition can be that the number of training times is reached, or the difference between the current loss function and the previous loss function is less than the preset value, or the loss function value is less than the first threshold. This application does not limit this.

[0122] S105: inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested;

[0123] In this step, firstly, an image of the part to be tested is obtained as the image to be tested. It is understandable that the image to be tested can be obtained through a camera or a stored photo. For example, the two parts to be tested are photographed using a mobile phone camera to obtain the image to be tested. The image to be tested is input into the target segmentation model, and the target segmentation model can accurately segment the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested.

[0124] S106: Determine whether the part to be tested has a misalignment defect based on the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested.

[0125] In this step, after obtaining the edge positions of the part area of ​​the part to be measured and the reference area of ​​the part to be measured, by comparing the edge positions of the part area of ​​the part to be measured and the reference area of ​​the part to be measured with the defect-free image of the defect-free part, it can be determined whether the part to be measured has a misalignment defect.

[0126] The offset defect detection method provided by the present application first obtains a training set for the acquired pre-annotated defect image, processes the pre-annotated defect image to obtain processed data, uses the processed data to optimize the initial loss function of the pre-constructed initial segmentation model to obtain the target loss function, and then trains the initial segmentation model based on the training set and the target loss function to obtain the target segmentation model; inputs the image to be tested of the part to be tested into the target segmentation model to obtain the edge position of the part area to be tested and the reference area of ​​the part to be tested; based on the edge position of the part area to be tested and the reference area of ​​the part to be tested, determines whether the part to be tested has an offset defect. The present application updates the loss function of the existing segmentation model to obtain the target segmentation model, so that the edge area of ​​the product can be accurately extracted and segmented using the target segmentation model, thereby improving the accuracy of detecting offset defects.

[0127] In some embodiments, the processing of the pre-labeled defect image to obtain processed data includes:

[0128] Adjusting the pre-marked defect image to a defect image of a preset size;

[0129] Performing edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0130] Performing a morphological dilation operation on the edge gradient image to obtain a dilated image;

[0131] The dilated image is binarized based on a preset binarization threshold to obtain a binarized image; wherein all the obtained binarized images are used as processing data.

[0132] Specifically, the preset size in this application is 3x3. In this application, a 3x3 sobel operator can be used to process the input image to obtain an edge gradient map G. It can be understood that G roughly shows the location where the gradient changes in the image exist in the form of a grayscale image, and the uniform image area without edges is mainly presented in black in G. Figure 3 As shown, a morphological dilation operation is performed on G using a window size of 7x7, and then G is binarized with a binarization threshold. The binarization threshold used in the embodiment of the present application is 10, and an edge area indication map G', that is, a binarized image G', is obtained; multiple binarized images G' can be used as processing data to optimize the initial loss function, thereby obtaining a target loss function.

[0133] In some embodiments, the optimizing the initial loss function of the pre-built initial segmentation model based on the processed data to obtain the target loss function includes:

[0134] Obtaining an initial loss function; the initial loss function is used to characterize the difference between each pixel point of the segmentation result obtained after the initial segmentation model calculates the defect image and the pre-labeled defect image;

[0135] Determine, based on the initial loss function, the pixel position of the defect image in the processed data whose pixel value is a preset value;

[0136] Determine a pixel difference value between a pixel at the pixel position and a pixel of a pre-marked defect image, and discard pixel difference values ​​whose pixel difference values ​​are lower than a first threshold;

[0137] The initial loss function is updated based on the preset magnification coefficient and the difference value of the remaining pixel points after discarding to obtain the target loss function.

[0138] Specifically, in this application, the initial loss function is optimized with the help of the edge area indicator graph G' in the data set obtained above, specifically,

[0139] For the pixel position with a value of 0 in the binary image G', the pixel difference e between the pixel position of the corresponding position and the pixel of the pre-annotated defect image is calculated during the loss function calculation process. i,j Truncate, when e i,j When it is less than 0.1, it will no longer participate in the calculation of the loss function, thereby reducing the impact of uniform texture areas on the loss value. Avoid over-optimization of the model on the segmentation effect of non-edge areas. It should be noted that for the pixel position with a value of 1 in the binary image G', e is not used. i,j By restricting the loss function, and further adding a factor of 2 to the loss function calculation, the model can further enhance the optimization effect of the boundary. This is because 2 times can make the loss value of this position larger, and the model will pay more attention to the segmentation effect of this position, thereby enhancing the optimization effect of the model on the boundary. Specifically, the target loss function obtained after the update is

[0140]

[0141] e i,j =(Seg i,j -Mask i,j ) 2 ,(i=0,...w,j=0,...,h)

[0142] Among them, G′ i，j To process the pixel at the pixel position ij of the defect image in the data, e' i，j It is the difference between the analysis result of the target segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image.

[0143] It can be understood that the target loss function obtained in the present application can effectively improve the segmentation effect of the segmentation model at the edge position, thereby completely segmenting areas such as edge contours.

[0144] In some embodiments, the step of inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part region of the part to be tested and the reference region of the part to be tested includes:

[0145] Obtaining the reference area contour of the defect-free part image;

[0146] Inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part area of ​​the part to be tested;

[0147] Determine the affine transformation relationship between the defect-free image and the image to be tested based on the reference area contour and the image to be tested using a contour matching algorithm, so as to map the reference area contour to a corresponding position in the image to be tested, and obtain a determination map;

[0148] Based on the determination diagram, it is determined whether the part to be tested has a misalignment defect.

[0149] Specifically, in the present application, a defect-free part image is first obtained from a defect-free data set, or a defect-free part is photographed to obtain a defect-free part image, and then, as Figure 4 As shown in (a), the reference area contour C is extracted from the defect-free part image. 标准基准 ; Then obtain the image of the part to be tested and input it into the target segmentation model to obtain Figure 4 (b) The measured contour C of the part to be measured 待测基准 And the outer contour C of the part area of ​​the part to be tested 零件 , using the contour matching algorithm using C 标准基准 With C 待测基准 Get the affine transformation relationship between the standard image and the image to be tested, and then convert C 标准基准 Map to the corresponding position in the image to be tested to obtain the complete uncovered reference area outline, that is, the judgment map; Figure 4 As shown in (c), finally, it is determined whether the part to be tested has a misalignment defect based on the judgment diagram.

[0150] In some embodiments, determining whether the part to be tested has a misalignment defect based on the determination diagram includes:

[0151] Determine the outer contour of the part area in the determination diagram;

[0152] Determine whether each contour point in the outer contour of the part region exceeds the contour range of the reference region;

[0153] Counting all contour points beyond the contour range of the reference area, and calculating the distance between the contour points and the contour of the reference area;

[0154] Determine whether the part to be tested has a displacement defect according to the distance value.

[0155] It is understandable that first determine C 零件 For each contour point in C 零件 For each contour point in C, calculate whether it is completely wrapped in C 标准基准 If there is a value exceeding C 标准基准 If the contour range is not large, the distance from the point to C is given. 标准基准 Finally, according to the distance from the contour point to C 标准基准 The distance between the tested part and the measured part is used to determine whether the part has a displacement defect.

[0156] In some embodiments, determining whether the part to be tested has a misalignment defect according to the distance value includes:

[0157] The maximum value among all the distance values ​​is determined, and the maximum value is compared with a second threshold value. When the maximum value exceeds the second threshold value, it is determined that the part to be tested has a misalignment defect.

[0158] Specifically, in this application, the C 标准基准 The contour points of the contour range include multiple, statistics C 零件 C 标准基准 For the points in the contour range, the distance values ​​are arranged from large to small, and then the maximum value is compared with the second threshold. If the maximum value exceeds the threshold, it means that there is a misalignment defect in the part to be tested.

[0159] In some embodiments, determining whether the part to be tested has a misalignment defect according to the distance value includes:

[0160] The number of contour points having distance values ​​is determined, and the number is compared with a number threshold. When the number exceeds the number threshold, it is determined that the part to be tested has a misalignment defect.

[0161] Alternatively, we can count the number of 标准基准 The number of contour points in the contour range, and then set the number threshold for the range, for example, 10 pixels are used as the threshold, and if it exceeds C 标准基准 When there are more than 10 pixels, it is considered that the part to be tested has a misaligned defect.

[0162] In some embodiments, obtaining the reference area contour of the defect-free part image includes:

[0163] Inputting the defect-free image into the target segmentation model to obtain a first edge of a reference area of ​​the defect-free image;

[0164] Binarization is performed on the first edge to obtain the reference area contour.

[0165] Specifically, in this application, a standard defect-free image is selected, and the edge of the reference area is segmented using the segmentation model, the segmented edge area is binarized, and the skeleton extraction algorithm is used to refine it to obtain a standard reference area contour C standard reference. Figure 4 (a) shown.

[0166] In some embodiments, the step of inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part region of the part to be tested includes:

[0167] Inputting the acquired image of the part to be tested into the target segmentation model;

[0168] The target segmentation model uses a skeleton extraction algorithm to extract the contour to be measured of the part to be measured and the outer contour of the part area of ​​the part to be measured.

[0169] Specifically, in the present application, a skeleton extraction algorithm is used to extract the contour to be measured of the part to be measured and the outer contour of the part area of ​​the part to be measured.

[0170] like Figure 5 As shown, an embodiment of the present application provides a device for detecting offset defects, the device comprising:

[0171] An acquisition module 501 is used to acquire pre-labeled defect images to obtain a training set;

[0172] An acquisition module 502 is used to process the pre-marked defect image to obtain processed data;

[0173] An optimization module 503 is used to optimize the initial loss function of the pre-built initial segmentation model based on the processed data to obtain a target loss function;

[0174] A training module 504 is used to train the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model;

[0175] An input module 505 is used to input the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested;

[0176] The detection module 506 is used to determine whether the part to be tested has a misalignment defect based on the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested.

[0177] In some embodiments, the acquisition module includes:

[0178] An adjusting unit, used for adjusting the pre-marked defect image to a defect image of a preset size;

[0179] A processing unit, configured to perform edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0180] An operation unit, used for performing a morphological dilation operation on the edge gradient image to obtain a dilated image;

[0181] The binarization unit is used to perform binarization processing on the dilated image based on a preset binarization threshold value to obtain a binarized image; wherein all the obtained binarized images are used as processing data.

[0182] In some embodiments, the optimization module includes:

[0183] A first acquisition unit, used to acquire the initial loss function, where the initial loss function is used to characterize the difference between each pixel point of the segmentation result obtained after the initial segmentation model calculates the defect image and the pre-labeled defect image;

[0184] A determining unit, configured to determine, based on the initial loss function, a pixel point position of a defect image in the processed data whose pixel value is a preset value;

[0185] a discarding unit, configured to determine a pixel difference value between a pixel at the pixel position and a pixel of a pre-marked defect image, and discard a pixel difference value whose pixel difference value is lower than a first threshold;

[0186] An updating unit is used to update the initial loss function based on a preset magnification coefficient and the difference value of the remaining pixel points after discarding to obtain a target loss function.

[0187] In some embodiments, the target loss function is calculated in the following manner:

[0188]

[0189] e i,j =(Seg i,j -Mask i,j ) 2 ,(i=0,...w,j=0,...,h)

[0190] Among them, Loss is the target loss function, i is the horizontal coordinate of each pixel in the defect image, j is the vertical coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image. i，j The analysis result of the initial segmentation model is the pixel at the pixel position ij, Mask i，j is the pixel of the pre-annotated defect image at the pixel position ij, G′ i，j For the training defect image at the pixel position ij, e' i，j It is the difference between the analysis result of the target segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image.

[0191] In some embodiments, the input module includes:

[0192] A second acquisition unit is used to acquire the reference area contour of the defect-free part image;

[0193] An input unit, used for inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part area of ​​the part to be tested;

[0194] A transformation unit, configured to determine an affine transformation relationship between the defect-free image and the image to be tested based on the reference area contour and the image to be tested by using a contour matching algorithm, so as to map the reference area contour to a corresponding position in the image to be tested, and obtain a determination map;

[0195] A determination unit is used to determine whether the part to be tested has a misalignment defect based on the determination diagram.

[0196] In some embodiments, the determination unit includes:

[0197] A first determination subunit, used to determine the outer contour of the part area in the determination diagram;

[0198] A second determination subunit is used to determine whether each contour point in the outer contour of the part area exceeds the contour range of the reference area;

[0199] A calculation unit, used for counting all contour points beyond the contour range of the reference area, and calculating the distance between the contour points and the contour of the reference area;

[0200] The third determination subunit is used to determine whether the part to be tested has a displacement defect according to the distance value.

[0201] In some embodiments, the third determining subunit includes:

[0202] The first determination super subunit is used to determine the maximum value of all distance values, compare the maximum value with a second threshold value, and when the maximum value exceeds the second threshold value, determine that the part to be tested has a misalignment defect.

[0203] In some embodiments, the third determining subunit includes:

[0204] The second determination super subunit is used to determine the number of contour points with distance values, compare the number with a number threshold, and when the number exceeds the number threshold, determine that the part to be tested has a misalignment defect.

[0205] In some embodiments, the second acquisition unit includes:

[0206] A segmentation subunit, configured to input the defect-free image into the target segmentation model to obtain a first edge of a reference area of ​​the defect-free image;

[0207] The processing subunit is used to perform binarization processing on the first edge to obtain the reference area contour.

[0208] In some embodiments, the input unit includes:

[0209] The extraction subunit is used to extract the contour to be measured of the part to be measured and the outer contour of the part area of ​​the part to be measured by using a skeleton extraction algorithm.

[0210] It should be noted that the misaligned defect detection device in the embodiment of the present application, since the principle of solving the problem by the misaligned defect detection device is similar to the aforementioned misaligned defect detection method, the implementation process, implementation principle and beneficial effects of the misaligned defect detection device can all be referred to the description of the implementation process, implementation principle and beneficial effects of the aforementioned method, and the repeated parts will not be repeated.

[0211] An embodiment of the present application provides an electronic device, including:

[0212] at least one processor; and

[0213] a memory communicatively connected to the at least one processor; wherein,

[0214] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the above embodiments.

[0215] An embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in any of the above embodiments.

[0216] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0217] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0218] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0219] Multiple components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0220] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the offset defect detection method. For example, in some embodiments, the offset defect detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the offset defect detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the offset defect detection method in any other appropriate manner (e.g., by means of firmware).

[0221] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0223] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0225] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0226] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0227] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0228] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0229] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for detecting offset defects, characterized in that: The method comprises: Obtain pre-labeled defect images to obtain a training set; Processing the pre-marked defect image to obtain processed data; Optimizing an initial loss function of a pre-constructed initial segmentation model based on the processed data to obtain a target loss function; Training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model; Inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested; Based on the edge positions of the part area of ​​the part to be measured and the reference area of ​​the part to be measured, it is determined whether the part to be measured has a misalignment defect.

2. The method according to claim 1, characterized in that The step of processing the pre-marked defect image to obtain processed data includes: Adjusting the pre-marked defect image to a defect image of a preset size; Performing edge gradient processing on the defect image of the preset size to obtain an edge gradient map; Performing a morphological dilation operation on the edge gradient image to obtain a dilated image; The dilated image is binarized based on a preset binarization threshold to obtain a binarized image; wherein all the obtained binarized images are used as processing data.

3. The method according to claim 1, characterized in that: The optimizing the initial loss function of the pre-built initial segmentation model based on the processed data to obtain the target loss function includes: Acquire the initial loss function, where the initial loss function is used to characterize the difference between each pixel point of the segmentation result obtained after the initial segmentation model calculates the defect image and the pre-labeled defect image; Determine, based on the initial loss function, the pixel position of the defect image in the processed data whose pixel value is a preset value; Determine a pixel difference value between a pixel at the pixel position and a pixel of a pre-marked defect image, and discard pixel difference values ​​whose pixel difference values ​​are lower than a first threshold; The initial loss function is updated based on the preset magnification coefficient and the difference value of the remaining pixel points after discarding to obtain the target loss function.

4. The method according to claim 3, characterized in that The target loss function is calculated in the following way: have been i,j =(Seg i,j -Mask i,j ) 2 ,(i=0,...w,j=0,...,h) Among them, Loss is the target loss function, i is the horizontal coordinate of each pixel in the defect image, j is the vertical coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image. i，j The analysis result of the initial segmentation model is the pixel at the pixel position ij, Mask i，j is the pixel of the pre-annotated defect image at the pixel position ij, G′ i，j For the training defect image at the pixel position ij, e' i，j It is the difference between the analysis result of the target segmentation model and the pixel point at the pixel position ij of the pre-annotated defect image.

5. The method according to claim 1, characterized in that The step of inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested includes: Obtaining the reference area contour of the defect-free part image; Inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour to be tested of the part to be tested and the outer contour of the part area of ​​the part to be tested; Determine the affine transformation relationship between the defect-free image and the image to be tested based on the reference area contour and the image to be tested using a contour matching algorithm, so as to map the reference area contour to a corresponding position in the image to be tested, and obtain a determination map; Based on the determination diagram, it is determined whether the part to be tested has a misalignment defect.

6. The method according to claim 5, characterized in that The determining whether the part to be tested has a misalignment defect based on the determination diagram includes: Determine the outer contour of the part area in the determination diagram; Determine whether each contour point in the outer contour of the part region exceeds the contour range of the reference region; Counting all contour points beyond the contour range of the reference area, and calculating the distance between the contour points and the contour of the reference area; Determine whether the part to be tested has a displacement defect according to the distance value.

7. The method according to claim 6, characterized in that The step of determining whether the part to be tested has a displacement defect according to the distance value includes: The maximum value among all the distance values ​​is determined, and the maximum value is compared with a second threshold value. When the maximum value exceeds the second threshold value, it is determined that the part to be tested has a misalignment defect.

8. A device for detecting offset defects, characterized in that: The device comprises: An acquisition module is used to acquire pre-labeled defect images to obtain a training set; A processing module, used for processing the pre-marked defect image to obtain processing data; An optimization module, used for optimizing an initial loss function of a pre-built initial segmentation model based on the processed data to obtain a target loss function; A training module, used for training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model; An input module, used for inputting the image of the part to be tested into the target segmentation model to obtain the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested; The detection module is used to determine whether the part to be tested has a misalignment defect based on the edge position of the part area of ​​the part to be tested and the reference area of ​​the part to be tested.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images