Methods, devices, electronic equipment and storage media for detecting misalignment defects

Abstract

This application provides a method, apparatus, electronic device, and storage medium for detecting misalignment defects. The method includes acquiring pre-labeled defect images to obtain a training set; processing the defect images to obtain processed data; optimizing the initial loss function of a pre-constructed initial segmentation model using the processed data to obtain a target loss function; training the initial segmentation model to obtain a target segmentation model; inputting the image to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested; and determining whether the part to be tested has a misalignment defect based on the edge positions of the part region and the reference region of the part to be tested. This application updates the loss function of existing segmentation models to obtain the target segmentation model, thereby enabling accurate extraction and segmentation of the edge regions of products and improving the accuracy of misalignment defect detection.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device and storage medium for detecting misalignment defects. Background Technology

[0002] In modern industrial production, product quality requirements are becoming increasingly stringent. Traditional industrial misalignment defect detection mainly focuses on identifying surface defects, such as cracks, dents, or uneven coloring. These defects often directly affect the appearance and basic functions of products, and are therefore easily noticed. However, in addition to surface defects, there is a more common type of defect—positional misalignment, or simply misalignment.

[0003] In related technologies, an edge of a part is extracted using image processing techniques, and then the edge of a reference area is located in a similar manner. The distance between these two edges is calculated to determine if the part is misaligned. If the two edges are very close or very far apart, it indicates a misalignment defect. However, image detection is prone to problems due to significant variations in incoming materials during production. Inconsistencies in surface reflectivity, wrinkle levels, and milling quality in the reference area lead to large fluctuations in pixel values ​​between the part and the reference area, making it difficult to stably obtain edge positions using a single set of parameters. This results in over-detection and under-detection issues. Alternatively, object detection models can be used to detect misalignment defects. Currently, the main detection models used are general object detection models to identify parts with misalignment features. However, traditional supervised training-based detection methods require excessive training samples and struggle to achieve high generalization ability in highly complex and variable production environments. A certain number of defect samples need to be collected for each misalignment direction to achieve good detection accuracy, resulting in insufficient precision in existing misalignment detection results. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for detecting misalignment defects, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0005] According to a first aspect of this application, a method for detecting misalignment defects is provided, the method comprising:

[0006] Obtain pre-labeled defect images to obtain the training set;

[0007] The pre-annotated defect image is processed to obtain processed data;

[0008] Based on the processed data, the initial loss function of the pre-constructed initial segmentation model is optimized to obtain the target loss function;

[0009] The initial segmentation model is trained based on the training set and the target loss function to obtain the target segmentation model;

[0010] The image of the part to be tested is input into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested.

[0011] Based on the edge positions of the part area and the reference area of ​​the part under test, it is determined whether the part under test has a misalignment defect.

[0012] In one possible implementation, processing the pre-annotated defect image to obtain processed data includes:

[0013] Adjust the pre-labeled defect image to a defect image of a preset size;

[0014] The defect image of the preset size is subjected to edge gradient processing to obtain an edge gradient map;

[0015] A morphological dilation operation is performed on the edge gradient map to obtain a dilated image;

[0016] The dilated image is binarized based on a preset binarization threshold to obtain a binarized image; wherein, all the obtained binarized images are used as processing data.

[0017] In one possible implementation, optimizing the initial loss function of the pre-constructed initial segmentation model based on the processed data to obtain the target loss function includes:

[0018] Obtain the initial loss function, which is used to characterize the difference between each pixel point between the segmentation result obtained by the initial segmentation model after calculating the defect image and the pre-labeled defect image;

[0019] Based on the initial loss function, determine the pixel positions of the defective images in the processed data with pixel values ​​of preset values;

[0020] Determine the pixel difference value between the pixel at the pixel location and the pixel in the pre-labeled defect image, and discard pixel difference values ​​that are lower than a first threshold.

[0021] The initial loss function is updated based on the preset magnification coefficient and the difference values ​​of the remaining pixels after discarding, to obtain the target loss function.

[0022] In one possible implementation, the target loss function is calculated in the following manner:

[0023]

[0024] e i,j =(Seg i,j -Mask i,j ) 2 (i=0,...w,j=0,...,h)

[0025] Where Loss is the target loss function, i is the x-coordinate of each pixel in the defect image, j is the y-coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j Seg represents the difference between the analysis results of the initial segmentation model and the pre-annotated defect image at the pixel position ij. i，j The Mask represents the pixel at position ij in the analysis results of the initial segmentation model. i，j For the pre-annotated defect image, at pixel position ij, G i ' ，j To train the defect image at pixel position ij, e' i，j The difference between the analysis results of the target segmentation model and the pre-labeled defect image at the pixel position ij.

[0026] In one possible implementation, the step of inputting the image of the part to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested includes:

[0027] Obtain the reference region contour of the image of the defect-free part;

[0028] The acquired image of the part to be tested is input into the target segmentation model to obtain the outline of the part to be tested and the outer outline of the part region of the part to be tested.

[0029] A contour matching algorithm is used to determine the affine transformation relationship between the defect-free image and the image under test based on the contour of the reference region and the contour to be tested, so as to map the contour of the reference region to the corresponding position in the image under test and obtain a judgment image.

[0030] Based on the determination diagram, it is determined whether the part under test has a misalignment defect.

[0031] In one possible implementation, determining whether the part under test has a misalignment defect based on the determination map includes:

[0032] Determine the outer contour of the part area in the determination diagram;

[0033] Determine whether each contour point in the outer contour of the part area exceeds the contour range of the reference area;

[0034] Count all contour points that are outside the contour range of the reference region, and calculate the distance between the contour points and the contour of the reference region;

[0035] The distance value is used to determine whether the part under test has a misalignment defect.

[0036] In one possible implementation, determining whether the part under test has a misalignment defect based on the distance value includes:

[0037] The maximum value among all distance values ​​is determined, and the maximum value is compared with a second threshold. When the maximum value exceeds the second threshold, it is determined that the part under test has a misalignment defect.

[0038] In one possible implementation, determining whether the part under test has a misalignment defect based on the distance value includes:

[0039] The number of contour points with distance values ​​is determined, and the number is compared with a number threshold. When the number exceeds the number threshold, it is determined that the part under test has a misalignment defect.

[0040] In one possible implementation, obtaining the reference region contour of the defect-free part image includes:

[0041] The defect-free image is input into the target segmentation model to obtain the first edge of the reference region of the defect-free image;

[0042] The first edge is binarized to obtain the contour of the reference region.

[0043] In one possible implementation, the step of inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour of the part to be tested and the outer contour of the part region of the part to be tested includes:

[0044] The acquired image of the part to be tested is input into the target segmentation model;

[0045] The target segmentation model uses a skeleton extraction algorithm to extract the contour of the part to be tested and the outer contour of the part region of the part to be tested.

[0046] According to a second aspect of this application, a misalignment defect detection device is provided, the device comprising:

[0047] The acquisition module is used to acquire pre-annotated defect images to obtain a training set;

[0048] The processing module is used to process the pre-annotated defect image to obtain processed data;

[0049] The optimization module is used to optimize the initial loss function of the pre-constructed initial segmentation model based on the processed data to obtain the target loss function;

[0050] The training module is used to train the initial segmentation model based on the training set and the target loss function to obtain the target segmentation model;

[0051] The input module is used to input the image of the part to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested.

[0052] The detection module is used to determine whether the part under test has a misalignment defect based on the edge position of the part area and the reference area of ​​the part under test.

[0053] In one possible implementation, the acquisition module includes:

[0054] An adjustment unit is used to adjust the pre-labeled defect image to a defect image of a preset size;

[0055] The processing unit is used to perform edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0056] The operation unit is used to perform morphological dilation operation on the edge gradient map to obtain a dilated image;

[0057] The binarization unit is used to perform binarization processing on the dilated image based on a preset binarization threshold to obtain a binarized image; wherein, all the obtained binarized images are used as processing data.

[0058] In one possible implementation, the optimization module includes:

[0059] The first acquisition unit is used to acquire the initial loss function, which is used to characterize the difference between each pixel point between the segmentation result obtained by the initial segmentation model after calculating the defect image and the pre-labeled defect image.

[0060] The determining unit is used to determine the position of the pixel point with a preset value in the defective image in the processed data based on the initial loss function;

[0061] The discard unit is used to determine the pixel difference value between the pixel at the pixel location and the pixel in the pre-labeled defect image, and discard the pixel difference value that is lower than a first threshold.

[0062] The update unit is used to update the initial loss function based on a preset magnification coefficient and the difference values ​​of the remaining pixels after discarding, so as to obtain the target loss function.

[0063] In one possible implementation, the target loss function is calculated in the following manner:

[0064]

[0065] e i,j =(Seg i,j -Mask i,j ) 2 (i=0,...w,j=0,...,h)

[0066] Where Loss is the target loss function, i is the x-coordinate of each pixel in the defect image, j is the y-coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j Seg represents the difference between the analysis results of the initial segmentation model and the pre-annotated defect image at the pixel position ij. i，j The Mask represents the pixel at position ij in the analysis results of the initial segmentation model. i，j For the pre-annotated defect image, at pixel position ij, G i ' ，j To train the defect image at pixel position ij, e' i，j The difference between the analysis results of the target segmentation model and the pre-labeled defect image at the pixel position ij.

[0067] In one possible implementation, the input module includes:

[0068] The second acquisition unit is used to acquire the reference area contour of the image of the defect-free part;

[0069] The input unit is used to input the acquired image of the part to be tested into the target segmentation model to obtain the outline of the part to be tested and the outer outline of the part region of the part to be tested.

[0070] The transformation unit is used to determine the affine transformation relationship between the defect-free image and the image under test based on the contour matching algorithm of the reference region contour and the contour to be tested, so as to map the reference region contour to the corresponding position in the image under test and obtain the judgment image.

[0071] The determination unit is used to determine whether the part under test has a misalignment defect based on the determination diagram.

[0072] In one possible implementation, the determination unit includes:

[0073] The first determining subunit is used to determine the outer contour of the part area in the determination diagram;

[0074] The second determining subunit is used to determine whether each contour point in the outer contour of the part region exceeds the contour range of the reference region.

[0075] A calculation unit is used to count all contour points that exceed the contour range of the reference region and calculate the distance between the contour points and the contour of the reference region.

[0076] The third determining subunit is used to determine whether the part under test has a misalignment defect based on the distance value.

[0077] In one possible implementation, the third determining subunit includes:

[0078] The first determining super sub-unit is used to determine the maximum value among all distance values, compare the maximum value with a second threshold, and determine that the part under test has a misalignment defect when the maximum value exceeds the second threshold.

[0079] In one possible implementation, the third determining subunit includes:

[0080] The second determining super sub-unit is used to determine the number of contour points with distance values, compare the number with a number threshold, and determine that the part under test has a misalignment defect when the number exceeds the number threshold.

[0081] In one possible implementation, the second acquisition unit includes:

[0082] A segmentation subunit is used to input the defect-free image into the target segmentation model to obtain the first edge of the reference region of the defect-free image;

[0083] The processing subunit is used to perform binarization processing on the first edge to obtain the contour of the reference region.

[0084] In one possible implementation, the input unit includes:

[0085] The extraction subunit is used to extract the contour of the part to be tested and the outer contour of the part area of ​​the part to be tested using a skeleton extraction algorithm.

[0086] According to a third aspect of this application, an electronic device is provided, comprising:

[0087] At least one processor; and

[0088] A memory communicatively connected to the at least one processor; wherein,

[0089] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0090] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform the methods described in this application.

[0091] In this application, a training set is first obtained by acquiring pre-labeled defect images. The pre-labeled defect images are then processed to obtain processed data. The initial loss function of a pre-constructed initial segmentation model is optimized using the processed data to obtain a target loss function. The initial segmentation model is then trained based on the training set and the target loss function to obtain a target segmentation model. The image of the part to be tested is input into the target segmentation model to obtain the edge positions of the part's region and reference region. Based on the edge positions of the part's region and reference region, it is determined whether the part to be tested has a misalignment defect. This application updates the loss function of existing segmentation models to obtain the target segmentation model. This target segmentation model can accurately extract and segment the edge regions of products, improving the accuracy of detecting misalignment defects.

[0092] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0093] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, wherein:

[0094] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0095] Figure 1 This paper illustrates the implementation flow of the misalignment defect detection method according to an embodiment of this application. Figure 1 ;

[0096] Figure 2 A schematic diagram of the structure of the misalignment defect according to an embodiment of this application is shown;

[0097] Figure 3 This illustration shows a flowchart of the edge extraction process in the segmentation model of an embodiment of this application;

[0098] Figure 4 This paper illustrates the implementation flow of the misalignment defect detection method according to an embodiment of this application. Figure 2 ;

[0099] Figure 5A schematic diagram of the structural composition of the misalignment defect detection device according to an embodiment of this application is shown;

[0100] Figure 6 A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0101] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0102] Misalignment defects refer to problems that do not conform to production rules during product assembly or installation. Generally, it means that a certain area of ​​a part exceeds the permissible installation reference area. For example, a piece of tape that should be completely inside a groove may have a corner extending beyond it during installation, which is a misalignment defect. This problem may not immediately affect the product's appearance, but it can lead to manufacturing failures or performance degradation in subsequent processing or actual use. Because misalignment usually involves the relative position or assembly method between components, it is often difficult to detect using traditional image processing techniques.

[0103] It should be noted that a segmentation model is used because it doesn't require collecting too many defect samples. Only about 60 samples are needed for the model to achieve a certain level of segmentation for both the part area and the reference area, enabling offset detection. In contrast, traditional target detection models require approximately 200 labeled defect samples to achieve the same accuracy. The reference area includes the regions of grooves, positioning lines, bosses, etc.

[0104] However, traditional segmentation models generally don't perform well at edge locations because the area occupied by edge regions is relatively small, while the area occupied by non-edge regions is much larger. The model's loss function calculation is dominated by the latter, causing the model to pay less attention to the accurate segmentation of edge regions. This application optimizes the training process of the segmentation model, enabling it to focus primarily on the accurate segmentation of edge regions.

[0105] The misalignment defect detection method provided in this application uses a segmentation model to segment the part position and the edge position of the reference area. Then, image processing algorithms can be used to analyze whether the part exceeds the reference area and calculate specific quantitative indicators such as the width of the exceeding area, thereby determining that the part under test has a misalignment defect.

[0106] The following description, in conjunction with the accompanying drawings, details the method, apparatus, electronic equipment, and storage medium for detecting misalignment defects provided in this application.

[0107] This application provides a method for detecting misalignment defects, such as... Figure 1 As shown, the method includes:

[0108] S101: Obtain pre-annotated defect images to obtain the training set;

[0109] In this step, it is first necessary to explain the misalignment defect, such as... Figure 2 As shown, where, Figure 2 In the image, a defect-free image represents an OK condition, while an offset image represents a defective product.

[0110] This application first obtains some defect images from the defect dataset, then annotates the defects on the defect images to obtain annotated defect images, and uses these annotated defect images as the training set.

[0111] S102: Process the pre-annotated defect image to obtain processed data;

[0112] It is understandable that after obtaining the labeled defect image, some preprocessing can be performed on the defect image, such as size adjustment and grayscale processing, to obtain processed data. In this application, the processed data is used to improve the initial loss function.

[0113] S103: Optimize the initial loss function of the pre-constructed initial segmentation model based on the processed data to obtain the target loss function;

[0114] The segmentation model is used in this step because it doesn't require collecting too many defect samples; only about 60 samples are needed for the model to achieve a certain segmentation effect on both the part area and the reference area, enabling offset detection. In contrast, traditional target detection models require about 200 labeled defect samples to achieve the same accuracy. Specifically, the reference area is the region where parts such as grooves, positioning lines, and bosses are located.

[0115] The segmentation model in this application uses U-Net. The segmentation model calculates the difference between each pixel in the segmentation result Seg and the labeled image Mask and integrates them as the initial loss function. In this application, the initial loss function is optimized to obtain the target loss function. The target loss function can focus on edge regions and thus achieve accurate segmentation.

[0116] Specifically, the initial loss function in this application is:

[0117]

[0118] e i,j=(Seg i,j -Mask i,j ) 2 (i=0,...w,j=0,...,h)

[0119] Where i is the x-coordinate of each pixel in the defect image, j is the y-coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j Mask is used to compare the differences between the initial segmentation model's analysis results and the pre-annotated defect image at the pixel location ij. i，j For the pre-annotated defect image, at pixel position ij, G i ' ，j To train the target loss function, we need to find the pixel at position ij in the defect image. After optimizing the initial loss function, we obtain the optimized target loss function.

[0120] S104: Train the initial segmentation model based on the training set and the target loss function to obtain the target segmentation model;

[0121] In this step, the initial segmentation model is trained using the target loss function on the pre-labeled defect images until the initial segmentation model meets the convergence condition. It can be understood that the convergence condition can be reaching the training number, or the difference between the current loss function and the previous loss function is less than a preset value, or the loss function value is less than a first threshold. This application does not limit this.

[0122] S105: Input the image of the part to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested;

[0123] In this step, the first step is to acquire an image of the part to be tested as the test image. This can be obtained through a camera or stored photographs; for example, by using a mobile phone camera to photograph the two parts to be tested. The test image is then input into a target segmentation model, which can accurately segment the edge positions of the part's component area and its reference area.

[0124] S106: Based on the edge positions of the part area to be tested and the reference area to be tested, determine whether the part to be tested has a misalignment defect.

[0125] In this step, after obtaining the edge positions of the part area and the reference area of ​​the part to be tested, by comparing the edge positions of the part area and the reference area of ​​the part to be tested with the defect-free image of the defect-free part, it is possible to determine whether the part to be tested has a misalignment defect.

[0126] The method for detecting misalignment defects provided in this application first obtains a training set from pre-labeled defect images. The pre-labeled defect images are then processed to obtain processed data. This processed data is used to optimize the initial loss function of a pre-constructed initial segmentation model, resulting in a target loss function. The initial segmentation model is then trained based on the training set and the target loss function to obtain a target segmentation model. The image of the part to be tested is input into the target segmentation model to obtain the edge positions of the part's region and reference region. Based on the edge positions of the part's region and reference region, it is determined whether the part to be tested has a misalignment defect. This application updates the loss function of existing segmentation models to obtain the target segmentation model. This target segmentation model can accurately extract and segment the edge regions of products, improving the accuracy of misalignment defect detection.

[0127] In some embodiments, processing the pre-annotated defect image to obtain processed data includes:

[0128] Adjust the pre-labeled defect image to a defect image of a preset size;

[0129] The defect image of the preset size is subjected to edge gradient processing to obtain an edge gradient map;

[0130] A morphological dilation operation is performed on the edge gradient map to obtain a dilated image;

[0131] The dilated image is binarized based on a preset binarization threshold to obtain a binarized image; wherein, all the obtained binarized images are used as processing data.

[0132] Specifically, this application uses a preset size of 3x3. A 3x3 Sobel operator can be used to process the input image to obtain an edge gradient map G. It can be understood that G, in grayscale form, roughly represents the locations of gradient changes in the image, while non-edge, uniform image regions are primarily represented as black in G. For example... Figure 3 As shown, morphological dilation is performed on G using a 7x7 window, and then G is binarized with a binarization threshold. In this embodiment, the binarization threshold is 10, resulting in an edge region indication map G', i.e., a binarized image G'. Multiple binarized images G' can be used as processing data to optimize the initial loss function, thereby obtaining the target loss function.

[0133] In some embodiments, optimizing the initial loss function of the pre-built initial segmentation model based on the processed data to obtain the target loss function includes:

[0134] Obtain an initial loss function; the initial loss function is used to characterize the difference between each pixel point between the segmentation result obtained by the initial segmentation model after calculating the defect image and the pre-labeled defect image;

[0135] Based on the initial loss function, determine the pixel positions of the defective images in the processed data with pixel values ​​of preset values;

[0136] Determine the pixel difference value between the pixel at the pixel location and the pixel in the pre-labeled defect image, and discard pixel difference values ​​that are lower than a first threshold.

[0137] The initial loss function is updated based on the preset magnification coefficient and the difference values ​​of the remaining pixels after discarding, to obtain the target loss function.

[0138] Specifically, in this application, the initial loss function is optimized using the edge region indicator map G' from the dataset obtained above. Specifically,

[0139] For a pixel location with a value of 0 in the binarized image G', during the loss function calculation, the pixel difference e between the corresponding pixel location and the pre-labeled defect image is calculated. i,j Truncation is performed when e i,j Values ​​less than 0.1 are no longer included in the loss function calculation, thus reducing the impact of uniform texture regions on the loss value. This avoids the model over-optimizing the segmentation effect on non-edge regions. It should be noted that for pixels with a value of 1 in the binarized image G', e is not included. i,j By imposing restrictions and further adding a factor of 2 to the loss function calculation, the model's optimization effect on the boundaries can be further enhanced. This is because a factor of 2 results in a larger loss value at that location, causing the model to focus more on the segmentation effect at that position, thus strengthening the model's optimization effect on the boundaries. Specifically, the updated objective loss function is...

[0140]

[0141] e i,j =(Seg i,j -Mask i,j ) 2 (i=0,...w,j=0,...,h)

[0142] Among them, G i ' ，j To process the pixel at position ij in the defect image data, e' i，j The difference between the analysis results of the target segmentation model and the pre-labeled defect image at the pixel position ij.

[0143] It is understandable that the target loss function obtained in this application can effectively improve the segmentation effect of the segmentation model at the edge position, thereby enabling complete segmentation of regions such as edge contours.

[0144] In some embodiments, the step of inputting the image of the part to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested includes:

[0145] Obtain the reference region contour of the image of the defect-free part;

[0146] The acquired image of the part to be tested is input into the target segmentation model to obtain the outline of the part to be tested and the outer outline of the part region of the part to be tested.

[0147] A contour matching algorithm is used to determine the affine transformation relationship between the defect-free image and the image under test based on the contour of the reference region and the contour to be tested, so as to map the contour of the reference region to the corresponding position in the image under test and obtain a judgment image.

[0148] Based on the determination diagram, it is determined whether the part under test has a misalignment defect.

[0149] Specifically, this application first obtains images of defect-free parts from a defect-free dataset, or obtains images of defect-free parts by taking photographs of them, and then, as... Figure 4 As shown in (a), the reference region contour C is extracted from the image of the defect-free part. 标准基准 Next, acquire the image of the part to be tested, input it into the target segmentation model, and obtain the following result: Figure 4 (b) shows the measured profile C of the part to be measured. 待测基准 And the outer contour C of the part area of ​​the part to be tested. 零件 Using contour matching algorithm with C 标准基准 With C 待测基准 Obtain the affine transformation relationship between the standard image and the image to be tested, and then C 标准基准 Mapping this to the corresponding position in the image under test yields the complete outline of the uncovered reference region, i.e., the decision map; such as... Figure 4 As shown in (c), finally, the presence of a misalignment defect in the part under test is determined according to the determination diagram.

[0150] In some embodiments, determining whether the part under test has a misalignment defect based on the determination map includes:

[0151] Determine the outer contour of the part area in the determination diagram;

[0152] Determine whether each contour point in the outer contour of the part area exceeds the contour range of the reference area;

[0153] Count all contour points that are outside the contour range of the reference region, and calculate the distance between the contour points and the contour of the reference region;

[0154] The distance value is used to determine whether the part under test has a misalignment defect.

[0155] Understandably, C should be determined first. 零件 For each contour point in C 零件 For each contour point in the matrix, calculate whether it is completely enclosed within C. 标准基准 If there exists that exceeds C 标准基准 In the case of the outline range, the distance from that point to C is given. 标准基准 The distance. Finally, based on the contour point to C. 标准基准 The distance is used to determine whether the part under test has a misalignment defect.

[0156] In some embodiments, determining whether the part under test has a misalignment defect based on the distance value includes:

[0157] The maximum value among all distance values ​​is determined, and the maximum value is compared with a second threshold. When the maximum value exceeds the second threshold, it is determined that the part under test has a misalignment defect.

[0158] Specifically, this application exceeds C 标准基准 The contour points within the contour range include multiple points, statistically C 零件 China Super League C 标准基准 Points within the contour range are arranged in descending order of distance values. The maximum value is then compared with a second threshold. If the maximum value exceeds the threshold, it indicates that the part under test has a misalignment defect.

[0159] In some embodiments, determining whether the part under test has a misalignment defect based on the distance value includes:

[0160] The number of contour points with distance values ​​is determined, and the number is compared with a number threshold. When the number exceeds the number threshold, it is determined that the part under test has a misalignment defect.

[0161] Alternatively, statistics can be collected exceeding C. 标准基准 The number of contour points within the contour range is determined, and then a threshold is set for the number of points, for example, with a threshold of 10 pixels. Points exceeding C... 标准基准 If there are more than 10 pixels, the part under test is considered to have a misalignment defect.

[0162] In some embodiments, the process of obtaining the reference region contour of the defect-free part image includes:

[0163] The defect-free image is input into the target segmentation model to obtain the first edge of the reference region of the defect-free image;

[0164] The first edge is binarized to obtain the contour of the reference region.

[0165] Specifically, this application selects a standard, defect-free image, uses a segmentation model to segment the edges of the reference region, binarizes the segmented edge regions, and uses a skeleton extraction algorithm to refine them, resulting in a standard reference region contour C. 标准基准 .like Figure 4 As shown in (a).

[0166] In some embodiments, the step of inputting the acquired image of the part to be tested into the target segmentation model to obtain the contour of the part to be tested and the outer contour of the part region of the part to be tested includes:

[0167] The acquired image of the part to be tested is input into the target segmentation model;

[0168] The target segmentation model uses a skeleton extraction algorithm to extract the contour of the part to be tested and the outer contour of the part region of the part to be tested.

[0169] Specifically, in this application, a skeleton extraction algorithm is used to extract the contour of the part to be tested and the outer contour of the part area of ​​the part to be tested.

[0170] like Figure 5 As shown in the figure, this application provides a misalignment defect detection device, the device comprising:

[0171] The acquisition module 501 is used to acquire pre-annotated defect images to obtain a training set;

[0172] The acquisition module 502 is used to process the pre-annotated defect image to obtain processed data;

[0173] The optimization module 503 is used to optimize the initial loss function of the pre-constructed initial segmentation model based on the processed data to obtain the target loss function;

[0174] Training module 504 is used to train the initial segmentation model based on the training set and the target loss function to obtain the target segmentation model;

[0175] The input module 505 is used to input the image of the part to be tested into the target segmentation model to obtain the edge positions of the part region and the reference region of the part to be tested.

[0176] The detection module 506 is used to determine whether the part under test has a misalignment defect based on the edge position of the part area under test and the reference area of ​​the part under test.

[0177] In some embodiments, the acquisition module includes:

[0178] An adjustment unit is used to adjust the pre-labeled defect image to a defect image of a preset size;

[0179] The processing unit is used to perform edge gradient processing on the defect image of the preset size to obtain an edge gradient map;

[0180] The operation unit is used to perform morphological dilation operation on the edge gradient map to obtain a dilated image;

[0181] The binarization unit is used to perform binarization processing on the dilated image based on a preset binarization threshold to obtain a binarized image; wherein, all the obtained binarized images are used as processing data.

[0182] In some embodiments, the optimization module includes:

[0183] The first acquisition unit is used to acquire the initial loss function, which is used to characterize the difference between each pixel point between the segmentation result obtained by the initial segmentation model after calculating the defect image and the pre-labeled defect image.

[0184] The determining unit is used to determine the position of the pixel point with a preset value in the defective image in the processed data based on the initial loss function;

[0185] The discard unit is used to determine the pixel difference value between the pixel at the pixel location and the pixel in the pre-labeled defect image, and discard the pixel difference value that is lower than a first threshold.

[0186] The update unit is used to update the initial loss function based on a preset magnification coefficient and the difference values ​​of the remaining pixels after discarding, so as to obtain the target loss function.

[0187] In some embodiments, the target loss function is calculated in the following manner:

[0188]

[0189] e i,j =(Seg i,j -Mask i,j ) 2 (i=0,...w,j=0,...,h)

[0190] Where Loss is the target loss function, i is the x-coordinate of each pixel in the defect image, j is the y-coordinate of each pixel in the defect image, n is the number of images, w is the width of the defect image, h is the height of the defect image, and e is the height of the defect image. i,j Seg represents the difference between the analysis results of the initial segmentation model and the pre-annotated defect image at the pixel position ij. i，j The Mask represents the pixel at position ij in the analysis results of the initial segmentation model. i，j For the pre-annotated defect image, at pixel position ij, G i ' ，j To train the defect image at pixel position ij, e' i，j The difference between the analysis results of the target segmentation model and the pre-labeled defect image at the pixel position ij.

[0191] In some embodiments, the input module includes:

[0192] The second acquisition unit is used to acquire the reference area contour of the image of the defect-free part;

[0193] The input unit is used to input the acquired image of the part to be tested into the target segmentation model to obtain the outline of the part to be tested and the outer outline of the part region of the part to be tested.

[0194] The transformation unit is used to determine the affine transformation relationship between the defect-free image and the image under test based on the contour matching algorithm of the reference region contour and the contour to be tested, so as to map the reference region contour to the corresponding position in the image under test and obtain the judgment image.

[0195] The determination unit is used to determine whether the part under test has a misalignment defect based on the determination diagram.

[0196] In some embodiments, the determination unit includes:

[0197] The first determining subunit is used to determine the outer contour of the part area in the determination diagram;

[0198] The second determining subunit is used to determine whether each contour point in the outer contour of the part region exceeds the contour range of the reference region.

[0199] A calculation unit is used to count all contour points that exceed the contour range of the reference region and calculate the distance between the contour points and the contour of the reference region.

[0200] The third determining subunit is used to determine whether the part under test has a misalignment defect based on the distance value.

[0201] In some embodiments, the third determining subunit includes:

[0202] The first determining super sub-unit is used to determine the maximum value among all distance values, compare the maximum value with a second threshold, and determine that the part under test has a misalignment defect when the maximum value exceeds the second threshold.

[0203] In some embodiments, the third determining subunit includes:

[0204] The second determining super sub-unit is used to determine the number of contour points with distance values, compare the number with a number threshold, and determine that the part under test has a misalignment defect when the number exceeds the number threshold.

[0205] In some embodiments, the second acquisition unit includes:

[0206] A segmentation subunit is used to input the defect-free image into the target segmentation model to obtain the first edge of the reference region of the defect-free image;

[0207] The processing subunit is used to perform binarization processing on the first edge to obtain the contour of the reference region.

[0208] In some embodiments, the input unit includes:

[0209] The extraction subunit is used to extract the contour of the part to be tested and the outer contour of the part area of ​​the part to be tested using a skeleton extraction algorithm.

[0210] It should be noted that the principle of the offset defect detection device in this application is similar to that of the aforementioned offset defect detection method. Therefore, the implementation process, implementation principle, and beneficial effects of the offset defect detection device can be found in the description of the implementation process, implementation principle, and beneficial effects of the aforementioned method. Repeated descriptions will not be repeated.

[0211] This application provides an electronic device, including:

[0212] At least one processor; and

[0213] A memory communicatively connected to the at least one processor; wherein,

[0214] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any of the above embodiments.

[0215] This application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to perform the methods described in any of the above embodiments.

[0216] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0217] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0218] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of the electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0219] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0220] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the misalignment defect detection method. For example, in some embodiments, the misalignment defect detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the misalignment defect detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the misalignment defect detection method by any other suitable means (e.g., by means of firmware).

[0221] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0222] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0223] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0224] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0225] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0226] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0227] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0228] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0229] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of detecting a misregistration defect, the method comprising: The method comprises: acquiring a pre-labeled defect image to obtain a training set; processing the pre-labeled defect image to obtain processing data; optimizing an initial loss function of a pre-constructed initial segmentation model based on the processing data to obtain a target loss function; training the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model; inputting a to-be-tested image of a to-be-tested part into the target segmentation model to obtain edge positions of a part region of the to-be-tested part and a reference region of the to-be-tested part; determining whether the to-be-tested part has a misalignment defect based on the edge positions of the part region of the to-be-tested part and the reference region of the to-be-tested part; The method comprises: acquiring the initial loss function, wherein the initial loss function is used to represent a difference between a segmentation result obtained by the initial segmentation model after performing calculation on a defect image and the pre-labeled defect image; determining pixel point positions of the defect image in the processing data whose pixel values are preset values based on the initial loss function; determining a pixel point difference value between a pixel point at the pixel point position and a pixel point of the pre-labeled defect image, discarding a pixel point difference value lower than a first threshold value; updating the initial loss function based on a preset magnification coefficient and the remaining pixel point difference values after discarding to obtain a target loss function; wherein, Loss is a target loss function, i is the horizontal coordinate of each pixel point in the defect image, j is the vertical coordinate of each pixel point in the defect image, n is the number of images, w is the width of the defect image, and h is the height of the defect image, is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel point position ij of the pre-labeled defect image, is the pixel point at the pixel point position ij of the analysis result of the initial segmentation model, is the pixel point at the pixel point position ij of the pre-labeled defect image, is the pixel point at the pixel point position ij of the defect image in the training interface, is the difference between the analysis result of the target segmentation model and the pixel point at the pixel point position ij of the pre-labeled defect image.

2. The method of claim 1, wherein, The target loss function is calculated in the following manner: The method comprises: adjusting the pre-labeled defect image into a defect image of a preset size; performing edge gradient processing on the defect image of the preset size to obtain an edge gradient image; performing a morphological dilation operation on the edge gradient image to obtain a dilated image; 3. The method of claim 1, wherein, performing binarization processing on the dilated image based on a preset binarization threshold to obtain a binarized image; wherein all obtained binarized images are used as processing data. The method comprises: acquiring a reference region contour of a defect-free part image; inputting the to-be-tested image of the to-be-tested part into the target segmentation model to obtain a to-be-tested contour of the to-be-tested part and an outer contour of the part region of the to-be-tested part; determining an affine transformation relationship between the defect-free image and the to-be-tested image based on the reference region contour and the to-be-tested contour by using a contour matching algorithm, so as to map the reference region contour to a corresponding position in the to-be-tested image to obtain a judgment image; 4. The method of claim 3, wherein, determining whether the to-be-tested part has a misalignment defect based on the judgment image. The method comprises: determining an outer contour of the part region in the judgment image; determining whether each contour point in the outer contour of the part region exceeds the reference region contour range; counting distances between all contour points beyond the contour of the reference region and the contour of the reference region; determining whether the part under test has a misalignment defect according to the distance values.

5. The method of claim 4, wherein, The method for determining whether the part under test has a misalignment defect according to the distance values comprises: determining a maximum value among all distance values, comparing the maximum value with a second threshold value, and determining that the part under test has a misalignment defect when the maximum value exceeds the second threshold value.

6. The method of claim 4, wherein, The method for determining whether the part under test has a misalignment defect according to the distance values comprises: determining a number of contour points having distance values, comparing the number with a number threshold value, and determining that the part under test has a misalignment defect when the number exceeds the number threshold value.

7. The method of claim 3, wherein, The method for obtaining the contour of the reference region of the image of the defect-free part comprises: inputting the image of the defect-free part into the target segmentation model to obtain a first edge of the reference region of the image of the defect-free part; performing binaryzation processing on the first edge to obtain the contour of the reference region.

8. The method of claim 3, wherein, The method for inputting the image of the part under test into the target segmentation model to obtain the contour of the part under test and the outer contour of the part region of the part under test comprises: inputting the image of the part under test into the target segmentation model; The target segmentation model extracts the contour of the part under test and the outer contour of the part region of the part under test by using a skeleton extraction algorithm.

9. A misregistration defect detection apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire a pre-labeled defect image to obtain a training set; a processing module configured to process the pre-labeled defect image to obtain processing data; an optimization module configured to optimize an initial loss function of an initial segmentation model based on the processing data to obtain a target loss function; a training module configured to train the initial segmentation model based on the training set and the target loss function to obtain a target segmentation model; an input module configured to input an image of a part under test into the target segmentation model to obtain edge positions of a part region of the part under test and a reference region of the part under test; a detection module configured to determine whether the part under test has a misalignment defect based on the edge positions of the part region of the part under test and the reference region of the part under test. The optimization module comprises: a first acquisition unit configured to acquire the initial loss function, the initial loss function being used to represent a difference between a segmentation result obtained by the initial segmentation model after performing calculation on a defect image and a pre-labeled defect image; a determination unit configured to determine pixel point positions of the defect image in the processing data whose pixel values are preset values based on the initial loss function; a discard unit configured to determine pixel point difference values between the pixel points at the pixel point positions and pixel points of the pre-labeled defect image, and discard pixel point difference values lower than a first threshold value; and an update unit configured to update the initial loss function based on a preset magnification coefficient and the remaining pixel point difference values after discarding to obtain a target loss function. The target loss function is calculated in the following manner: wherein, Loss is a target loss function, i is the horizontal coordinate of each pixel point in the defect image, j is the vertical coordinate of each pixel point in the defect image, n is the number of images, w is the width of the defect image, and h is the height of the defect image, is the difference between the analysis result of the initial segmentation model and the pixel point at the pixel point position ij of the pre-labeled defect image, is the pixel point of the analysis result of the initial segmentation model at the pixel point position ij, is the pixel point of the pre-labeled defect image at the pixel point position ij, is the pixel point of the defect image in the training interface at the pixel point position ij, is the difference between the analysis result of the target segmentation model and the pixel point at the pixel point position ij of the pre-labeled defect image.

10. The apparatus of claim 9, wherein, The acquisition module comprises: An adjusting unit is configured to adjust the pre-labeled defect image into a defect image with a preset size; A processing unit is configured to perform edge gradient processing on the defect image with the preset size to obtain an edge gradient map; An operation unit is configured to perform a morphological dilation operation on the edge gradient map to obtain a dilated image; A binarization unit is configured to perform binarization processing on the dilated image based on a preset binarization threshold to obtain a binarized image; and all obtained binarized images are taken as processing data.

11. The apparatus of claim 9, wherein, The input module comprises: A second acquisition unit is configured to acquire a reference region contour of a defect-free part image; An input unit is configured to input an acquired to-be-tested image of a to-be-tested part into the target segmentation model to obtain a to-be-tested contour of the to-be-tested part and an outer contour of a part region of the to-be-tested part; A transformation unit is configured to determine an affine transformation relationship between a defect-free image and a to-be-tested image based on the reference region contour and the to-be-tested contour by using a contour matching algorithm, to map the reference region contour to a corresponding position in the to-be-tested image to obtain a judgment map; A judgment unit is configured to determine whether the to-be-tested part has a misalignment defect based on the judgment map.

12. The apparatus of claim 11, wherein, The judgment unit comprises: A first determination sub-unit is configured to determine an outer contour of a part region in the judgment map; A second determination sub-unit is configured to determine whether each contour point in the outer contour of the part region exceeds the reference region contour range; A calculation unit is configured to count all contour points exceeding the reference region contour range, and calculate distances between the contour points and the reference region contour; A third determination sub-unit is configured to determine whether the to-be-tested part has a misalignment defect according to the distance value.

13. The apparatus of claim 12, wherein, The third determination sub-unit comprises: A first determination exceeding sub-unit is configured to determine a maximum value in all distance values, compare the maximum value with a second threshold value, and determine that the to-be-tested part has a misalignment defect when the maximum value exceeds the second threshold value.

14. The apparatus of claim 12, wherein, The third determination sub-unit comprises: A second determination exceeding sub-unit is configured to determine a number of contour points with distance values, compare the number with a number threshold value, and determine that the to-be-tested part has a misalignment defect when the number exceeds the number threshold value.

15. The apparatus of claim 11, wherein, The second acquisition unit comprises: A segmentation sub-unit is configured to input the defect-free image into the target segmentation model to obtain a first edge of a reference region of the defect-free image; A processing sub-unit is configured to perform binarization processing on the first edge to obtain the reference region contour.

16. The apparatus of claim 11, wherein, The input unit comprises: An extraction sub-unit is configured to extract the to-be-tested contour of the to-be-tested part and the outer contour of the part region of the to-be-tested part by using a skeleton extraction algorithm.

17. An electronic device, comprising: comprise: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

18. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable the computer to perform the method of any one of claims 1-8.

Images