Double-model fused object surface defect detection method and device and medium

Abstract

The invention relates to a double-model fusion object surface defect detection method and device and a medium, and belongs to the technical field of industrial defect detection.The method comprises the steps that S1, a target object defect image is obtained, a data set is made, and models are trained; s2, inputting a target object image, and synchronously using target detection and a semantic segmentation model to detect a target object; s3, the detection result is judged, if only one model detects defects or no defects are detected, the detection result is output, and the process is ended; s4, if the condition of S3 is not met, calculating the ratio of the area of an intersection region in the detection results of the two models to the respective detection results; and S5, comparing the two ratios with a set intersection threshold value, reserving a result meeting the requirement, repeating the steps S4 and S5, and outputting a final detection result. According to the invention, the detection results of two different models are unified, joint detection is realized, and the accuracy of surface defect detection of the button cell is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of industrial defect detection, and in particular to a double-model fusion object surface defect detection method, device and medium. BACKGROUND

[0002] Commonly used power storage devices such as button batteries will produce defects such as pits, black powder, scratches, liquid leakage and wear on the surface during production. These surface defects will affect the quality and performance of the button battery. In order to ensure the quality and safety of the button battery, surface defect detection becomes an essential part of the production process. Button battery surface defects are diverse and complex, and all defects can occur on the positive and negative surfaces. The negative surface of the button battery is designed to be uniform, and the positive surface is engraved with text and pattern information. The different positive and negative background features increase the difficulty of defect detection.

[0003] A button battery negative shell defect detection method based on machine vision is disclosed in Chinese patent No. CN111766245A. This method uses traditional image processing algorithms to complete button battery negative shell defect detection at five workstations. Since the positive and negative background features of the button battery are different, this method can only detect defects in the negative area and will lose its effectiveness in the positive area.

[0004] A button battery positive pit detection method is disclosed in Chinese patent No. CN110108710A. This method uses a combination of reflective gratings and ring-shaped light sources to design an image acquisition device. This light source combination improves the contrast between the button battery pits and the normal area. Finally, a digital image processing method is used to identify the defect area. This method can only detect one type of positive pit defect and cannot meet the detection requirements of multiple types of button battery surface defects.

[0005] A glass panel defect detection method based on a semantic segmentation and target detection fusion model is disclosed in Chinese patent No. CN113724231A. This method first uses a semantic segmentation model to detect the image, superimposes the obtained mask on the original image to output an image, and finally uses an image classification and target detection network to detect defects. This method combines a semantic segmentation model, an image classification model and a target detection model in series. Since the accuracy of each model is difficult to reach 100%, the use of a series connection will gradually reduce the detection accuracy, so the stability of this method is poor.

[0006] In summary, the existing technology has the following problems: the target object has many types of defects, diverse features, significant differences in the same defect features (large intra-class differences), and cross-scale features, which result in low accuracy of a single deep learning model (image classification, target detection, semantic segmentation, etc.). SUMMARY

[0007] To solve the above problems, the application provides a double-model fusion object surface defect detection method based on target detection and semantic segmentation, a device and a medium, which detect the surface defects of button cells, solve the problem that the existing detection methods cannot detect all types of defects, and also solve the problem that the output information of the target detection model and the semantic segmentation model is inconsistent, thereby realizing joint detection of two different models.

[0008] The steps for solving the above problems are as follows:

[0009] S1, collect target object defect images, make the target object defect images into a target detection data set and a semantic segmentation data set, and train a target detection model and a semantic segmentation model;

[0010] S2, simultaneously detect the target object images by using the target detection model and the semantic segmentation model trained in S1, and output the detection results of the two models, wherein the detection results of the two models both include the position, area, confidence and predicted category of defects;

[0011] S3, judge the detection results of the two models in S2, output the results if the judgment condition is met, and end the detection process if the judgment condition is not met, and enter S4;

[0012] S4, calculate the ratio of the intersection area to the area of each detection result in the detection results of the two models in S2;

[0013] S5, set an intersection threshold, compare the ratio in S4 with the set intersection threshold, and then perform cyclic judgment according to the confidence and the predicted category in S2, and output the final detection result.

[0014] Further, the target detection model in S1 is YOLO v5, and the semantic segmentation model in S1 is DeepLabV3+.

[0015] Further, the target detection model in S2 can directly output the position, area, confidence and predicted category of defects in the detection result, and the semantic segmentation model in S2 can directly output the position and predicted category of defects in the detection result.

[0016] Further, the calculation process of the area and confidence of defects in the detection result of the semantic segmentation model in S2 is as follows:

[0017] a, extract the confidence corresponding to each pixel point of the input image from the three-dimensional tensor at the end of the semantic segmentation model, wherein the channel number of the three-dimensional tensor represents the predicted category of each pixel point, and the confidence is converted into a two-dimensional confidence matrix;

[0018] b. Convert the two-dimensional confidence matrix into a single-channel image. The connected components outside the background region in the single-channel image represent the detection results. Set a connected component threshold. Connected components with an area smaller than the set connected component threshold are considered noise in the detection results. The noise is filtered out, and the retained connected components are used as the final detection results of the semantic segmentation model. The area of ​​the detection results is represented by the bounding rectangle of each connected component.

[0019] c. Apply a two-dimensional Gaussian filter to the two-dimensional confidence matrix, and then calculate the confidence score of each defect in the semantic segmentation model detection result based on the two-dimensional confidence matrix. The specific formula for the two-dimensional Gaussian filter is as follows:

[0020]

[0021] Where (i,j) represents the position of the midpoint of the Gaussian kernel, k is used to control the size of the convolution kernel, σ is the standard deviation, and e is a constant.

[0022] Furthermore, the detection result of the target detection model in S2 is represented by M. i M indicates i The area is M i A. Confidence level M i C. Prediction category is M i F, where i represents the i-th detection result of the object detection model;

[0023] The detection results of the semantic segmentation model in S2 are represented by Y. j Y indicates j The area is Y j A. Confidence level Y j C. Predicted category is Y j F, where j represents the j-th detection result of the semantic segmentation model.

[0024] Furthermore, the judgment condition in S3 is: neither the target detection model nor the semantic segmentation model detects the defect, or only one of the two models detects the defect.

[0025] Furthermore, the area of ​​the intersecting region in S4 is represented by U. ij This means that the detection results of the two models are mapped onto the same image, through U ij M i A and Y j A calculates the ratio of the area of ​​the intersecting region to the area of ​​the detection results of each of the two models, using the following formula:

[0026]

[0027]

[0028] Among them SM ij M represents the ratio of the area of ​​the intersecting region to the area detected by the target detection model. i A represents the area detected by the object detection model; SY ij Y represents the ratio of the area of ​​the intersecting region to the area detected by the semantic segmentation model. j A represents the area of ​​the detection result of the semantic segmentation model.

[0029] Furthermore, in S5, the intersection threshold is set to T, and SM ij If the value is greater than T, it means that the intersection region in S4 is equivalent to the detection position of the i-th result of the target detection model;

[0030] SY ij If the value is greater than T, it means that the intersection region in S4 is equivalent to the detection position of the j-th result of the semantic segmentation model;

[0031] According to SM ij SY ij The relationship with T can be divided into the following four cases:

[0032] Scenario 1, SM ij and SY ij All are greater than or equal to T;

[0033] Scenario 2, SM ij and SY ij All are less than T;

[0034] Scenario 3, SM ij Greater than or equal to T, SY ij Less than T;

[0035] Scenario 4, SM ij Less than T, SY ij Greater than or equal to T;

[0036] The loop judgment in S5 represents each detection result M of the target detection model. i Each detection result Y from the semantic segmentation model needs to be compared with the result Y. j Make a location determination.

[0037] The present invention also provides an object surface defect detection device, including one or more processors, for implementing the object surface defect detection method as described above.

[0038] The present invention also provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the object surface defect detection method as described above.

[0039] The advantages of this invention are:

[0040] 1) Optimizing the detection results of the semantic segmentation model can filter out noise with an area smaller than the threshold in the detection results, and at the same time calculate the confidence of each detection result of the semantic segmentation model.

[0041] 2) This invention unifies the output results of the target detection model and the semantic segmentation model, realizes joint detection of the two models, and improves the accuracy of button battery surface defect detection. Attached Figure Description

[0042] Figure 1 This is one of the flowcharts of a dual-model fusion method for detecting surface defects in objects according to the present invention;

[0043] Figure 2 This is the second flowchart of a dual-model fusion method for detecting surface defects in objects according to the present invention.

[0044] Figure 3 This is a flowchart illustrating the calculation of the area and confidence score of defects in the semantic segmentation model detection results of a dual-model fusion method for detecting surface defects of an object according to the present invention.

[0045] Figure 4 This is a schematic diagram of the structure of a surface defect detection device according to the present invention. Detailed Implementation

[0046] The embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. These embodiments are provided to better understand the present invention. Other embodiments obtained by those skilled in the art based on the embodiments of the present invention are all within the protection scope of the present invention.

[0047] This invention provides a method for detecting surface defects of objects using dual-model fusion. The following detailed explanation uses the detection of a button battery as an example:

[0048] like Figure 1 As shown, the detection method includes the following steps:

[0049] S1. Collect defective images of button batteries, and create object detection and semantic segmentation datasets from these images. Train the object detection and semantic segmentation models accordingly.

[0050] In this embodiment, a monochrome area array camera is used to acquire defect images of button batteries. The defect images of button batteries are divided into five categories: dents, black powder, scratches, leakage, and wear, distinguishing between positive and negative electrodes, for a total of 10 types of defects. Black powder defects are further divided into blocky and scattered defects, and leakage defects are further divided into sheet-like and diffused defects.

[0051] In this embodiment, the target detection dataset is created using the Labelimg software. The defect types are: positive electrode pit, negative electrode pit, positive electrode scattered black powder, negative electrode scattered black powder, positive electrode divergent leakage, negative electrode divergent leakage, positive electrode wear, and negative electrode wear, for a total of eight defects.

[0052] In this embodiment, the semantic segmentation dataset was created using Labelme software. The defect types are: positive electrode block black powder, negative electrode block black powder, positive electrode sheet leakage, negative electrode sheet leakage, positive electrode scratch, and negative electrode scratch, for a total of six defects.

[0053] In this embodiment, OpenCV is used to enhance and expand the object detection dataset and semantic segmentation dataset. The enhancement strategies are rotation, translation, and scaling.

[0054] In this embodiment, the object detection model used is YOLO v5, and its network structure has been optimized by adding a CBAM attention mechanism between the Backbone and Neck. The YOLO v5 model is then trained using an object detection dataset. Alternatively, other similar models can also be used.

[0055] In this embodiment, the semantic segmentation model used is DeepLabV3+, and its network structure has been optimized by replacing the Xception network in the backbone with MobileNetV2. The DeepLabV3+ model is then trained using a semantic segmentation dataset. Alternatively, other similar models can also be used.

[0056] S2 uses the object detection model and semantic segmentation model trained in S1 to simultaneously detect defects in the button battery image, outputting the detection results of both models. Both detection results include the location, area, confidence level, and predicted category of the defects, specifically:

[0057] In this embodiment, a target detection model is used to detect the input button battery image, and the location, area, confidence level, and predicted category of the defect can be directly output.

[0058] In this embodiment, a semantic segmentation model is used to detect the input button battery image, which can directly output the location and predicted category of the defect.

[0059] like Figure 3 As shown, the calculation process for the area and confidence score of defects in the detection results of the semantic segmentation model is as follows:

[0060] a) Obtain a three-dimensional tensor from the end of the semantic segmentation model. The three-dimensional tensor stores the confidence of the predicted class corresponding to each pixel of the input image. Extract the highest confidence corresponding to each pixel to obtain a two-dimensional confidence matrix. Count the number of channels of the three-dimensional tensor to which each confidence belongs in the two-dimensional confidence matrix. Determine the predicted class of the pixel based on the number of channels.

[0061] b. Convert the two-dimensional confidence matrix into a single-channel image. Set the confidence threshold to 100. Connected regions in the single-channel image whose area is less than the confidence threshold are considered noise. The noise is masked out, and the retained connected regions are used as the final detection results of the semantic segmentation model. The area of ​​the detection results of the semantic segmentation model is represented by the bounding rectangle of each connected region.

[0062] c. Apply a two-dimensional Gaussian filter to the two-dimensional confidence matrix corresponding to each preserved connected component. Then calculate the sum of the two-dimensional confidence matrices. Divide the sum by the area of ​​each preserved connected component to obtain the confidence score of the semantic segmentation model's detection result. The specific formula for the two-dimensional Gaussian filter is as follows:

[0063]

[0064] Where (i,j) represents the position of the midpoint of the Gaussian kernel, k is used to control the size of the convolution kernel, σ is the standard deviation, and e is a constant.

[0065] In this embodiment, S2 unifies the detection results of the target detection model and the semantic segmentation model. The two models can synchronously output the location, area, confidence level and predicted category of each defect in the detected image.

[0066] S3, as Figure 2 As shown, the detection results of the two models in S2 are judged. If the judgment condition is met, the result is output and the detection process ends. Specifically:

[0067] In this embodiment, the judgment condition in S3 is: neither the object detection model nor the semantic segmentation model detects a defect, or only one of the object detection model and the semantic segmentation model detects a defect. That is, both "neither the object detection model nor the semantic segmentation model detects a defect" and "only one of the object detection model and the semantic segmentation model detects a defect" are met. If the judgment condition is met, the result is directly output, and the detection process ends; if the judgment condition in S3 is not met, the detection processes in S4 and S5 continue.

[0068] S4, such as Figure 3 As shown, if the judgment condition in S3 is not met, it is necessary to calculate the ratio of the area of ​​the intersecting region in the detection results of the two models to the area of ​​their respective detection results based on the detection results of the two models in S2. Specifically:

[0069] In this embodiment, the detection result of the target detection model is represented by M. i M indicates i The area is M i A. Confidence level M i C. Prediction Category M i F, where i represents the i-th detection result of the object detection model.

[0070] In this embodiment, the detection result of the semantic segmentation model is represented by Y. j Y indicates j The area is Y j A. Confidence level Y j C. Predicted Category Y j F, where j represents the j-th detection result of the semantic segmentation model.

[0071] In this embodiment, the area of ​​the intersecting region in S4 is represented by U. ij It means that through U ij M i A and Y j A calculates the ratio of the area of ​​the intersecting region to the area of ​​the detection results of each of the two models, using the following formula:

[0072]

[0073]

[0074] Among them SM ij M represents the ratio of the area of ​​the intersecting region to the area detected by the target detection model. i A represents the area detected by the object detection model, SY ij Y represents the ratio of the area of ​​the intersecting region to the area detected by the semantic segmentation model. j A represents the area of ​​the detection result of the semantic segmentation model.

[0075] S5, such as Figure 2 As shown, an intersection threshold is set, the ratio in S4 is compared with the set intersection threshold, and then the confidence level and predicted category in S2 are used for iterative judgment to output the final detection result, specifically:

[0076] In this embodiment, the intersection threshold T in S5 is set to 0.3, SM ij A value greater than 0.3 indicates that the intersection region in S4 is equivalent to the detection position of the i-th result of the target detection model; SY ij A value greater than 0.3 indicates that the intersection region in S4 is equivalent to the detection location of the j-th result of the semantic segmentation model; according to SM ij SY ijBased on the relationship with T, the detection results in S5 can be divided into four cases:

[0077] Scenario 1, SM ij and SY ij All are greater than or equal to T;

[0078] Scenario 2, SM ij and SY ij All are less than T;

[0079] Scenario 3, SM ij Greater than or equal to T, SY ij Less than T;

[0080] Scenario 4, SM ij Less than T, SY ij Greater than or equal to T.

[0081] In this embodiment, Case 1 indicates that the target detection model and the semantic segmentation model detect the same location. In this case, the comparison M... i C and Y j C. Retain detection results with high confidence.

[0082] Case 2 indicates that the object detection model and the semantic segmentation model detect different locations, so two detection results are retained.

[0083] Case 3 indicates that the detection location of the semantic segmentation model includes the detection location of the object detection model. The detection result of the semantic segmentation model is retained, while the detection result of the object detection model is discarded.

[0084] Case 4 indicates that the detection location of the object detection model includes the detection location of the semantic segmentation model. The detection result of the object detection model is retained, while the detection result of the semantic segmentation model is discarded.

[0085] In this embodiment, after determining cases 1 to 4, step S4 is performed again to calculate M. (i+1) With Y j SM (i+1)j and SY (i+1)j Each detection result M of the target detection model i Each detection result Y from the semantic segmentation model needs to be compared with the result Y. j Perform positional judgments; discarded detection results do not need to be judged in the next round. After all judgments are completed, output the final result.

[0086] It should be noted that the surface defect detection method of the present invention can be used not only to detect button batteries, but also to detect surface defects of other objects.

[0087] Corresponding to the aforementioned embodiment of a dual-model fusion method for detecting surface defects in objects, the present invention also provides an embodiment of an object surface defect detection device.

[0088] See Figure 4 The present invention provides an object surface defect detection device, which includes one or more processors for implementing the object surface defect detection method in the above embodiments.

[0089] An embodiment of the object surface defect detection device of the present invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, including a surface defect detection device for objects according to the present invention. Except for... Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing-capable manufacturing process in which the device is located in the embodiment may also include other hardware depending on the actual function of the data processing-capable device, which will not be described in detail here.

[0090] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This invention also provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the object surface defect detection method described in the above embodiments.

[0093] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0094] The above are merely preferred embodiments of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. It should be noted that for those skilled in the art, alternative solutions that can be easily conceived are all within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for detecting surface defects of an object using dual-model fusion, characterized in that, The method includes the following steps: S1. Collect images of defects in the target object, and create target detection datasets and semantic segmentation datasets from these images. Train the target detection model and the semantic segmentation model using these datasets. S2 uses the object detection model and semantic segmentation model trained in S1 to detect the target object image simultaneously, and outputs the detection results of the two models. Both detection results include the location, area, confidence level and predicted category of the defect. S3: Judge the detection results of the two models in S2. If the judgment conditions are met, output the result and the detection process ends. If not, proceed to S4. S4, based on the detection results of the two models in S2, calculate the ratio of the area of ​​the intersecting region in the detection results of the two models to the area of ​​their respective detection results; S5, set the intersection threshold, compare the ratio in S4 with the set intersection threshold, and then perform iterative judgment based on the confidence level and predicted category in S2 to output the final detection result.

2. The method for detecting surface defects of an object using dual-model fusion according to claim 1, characterized in that, The target detection model in S1 is YOLOv5, and the semantic segmentation model in S1 is DeepLabV3+.

3. The method for detecting surface defects of an object using dual-model fusion according to claim 1, characterized in that, The calculation process for the area and confidence score of defects in the detection results of the semantic segmentation model in S2 is as follows: a. Extract the confidence level of each pixel in the input image from the three-dimensional tensor at the end of the semantic segmentation model. The number of channels in the three-dimensional tensor represents the predicted category of each pixel. Convert the confidence level into a two-dimensional confidence matrix. b. Convert the two-dimensional confidence matrix into a single-channel image. The connected components outside the background region in the single-channel image represent the detection results. Set a connected component threshold. Connected components with an area smaller than the set connected component threshold are considered noise in the detection results. The noise is filtered out, and the retained connected components are used as the final detection results of the semantic segmentation model. The area of ​​the detection results is represented by the bounding rectangle of each connected component. c. Apply a two-dimensional Gaussian filter to the two-dimensional confidence matrix, and then calculate the confidence score of each defect in the semantic segmentation model detection result based on the two-dimensional confidence matrix. The specific formula for the two-dimensional Gaussian filter is as follows: Where (i,j) represents the position of the midpoint of the Gaussian kernel, k is used to control the size of the convolution kernel, σ is the standard deviation, and e is a constant.

4. The method for detecting surface defects of an object using dual-model fusion according to claim 1, characterized in that, The detection results of the target detection model in S2 are represented by M. i M indicates i The area is M i A. Confidence level M i C. Prediction category is M i F, where i represents the i-th detection result of the object detection model; The detection results of the semantic segmentation model in S2 are represented by Y. j Y indicates j The area is Y j A. Confidence level Y j C. Predicted category is Y j F, where j represents the j-th detection result of the semantic segmentation model.

5. The method for detecting surface defects of an object using dual-model fusion according to claim 1, characterized in that, The judgment condition in S3 is: neither the target detection model nor the semantic segmentation model detects a defect, or only one of the two models detects a defect.

6. The method for detecting surface defects of an object using dual-model fusion according to claim 1, characterized in that, The area of ​​the intersecting region in S4 is represented by U. ij This means that the detection results of the two models are mapped onto the same image, through U ij M i A and Y j A calculates the ratio of the area of ​​the intersecting region to the area of ​​the detection results of each of the two models, using the following formula: Among them SM ij M represents the ratio of the area of ​​the intersecting region to the area detected by the target detection model. i A represents the area detected by the object detection model; SY ij Y represents the ratio of the area of ​​the intersecting region to the area detected by the semantic segmentation model. j A represents the area of ​​the detection result of the semantic segmentation model.

7. The method for detecting surface defects of an object using dual-model fusion according to claim 6, characterized in that, In S5, the intersection threshold is set to T, SM ij If the value is greater than T, it means that the intersection region in S4 is equivalent to the detection position of the i-th result of the target detection model; SY ij If the value is greater than T, it means that the intersection region in S4 is equivalent to the detection position of the j-th result of the semantic segmentation model; According to SM ij SY ij The relationship with T can be divided into the following four cases: Scenario 1, SM ij and SY ij All are greater than or equal to T; Scenario 2, SM ij and SY ij All are less than T; Scenario 3, SM ij Greater than or equal to T, SY ij Less than T; Scenario 4, SM ij Less than T, SY ij Greater than or equal to T; The loop judgment in S5 represents each detection result M of the target detection model. i Each detection result Y from the semantic segmentation model needs to be compared with the result Y. j Make a location determination.

8. A surface defect detection device for an object, characterized in that, It includes one or more processors for implementing the object surface defect detection method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the object surface defect detection method according to any one of claims 1-7.

Images