Glue defect detection method and device, storage medium and electronic equipment

Abstract

The invention provides a glue defect detection method and device, a storage medium and electronic equipment. The method comprises the following steps: inputting a to-be-detected image into a pre-trained glue position identification model, and obtaining a glue position identification result corresponding to the to-be-detected image output by the glue position identification model, the glue position identification model is obtained by inputting a training sample set formed by a plurality of sample images and corresponding glue position labels into an initial to-be-trained network for training; and if the glue position identification result satisfies a preset rule, determining that the to-be-detected image has glue breakage. According to the invention, the glue position in the to-be-detected image is identified by using the glue position identification model obtained by training a large number of sample glue images and the corresponding glue position labels, so that the accuracy of glue position identification is improved; and the reliable robustness of the glue position identification model also prevents the identification result from being influenced by the environment when the image to be detected is shot.

Description

Technical Field

[0001] The present invention relates to the technical field of glue application position detection on chip - carrying plates, and more specifically, to glue application position detection based on images, especially a glue defect detection method, device, storage medium, and electronic device based on deep learning. Background Art

[0002] When using AA glue application on chip - carrying plates, problems such as broken glue and glue width not meeting specifications often occur, resulting in customer complaints. Therefore, there is an urgent need to develop a glue application detection software to detect whether there is broken glue on the target plate and calculate the glue width. Figure 1 A comparison diagram of the complete glue application image and the broken - glue image provided by the prior art is as Figure 1 shown. In the left complete glue application image, there is no broken glue at the position indicated by the arrow, while in the right broken - glue image, there is a broken - glue phenomenon at the position indicated by the arrow. Due to the broken - glue situation in the right image, defect detection for glue breakage is required.

[0003] Figure 2 Images collected and processed by the glue defect detection algorithm in the prior art are Figure 2 From left to right, the four images represent the result diagrams corresponding to the four steps of the glue defect detection algorithm in the prior art, as Figure 2 shown. The currently used algorithm steps are as follows: First step, collect the picture of the plate before glue application ( Figure 2 the second - from - left image in Figure 2 ); Second step, collect the picture of the plate after glue application ( Figure 2 the left - most image in Figure 2 ); Third step, through the same pre - processing (image enhancement) of the images before and after glue application, and then use the glue application image ( Figure 2 the left - most image in Figure 2 ) to "subtract" the image without glue application ( Figure 2 the second - from - left image in Figure 2 ), the contour image of the glue on the plate can be obtained ( Figure 2 the third - from - left image in Figure 2 ); Fourth step, binarize the contour image ( Figure 2 the third - from - left image in Figure 2 ), that is, set the gray - scale value of the pixels on the contour image to 0 or 1 (0 represents black, 1 represents white) to obtain a binarized image ( Figure 2 the fourth - from - left image in

[0004] This algorithm is affected by the shooting environment and has problems such as being unable to accurately identify or not being able to identify the glue position. Summary of the Invention

[0005] The present invention provides a glue defect detection method, device, storage medium and electronic device to solve the problems in the prior art that the method for detecting glue defects on a board is affected by the shooting environment of the image to be detected and cannot recognize or accurately recognize broken glue. By using a glue position recognition model trained with a large number of sample glue images and corresponding glue position labels to recognize the glue position in the image to be detected, the accuracy of glue position recognition is improved, and the reliable robustness of the glue position recognition model also avoids the influence of the recognition result by the environment when the image to be detected is taken.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or be learned in part through the practice of the present application.

[0007] In a first aspect, the present invention provides a glue defect detection method, including:

[0008] Input the image to be detected into a pre-trained glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model, where the glue position recognition model is obtained by training an initial network to be trained with a training sample set composed of a plurality of sample images and corresponding glue position labels;

[0009] If the glue position recognition result meets a preset rule, it is determined that there is broken glue in the image to be detected.

[0010] Optionally, the glue position recognition result is represented by a binary image. Based on the rule that if the glue position recognition result meets a preset rule, it is determined that there is broken glue in the image to be detected, specifically including:

[0011] Based on the binary image corresponding to the glue position recognition result, determine the largest contour of the glue in the binary image; where the largest contour is the continuous glue area with the largest area;

[0012] If the perimeter of the circumscribed rectangle of the largest contour, the glue width of the glue in the largest contour, and the true perimeter of the largest contour meet preset conditions, it is determined that there is broken glue.

[0013] Optionally, according to the glue defect detection method provided by the present invention, the rule that if the perimeter of the circumscribed rectangle of the largest contour, the glue width of the glue in the largest contour, and the true perimeter of the largest contour meet preset conditions, it is determined that there is broken glue, specifically includes:

[0014] If the first condition and / or the second condition are met, it is determined that there is broken glue in the glue;

[0015] Among them, the first condition includes that there is a value less than or equal to zero in the glue width of the glue in the maximum contour, and the second condition includes that the ratio of the true perimeter of the maximum contour to the perimeter of the circumscribed rectangle of the maximum contour is greater than a preset threshold.

[0016] Optionally, according to the glue defect detection method provided by the present invention, the glue position recognition model is constructed using the semantic segmentation model in yolov8.

[0017] Optionally, according to the glue defect detection method provided by the present invention, the glue position recognition model includes a feature extraction network, a feature pyramid network, and a segmentation branch network connected in sequence;

[0018] Correspondingly, the inputting the image to be detected into the glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model specifically includes:

[0019] Inputting the image to be detected into the feature extraction network to obtain multiple feature maps of different scales of the image to be detected output by the feature extraction network;

[0020] Inputting the multiple feature maps of different scales into the feature pyramid network to obtain a fused feature map output by the feature pyramid network;

[0021] Inputting the fused feature map into the segmentation branch network to obtain the glue position recognition result output by the segmentation branch network.

[0022] Optionally, according to the glue defect detection method provided by the present invention, the loss function in the training process of the glue position recognition model is constructed based on the cross-entropy loss formula.

[0023] Optionally, according to the glue defect detection method provided by the present invention, the glue position label corresponding to the sample image is manually marked.

[0024] In a second aspect, the present invention provides a glue defect detection device, and the device includes:

[0025] An identification module, configured to input an image to be detected into a glue position recognition model to obtain a glue position recognition result corresponding to the image to be detected output by the glue position recognition model, where the glue position recognition model is obtained by training an initial network to be trained by inputting a training sample set composed of a plurality of sample images and corresponding glue position labels;

[0026] A judgment module, configured to determine that there is a glue break in the image to be detected if the glue position recognition result meets a preset rule.

[0027] In a third aspect, the present invention provides a computer-readable storage medium, in which at least one program code is stored, and the at least one program code is loaded and executed by a processor to implement the operations performed by the glue defect detection method as described in the first aspect.

[0028] In a fourth aspect, the present invention provides an electronic device, which includes one or more processors and one or more memories. At least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the glue defect detection method as described in the first aspect.

[0029] The glue defect detection method provided by the present invention uses a glue position recognition model trained with a large number of sample glue images and corresponding glue position labels to recognize the glue position in the image to be measured, improving the accuracy of glue position recognition. Moreover, the reliable robustness of the glue position recognition model also avoids the influence of the recognition result by the environment when the image to be measured is taken.

[0030] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 It is a comparison diagram of the complete glue application image and the broken glue image provided by the prior art;

[0033] Figure 2 It is the image collected and processed in the glue defect detection algorithm in the prior art;

[0034] Figure 3 It is a schematic flowchart of the glue defect detection method of the present invention;

[0035] Figure 4 It is a binary comparison diagram corresponding to the glue position recognition result provided by the present invention;

[0036] Figure 5 It is a binary image corresponding to the glue position recognition result provided by the present invention;

[0037] Figure 6 It is a schematic diagram of the circumscribed rectangle of the maximum contour provided by the present invention;

[0038] Figure 7 Schematic diagram of the true perimeter of the maximum contour provided by the present invention;

[0039] Figure 8 Flowchart of the method for judging whether the glue is broken provided by the present invention;

[0040] Figure 9 Schematic diagram of the process of labeling with lableme software provided by the present invention;

[0041] Figure 10 Schematic diagram of the process of converting a json file to a txt file provided by the present invention;

[0042] Figure 11 Schematic diagram of the structure of the glue defect detection device provided by the present invention;

[0043] Figure 12 Schematic diagram of the physical structure of an electronic device provided by the present invention. Detailed implementation manners

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0045] In addition, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of this application. However, those skilled in the art will realize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be used. In other cases, well-known methods, devices, implementations, or operations are not shown or described in detail to avoid obscuring aspects of this application.

[0046] The block diagrams shown in the drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are merely illustrative and not necessarily include all the contents and operations / steps, nor are they necessarily executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0048] It should be noted that: "a plurality of" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the front and rear associated objects.

[0049] The following will be combined with Figures 3 - 11 to describe the glue defect detection method, device, storage medium, and electronic device of the present invention.

[0050] Figure 3 is a schematic flowchart of the glue defect detection method of the present invention, and the execution subject is a glue defect detection device. As Figure 3 shown, the method of the present invention includes the following steps:

[0051] Step 310, input the image to be detected into a pre-trained glue position recognition model, and obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model. Among them, the glue position recognition model is obtained by training an initial network to be trained by inputting a training sample set composed of a plurality of sample images and corresponding glue position labels.

[0052] Here, the image to be detected is the image of the board to be judged whether there is a glue breakage phenomenon. The glue on the board is used to paste the chip on the board. Since the chip is usually rectangular, the shape of the normal glue application is a non-interrupted rectangle with a certain width.

[0053] Specifically, input the image to be detected into the glue position recognition model, and the glue position recognition model outputs the glue position recognition result corresponding to the image to be detected. Here, the glue position recognition result is represented by a matrix graph of 0 and 1 values, that is, represented by a binary graph. The length and width of the matrix are the length and width of the image to be detected respectively. For example: use the coordinates of the 1 value in the matrix to represent the pixel position of the glue in the image to be detected, and use the coordinates of the 0 value in the matrix to represent the pixel position in the image to be detected that is not glue.

[0054] The glue position recognition model here is a pre-trained model, which is used to determine the position of the glue in the input glue image based on the input glue image and output the glue position recognition result. Therefore, before performing step 310, the glue position recognition model can also be pre-trained, and specifically, the glue position recognition model can be trained in the following way: First, collect a large number of sample glue images, and at the same time determine the glue position label corresponding to each sample glue image. Among them, a large number of sample glue images can be plate images collected from various shooting environments, such as backlight scenes, daylight scenes, indoor scenes, outdoor scenes, etc. In this way, the glue position recognition model trained by the sample glue images collected under various shooting environments can obtain accurate glue position recognition results without being affected by the shooting environment of the image to be detected during use; and the glue position label corresponding to the sample glue image can be manually calibrated. Train the initial model based on a large number of sample images and the corresponding glue position labels to obtain the glue position recognition model. Among them, the initial model can be a single neural network model or a combination of multiple neural network models. For example, use the semantic segmentation model or UNet model in yolov8. The embodiments of the present invention do not specifically limit the type and structure of the initial model.

[0055] Step 320, if the glue position recognition result meets the preset rule, it is determined that there is glue breakage in the image to be detected.

[0056] Specifically, a binary matrix corresponding to the glue position recognition result can be obtained to get a binary image, that is, the shape of the white glue is displayed on the black background image. Based on this binary image, further judgment on whether there is glue breakage can be carried out. A preset rectangular frame can be used to perform row scanning and column scanning along the glue position to find out whether there is a place where the glue width is zero. If there is a place where the glue width is zero, it means that there is a discontinuous glue shape; it can also be judged by the relationship between the true perimeter of the maximum glue contour, the perimeter of the circumscribed rectangle of the maximum glue contour, and the glue width of the glue in the maximum contour. Among them, the maximum glue contour is the continuous glue area with the largest area; therefore, the method for judging whether there is glue breakage in the image to be detected is not specifically limited here.

[0057] The glue defect detection method provided by the embodiments of the present invention uses the glue position recognition model trained by a large number of sample glue images and the corresponding glue position labels to recognize the glue position in the image to be detected, improving the accuracy of glue position recognition. Moreover, the reliable robustness of the glue position recognition model also avoids the influence of the recognition result by the environment during the shooting of the image to be detected.

[0058] Based on any of the above embodiments, in this method, if the glue position recognition result meets a preset rule, it is determined that there is glue breakage in the image to be detected, which specifically includes:

[0059] Based on the glue position recognition result, it is determined whether there is glue breakage in the image to be detected, which specifically includes:

[0060] Based on the binary image corresponding to the glue position recognition result, the maximum contour of the glue in the binary image is determined; wherein, the maximum contour is the continuous glue area with the largest area;

[0061] If the perimeter of the circumscribed rectangle of the maximum contour, the glue width of the glue in the maximum contour, and the true perimeter of the maximum contour meet the preset conditions, it is determined that there is glue breakage.

[0062] Specifically, Figure 4 is the binary comparison image corresponding to the glue position recognition result provided by the present invention, Figure 4 in which the right figure is the binary image of the glue position output after the glue position recognition of the broken glue plate, Figure 4 and the left figure is the binary image of the glue position output after the glue position recognition of the complete glue plate. It can be seen that Figure 4 there is an obvious discontinuous position at the lower right of the glue in the right figure.

[0063] The following describes the method for processing the binary image of the glue position to realize the automatic judgment of whether there is a glue breakage position.

[0064] First, the method for measuring the glue width in the image is described. Figure 5 is the binary image corresponding to the glue position recognition result provided by the present invention. As Figure 5 shown, in the binary image of the glue position, a ROI (Region of Interest) area A is taken at the center of each of the upper, lower, left, and right side lengths. The size of the ROI area A can be set according to experience. For example, ROI_width = 64 and ROI_height = 64 are set. Then, the average glue widths within the ROI area A corresponding to the upper, lower, left, and right side lengths are calculated respectively. The calculation method of the glue width is as follows (where the pixel brightness value of the black area in the binary image of the glue position is 0, and the pixel brightness value of the white area is 1):

[0065] 1. Calculate the average glue width within the upper side area (the calculation method of the lower side glue width is the same).

[0066] Step 1: Traverse all pixel brightness values within this ROI area, and count the total number of pixels with a brightness value of 255, denoted as count. Step 2: Based on the formula average_glue_width = count / ROI_width, determine the average glue width average_glue_width within the upper area.

[0067] 2. Calculate the average glue width within the left area (the calculation method for the right glue width is the same)

[0068] Step 1: Traverse all pixel brightness values within this ROI area, and count the total number of pixels with a brightness value of 255, denoted as count. Step 2: Based on the formula average_glue_width = count / ROI_height, determine the average glue width average_glue_width within the left area.

[0069] Then, detect whether there is a glue break in the binary image of the glue position. First, use the findContours function in the OpenCV vision library to find the largest contour in the image. Here, the largest contour is the continuous glue area that contains the largest area in the binary image of the glue position. Among them, the FindContours function is a function in the OpenCV vision library used to retrieve contours from a binary image and return the number of detected contours.

[0070] Figure 6 It is a schematic diagram of the circumscribed rectangle of the largest contour provided by the present invention, as Figure 6 shown, Figure 6 The leftmost figure is a complete and continuous glue pattern. Therefore, the largest continuous glue area with the largest area is the contour of the entire rectangular glue, that is, the largest contour, and the smallest rectangle including the entire rectangular glue is the circumscribed rectangle of the largest contour; Figure 6 The middle figure is a glue pattern with a section of broken glue at the lower right corner. Therefore, the largest continuous glue area with the largest area is the rectangular glue with a break at the lower right corner, and the smallest rectangle including this rectangular glue is the circumscribed rectangle of the largest contour; Figure 6 The rightmost figure is a glue pattern with multiple breaks. Assume that the largest continuous glue area found by the findContours function is a section of glue area at the lower left corner. Then Figure 6 the largest contour in the rightmost figure includes a section of glue area at the lower left corner, and the smallest rectangle including this section of glue area is Figure 6 the circumscribed rectangle of the largest contour in the rightmost figure. The perimeter of the circumscribed rectangle of the largest contour is denoted as arcLength_Rect.

[0071] Furthermore, the maximum contour extracted previously is fitted using the polygon fitting technique to obtain the accurate perimeter of the maximum contour, i.e., the true perimeter, denoted as arcLength_actual. Among them, the polygon fitting technique is used to fit irregular contours or boundary lines into simpler geometric shapes.

[0072] Figure 7 It is a schematic diagram of the true perimeter of the maximum contour provided by the present invention, as Figure 7 shown, Figure 7 The left figure shows the true perimeter of the maximum contour in the complete glue pattern. Since there is no glue breakage in the complete glue pattern, the true perimeter is the perimeter of the outer contour of the glue, represented by the gray line; Figure 7 The right figure shows the true perimeter of the maximum contour in the glue breakage pattern. Since there is a glue breakage at the lower right corner, the true perimeter is the sum of the outer perimeters of the glue area, represented by the gray line.

[0073] Finally, based on the perimeter of the circumscribed rectangle of the maximum contour arcLength_Rect, the true perimeter of the maximum contour arcLength_actual, and the glue width of the glue in the maximum contour, a judgment is made on whether there is glue breakage. It can be seen from Figure 6 and Figure 7 that if there is no glue breakage, the perimeter of the circumscribed rectangle of the maximum contour arcLength_Rect is equal to the true perimeter of the maximum contour arcLength_actual. If there is glue breakage, the true perimeter of the maximum contour arcLength_actual will be greater than the perimeter of the circumscribed rectangle of the maximum contour arcLength_Rect. Therefore, the size relationship between arcLength_Rect and arcLength_actual can be set to determine whether there is glue breakage. The specific setting method, for example, is that the difference between the two exceeds a preset threshold, or the ratio between the two exceeds a preset value, which is not specifically limited here.

[0074] Based on any of the above embodiments, in this method, if the perimeter of the circumscribed rectangle of the maximum contour, the glue width of the glue in the maximum contour, and the true perimeter of the maximum contour meet the preset conditions, it is determined that there is glue breakage. The preset conditions include a first condition and / or a second condition.

[0075] Among them, the first condition includes that the glue width of the glue in the maximum contour has a value less than or equal to zero, and the second condition includes that the ratio of the true perimeter of the maximum contour divided by the perimeter of the circumscribed rectangle of the maximum contour is greater than a preset threshold.

[0076] Here, the judgment method for whether there is glue breakage is further limited. Figure 8 It is a flowchart of the judgment method for whether there is glue breakage provided by the present invention, asFigure 8 As shown in Figure 8 , the judgment method is as follows:

[0077] Step 3201: If the ratio of the actual perimeter arcLength_actual of the largest contour to the perimeter arcLength_Rect of the circumscribed rectangle of the largest contour is greater than a preset threshold (the preset threshold is preferably 1.5 and can be set to other values), it is determined that there is a glue breakage phenomenon;

[0078] Step 3202: If there are values less than zero in the glue widths of the top, bottom, left, and right in the largest contour (the calculation algorithm of the glue width has been described above), it is determined that there is a glue breakage phenomenon;

[0079] Step 3203: If the ratio of the actual perimeter arcLength_actual of the largest contour to the perimeter arcLength_Rect of the circumscribed rectangle of the largest contour is less than or equal to the preset threshold, and there are values greater than or equal to zero in the glue widths of the top, bottom, left, and right in the largest contour, that is, the conditions in the above steps 1 and 2 are not satisfied, it is determined that there is no glue breakage phenomenon, that is, a complete glue pattern.

[0080] Based on the above embodiments, in this method, the glue position recognition model is constructed using the semantic segmentation model in yolov8.

[0081] Specifically, yolov8 is the latest version of the YOLO (You Only Look Once) series. It is not a model specifically for semantic segmentation but an object detection model. yolov8 can perform semantic segmentation by using the prediction results of the segmentation branch of yolov8. The semantic segmentation model in yolov8 can make predictions at multiple image scales and can effectively detect and segment objects of different sizes.

[0082] The method provided by the embodiments of the present invention, by further providing that the model category used during the training of the glue position recognition model is the semantic segmentation model in yolov8, utilizes the semantic segmentation function of the semantic segmentation model in yolov8, can make predictions at multiple image scales, and effectively detects and segments objects of different sizes.

[0083] Based on any of the above embodiments, in this method, the glue position recognition model includes a feature extraction network, a feature pyramid network, and a segmentation branch network connected in sequence;

[0084] Correspondingly, inputting the image to be detected into the glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model specifically includes:

[0085] Input the image to be detected into the feature extraction network to obtain multiple feature maps of different scales of the image to be detected output by the feature extraction network;

[0086] Input the multiple feature maps of different scales into the feature pyramid network to obtain a fused feature map output by the feature pyramid network;

[0087] Input the fused feature map into the segmentation branch network to obtain the glue position recognition result output by the segmentation branch network.

[0088] Specifically, the feature extraction network performs feature extraction on the input image to be detected at different scales, and then obtains and outputs multiple feature maps of different scales of the image to be detected to the feature pyramid network. Among them, the feature extraction network is usually a backbone network for extracting features from the input image. Here, the CSPDarknet53 network can be used as the feature extraction network; the feature pyramid network fuses the multiple feature maps of different scales input, and then obtains and outputs the fused feature map of the image to be detected to the segmentation branch network. Among them, the feature pyramid network (FPN, Feature Pyramid Network) can utilize both high-level semantic information and low-level position information when fusing feature maps of different scales, which is beneficial to detecting and segmenting targets of different sizes; the segmentation branch network judges the confidence of the input feature fusion map, and then judges whether each pixel point in the image to be detected corresponding to the feature fusion map is a glue area, and outputs the glue position recognition result. Here, the segmentation branch network can be trained based on the sample fusion feature map of the sample glue image and the glue position label corresponding to the sample glue image.

[0089] Based on any of the above embodiments, in this method, the loss function in the training process of the glue position recognition model is constructed based on the cross-entropy loss formula.

[0090] Specifically, during the training process, the loss function is used to calculate the difference between the prediction result and the true label, and then the parameters to be adjusted in the model network are adjusted based on this difference until at least one of the conditions that this difference is less than the preset threshold or the number of iterations exceeds the preset value is satisfied.

[0091] The formula of the cross-entropy loss function can be expressed by the following formula:

[0092]

[0093] Among them, is the value of the cross-entropy loss function; y is the true label, usually a one-hot encoded vector, where the value at the index of the true class is 1 and the values at other positions are 0; It is the predicted output of the glue position recognition model, usually a probability distribution vector processed by a softmax function; m represents the class index, where m = 2, and there are only two types of discrimination, one is the glue area and the other is the non - glue area; log represents the natural logarithm.

[0094] In practical applications, for numerical stability, a very small constant ∈ is usually added to avoid the numerical value in the logarithmic function becoming negative infinity. Therefore, the formula of the cross - entropy loss function can be modified as follows:

[0095]

[0096] Among them, ∈ is a constant close to 0.

[0097] Here, further explanation is made for the training of the glue position recognition model. In addition to using the cross - entropy loss function to calculate the difference between the prediction result and the true label during the model training process, and adjusting the parameters to be adjusted in the model network based on this difference feedback, after the model training is completed, the performance of the trained model is also evaluated by the intersection over union (IoU). The IoU can measure the overlap degree between the predicted glue bounding box and the true glue bounding box, and evaluate the accuracy of the model's detection and segmentation; when the IoU does not meet the preset conditions, at least one of the network structure and hyperparameters is adjusted and retrained; and during the training process, the network structure and hyperparameters can also be optimized and adjusted to improve the performance and accuracy of the model.

[0098] Based on any of the above embodiments, in this method, the glue position label corresponding to the sample image is manually labeled.

[0099] Specifically, the acquisition method of the training set during the training of the glue position recognition model is as follows: collect a batch of complete glue - painting and glue - breaking images, and then label these images to mark the position of the glue. Here, the lableme software can be used for labeling. Figure 9 This is a schematic diagram of the labeling process using the lableme software provided by the present invention. As Figure 9 shown, the contour area of the glue is drawn by connecting lines, and then the lableme software can save the points in these contour areas as a json file (including the position coordinates of the points, the class label, and the width and height of the image), and then convert the json file into a txt file that can be directly read by yolov8 (including the position coordinates of the points and the class label). Figure 10 This is a schematic diagram of the process of converting the json file to the txt file provided by the present invention. As Figure 10As shown, in the Jason file, "lable": "A" indicates that the type of the pixel point of the current record is A (here, the glue position area and the non - glue position area are distinguished and represented by type A and type B respectively). 159.4188034188034 and 166.68376068376068 in "points" represent the coordinate positions of a pixel point in the image of type A. The width and height of the image are 512 pixels and 512 pixels respectively. When converting this Jason file to a txt file, the number "0" represents type A, and the number "1" represents type B. The numbers 0.3113648504273504 and 0.3255542200854701 respectively correspond to the normalized values of 159.4188034188034 and 166.68376068376068 in the Jason file.

[0100] Based on the image dataset and the txt labels of the pixel points within the contour position area, use YOLOv8 for the training and optimization of semantic segmentation to improve the recognition and positioning accuracy of glue objects. Using semantic segmentation technology, each pixel in the image will be labeled correspondingly. For example, the output result is a matrix of 0 - 1 values. Among them, the "1" value indicates that the pixel point at the corresponding position in the matrix is of the glue type, and the "0" value indicates that the pixel point at the corresponding position in the matrix is of the non - glue type. Therefore, usually, the output result can be converted into a binary image, that is, a white glue area is shown on a black - background image.

[0101] The following introduces the device embodiments of the present application, which can be used to execute the glue defect detection method in the above - mentioned embodiments of the present application. For the details not disclosed in the device embodiments of the present application, please refer to the embodiments of the glue defect detection method above.

[0102] Based on any of the above - mentioned embodiments, Figure 11 is a schematic structural diagram of the glue defect detection device provided by the present invention. As Figure 11 shown, the device includes an identification module 1010 and a judgment module 1020, where,

[0103] The identification module 1010 is configured to input the image to be detected into a pre - trained glue position recognition model, and obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model. The glue position recognition model is obtained by training an initial network to be trained with a training sample set composed of multiple sample images and corresponding glue position labels;

[0104] The judgment module 1020 is configured to determine that there is a glue break in the image to be detected if the glue position recognition result meets a preset rule.

[0105] The glue defect detection device provided by the embodiment of the present invention uses a glue position recognition model trained with a large number of sample glue images and corresponding glue position labels to recognize the glue position in the image to be detected, improving the accuracy of glue position recognition. Moreover, the reliable robustness of the glue position recognition model also avoids the influence of the recognition result by the environment during the shooting of the image to be detected.

[0106] Based on any of the above embodiments, in the device, the judgment module is specifically used for:

[0107] Based on the binary image corresponding to the glue position recognition result, determine the largest contour of the glue in the binary image; wherein, the largest contour is a continuous glue area with the largest area;

[0108] If the perimeter of the circumscribed rectangle of the largest contour, the glue width of the glue in the largest contour, and the true perimeter of the largest contour meet the preset conditions, it is determined that there is a glue break.

[0109] Based on any of the above embodiments, in the device, the step of if the perimeter of the circumscribed rectangle of the largest contour, the glue width of the glue in the largest contour, and the true perimeter of the largest contour meet the preset conditions, it is determined that there is a glue break specifically includes:

[0110] If the first condition and / or the second condition are satisfied, it is determined that there is a glue break;

[0111] Wherein, the first condition includes that the glue width of the glue in the largest contour has a value less than or equal to zero, and the second condition includes that the ratio of the true perimeter of the largest contour to the perimeter of the circumscribed rectangle of the largest contour is greater than a preset threshold.

[0112] Based on any of the above embodiments, in the device, the glue position recognition model is constructed by using the semantic segmentation model in yolov8.

[0113] Based on any of the above embodiments, in the device, the glue position recognition model includes a feature extraction network, a feature pyramid network, and a segmentation branch network connected in sequence;

[0114] Correspondingly, the step of inputting the image to be detected into the glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model specifically includes:

[0115] Input the image to be detected into the feature extraction network to obtain multiple feature maps of different scales of the image to be detected output by the feature extraction network;

[0116] Input the multiple feature maps of different scales into the feature pyramid network to obtain a fused feature map output by the feature pyramid network;

[0117] Input the fused feature map into the segmentation branch network to obtain the glue position recognition result output by the segmentation branch network.

[0118] Based on any of the above embodiments, in this device, the loss function in the training process of the glue position recognition model is constructed based on the cross-entropy loss formula.

[0119] Based on any of the above embodiments, in this device, the glue position label corresponding to the sample image is manually marked.

[0120] Figure 12 An example of the physical structure diagram of an electronic device is shown as Figure 12 shown. The electronic device may include: a processor 1110, a communication interface 1120, a memory 1130, and a communication bus 1140. Among them, the processor 1110, the communication interface 1120, and the memory 1130 complete mutual communication through the communication bus 1140. The processor 1110 can call the logical instructions in the memory 1130 to execute the glue defect detection method, which includes: inputting the image to be detected into a pre-trained glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model, where the glue position recognition model is obtained by training an initial network to be trained with a training sample set composed of multiple sample images and corresponding glue position labels; if the glue position recognition result meets a preset rule, it is determined that the image to be detected has glue breakage.

[0121] In addition, when the logical instructions in the above-mentioned memory 1230 are implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0122] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the glue defect detection method provided by each of the above methods. The method includes: inputting an image to be detected into a pre-trained glue position recognition model to obtain a glue position recognition result corresponding to the image to be detected output by the glue position recognition model, where the glue position recognition model is obtained by training an initial network to be trained with a training sample set composed of a plurality of sample images and corresponding glue position labels; if the glue position recognition result meets a preset rule, it is determined that the image to be detected has a glue breakage.

[0123] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the glue defect detection method provided by each of the above methods. The method includes: inputting an image to be detected into a pre-trained glue position recognition model to obtain a glue position recognition result corresponding to the image to be detected output by the glue position recognition model, where the glue position recognition model is obtained by training an initial network to be trained with a training sample set composed of a plurality of sample images and corresponding glue position labels; based on the glue position recognition result, it is determined whether the image to be detected has a glue breakage.

[0124] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

[0125] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, also by hardware. Based on such an understanding, the above technical solutions, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A glue defect detection method, characterized in that: include: Inputting the image to be detected into a pre-trained glue position recognition model to obtain a glue position recognition result corresponding to the image to be detected output by the glue position recognition model, wherein the glue position recognition model is obtained by inputting a training sample set consisting of a plurality of sample images and corresponding glue position labels into an initial network to be trained; If the glue position recognition result meets the preset rule, it is determined that glue breakage exists in the image to be detected.

2. The glue defect detection method according to claim 1, characterized in that: The glue position recognition result is represented by a binary image, and the step of determining whether glue breakage exists in the image to be detected based on the fact that the glue position recognition result satisfies a preset rule specifically includes: Based on the binary image corresponding to the glue position recognition result, determining the maximum outline of the glue in the binary image; wherein the maximum outline is a continuous glue area with the largest area; If the perimeter of the circumscribed rectangle of the maximum contour, the glue width of the glue in the maximum contour, and the actual perimeter of the maximum contour meet preset conditions, it is determined that glue breakage exists.

3. The glue defect detection method according to claim 2, characterized in that: If the perimeter of the circumscribed rectangle of the maximum contour, the glue width of the glue in the maximum contour, and the true perimeter of the maximum contour meet preset conditions, then it is determined that there is glue breakage, specifically including: If the first condition and / or the second condition is met, it is determined that the glue is broken; The first condition includes that the glue width of the glue in the maximum contour has a value less than or equal to zero, and the second condition includes that the ratio of the true perimeter of the maximum contour divided by the perimeter of the circumscribed rectangle of the maximum contour is greater than a preset threshold.

4. The glue defect detection method according to any one of claims 1 to 3, characterized in that: The glue position recognition model is constructed using the semantic segmentation model in yolov8.

5. The glue defect detection method according to claim 4, characterized in that: The glue position recognition model includes a feature extraction network, a feature pyramid network and a segmentation branch network connected in sequence; Correspondingly, the step of inputting the image to be detected into the glue position recognition model to obtain the glue position recognition result corresponding to the image to be detected output by the glue position recognition model specifically includes: Inputting the image to be detected into the feature extraction network to obtain a plurality of feature maps of different scales of the image to be detected output by the feature extraction network; Inputting the plurality of feature maps of different scales into the feature pyramid network to obtain a fused feature map output by the feature pyramid network; The fused feature map is input into the segmentation branch network to obtain the glue position recognition result output by the segmentation branch network.

6. The glue defect detection method according to any one of claims 1 to 3, characterized in that: The loss function in the glue position recognition model training process is constructed based on the cross entropy loss formula.

7. The glue defect detection method according to any one of claims 1 to 3, characterized in that: The glue position labels corresponding to the sample images are manually marked.

8. A glue defect detection device, characterized in that: include: A recognition module, used for inputting the image to be detected into a pre-trained glue position recognition model, and obtaining a glue position recognition result corresponding to the image to be detected output by the glue position recognition model, wherein the glue position recognition model is obtained by inputting a training sample set consisting of a plurality of sample images and corresponding glue position labels into an initial network to be trained; The judgment module is used to determine whether glue breakage exists in the image to be detected if the glue position recognition result meets a preset rule.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed by the glue defect detection method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device includes one or more processors and one or more memories, wherein at least one program code is stored in the one or more memories, and the at least one program code is loaded and executed by the one or more processors to implement the operations performed by the glue defect detection method as described in any one of claims 1 to 7.

Images