A detection method suitable for detecting the outline of light-colored glue

Abstract

The present invention discloses a detection method suitable for detecting the outline of light-colored glue, which relates to the field of machine vision detection and comprises the following steps: using a CCD to collect images of an object to be inspected after gluing, marking the outlines of glue of different categories, and adding background labels to obtain original data; constructing a deep learning model, taking the original data as input data and binarizing it; converting the binarized image into a mask image; classifying objects in the image according to pixels to obtain a classified mask image; sending the classified mask image into a detection module for filtering, separating the filtered mask image to obtain a separated mask image; training the deep learning model according to the separated mask image, and outputting the final weight after the training is completed; transmitting the real-time collected pictures into the trained deep learning model for inference and prediction. The present invention improves the detection efficiency and the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of machine vision detection, and in particular to a detection method suitable for detecting the contour of light-colored glue. Background Art

[0002] The dispensing process is commonly found in all aspects of industrial production, such as some common bonding operations of electronic components. This process plays a decisive role in subsequent work, and the quality of the bonding effect indirectly determines the product yield.

[0003] However, the current process of judging whether the dispensing effect is good or bad is mainly based on two methods. The first is manual visual inspection, and the second is detection through traditional visual means. This method based on manual visual inspection has the following problems and shortcomings. First, its efficiency is relatively low, because when the human eye is inspected, the workpiece needs to be placed under the light, and the outline of the glue is found through a certain reflection angle. Then, it is compared with the standard document to determine whether the current workpiece is qualified. Moreover, as the working time increases, the fatigue level of the human eye will increase, and its evaluation criteria are somewhat subjective. This can easily lead to inconsistent standards for the screened workpieces, which has a certain degree of negative impact on industrial production. Secondly, its replication and promotion are extremely limited. If a new production line requires this inspection process, a new employee needs to be retrained to be responsible, which greatly reduces production efficiency. The second method based on traditional vision is derived and improved from the manual visual inspection method. It uses the idea of ​​machines replacing humans, and uses industrial CCD cameras to replace human eye imaging. It combines traditional visual solutions such as threshold segmentation and template matching to detect glue contours, but this method has the following disadvantages: First, traditional vision has extremely high requirements for the detection environment (i.e., external factors such as ambient light), and its working stability is greatly limited. For example, on sunny days, the external light intensity is high, and some thresholds set in this time period may become invalid during cloudy days. Its dependence on the environment limits its ability to be quickly promoted; second, its detection content is extremely limited. After experiments, the current traditional vision solution will greatly reduce its contour detection ability when facing light-colored or transparent glue. Since the current mainstream traditional solution is based on the method of threshold segmentation of the image, The glue outline is further distinguished from the current background to find its outline. However, when facing light-colored or transparent glue, this method cannot be distinguished well by traditional vision because the target grayscale is close to the background grayscale after the image is converted to grayscale. In addition, this solution based on traditional vision usually requires a relatively clean imaging surface and a relatively clean solid-color background for the object to be detected. This makes it extremely unsuitable for industrial production. Thirdly, the replication and promotion efficiency of traditional vision solutions is extremely low. Due to different production lines and different imaging light paths, the imaging effect cannot be 100% unified. For new environments, the corresponding threshold needs to be reset for normal use. Secondly, when a new type of glue comes, the detection algorithm needs to be reset, which makes it difficult to achieve generalization and compatibility. In industrial scenarios with diversified production, this shortcoming greatly limits its application in the industrial field.

[0004] The existing technology has low detection efficiency, strong dependence on environmental factors, and low detection accuracy. Summary of the Invention

[0005] In order to solve the problems of low detection efficiency, strong dependence on environmental factors and low detection accuracy in the above existing technologies, the present invention proposes a detection method suitable for detecting the outline of light-colored glue, which improves detection efficiency, reduces dependence on environmental factors and improves detection accuracy.

[0006] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0007] A method for detecting the outline of light-colored glue comprises the following steps:

[0008] An industrial CCD is used to capture images of the objects to be inspected after dispensing glue. The contours of different types of glue are marked manually, and background labels are added to obtain the original data.

[0009] A deep learning model based on the field of semantic segmentation is constructed, and the original data is used as input data, the input data is binarized, and the binarized image is used as the fourth dimension information of the input data; the binarized image is converted into a mask image; according to the mask image, the objects in the image are classified into different categories according to the pixels, and a classified mask image is obtained; the classified mask image is sent to the detection module for filtering, and the filtered mask image is separated to obtain a separated mask image.

[0010] The deep learning model is trained based on the separated mask image, and the final weights of the deep learning model are output after the training is completed.

[0011] The real-time collected images are transferred into the trained deep learning model for inference and prediction.

[0012] The working principle of the present invention is as follows:

[0013] Use CCD to capture images of the object to be tested, obtain raw data after annotation, build a deep learning model based on the semantic segmentation field, use the raw data as input data to train the deep learning model, and transmit the real-time captured images to the trained deep learning model for inference and prediction.

[0014] Preferably, when collecting images of the object to be inspected after dispensing glue, the light source used is a strip light, the object to be inspected is illuminated from the side, and the industrial CCD is controlled by a sensor to take pictures.

[0015] The sensor is a through-beam laser sensor. When the object to be inspected triggers the sensor, the industrial computer sends an electrical signal to the host computer. After receiving the electrical signal, the host computer sends a soft contact instruction to the industrial CCD, and then obtains the captured image through the callback function built into the industrial CCD.

[0016] Preferably, the target area is the area where the glue is located. Before capturing the image of the glue contour of the object to be measured, the object to be measured is positioned by target detection, and the target area is determined by locating the bonding part of the object to be measured.

[0017] Furthermore, the data generated by the annotation is a mask image with pixel values ​​between 0 and 255; the target grayscale values ​​of different categories are different.

[0018] Preferably, the raw data is pre-processed before being input into the deep learning model.

[0019] The data pre-processing achieves data enhancement by rotating, cropping, and contrast enhancement of the original image.

[0020] Preferably, the deep learning model includes a feature extraction module, a feature fusion module, a feature pyramid module, and an output module.

[0021] The binarized image is subjected to a feature extraction module to extract high-level features and low-level features; the feature fusion module fuses the high-level features and the low-level features to obtain a first fused feature, wherein the first fused features are multiple and have first fused features of different scales; the feature pyramid module fuses the first fused features to obtain a second fused feature; the second fused feature is upsampled to obtain a feature map with the same scale as the binarized image; the pixels of the feature map are regressed and classified and assigned corresponding labels, thereby classifying the objects in the image at the pixel level to obtain a classified mask image; the output module includes a classification layer for outputting the classified mask image.

[0022] Furthermore, the steps of extracting high-level features and low-level features by the feature extraction module are as follows:

[0023] Use 1*1 convolution to increase the dimension of the binarized image, use 3*3 depth-wise separable convolution to extract features, and then use a 1*1 convolution to reduce the dimension to obtain low-level features and high-level features.

[0024] Furthermore, the training of the deep learning model includes freezing training and thawing training.

[0025] The frozen training method is as follows: freeze the output module, set the initial learning rate to 5e-4, and train for 50 epochs;

[0026] The unfreezing training method is as follows: freeze all modules except the output module, set the learning rate to 5e-5, and train for 30 epochs.

[0027] Preferably, the steps of filtering the mask image are as follows:

[0028] Filter out pixels with background labels and count the category distribution of the remaining pixels. Define the pixel category with the largest proportion as the detection target, and separate different categories of glue through template matching.

[0029] The method for separating the filtered mask image is as follows:

[0030] Different types of objects require different numbers of glue dispensing positions. The number of classification blocks in the target area is determined based on the pre-set number of glue dispensing positions. The first k mask images are taken from the largest to the smallest according to the area of ​​the target area, where k is the number of glue dispensing positions.

[0031] Preferably, the performance of the deep learning model is evaluated based on the classification accuracy of each pixel and the classification IOU with the original label file, an IOU threshold is set, and the model weight with the highest classification accuracy under the IOU threshold is selected.

[0032] Beneficial effects of the present invention:

[0033] 1. The deep learning model performs inference and prediction on real-time collected images, improving detection efficiency and reducing dependence on environmental factors.

[0034] 2. Use binarization processing to make the glue contour more precise and improve the accuracy of detection.

[0035] 3. Through two training processes, when facing complex working environments, the glue contour can be better located and detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the process of the present invention.

[0037] Figure 2 This is the glue outline effect extracted by traditional method.

[0038] Figure 3 This is the glue contour rendering extracted by the present invention. DETAILED DESCRIPTION

[0039] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0040] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0041] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0042] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0043] Example 1

[0044] In this embodiment, if Figure 1 As shown, a detection method suitable for detecting the outline of light-colored glue includes the following steps:

[0045] An industrial CCD is used to capture images of the objects to be inspected after dispensing glue. The contours of different types of glue are marked manually, and background labels are added to obtain the original data.

[0046] A deep learning model based on the field of semantic segmentation is constructed, and the original data is used as input data, the input data is binarized, and the binarized image is used as the fourth dimension information of the input data; the binarized image is converted into a mask image; according to the mask image, the objects in the image are classified into different categories according to the pixels, and a classified mask image is obtained; the classified mask image is sent to the detection module for filtering, and the filtered mask image is separated to obtain a separated mask image.

[0047] The deep learning model is trained based on the separated mask image, and the final weights of the deep learning model are output after the training is completed.

[0048] The real-time collected images are transferred into the trained deep learning model for inference and prediction.

[0049] The working principle of the present invention is as follows:

[0050] Use CCD to capture images of the object to be tested, obtain raw data after annotation, build a deep learning model based on the semantic segmentation field, use the raw data as input data to train the deep learning model, and transmit the real-time captured images to the trained deep learning model for inference and prediction.

[0051] Example 2

[0052] In this embodiment, a detection method suitable for detecting the outline of light-colored glue is characterized by comprising the following steps:

[0053] A 5-megapixel industrial CCD is used to capture images of the objects to be inspected after glue dispensing. Through manual labeling, the contours of different types of glue are marked, and background labels are added to obtain the original data.

[0054] A deep learning model based on semantic segmentation is constructed. The raw data is used as input, binarized, and the resulting binarized image is used as the fourth dimension of the input data. The binarized image is converted into a mask image. Based on the mask image, objects in the image are classified into different categories based on their pixels, resulting in a classified mask image. The classified mask image is then fed into the detection module for filtering. The filtered mask image is then separated, and the target area is distinguished from the background based on the pre-set number of dispensing locations.

[0055] Train the deep learning model and output the final weights of the deep learning model after training. Encapsulate the inference functions of the deep learning model into a DLL file. The host computer calls the DLL open interface function to transmit the real-time collected images into the deep learning model for inference and prediction and return the results.

[0056] In this embodiment, the original data is divided into a training set and a test set. The training set is used to train the deep learning model, and the test set is used to test the trained deep learning model.

[0057] In this embodiment, when collecting images of the object to be inspected after dispensing glue, the light source used is a strip light, which illuminates the object to be inspected from the side, and the industrial CCD is controlled by the sensor to take the picture.

[0058] The sensor is a through-beam laser sensor. When the object to be inspected triggers the sensor, the industrial computer sends an electrical signal to the host computer. After receiving the electrical signal, the host computer sends a soft contact instruction to the industrial CCD, and then obtains the captured image through the callback function built into the industrial CCD.

[0059] In this embodiment, the target area is the area where the glue is located. Before capturing the image of the glue contour of the object to be measured, the object to be measured is positioned by target detection, and the target area is determined by locating the bonding part of the object to be measured.

[0060] In this embodiment, the detection method of the detection module is as follows:

[0061] The value of each pixel is changed from the original grayscale value to the category judged by the deep learning model. The number of categories corresponds to the set number of classes. The number of pixels of other categories except background pixels is calculated through the mask image, and the area contour is calculated; the contour is the junction of background pixels and non-background pixels.

[0062] In this embodiment, the number of images collected in the dataset is 500, including 6 types of glue. Different types of glue correspond to labels 0, 1, 2, 3, 4, and 5 respectively, and the background class label is 6.

[0063] More specifically, the software used for annotation is labelme, and the data generated by the annotation is a mask image; the mask image is an 8-bit deep binary image with pixel values ​​between 0-255; the target area is a set grayscale value, and the grayscale values ​​of different categories of glue areas are different.

[0064] In this embodiment, the raw data is pre-processed before being input into the deep learning model;

[0065] The data pre-processing achieves data enhancement and expands sample diversity by rotating, cropping, and contrast enhancing the original image.

[0066] In this embodiment, the deep learning model includes a feature extraction module, a feature fusion module, a feature pyramid module, and an output module.

[0067] The deep learning model is developed using DeeplabV3+ and Mobilenetv2 is used as the backbone network.

[0068] The binarized image is subjected to a feature extraction module to extract high-level features and low-level features; the feature fusion module fuses the high-level features and the low-level features to obtain a first fused feature, wherein the first fused features are multiple and have first fused features of different scales; the feature pyramid module fuses the first fused features to obtain a second fused feature; the second fused feature is upsampled to obtain a feature map with the same scale as the binarized image; the pixels of the feature map are regressed and classified and assigned corresponding labels, thereby classifying the objects in the image at the pixel level to obtain a classified mask image; the output module includes a classification layer for outputting the classified mask image.

[0069] More specifically, the steps of extracting high-level features and low-level features by the feature extraction module are as follows:

[0070] Use 1*1 convolution to increase the dimension of the binarized image, use 3*3 depth-separable convolution to extract features, and then use a 1*1 convolution to reduce the dimension to obtain low-level features and high-level features;.

[0071] Preferably, the steps of filtering the mask image are as follows:

[0072] Filter out pixels with background labels and count the category distribution of the remaining pixels. Define the pixel category with the largest proportion as the detection target, and separate different categories of glue through template matching.

[0073] The method for separating the filtered mask image is as follows:

[0074] Different types of objects require different numbers of glue dispensing positions. The number of classification blocks in the target area is determined based on the pre-set number of glue dispensing positions. The first k mask images are taken from the largest to the smallest according to the area of ​​the target area, where k is the number of glue dispensing positions.

[0075] In this embodiment, during model training, the glue category corresponding to each classification block is predicted based on the separated mask image.

[0076] In this embodiment, for model training, the hardware equipment is two 3090 graphics cards, and the training of the deep learning model includes frozen training and unfrozen training.

[0077] The frozen training method is as follows: the output module is frozen, the initial learning rate is 5e-4, and training is performed for 50 epochs.

[0078] The unfreezing training method is as follows: freeze all modules except the output module, set the learning rate to 5e-5, and train for 30 epochs.

[0079] In this embodiment, the performance of the deep learning model is evaluated based on the classification accuracy of each pixel point and the classification IOU with the original label file, an IOU threshold is set, the model weight with the highest classification accuracy under the IOU threshold is selected, and the model weight is compiled into a DLL file using C++.

[0080] In this embodiment, if Figure 2 As shown in the figure, the distinction between the contour and background extracted by the traditional method is not clear and the accuracy is low.

[0081] In this embodiment, Figure 3 As shown, the effect of the extracted glue contour can be clearly seen.

[0082] Example 3

[0083] In this embodiment, a 5-megapixel industrial CCD camera is used for image acquisition, and the light source used is a strip light for side lighting. In the actual production process, the dispensing parts are photographed through sensor control. After obtaining the required pictures, we will annotate them, use dedicated annotation software to distinguish the foreground and background at the pixel level, and save the annotation information locally.

[0084] Secondly, the deep learning model is modeled and trained using DeeplabV3+. First, the path for reading local image data and label data is set. Then, the training parameters such as the learning rate and number of iterations are set during the training process, and then the training is started. After the training is completed, the final weight is output.

[0085] Finally, the reasoning part of the deep learning model is packaged into an externally callable form. The functions of the model reasoning part are encapsulated into a DLL file using C++, and then called in the host computer to realize online detection.

[0086] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A detection method suitable for detecting the outline of light-colored glue, characterized in that: The following steps are involved: Use industrial CCD to capture images of the objects to be inspected after dispensing glue. Through manual annotation, mark the contours of different types of glue and add background labels to obtain the original data. A deep learning model based on semantic segmentation is constructed. The original data is used as input data, and the input data is binarized. The binarized image is used as the fourth dimension of the input data. The binarized image is converted into a mask image. Objects in the image are classified into different categories according to the mask image according to the pixels, and a classified mask image is obtained. The classified mask image is sent to the detection module for filtering, and the filtered mask image is separated to obtain a separated mask image. The steps for filtering the mask image are as follows: Filter out pixels with background labels and count the category distribution of the remaining pixels. Define the pixel category with the largest proportion as the detection target, and separate different categories of glue through template matching. The method for separating the filtered mask image is as follows: Different types of objects require different numbers of dispensing locations. The number of classification blocks in the target area is determined based on the pre-set number of dispensing locations. The first k mask images are taken from the largest to the smallest area of ​​the target area, where k is the number of dispensing locations. The deep learning model is trained based on the separated mask image, and the final weight of the deep learning model is output after the training is completed; The real-time collected images are transferred into the trained deep learning model for inference and prediction.

2. A detection method suitable for detecting the outline of light-colored glue according to claim 1, characterized in that: When collecting images of objects to be inspected after dispensing glue, the light source used is a strip light, which illuminates the objects to be inspected from the side, and the industrial CCD is controlled by the sensor to take pictures; The sensor is a through-beam laser sensor. When the object to be inspected triggers the sensor, the industrial computer sends an electrical signal to the host computer. After receiving the electrical signal, the host computer sends a soft contact instruction to the industrial CCD, and then obtains the captured image through the callback function built into the industrial CCD.

3. A detection method suitable for detecting the outline of light-colored glue according to claim 1, characterized in that: The target area is the area where the glue is located. Before capturing the image of the glue contour of the object to be tested, the object to be tested is positioned by target detection, and the target area is determined by locating the bonding part of the object to be tested.

4. A detection method suitable for detecting the outline of light-colored glue according to claim 3, characterized in that: The data generated by the annotation is a mask image with pixel values ​​between 0 and 255; the grayscale values ​​of target areas of different categories are different.

5. The detection method for detecting the outline of light-colored glue according to claim 1, characterized in that: Before the raw data is input into the deep learning model, the raw data is pre-processed; The data pre-processing achieves data enhancement by rotating, cropping, and contrast enhancement of the original image.

6. The detection method for detecting the outline of light-colored glue according to claim 1, characterized in that: The deep learning model includes a feature extraction module, a feature fusion module, a feature pyramid module, and an output module; The binarized image is used to extract high-level features and low-level features through the feature extraction module; feature The fusion module fuses the high-level features and the low-level features to obtain a first fusion feature, wherein the first fusion feature is a plurality of first fusion features of different scales; The feature pyramid module fuses the first fused features to obtain the second fused features; upsamples the second fused features to obtain a feature map with the same scale as the binarized image; regresses and classifies the pixels of the feature map and assigns corresponding labels, thereby classifying the objects in the image at the pixel level and obtaining a classified mask image; The output module includes a classification layer for outputting a classified mask image.

7. A detection method for detecting the outline of light-colored glue according to claim 6, characterized in that: The steps of extracting high-level features and low-level features by the feature extraction module are as follows: Use 1*1 convolution to increase the dimension of the binarized image, use 3*3 depth-wise separable convolution to extract features, and then use a 1*1 convolution to reduce the dimension to obtain low-level features and high-level features.

8. The detection method for detecting the outline of light-colored glue according to claim 6, characterized in that: The training of deep learning models includes freezing training and unfreezing training; The frozen training method is as follows: freeze the output module and the initial learning rate is 5e -4 , train for 50 epochs; The unfreezing training method is as follows: freeze all modules except the output module and set the learning rate to 5e -5 , train for 30 epochs.

9. The detection method for detecting the outline of light-colored glue according to claim 1, characterized in that: The performance of the deep learning model is evaluated based on the classification accuracy of each pixel and the classification IOU with the original label file. The IOU threshold is set and the model weight with the highest classification accuracy under the IOU threshold is selected.

Images