Defect detection method and system for optical diffraction element based on deep learning and medium

Abstract

The invention discloses an optical diffraction element defect detection method and system based on deep learning and a medium. The method comprises the following steps: acquiring a first to-be-detected image, preprocessing the first to-be-detected image to obtain a first ROI sub-image of the first to-be-detected image, and establishing a geometric rectangular coordinate system; a first deep learning model deployed in advance is loaded to perform semantic segmentation on the first ROI subgraph, the first deep learning model is used for identifying the first dispensing area and outputting a first feature graph, the first deep learning model is based on a Unit + + network structure, and a multi-scale large-kernel convolution module is adopted by both an encoder and a decoder of the first deep learning model; detecting a first defect in the first ROI sub-graph by adopting a first 2D traditional algorithm; the first feature map detects a second defect and a third defect by using a second 2D traditional algorithm. According to the method, a deep learning algorithm is combined with a 2D traditional algorithm, so that the detection speed and accuracy of the optical diffraction element can be improved.

Description

Technical Field

[0001] The present invention relates to the field of non-contact product appearance detection, and in particular to a method, system and medium for detecting defects of an optical diffraction element based on deep learning. Background Art

[0002] In recent years, with the rapid development of science and technology, smartphones have almost become a necessity in people's lives. Camera modules are an indispensable part of smartphones, and the application of diffractive optical elements (DOEs) in camera modules is becoming increasingly important. DOEs can shape laser beams into specific patterns for cameras to perform depth perception and 3D modeling. In the assembly process of DOEs, glue bonding is a key step, the purpose of which is to achieve sealing, fixing, waterproofing and dustproofing of components. Therefore, the development of dispensing technology has become particularly important. With the continuous advancement of manufacturing processes, the requirements for the fineness of dispensing quality are also constantly increasing.

[0003] In the actual production process, the DOE surface may be dirty or the sealing effect may be unstable due to factors such as ambient temperature, dispensing speed, and machine stability. In order to ensure the precision of the camera module, the dispensing process must meet high standards, including precise control of the shape, width, and length of the glue, while avoiding the lack, omission, or overflow of the glue.

[0004] Although the factory has adopted automation technology, the factory's defect detection method for DOE packaging mostly relies on manual sampling inspection, using a magnifying glass or two-dimensional visual inspection equipment for measurement. This method is random, and due to the complexity and time-consuming nature of manual operation, it will seriously affect production efficiency in large-scale production. In addition, when using two-dimensional visual inspection equipment for measurement, it is difficult to set a suitable threshold to accurately identify the edge due to the small grayscale difference between the edge of the dispensing and the background, resulting in large errors in the measurement results. Therefore, products using traditional inspection methods may have a large number of undetected defective products, or there is a risk of mistakenly judging qualified products as defective products. Summary of the invention

[0005] In order to overcome the above-mentioned shortcomings, the purpose of the present invention is to provide an optical diffraction element defect detection method, system and medium based on deep learning, which can improve the speed and accuracy of optical diffraction element detection.

[0006] In order to achieve the above objectives, the technical solution adopted by the present invention is: a method for detecting defects of optical diffraction elements based on deep learning, comprising: Acquire a first image to be tested, preprocess the first image to be tested to obtain a first ROI region sub-image of the first image to be tested and establish a geometric rectangular coordinate system; The first deep learning model deployed in advance is loaded to perform semantic segmentation on the first ROI region sub-image. The first deep learning model is used to identify the first dispensing area and output the first feature map. The first deep learning model is based on the Unet++ network structure. The encoder and decoder of the first deep learning model both adopt multi-scale large-kernel convolution modules. The first ROI region sub-image uses a first 2D traditional algorithm to detect a first defect; The first feature map uses a second 2D traditional algorithm to detect the second defect and the third defect. The beneficial effects of the present invention are: 1. The first deep learning model can maintain extremely high segmentation accuracy without scaling the image by improving the UNet++ network structure. The accuracy error of repeated measurements in the experiment is controlled within the range of ±1 pixel.

[0007] 2. The first deep learning model uses the MSLK module to replace the two convolution modules in the UNet++ network structure. The use of large-kernel convolution can obtain a larger receptive field and linear complexity, which not only realizes the adaptability of the spatial dimension, but also realizes the adaptability of the channel dimension, and enhances the network's attention to important channels and regions in the input feature map, thereby improving the feature representation ability.

[0008] 3. Combining traditional 2D algorithm detection with deep learning algorithm detection greatly shortens the image data processing time, improves the production efficiency of the entire process from material input to material output, and reduces the average processing time of a single particle to 1.3s.

[0009] Further, preprocessing the first image to be tested to obtain a first ROI region sub-image of the first image to be tested and establishing a geometric rectangular coordinate system specifically includes: Performing rough positioning on the first image to be tested, using the template positioning in the 2D tool to perform ROI positioning, and cropping to form a first ROI area sub-image; The first ROI area sub-image is precisely positioned, and the 2D line finding tool is used to obtain the four sides of the product and establish a geometric rectangular coordinate system based on two perpendicular sides of the four sides.

[0010] Furthermore, the first deep learning model is pre-trained, and the training method specifically includes: Acquire a first data set and divide the first data set into a first training set and a first validation set, wherein the first data set includes a plurality of ROI region sub-images, wherein the plurality of ROI region sub-images have a second defect, a third defect, or no defect; Defect annotation is performed on the first glue dispensing area of ​​the ROI area sub-image to form an annotation image; Construct an initial model based on the Unet++ network structure. The initial model is a nested encoder-decoder structure. The encoder consists of two multi-scale large-kernel convolution modules and maximum pooling downsampling. The decoder consists of transposed convolution upsampling and two multi-scale large-kernel convolution modules. Skip connections are used between the same layers to fuse semantic information at different stages. Input the first training set into the initial model for training and save the trained initial model; The first validation set is input into the trained initial model for validation, and the initial model with the best performance on the first validation set is saved as the first deep learning model.

[0011] Specifically, the multi-scale large kernel convolution module includes a series of 3×3 convolutions, a convolution attention mechanism unit, three parallel branches of multi-scale large kernel convolutions, a splicing unit, and a 1×1 convolution. The multi-scale large kernel convolution module uses residual connections. The processing of the input image by the multi-scale large kernel convolution module specifically includes: The input image is processed by a 3×3 convolution and a convolutional attention mechanism unit respectively; Image features are extracted using multi-scale large kernel convolutions with three branches, and two depth-wise strip convolutions are used in each branch to approximate standard depth-wise convolutions with large kernels; Concatenate the outputs of large kernel convolutions, use 1×1 convolution to adjust the number of channels, and model the relationship between different channels; The input graph is connected through residual connections to form the output image.

[0012] Furthermore, the convolutional attention mechanism unit includes a channel attention mechanism and a spatial attention mechanism connected in series, and the output of the channel attention mechanism serves as the input of the spatial attention mechanism.

[0013] Specifically, the first ROI region sub-image uses the first 2D traditional algorithm to detect the first defect, which specifically includes: On the first ROI region sub-image, based on the coordinate system, the four sides are indented inward by a fixed distance to establish ROI region 1 where defect 1 is located; Searching for the first defect region within the ROI region 1 by using an image analysis tool; Determining whether the area of ​​the first defect is smaller than a set threshold; If so, the first defect does not exist; if not, the first defect exists.

[0014] Further, the first feature map uses the second 2D traditional algorithm to detect the second defect and the third defect specifically including: On the first feature map, with the coordinate system as the reference, the four edges are indented inward by a fixed distance, and four ROI regions of the same size are established in different first dispensing areas; The area of ​​glue was searched in two of the four ROI regions using image analysis tools; Compare the area of ​​glue in the four ROI regions 2 with the set range; If it is within the set range, there is no second defect and third defect; if it is less than the set range, it is the second defect; if it is greater than the set threshold, it is the third defect.

[0015] Furthermore, the detection method also includes the following steps: Acquire a second image to be tested, and preprocess the second image to be tested to obtain a second ROI region subimage of the second image to be tested; Loading a pre-deployed second deep learning model to perform semantic segmentation on a second ROI region sub-image, the second deep learning model is used to identify a second dispensing region and output a second feature map, and the second deep learning model and the first deep learning model are formed based on the same initial model training; The second feature map uses the third 2D traditional algorithm to detect the fourth defect.

[0016] The present invention also discloses an optical diffraction element defect detection system based on deep learning, and the system adopts the above-mentioned detection method.

[0017] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned detection method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of a testing method in an embodiment of the present invention; Figure 2 Flowchart of the training process of the first deep learning model in an embodiment of the present invention; Figure 3 This is a first 2D traditional algorithm detection flow chart in an embodiment of the present invention; Figure 4 This is a second 2D traditional algorithm detection flow chart in an embodiment of the present invention; Figure 5 is a flowchart of the training process of the second deep learning model in an embodiment of the present invention; Figure 6 This is a third 2D traditional algorithm detection flow chart in an embodiment of the present invention; Figure 7 is a schematic diagram of an initial model in an embodiment of the present invention; Figure 8 Schematic diagram of the MSLK module in an embodiment of the present invention; Fig. 9 is a schematic diagram of a CBAM unit in an embodiment of the present invention; Fig.10Schematic diagram of a channel attention mechanism in an embodiment of the present invention; Fig.11 Schematic diagram of a spatial attention mechanism in an embodiment of the present invention; Fig.12 The first ROI region sub-image and geometric rectangular coordinate system in the embodiment of the present invention; Fig.13 is a schematic diagram of a first characteristic graph in an embodiment of the present invention; Fig.14 is a schematic diagram of a second characteristic graph in an embodiment of the present invention; Fig.15 is a schematic diagram of a first defect in an embodiment of the present invention; Fig.16 Schematic diagram of the second defect and the third defect in an embodiment of the present invention; Fig.17 Schematic diagram of the fourth defect in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0020] The present invention discloses a deep learning-based optical diffraction element defect detection method, which is used for detecting four defects of an optical diffraction element. The four defects are respectively a first defect (DOE dirtiness), a second defect (Sealing glue leakage), a third defect (Sealing glue overflow) and a fourth defect (Sealing glue deficiency).

[0021] See attached Figure 1 As shown, the detection method includes: S100 , obtaining a first image to be tested, preprocessing the first image to be tested to obtain a first ROI region sub-image of the first image to be tested, and establishing a geometric rectangular coordinate system.

[0022] The first image to be tested is acquired by a first image acquisition device, which includes a camera and coaxial light. The coaxial light is used to illuminate the product. The camera lens has 5 million pixels, and the pixel size of the captured image is 2448*2048.

[0023] In this embodiment, since the background of the first image to be tested taken by the camera is complex and the area to be tested is random, it is necessary to perform preprocessing of cropping and positioning on the first image to be tested to ensure that the position of the area to be tested is fixed, and obtain a first ROI area sub-image of the area to be tested. The first ROI area sub-image is shown in FIG. Fig.12 shown.

[0024] S200, loading a first deep learning model deployed in advance to perform semantic segmentation on a first ROI region sub-image, wherein the first deep learning model is used to identify a first dispensing area and output a first feature map.

[0025] The first deep learning model is based on the Unet++ network structure. The encoder and decoder of the first deep learning model both use multi-scale large kernel convolution modules, referred to as MSLK modules. The first deep learning model is a pre-trained AI model. Through the first deep learning model, the first dispensing area can be quickly identified. At the same time, the first deep learning model uses the MSLK module to replace the two convolution modules in the UNet++ network structure. The use of large kernel convolution can obtain a larger receptive field and linear complexity, which not only realizes the adaptability of the spatial dimension, but also realizes the adaptability of the channel dimension, and enhances the network's attention to important channels and regions in the input feature map, thereby improving the representation ability of features.

[0026] The model is used to detect the specific area of ​​the first dispensing for the first ROI region sub-image. The model infers the input image data and uses the forward propagation of the neural network to generate the first feature map. Fig.13 As shown, this feature map is a binary image, where the value 0 represents the background and the value 255 represents the glue spot position, thereby completing the detection of the specific area of ​​the first glue spot.

[0027] S300 , using a first 2D traditional algorithm to detect a first defect in a first ROI region sub-image.

[0028] For DOE contamination, the first 2D traditional algorithm can be directly used for detection.

[0029] S400 , the first feature map uses a second 2D traditional algorithm to detect a second defect and a third defect.

[0030] For Sealing glue leakage and Sealing glue overflow, after completing AI judgment, the first glue area detection is performed on its first feature map to determine whether Sealing glue leakage and Sealing glue overflow exist.

[0031] In one embodiment, S300 and S400 can be performed simultaneously, that is, the detection of DOE contamination and the detection of Sealing glue leakage and Sealing glue overflow are performed simultaneously, so as to reduce the detection time.

[0032] In this embodiment, the first deep learning model can maintain extremely high segmentation accuracy without scaling the image by improving the UNet++ network structure, and the precision error of repeated measurements in the experiment is controlled within the range of ±1 pixel. The first deep learning model uses the MSLK module to replace the two convolution modules in the UNet++ network structure. The use of large kernel convolution can obtain a larger receptive field and linear complexity, which not only realizes the adaptability of the spatial dimension, but also realizes the adaptability of the channel dimension, and enhances the network's attention to important channels and regions in the input feature map, thereby improving the representation ability of the feature. Therefore, the pixel accuracy of 0.00345mm can be achieved, so that the detection miss rate is 0, and the size measurement repetition accuracy is within ±0.0035mm. At the same time, the traditional 2D algorithm detection is combined with the deep learning algorithm detection, which greatly shortens the image data processing time, improves the complete process efficiency from feeding to discharging, and reduces the average processing time of a single particle to 1.3s. The detection method of this embodiment achieves zero misjudgment and zero leakage in an actual production environment, which not only improves the efficiency and accuracy of detection, but also saves costs for enterprises and improves product quality.

[0033] In steps S100-S400, the detection of three defects of DOE contamination, Sealing glue leakage and Sealing glue overflow of the optical diffraction element is completed through a first image to be tested, but the insufficient glue of Sealing is located on the other side of the optical diffraction element, so it cannot be detected by using the first image to be tested. The detection method also includes: S500 , obtaining a second image to be tested, and preprocessing the second image to be tested to obtain a second ROI region sub-image of the second image to be tested.

[0034] The second image to be tested is acquired by a second image acquisition device, which includes a camera, coaxial light and annular light. The coaxial light and annular light are combined to illuminate and can clearly show the defect of insufficient glue. The camera is the same as the first image acquisition device.

[0035] Also, because the background of the second image to be tested is complex, the second image to be tested needs to be preprocessed by cutting and positioning to obtain a second ROI region sub-image of the area to be tested. The process of preprocessing the second image to be tested is the same as the process of preprocessing the first image to be tested.

[0036] S600, loading a second deep learning model deployed in advance to perform semantic segmentation on the second ROI region sub-image, where the second deep learning model is used to identify the second dispensing area and output a second feature map.

[0037] The second deep learning model and the first deep learning model are formed based on the same initial model training, and the same initial model is trained using different training sets. The second feature map of the second dispensing area is shown in the attached Fig.14 shown.

[0038] S700. The second feature map uses the third 2D traditional algorithm to detect the fourth defect.

[0039] Steps S500-S700 complete the detection of insufficient glue in Sealing of the optical diffraction element through a second image to be tested. In one embodiment, S500-S700 can be performed simultaneously with S100-S400, or the detection of insufficient glue in Sealing can be performed first through S500-S700, and then the detection of DOE contamination, glue leakage in Sealing and glue overflow in Sealing can be performed through S100-S400.

[0040] In one embodiment, preprocessing the first image to be tested to obtain a first ROI region sub-image of the first image to be tested and establishing a geometric rectangular coordinate system specifically includes: The first image to be tested is roughly positioned, the ROI is positioned using the template positioning in the 2D tool, and the image is cut to form a first ROI region sub-image. The pixel size of the first ROI region sub-image is 1728*1440.

[0041] Precisely locate the first ROI area sub-image, use the 2D line finding tool to take the four sides of the product and establish a geometric rectangular coordinate system based on two perpendicular sides of the four sides. Fig.12 As shown, the four variables are L1, L2, L3 and L4, and a geometric rectangular coordinate system is established with the intersection of the extended lines of L4 and L3 as the origin O, L3 as the X-axis, and L4 as the Y-axis.

[0042] In one embodiment, the first deep learning model is pre-trained, see the attached Figure 2 As shown, the training method specifically includes: S21. Acquire a first data set and divide the first data set into a first training set and a first validation set.

[0043] The first data set includes a plurality of ROI region sub-images, wherein the plurality of ROI region sub-images have Sealing leaks, Sealing overflows, and no defects. The ROI region sub-images are obtained by the pre-processing step in step S100. Exemplarily, there are 300 ROI region sub-images, wherein 100 ROI region sub-images each have Sealing leaks, Sealing overflows, and no defects.

[0044] The first data set is divided into a first training set and a first validation set according to a set ratio, and the first training set and the first validation set are allocated in a ratio of 9:1 to the total number of images. The first validation set should contain defect types and distributions similar to those of the first training set, while avoiding duplication with the first training set to ensure the accuracy and reliability of the evaluation results.

[0045] S22, marking defects in the first glue spot area of ​​the ROI area sub-image to form a marked image.

[0046] For example, the first glue spot area in the ROI area sub-image is painted with a brush, strictly following the standard of one pixel. The annotation information includes the pixel size and position of the first glue spot area. After annotation, an annotation map with the same size as the original ROI area sub-image is generated. The annotation map can be compared with the model annotation map to verify the model.

[0047] S23, cutting the ROI region sub-image and the annotation image to form a small image.

[0048] In one embodiment, the pixel size of the ROI region sub-image formed by cutting is 1728*1440. The image is relatively large. If the entire image is directly used for training, insufficient graphics card memory will occur, resulting in training failure. Therefore, it is necessary to cut the ROI region sub-image into small images, and divide the ROI region sub-image and its corresponding annotation image into 2 rows and 2 columns. After cutting the ROI region sub-image and its corresponding annotation image, four small images and a small annotation image are generated. The pixel size of the small image after cutting is 864*720. The small image formed after cutting is input into the subsequent initial model for training.

[0049] Of course, in one embodiment, if the graphics card memory of the controller is sufficient, the cutting of the ROI region sub-image and the annotation image may not be performed.

[0050] S24. Build an initial model based on the Unet++ network structure.

[0051] The basic network UNet++ model is a nested encoder-decoder structure, in which the encoder consists of two convolution modules and maximum pooling downsampling, and the decoder consists of transposed convolution upsampling and two convolution modules. Each decoder not only contains feature maps of a single layer, but also introduces feature maps of other layers. The same layer uses jump connections to fuse semantic information at different stages. In this embodiment, a multi-scale large-core convolution module is used to replace the two convolution modules in the UNet++ model to build an initial model.

[0052] Set the initial model training parameters, including learning rate, batch size (batch_size), number of training rounds (epoch), etc. For example, according to factors such as the image pixel size and the number of first training sets, the initial learning rate is set to 0.001, the batch size is set to 16, and the number of training rounds is set to 180.

[0053] The initial model is shown in the attached Figure 7As shown in the figure, the downward solid arrow represents downsampling, the upward solid arrow represents upsampling, and the dotted arrow represents a jump connection. The purpose of downsampling is to increase the robustness to some small perturbations of the input image, such as image translation, rotation, etc., reduce the risk of overfitting, reduce the amount of calculation and increase the receptive field, so that the subsequent convolution kernel can learn more global information. The purpose of upsampling is to restore and decode the abstract features to the size of the original image, and finally obtain the segmentation result; the jump connection combines short connections with long connections. The short connection is to enable the model to be trained, and the long connection is to obtain more information.

[0054] See attached Figure 7 As shown, the specific steps of the initial model processing the image are as follows: 24a. The cut small image is input to the MSLK0.0 module, and the output feature map is 0.0; 24b, feature map 0.0 is downsampled and then input into the MSLK1.0 module, outputting feature Figure 1 .0; 24c. Features Figure 1 .0 is upsampled and concatenated with feature map 0.0, and then input into the MSLK0.1 module to output feature map 0.1; 24d. Features Figure 1 .0 is downsampled and then input into the MSLK2.0 module to output features Figure 2 .0; 24e. Features Figure 2 .0 after upsampling and feature Figure 1 .0 splicing, input into MSLK1.1 module together, output features Figure 1 .1; 24f. Features Figure 1 .1 is upsampled and concatenated with feature map 0.0 and feature map 0.1, and input into the MSLK0.2 module together, and outputs feature map 0.2; 24g, Features Figure 2 .0 is downsampled and then input into the MSLK3.0 module to output features Figure 3 .0; 24h, Features Figure 3 .0 after upsampling and feature Figure 2 .0 splicing, input into MSLK2.1 module together, output features Figure 2 .1; 24i, Features Figure 2 .1 After upsampling and feature Figure 1 .0. Features Figure 1 .1 splicing, input together into MSLK1.2 module, output features Figure 1 .2; 24j. Features Figure 1.2 is upsampled and concatenated with feature maps 0.0, 0.1, and 0.2, and input into the MSLK0.3 module to output feature map 0.3; 24k, Features Figure 3 .0 is downsampled and then input into the MSLK4.0 convolution module to output features Figure 4 .0; 241. Features Figure 4 .0 after upsampling and feature Figure 3 .0 splicing, input into MSLK3.1 module together, output features Figure 3 .1; 24m, Features Figure 3 .1 After upsampling and feature Figure 2 .0. Features Figure 3 .1 splicing, input together into MSLK2.2 module, output features Figure 2 .2; 24n, Features Figure 2 .2 After upsampling and feature Figure 1 .0. Features Figure 1 .1. Features Figure 1 .2 splicing, input together into MSLK1.3 module, output features Figure 1 .3; 24o. Features Figure 1 After upsampling, .3 is concatenated with feature maps 0.0, 0.1, 0.2, and 0.3, and input into the MSLK0.4 module to output feature map 0.4.

[0055] S25, inputting the first training set into the initial model for training and saving the trained initial model.

[0056] In each epoch, the back propagation algorithm is used to optimize the network weights, and the neural network weights and parameters are updated after each round to continuously optimize and update the model. During the training process, the changes in the loss function are monitored to prevent overfitting. In the segmentation task, the mean intersection over union (mIOU) and the loss value (loss) are important indicators for evaluating model performance. mIOU measures the similarity between the predicted result and the true label, and the purpose of the loss function is to quantify the prediction error of the model. If the model mIOU is low and the loss value is high after training, hyperparameters such as learning rate, batch size, weight initialization, etc. need to be adjusted.

[0057] Use the Dice Loss function and SGDM optimizer to monitor the training process and find the best combination of model parameters. Dice Loss is based on the Dice coefficient and aims to optimize the overlap between the model's prediction and the true label. The Dice coefficient (or F1 Score) calculation formula is as follows:

[0058] Where Intersection represents the intersection (intersecting area) of the predicted mask and the true mask. Union represents the union of the predicted mask and the true mask. Dice Loss is a loss function designed based on the Dice coefficient, and its calculation formula is:

[0059] in is a small constant used to prevent the denominator from being zero.

[0060] The mIOU calculation formula is as follows:

[0061] Where P ij It means predicting category i as category j.

[0062] S26. Input the first verification set into the trained initial model for verification, and save the initial model that performs best on the first verification set as the first deep learning model.

[0063] See attached Figure 8 As shown in the figure, the multi-scale large kernel convolution module includes a series of 3×3 convolutions, a convolutional attention mechanism unit, three parallel branches of multi-scale large kernel convolutions, a splicing unit and a 1×1 convolution. The multi-scale large kernel convolution module adopts residual connection.

[0064] The processing process of the multi-scale large kernel convolution module on the input image specifically includes: a. The input image is processed by a 3×3 convolution and a convolutional attention mechanism unit respectively.

[0065] b. Use three-branch multi-scale large kernel convolution to extract image features.

[0066] Two depth strip convolutions are used in each branch to approximate the standard depth convolution with a large kernel. Here, the kernel size of each branch is set to 7, 11, and 21 respectively. For example, in order to simulate the standard 2D convolution with a kernel size of 7×7, only a pair of 7×1 and 1×7 strip convolutions are needed. Therefore, compared with the standard 2D convolution, the computational complexity of the strip convolution is exponentially reduced, which also makes it possible to use large kernel convolution in the model. The use of large kernel convolution can obtain a larger receptive field and linear complexity, which not only realizes the adaptability of the spatial dimension, but also realizes the adaptability of the channel dimension, which is crucial for computer vision tasks.

[0067] c. Concatenate the outputs of the large kernel convolutions, use 1×1 convolution to adjust the number of channels, and model the relationship between different channels.

[0068] d. The input graph forms an output image through residual connection, which alleviates the gradient vanishing problem of the deep network and avoids model degradation.

[0069] See attached Fig. 9 As shown in the figure, the convolutional block attention module unit includes a channel attention mechanism and a spatial attention mechanism connected in series, and the output of the channel attention mechanism serves as the input of the spatial attention mechanism.

[0070] See attached Fig.10 As shown in the figure, in the channel attention mechanism, the input feature map is subjected to global maximum pooling and global average pooling respectively, and then sent to two fully connected layers (MLP), the two feature maps are added together, and then normalized by the Sigmoid activation function to obtain the final Channel Attention. See the attached Fig.11 As shown in the figure, in the spatial attention mechanism, global maximum pooling and global average pooling are performed first, but then the pooled outputs are concatenated for a convolution operation, and finally a Sigmoid activation function is used for normalization to obtain Spatial Attention. Therefore, the CBAM unit can not only better capture the relationship between different channels to improve the feature representation ability, but also better focus on important areas in the image to improve the modeling ability of spatial information.

[0071] In one embodiment, see the attached Figure 3 As shown, the first ROI region sub-image uses the first 2D traditional algorithm to detect the first defect, specifically including: S31. On the first ROI region sub-image, based on the coordinate system, the four sides are indented inward by a fixed distance to establish ROI region 1 where defect 1 is located.

[0072] S32, searching for the first defect region in ROI region 1 by using an image analysis tool.

[0073] The image analysis tool is the 2D tool Blob in the OpenCV open source library.

[0074] S33. Determine whether the area of ​​the first defect is smaller than a set threshold.

[0075] The threshold is set manually and can be changed according to different products.

[0076] S34. If so, the first defect does not exist.

[0077] S35, if not, then there is a first defect, and the product is now a substandard product.

[0078] See the attached for the defects of DOE contamination found. Fig.15 shown.

[0079] In one embodiment, see the attached Figure 4 As shown, the first feature map uses the second 2D traditional algorithm to detect the second defect and the third defect, specifically including: S41. On the first feature map, with the coordinate system as a reference, the four edges are indented inward by a fixed distance, and four ROI regions 2 of the same size are established in different first glue-dispensing areas.

[0080] See attached Fig.16 As shown, the four ROI regions 2 are ROI region A, ROI region B, ROI region C and ROI region D. Since glue overflow and glue leakage may occur in these four regions, the four ROI regions 2 are detected.

[0081] S42. Search for the area of ​​glue in two of the four ROI regions using an image analysis tool.

[0082] S43, judging whether the area of ​​the glue in the four ROI regions 2 is within a set range.

[0083] The setting range includes the maximum threshold and the minimum threshold.

[0084] S44. If it is within the set range, the second defect and the third defect do not exist.

[0085] S45, if it is not within the set range, determine whether the area of ​​the glue in the second of the four ROI regions is smaller than the set range.

[0086] S46. If it is less than the set range, it is the second defect.

[0087] At this time, the glue area is smaller than the minimum threshold, indicating that the glue is insufficient, that is, the sealing glue is leaking.

[0088] S47. If it is greater than the set range, it is the third defect.

[0089] At this time, the glue area is greater than the maximum threshold, indicating that there is too much glue, which is called sealing glue overflow.

[0090] In one embodiment, the training process of the second deep learning model is substantially the same as that of the first deep learning model, and the same initial model is used. Figure 5 As shown in Figure 2, the training process specifically includes: S61. Acquire a second data set and divide the second data set into a second training set and a second validation set.

[0091] The second data set includes multiple ROI area sub-images, wherein multiple ROI area sub-images have insufficient sealing and no defects. Exemplarily, the second training set is at least 100 ROI area sub-images cropped from the second image, wherein at least 50 images have insufficient sealing and qualified product images.

[0092] S62, marking defects in the second glue spotting area of ​​the ROI area sub-image to form a marked image.

[0093] The sealing glue part needs to be applied with a paintbrush.

[0094] S63, cutting the ROI region sub-image and the annotation image to form a small image.

[0095] S64, inputting the second training set into the constructed initial model for training and saving the trained initial model.

[0096] S65. Input the second verification set into the trained initial model for verification, and save the initial model with the best performance on the second verification set as the second deep learning model.

[0097] At this point, the initial model has been built and can be directly called without having to build it again. The second deep learning model infers the input image data and uses the forward propagation of the neural network to generate the prediction results of the pixel size and position of the sealing glue and generate the following Fig.14 The second characteristic diagram shown completes the intelligent detection.

[0098] Step S700 is substantially the same as step S400. Figure 6 As shown, specifically including: S71. On the second feature map, with the coordinate system as a reference, the four edges are indented inward by a fixed distance, and four ROI regions of the same size are established in different first glue-dispensing areas.

[0099] S72. Search for the area of ​​glue in three of the four ROI regions using an image analysis tool.

[0100] S73, judging whether the area of ​​the glue in the third of the four ROI regions is greater than a set threshold.

[0101] S74: If it is greater than the set threshold, the fourth defect does not exist.

[0102] S75. If it is not greater than the set threshold, there is a fourth defect. That is, insufficient glue in Sealing. For the abnormality of insufficient glue in Sealing, please refer to the attached Fig.17 shown.

[0103] In order to verify that the defect detection method of the optical diffraction element after packaging of the present invention is better than the traditional detection method, and that the first deep learning model and the second deep learning model in the present invention have better effects than other common semantic segmentation models (UNet, UNet++, HRNet, PSPNet), the following comparative experiments are carried out.

[0104] Table 1 is a comparison between the detection method of the present invention and the traditional 2D algorithm defect detection method, and the comparison is made under actual production environment conditions, wherein the time stated in the table is: the average processing time of the complete process from feeding to discharging of a single particle in production.

[0105] Table 1

[0106] It can be clearly seen from Table 1 that the detection time of the present invention is shorter than that of the traditional 2D algorithm, and the false positive rate and missed positive rate are lower.

[0107] Table 2 is a comparison of the segmentation results of the first deep learning model in the present invention and other common semantic segmentation models (UNet, UNet++, HRNet, PSPNet) on the Sealing dispensing dataset (custom light source one).

[0108] Table 2 Unit: %

[0109] It can be clearly seen from Table 2 that the first deep learning model has a higher average intersection-over-union ratio and a lower loss value, and the first deep learning model has a better segmentation effect.

[0110] The present invention uses a traditional 2D algorithm combined with a deep learning semantic segmentation method to extract the defect contour of the area after the optical diffraction element is packaged in the picture, and then calculates the defect area by a traditional image processing algorithm. The purpose is to effectively identify DOE contamination and post-packaging dispensing defects, thereby improving the precision of the camera module.

[0111] In one embodiment, the present invention further discloses an optical diffraction element defect detection system based on deep learning, and the system adopts the above-mentioned detection method.

[0112] The system includes a first image acquisition device, a second image acquisition device, a controller and a display device.

[0113] The present invention also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned detection method are implemented.

[0114] Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above-mentioned method embodiments when executed by a processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0115] The above implementation modes are only for illustrating the technical concept and features of the present invention, and their purpose is to enable people familiar with this technology to understand the content of the present invention and implement it, and they cannot be used to limit the protection scope of the present invention. Any equivalent changes or modifications made according to the spirit of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting defects in optical diffraction elements based on deep learning, characterized in that: include: Acquire a first image to be tested, preprocess the first image to be tested to obtain a first ROI region sub-image of the first image to be tested and establish a geometric rectangular coordinate system; Loading a first deep learning model deployed in advance to perform semantic segmentation on the first ROI region sub-image, the first deep learning model is used to identify the first dispensing area and output a first feature map, the first deep learning model is based on a Unet++ network structure, and both the encoder and the decoder of the first deep learning model use a multi-scale large-kernel convolution module; The first ROI region sub-image uses a first 2D traditional algorithm to detect a first defect; The first feature map uses a second 2D traditional algorithm to detect the second defect and the third defect.

2. The optical diffraction element defect detection method based on deep learning according to claim 1, characterized in that: Preprocessing the first image to be tested to obtain a first ROI region sub-image of the first image to be tested and establishing a geometric rectangular coordinate system specifically includes: Performing rough positioning on the first image to be tested, using the template positioning in the 2D tool to perform ROI positioning, and cropping to form a first ROI area sub-image; The first ROI region sub-image is precisely positioned, and the four sides of the product are taken using a 2D line finding tool and a geometric rectangular coordinate system is established based on two perpendicular sides of the four sides.

3. The optical diffraction element defect detection method based on deep learning according to claim 1, characterized in that: The first deep learning model is pre-trained, and the training method specifically includes: Acquire a first data set and divide the first data set into a first training set and a first validation set, wherein the first data set includes a plurality of ROI region sub-images, wherein the plurality of ROI region sub-images have a second defect, a third defect, or no defect; Performing defect marking on the first glue dispensing area of ​​the ROI area sub-image to form a marked image; Construct an initial model based on the Unet++ network structure. The initial model is a nested encoder-decoder structure. The encoder consists of two multi-scale large-kernel convolution modules and maximum pooling downsampling. The decoder consists of transposed convolution upsampling and two multi-scale large-kernel convolution modules. Skip connections are used between the same layers to fuse semantic information at different stages. Inputting the first training set into an initial model for training and saving the trained initial model; The first validation set is input into the trained initial model for validation, and the initial model that performs best on the first validation set is saved as the first deep learning model.

4. The optical diffraction element defect detection method based on deep learning according to claim 1, characterized in that: The multi-scale large kernel convolution module includes a series of 3×3 convolutions, a convolution attention mechanism unit, three parallel branches of multi-scale large kernel convolutions, a splicing unit and a 1×1 convolution. The multi-scale large kernel convolution module adopts a residual connection. The processing process of the multi-scale large kernel convolution module on the input image specifically includes: The input image is processed by a 3×3 convolution and a convolutional attention mechanism unit respectively; Image features are extracted using multi-scale large kernel convolutions with three branches, where two depth strip convolutions are used in each branch to approximate standard depth convolutions with large kernels; Concatenate the outputs of large kernel convolutions, use 1×1 convolution to adjust the number of channels, and model the relationship between different channels; The input graph is connected through residual connections to form the output image.

5. The optical diffraction element defect detection method based on deep learning according to claim 4, characterized in that: The convolutional attention mechanism unit includes a channel attention mechanism and a spatial attention mechanism connected in series, and the output of the channel attention mechanism serves as the input of the spatial attention mechanism.

6. The optical diffraction element defect detection method based on deep learning according to claim 2, characterized in that: The first ROI region sub-image uses the first 2D traditional algorithm to detect the first defect specifically including: On the first ROI region sub-image, based on the coordinate system, the four edges are indented inward by a fixed distance to establish a ROI region 1 where the defect 1 is located; Searching for the first defect region in ROI region 1 by using an image analysis tool; Determining whether the area of ​​the first defect is smaller than a set threshold; If so, the first defect does not exist; if not, the first defect exists.

7. The optical diffraction element defect detection method based on deep learning according to claim 2, characterized in that: The first feature map detects the second defect and the third defect according to the second 2D traditional algorithm specifically including: On the first feature map, based on the geometric rectangular coordinate system, the four edges are indented inward by a fixed distance, and four ROI regions 2 of the same size are established in different first glue dispensing areas; Searching for the area of ​​glue in two of the four ROI regions using an image analysis tool; Compare the area of ​​glue in the four ROI regions 2 with the set range; If it is within the set range, there is no second defect and third defect; if it is less than the set range, it is the second defect; if it is greater than the set threshold, it is the third defect.

8. The optical diffraction element defect detection method based on deep learning according to any one of claims 1 to 7, characterized in that: The detection method further comprises the following steps: Acquire a second image to be tested, and preprocess the second image to be tested to obtain a second ROI region subimage of the second image to be tested; Loading a pre-deployed second deep learning model to perform semantic segmentation on the second ROI region sub-image, the second deep learning model is used to identify the second dispensing area and output a second feature map, the second deep learning model and the first deep learning model are formed based on the same initial model training; The second feature map uses a third 2D traditional algorithm to detect a fourth defect.

9. An optical diffraction element defect detection system based on deep learning, characterized in that: The system adopts the detection method described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the detection method according to any one of claims 1 to 8 are implemented.

Images