Automatic focusing method and system for substrate glass defect image based on bright and dark field fusion
Through deep learning, the cross-fusion network of dual-branch features and combining bright and dark field image features, the autofocus problem under the cross-image of light and dark field of substrate glass is solved, and efficient and accurate defect image recognition is achieved, which significantly improves autofocus Accuracy.
Patent Information
- Application Number
- CN202411346194.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2024-12-20
- Estimated Expiration
- 2044-09-26
AI Technical Summary
The existing autofocus algorithm cannot be effectively applied to the cross-field imaging of substrate glass, and the clearest image of substrate glass defects cannot be found.
A two-branch feature cross-fusion network based on deep learning is adopted, combining bright and dark field image features, and through the spatial and channel attention cross-fusion module, high-throughput and accurate defect image autofocus is achieved.
The automatic focus accuracy of the substrate glass defect image is significantly improved, and the prediction accuracy and time complexity are excellent, reaching an accuracy of 96.5%, which is far higher than the existing methods.
Smart Images
Figure CN118870190B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of automatic focusing, and in particular relates to a method and system for automatic focusing of a substrate glass defect image based on bright and dark field fusion. Background Art
[0002] Liquid crystal displays have the characteristics of low power consumption, thin and light volume, large-format display, and full-color display, and are widely used in scenarios such as smartphones, medical imaging, and industrial control. As the core basic component of liquid crystal displays, the performance of substrate glass determines the key indicators of display devices such as resolution, transmittance, and viewing angle. The substrate glass is about 5 cm thick, and defects may appear at different thicknesses of the glass. Only by adjusting the array camera lens to the appropriate distance can the glass defects be clearly imaged.
[0003] Most autofocus tasks are based on the clarity evaluation function, which will produce the maximum response value at the most suitable position of the object distance, so as to find the clearest image. The clarity evaluation function is usually constructed using gradient, frequency and information entropy. For example, the Brenner algorithm calculates the sum of squares of adjacent pixel differences, the Tenengrad algorithm uses the Sobel operator to extract the gradient values in the horizontal and vertical directions respectively, the Laplacian algorithm calculates the second-order gradient of the image, the FFT, DCT and wavelet transform calculate the frequency characteristics of the image, and the pixel variance and Shannon entropy analyze the statistical characteristics of the image. At present, the autofocus work based on the clarity evaluation function has been very mature in natural images, but it performs poorly on substrate glass images. Recently, there has been initial progress in the use of deep learning to process autofocus tasks. This type of method regards the task as a classification or regression task, and directly uses the network to predict the clearest image, but the scene is still a natural image, and it cannot adapt to the substrate glass image under bright and dark field cross imaging.
[0004] In summary, traditional clarity evaluation functions based on image gradient, frequency, and information entropy are mostly designed for natural images and are not suitable for substrate glass under bright field and dark field imaging. Therefore, how to efficiently and accurately select the clearest substrate glass defect image has become an urgent problem to be solved. Summary of the invention
[0005] The purpose of the present invention is to address the problem that most of the current autofocus algorithms are designed for natural images and cannot be applied to substrate glass under bright and dark field cross imaging, and cannot find the clearest bright field and dark field images of substrate glass defects. The technical solution of the present invention proposes an ordinal regression technology based on a deep learning framework, and proposes a substrate glass defect image autofocus method and system based on bright and dark field fusion, aiming to find the clearest substrate glass defect bright field and dark field images. The method uses bright field and dark field images at the same time, and adopts a dual-branch feature cross fusion network to exchange feature information of bright field image sequences and dark field image sequences, so as to achieve high-throughput and accurate defect image autofocus, and can show excellent performance in terms of prediction accuracy and time complexity.
[0006] To this end, the present invention provides the following technical solutions:
[0007] In one aspect, the present invention provides a method for automatically focusing a substrate glass defect image based on bright and dark field fusion, comprising the following steps:
[0008] Acquire bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence;
[0009] Extracting image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map;
[0010] A dual-branch feature cross-fusion network is constructed by using several serially connected CIM modules, and then connecting global average pooling and 1*1 convolutional layers. The output of the previous CIM module is used as the input of the next CIM module. The CIM module is a 2-input 2-output spatial and channel attention cross-fusion module.
[0011] The bright field feature map and the dark field feature map are input into the dual-branch feature cross-fusion network to obtain the network output vectors corresponding to the bright field and the dark field. , , and use the network output vector , Determine the clearest bright field image and dark field image to achieve focus;
[0012] Among them, the camera is used to take bright field and dark field images of the substrate glass at different object distances to construct training samples, and the clearest bright field images and dark field images are used to make true value labels, which are then compared with the focusing results of the training samples to complete network training.
[0013] Further optionally, the processing process of each CIM module is:
[0014] First, the convolutional channel attention mechanism CAM is used to obtain the channel self-attention vector and channel cross-attention vector corresponding to the input bright field and dark field feature maps respectively;
[0015] Then, the channel self-attention vector corresponding to the bright field is multiplied by the bright field feature map channel by channel to obtain , the channel self-attention vector corresponding to the dark field is multiplied by the dark field feature map channel by channel to obtain , the channel-wise multiplication of the attention vector corresponding to the dark field and the bright field feature map is obtained , the channel-wise multiplication of the attention vector corresponding to the bright field and the dark field feature map is obtained ;
[0016] Next, for , , , Both use the convolutional spatial attention mechanism SAM to obtain the spatial self-attention matrix and and the spatial cross-attention matrix , ;
[0017] Finally, , , , One-to-one correspondence with , , , Multiply element by element to get , , , ,and then and Add together to get , and Add together to get ,vector , It is the output of CIM module.
[0018] Further optionally, the convolutional channel attention mechanism CAM is composed of a maximum pooling layer along the spatial dimension, an average pooling layer along the spatial dimension, a multi-layer perceptron MLP, and a Sigmoid function, and the obtained channel self-attention vector and channel cross-attention vector of the bright field are equal; the obtained channel self-attention vector and channel cross-attention vector of the dark field are equal;
[0019] The convolutional spatial attention mechanism SAM is composed of a maximum pooling layer along the channel dimension, an average pooling layer along the channel dimension, a convolutional layer, and a Sigmoid function.
[0020] Further optionally, after obtaining the dark field image of the substrate glass captured by the camera to obtain a dark field image sequence, the method further includes performing gradient guidance on the dark field image sequence, and then inputting the gradient-guided dark field image sequence into a feature map extraction network to obtain a dark field feature map;
[0021] The gradient guidance operation is as follows:
[0022] First, for the dark field image sequence, the Laplacian operator is used to calculate the second-order gradient channel by channel, and the gradient below the gradient threshold is set to 0, and finally the gradient map sequence is obtained;
[0023] Next, the variance of each gradient map is calculated in turn to obtain the variance vector;
[0024] Then, a two-layer fully connected encoding network is designed, and the variance vector is encoded. The encoding formula is as follows:
[0025] ,
[0026] ,
[0027] In the formula, and is the first layer network training parameter, and is the training parameter of the second layer network, is the first layer output, is the output vector of the encoding network, V is the variance vector;
[0028] Finally, the output vector The elements in are multiplied by the dark field image sequence to obtain the gradient-guided dark field image sequence.
[0029] Further optionally, after acquiring the bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence, and before extracting image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map, the method also includes: after acquiring the bright field image sequence and the dark field image sequence taken by the camera, using a Gamma transform to enhance the image of the bright field image sequence, and using a median filtering algorithm to sequentially process the images of the dark field image sequence.
[0030] Further optionally, the camera uses a cross imaging method to shoot bright field and dark field images of the substrate glass at different object distances to construct training samples, and selects an even number of times of object distance adjustment to shoot bright field images and an odd number of times of object distance adjustment to shoot dark field images; or selects an odd number of times of object distance adjustment to shoot bright field images and an even number of times of object distance adjustment to shoot dark field images;
[0031] Then, the clearest bright field image and dark field image found in the training sample are taken as adjacent images, and the serial numbers of the clearest bright field image and dark field image are recorded. and , then use the serial number and Determine the sequence number of each bright field image and The distance, the serial number J of each dark field image and Distance:
[0032] ,
[0033] ,
[0034] In the formula, is the bright field image serial number I and The distance is the dark field image sequence number J and Distance Then, based on the function corresponding to the distribution type, the distance vector , Converted into soft labels for training network model parameters, , is a distance vector formed by a bright field image sequence and a dark field image sequence;
[0035] If the Laplace distribution is selected, the normalized exponential function and the distance vector , Constructed soft label and It is expressed as:
[0036] ,
[0037] ,
[0038] In the formula, represents the normalized exponential function, is the true soft label of the bright field, is the true soft label of the dark field.
[0039] Optionally, when training the network, use KL divergence as the training loss, ADAM as the optimizer, and use Function processing and Get the vectors corresponding to the bright field and dark field , , so that the non-negative and one constraints are satisfied. The vector , The index corresponding to the maximum element value in corresponds to the bright field image and dark field image, which are the clearest bright field image and dark field image;
[0040] The overall loss Loss is expressed as:
[0041] ,
[0042] ,
[0043] ,
[0044] In the formula, express , express , N is the number of vector elements, i is the vector index, , is the weight of bright field classification loss and dark field classification loss.
[0045] In a second aspect, the present invention further provides a system based on the above-mentioned substrate glass defect image automatic focusing method based on bright and dark field fusion, comprising:
[0046] An image acquisition module is used to acquire the bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence;
[0047] A feature extraction module, used to extract image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map;
[0048] A fusion module is used to construct a dual-branch feature cross-fusion network by using several serially connected CIM modules, and then connecting the global average pooling and the 1*1 convolution layer. The output of the previous CIM module is used as the input of the next CIM module. The CIM module is a 2-input 2-output spatial and channel attention cross-fusion module.
[0049] Focus module, used to input the bright field feature map and dark field feature map into the double-branch feature cross-fusion network to obtain the network output vector corresponding to the bright field and dark field , , and use the network output vector , Determine the clearest bright field image and dark field image to achieve focus;
[0050] Among them, the camera is used to take bright field and dark field images of the substrate glass at different object distances to construct training samples, and the clearest bright field images and dark field images are used to make true value labels, which are then compared with the focusing results of the training samples to complete network training.
[0051] In a third aspect, the present invention further provides an electronic terminal, comprising:
[0052] one or more processors;
[0053] a memory storing one or more computer programs;
[0054] The processor calls the computer program to implement:
[0055] The above-mentioned steps of the substrate glass defect image automatic focusing method based on bright and dark field fusion.
[0056] In a fourth aspect, the present invention further provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement:
[0057] The above-mentioned steps of the substrate glass defect image automatic focusing method based on bright and dark field fusion. Beneficial Effects
[0058] Compared with the existing method, the advantages of the present invention are:
[0059] 1. The technical solution of the present invention proposes a model based on dual-branch feature cross-fusion, and designs a dual-branch feature cross-fusion module, which can exchange the feature information of the bright field image sequence and the dark field image sequence, so that the neural network can simultaneously extract and fuse the features of the bright field sequence and the dark field sequence. The dual-branch feature cross-fusion module provided by the present invention consists of two parts: spatial attention and channel attention, which are used to extract spatial features and sequence features respectively. It should be understood that when photographing the substrate glass, some defects are not clearly imaged under the irradiation of the bright field light source, but the dark field imaging is very obvious, but some defects are just the opposite and even not imaged directly in the dark field, so it is necessary to use both bright field features and dark field features. However, if the bright field image and the dark field image are simply stacked together by channel and then sent to the network, and the dual-branch strategy proposed by the present invention is not adopted, the prediction accuracy will be very low. Therefore, the technical solution of the present invention can effectively improve the autofocus accuracy of the substrate glass defect image.
[0060] 2. In the preferred embodiment of the present invention, image gradient guidance is proposed, which guides the network to understand the image clarity by calculating the variance of the second-order gradient of the dark field image sequence. Image gradient can reflect the edge sharpness of the image. Clear images often have very sharp edges, so it is reasonable to guide network prediction by gradient. In the substrate glass defect scene, the study found that some images did not meet the above rules very well, so it is preferred to add an encoding network to encode the gradient first and then introduce it into the network, so that the network can adaptively process the gradient information. Through experimental analysis, if the gradient guidance module is not introduced, the prediction accuracy on the test set is 92.1%, and the number of training rounds must be more than 60 rounds (the network can converge after 30 rounds of training after adding it). In summary, the use of gradient guidance can improve the prediction accuracy and generalization of the network, reduce the number of training rounds, and effectively improve the autofocus accuracy of substrate glass defect images. The accuracy rate on the constructed bright field and dark field data sets reached 96.5%, which is much higher than the existing autofocus method.
[0061] 3. The present invention preferably generates bright field images and dark field images crosswise, so the two best images must be adjacent. The present invention makes full use of this very important prior information. It should be understood that if the bright field sequence and the dark field sequence are processed separately, the prior information is not used and the prediction results of the two will not be adjacent, which is contrary to the facts.
[0062] 4. The present invention also proposes to transform the autofocus problem into an ordinal regression problem, construct soft labels through Laplace distribution, and use KL divergence as training loss to ensure the accuracy of network training. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] Figure 1 It is a flow chart of a substrate glass defect image automatic focusing method based on bright and dark field fusion disclosed in the present invention;
[0064] Figure 2a , Figure 2b They are schematic diagrams of shooting bright field and dark field sequence data sets respectively;
[0065] Figure 3a This is the effect picture before the bright field image Gamma calibration. Figure 3b This is the effect diagram after Gamma calibration of bright field image. Figure 3c This is the effect diagram of the dark field image before median filtering. Figure 3d This is the effect of the dark field image after median filtering;
[0066] Figure 4 This is a classification network architecture diagram of dual-branch feature cross-fusion and image gradient guidance;
[0067] Figure 5a , Figure 5b They are the prediction results of the dark field model and the bright field model respectively. DETAILED DESCRIPTION
[0068] The technical solution of the present invention proposes a method for automatically focusing substrate glass defect images based on bright-field and dark-field fusion and gradient guidance, which uses bright-field and dark-field cross-imaging to collect data on substrate glass, and adopts a double-branch feature cross-fusion network to perform high-throughput and accurate defect image automatic focusing, which can show excellent performance in terms of prediction accuracy and time complexity. The present invention will be further described below in conjunction with embodiments.
[0069] Embodiment 1:
[0070] The embodiment of the present invention preferably adopts a dual-branch feature cross-fusion and image gradient guided classification network for focusing, such as Figure 1 As shown, a substrate glass defect image auto-focusing method based on bright and dark field fusion is specifically provided, comprising the following steps:
[0071] Step 1: Adjust the distance between the substrate glass and the camera objective lens, use the area array camera for imaging, and then use the camera to capture bright field and dark field images of the substrate glass at different object distances;
[0072] In this embodiment, an area array camera is used for imaging. The area array camera is pulled by a vertical loading platform to adjust the distance between the substrate glass and the camera objective lens. The pulling platform descends 59 times in total, and 60 images are taken at different object distances, including 30 bright field images and 30 dark field images. Figure 2a , Figure 2b As shown, the instrument consists of an array camera, a motion traction mechanism, a bright field light source, a dark field light source and a substrate glass to be photographed.
[0073] Step 1.1: The area array camera moves downward every 0.1s. After the movement is completed, the light source is switched and the CMOS camera starts shooting.
[0074] Step 1.2: At even-numbered times (0, 2, 4, …, 58) of the drop, the bright field light source is turned on and a sequence of bright field images is taken. , the size of each image is 2560*2048, H and W are the length and width of the image, Bright field images taken for the descending even-numbered numbers 0 and 58 respectively.
[0075] Step 1.3: At the odd number of descents (1, 3, 5, ..., 59), the dark field light source is turned on and a dark field image sequence is taken. , the size of each image is 2560*2048, These are dark field images taken for the descending odd-order numbers 1 and 59. Since the amount of light entering the camera is small during dark field shooting, the digital gain of the CMOS needs to be adjusted to an appropriate value.
[0076] It should be noted that the present embodiment takes 60 photos, and other feasible embodiments are not limited to this; and the present embodiment preferably adopts a cross-shooting method so that the subsequent steps can determine that the clearest bright field image and dark field image are adjacent images. In other feasible embodiments, on the premise that the clearest bright field image and dark field image can be determined, the use of other shooting methods can be within the protection scope of the present invention and meet the requirements of the present invention.
[0077] Step 2: Use Gamma transformation to enhance the images of the bright field image sequence, and use the median filter algorithm to sequentially process the images of the dark field image sequence.
[0078] Among them, Figure 3a-3d As shown in the figure, the background of the bright field image is very similar to the defect. Gamma transformation is used to highlight the difference between the defect and the background and enhance the contrast of the bright field image sequence. The signal-to-noise ratio of the dark field image is low. Median filtering is used to reduce the noise intensity and retain the edge information.
[0079] On the one hand, the bright field image sequence is enhanced using the Gamma transform , the Gamma change formula is as follows:
[0080] ,
[0081] ,
[0082] in, It is the input image The preprocessed pixel value at the position, is the hyperparameter of gamma transformation. , the best correction effect can be achieved. is the pixel value after Gamma correction, is the defined intermediate parameter.
[0083] On the other hand, since the amount of light entering the camera during dark field shooting is small, the signal-to-noise ratio of the image is low and the brightness of the defect is weak. Therefore, the median filtering method is used to reduce the image noise intensity and retain the edge information of the image. The dark field image sequence is processed in sequence using the median filtering algorithm. The image size of the filter window is , the median filter formula is as follows:
[0084] ,
[0085] in, is the pixel in the filter window, is the center pixel of the filtered window, and the selected window size is 3*3. 2n+1 and 2m+1 represent the length and width of the filter window. Experiments show that a window size of 3*3 can achieve the best filtering effect, and the edge information of the image is lost the least. represents the image pixel position, ( , ) is the image pixel position within the filter window.
[0086] Then, stack the bright field and dark field sequences into a 3D tensor in the channel direction and with the subscript index of the image sequence. . Find the clearest bright field image and dark field image (these two images must be adjacent), and record the corresponding subscripts and As the true value for subsequent training, this embodiment finds the clearest bright field image and dark field image respectively through human eye observation. and truth value , Stored in mat format. It should be noted that stacking into tensors is the storage method for image sequences.
[0087] It should be understood that step 2 is a preferred solution of the technical solution of the present invention. In other feasible embodiments, other image preprocessing means or no image preprocessing are used to meet the technical idea of the present invention. The difference lies in the difference in effect.
[0088] Step 3: extracting image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map.
[0089] Since clear images often have large gradients, this embodiment preferably designs a gradient guidance module to improve algorithm performance.
[0090] First, for the dark field image sequence processed by the median filter algorithm in step 2 , use the Laplace operator to calculate the second-order gradient channel by channel, and set the gradient below the gradient threshold to 0, and finally get the gradient map sequence , the mathematical formula is as follows:
[0091] ,
[0092] ,
[0093] ,
[0094] Among them, Laplace is the Laplace operator, represents the convolution operation, It is an absolute value operation, threshold=20 is the gradient threshold, and the value in this embodiment is an example, which is an empirical value. is the second-order gradient, k is the subscript of the image sequence, Dark field image sequence The image sequence in is the dark field image of k.
[0095] Then calculate each gradient map in turn The variance of , the calculation formula is as follows:
[0096] ,
[0097] ,
[0098] Among them, H, W, , are the gradient image length, gradient image width, gradient image pixel mean and gradient image pixel variance respectively. is the gradient map at position The pixel value at .
[0099] In order to make full use of the gradient information, a two-layer fully connected module is designed to input the gradient vector Encoding allows the network to adaptively adjust the gradient information for dark field input image sequences The specific formula is as follows:
[0100] ,
[0101] ,
[0102] in, and is the first layer training parameter, and is the second layer training parameter, is the first layer output, is the output vector of the gradient encoding module. Elements in the dark field image sequence Multiply them together to get the gradient-guided dark field sequence , These are the dark field images with sequence numbers 1 and 59 after gradient guidance.
[0103] Finally, two Resnet50 networks (delete the last average pooling layer and fully connected layer) are used to extract the image features of the bright field and dark field sequence images respectively to obtain the bright field feature map , dark field feature map , the formula is as follows:
[0104] ,
[0105] ,
[0106] In the formula, This is a bright field image sequence enhanced by Gamma transformation.
[0107] It should be noted that the present embodiment preferably sets a gradient guidance module to improve the performance of the algorithm. In other feasible embodiments, directly using a feature extraction network to extract image features also meets the technical idea of the present invention, the difference lies in the difference in effect, such as directly using two Resnet50 networks to extract image features of bright field and dark field sequence images respectively.
[0108] Step 4: Use several serially connected CIM modules, and then connect the global average pooling and 1*1 convolutional layer to build a dual-branch feature cross-fusion network, such as Figure 4 As shown, the output of the previous CIM module serves as the input of the next CIM module.
[0109] In view of the problem that traditional clarity evaluation functions such as gradient, frequency, information entropy and existing deep learning algorithms for natural image focusing are not applicable to substrate glass images, the present invention designs a new network architecture to directly infer the clearest bright field image and dark field image labels. That is, a cross-fusion module is designed to share the features of bright field images and dark field images.
[0110] Since the bright field sequence and the dark field sequence are closely related and their information is complementary, the present invention designs a 2-input 2-output spatial and channel attention cross-fusion module CIM. The specific operation of the CIM module is as follows: first, CAM (Channel Attention Module, convolutional channel attention mechanism) is used to obtain , (It should be understood that the first CIM module corresponds to the bright field feature map and dark field feature map obtained in step 3.2; for other non-first CIM modules, the corresponding ones are the bright field feature map and dark field feature map input to the CIM module) channel self-attention vector , and the channel-cross-attention vector , CAM consists of a maximum pooling layer along the spatial dimension, an average pooling layer along the spatial dimension, a multi-layer perceptron MLP, and a Sigmoid function. The specific formula is as follows:
[0111] ,
[0112] ,
[0113] ,
[0114] In the above formula represents the training weights of the perceptron MLP, is the average pooling operation, is the maximum pooling operation, , . Similarly for:
[0115] ,
[0116] ,
[0117] Then and Multiply channel by channel to get , similarly and get , and get , and get Then use SAM (Spatial Attention Module, convolutional spatial attention mechanism) to obtain the spatial self-attention matrix and and the spatial cross-attention matrix , SAM consists of a maximum pooling layer along the channel dimension, an average pooling layer along the channel dimension, a convolutional layer, and a Sigmoid function. The specific formula is as follows:
[0118] ,
[0119] ,
[0120] ,
[0121] ,
[0122] In the above formula, It means that the input is first superimposed according to the channel dimension and then a 7*7 convolution operation is performed. After obtaining the four spatial attention matrices, they are respectively multiplied element by element with the four tensors processed by the corresponding CAM to obtain , Finally, and Add together to get , and similarly we get .
[0123] This embodiment is configured with three CIM modules, that is, three layers of CIM modules are connected in series, and the output of the CIM module is expressed as , (k=1,2,3), k represents the number of the CIM module, and is finally obtained from the output of the third CIM module , After global average pooling and 1*1 convolution, the network output is obtained. and , the formula is as follows:
[0124] ,
[0125] ,
[0126] Step 5: Network training.
[0127] The image sequence of this embodiment is generated as the traction mechanism moves downward. The distance between the front and rear objects in a frame is small. If it is directly converted into a traditional classification problem, this important prior will be lost. Therefore, based on soft labels, the autofocus problem is converted into an ordinal regression problem. Experiments show that this solution can complete the task quickly and accurately.
[0128] Step 5.1: Using the truth value , Constructing soft label sequence , , the soft label sequence is constructed as Laplace distribution. Among them, the soft label sequence should satisfy a certain statistical distribution. Laplace distribution is a unimodal distribution with few hyperparameters, so this distribution is used to construct soft labels. The formula is as follows:
[0129] ,
[0130] ,
[0131] ,
[0132] ,
[0133] In the formula, is the bright field image serial number I and The distance is the dark field image sequence number J and The distance It should be understood that softmax corresponds to the Laplace distribution. If other distributions (such as Gaussian distribution) are selected, the distance vector is processed by the corresponding function.
[0134] Step 5.2: Use Function processing and . Due to the obtained and The vector does not necessarily satisfy the non-negative and one constraints (the non-negative and one constraints refer to the elements in the quantity are all positive and add up to 1. Since we will introduce KL divergence as training loss later, the non-negative and one constraints must be satisfied). The formula is as follows:
[0135] ,
[0136] ,
[0137] Step 4.3: Use KL divergence as training loss, ADAM as optimizer, and learning rate -3, first-order moment exponential decay rate , the second-order moment exponential decay rate , weight decay term -3. Use an adaptive learning rate adjustment strategy when the training loss starts to fluctuate , there are 12,000 training samples (60 images per group). The overall training loss formula is:
[0138] ,
[0139] ,
[0140] ,
[0141] In the formula, express , express , N is the number of vector elements, i is the vector index, , is the weight of the bright field classification loss and the dark field classification loss. and .
[0142] The KL divergence set in this example describes the difference between probability distributions (one distribution is based on the soft labels of the Laplace distribution, and the other distribution is , ), and then use this as the model loss training model parameter. After the model is trained, the bright field and dark field images of the substrate glass are input into the model according to the above steps to obtain the vector , the index value corresponding to the largest element is used to determine the clearest bright field image and dark field image. Figure 5a-5b As shown, there are 30 bright field images and 30 dark field images. The images change from blurry to clear and then blurry again as the object distance changes. The model can predict the clearest image.
[0143] The embodiment of the present invention uses a test set to verify the accuracy of the algorithm. The test set has a total of 3500 groups, with an accuracy rate of 96.5% and an inference speed of about 100ms per group.
[0144] It should be noted that the above loss function is selected in the embodiment of the present invention. In other feasible embodiments, other loss functions are selected to meet the technical ideas of the present invention and fall within the protection scope of the present invention. In addition, in some embodiments, distance information may not be required. For example, if the clearest image is the 22nd one, a hard label, i.e. onehot encoding, is established. The encoding assigns 1 to the 22nd position of the vector and fills the others with 0, i.e., a hard label is constructed. The network model can also be trained by comparing the hard label with the network result.
[0145] Embodiment 2:
[0146] An embodiment of the present invention provides a system based on the above method, comprising: an image acquisition module, a feature extraction module, a fusion module and a focus module.
[0147] The image acquisition module is used to acquire the bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence;
[0148] A feature extraction module, used to extract image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map;
[0149] A fusion module is used to construct a dual-branch feature cross-fusion network by using several serially connected CIM modules, and then connecting the global average pooling and the 1*1 convolution layer. The output of the previous CIM module is used as the input of the next CIM module. The CIM module is a 2-input 2-output spatial and channel attention cross-fusion module.
[0150] Focus module, used to input the bright field feature map and dark field feature map into the double-branch feature cross-fusion network to obtain the network output vector corresponding to the bright field and dark field , , and use the network output vector , Determine the clearest brightfield and darkfield images to achieve focus.
[0151] In some embodiments, the model is pre-trained and then directly imported into the system; or the system also includes a training module for using an image acquisition module to acquire bright field and dark field images of the substrate glass at different object distances taken by a camera to construct training samples, and using the clearest bright field images and dark field images to produce true value labels, which are then compared with the focusing results of the training samples to complete network training.
[0152] Among them, please refer to the above method content for the specific implementation process of each module, the present invention will not be repeated here, and the division of the above functional modules is only for example illustration. In some embodiments, some functional modules can be merged, and some functional modules can be split. Each functional module can be implemented in software or hardware or a combination of software and hardware. Among them, the hardware and software equipment includes but is not limited to general-purpose computer equipment, programmable gate arrays, digital signal processors, microprocessors and their corresponding programming or burning software.
[0153] Embodiment 3:
[0154] This embodiment provides an electronic terminal, including: one or more processors; and a memory storing one or more computer programs; wherein the processor calls the computer program to implement: the steps of the above-mentioned substrate glass defect image automatic focusing method based on bright and dark field fusion.
[0155] As with Example 1, follow steps 1 to 5.
[0156] For the specific implementation process of each step, please refer to the description of the above method.
[0157] It should be understood that in the embodiments of the present invention, the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The memory may include a read-only memory and a random access memory, and provide instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the type of device.
[0158] Embodiment 4:
[0159] This embodiment provides a computer-readable storage medium storing a computer program, wherein the computer program is called by a processor to implement:
[0160] The above-mentioned steps of the substrate glass defect image automatic focusing method based on bright and dark field fusion.
[0161] As with Example 1, follow steps 1 to 5.
[0162] For the specific implementation process of each step, please refer to the description of the above method.
[0163] The readable storage medium is a computer-readable storage medium, which may be an internal storage unit of the software and hardware device described in any of the aforementioned embodiments, such as a hard disk or memory of a controller. The readable storage medium may also be an external storage device of the controller, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the controller. Furthermore, the readable storage medium may also include both an internal storage unit of the controller and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by the controller. The readable storage medium may also be used to temporarily store data that has been output or is to be output.
[0164] Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned readable storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0165] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is a device for realizing the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram with reference to the instructions executed by the processor according to the method, device (system) and computer program product of the embodiment of the present application. These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which realizes the function specified in one flow or multiple flows of the flow chart and / or one or multiple blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.
[0166] It should be emphasized that the examples described in the present invention are illustrative rather than restrictive, and therefore the present invention is not limited to the examples described in the specific embodiments. Any other embodiments derived by those skilled in the art based on the technical solution of the present invention that do not depart from the purpose and scope of the present invention, whether modified or replaced, also fall within the scope of protection of the present invention.
Claims
1. A method for automatically focusing substrate glass defect images based on bright and dark field fusion, characterized in that: The following steps are involved: Acquire bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence; Extracting image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map; A dual-branch feature cross-fusion network is constructed by using several serially connected CIM modules, and then connecting global average pooling and 1*1 convolutional layers. The output of the previous CIM module is used as the input of the next CIM module. The CIM module is a 2-input 2-output spatial and channel attention cross-fusion module. The bright field feature map and the dark field feature map are input into the dual-branch feature cross-fusion network to obtain the network output vectors corresponding to the bright field and the dark field. , , and use the network output vector , Determine the clearest bright field image and dark field image to achieve focus; The training samples are constructed by taking bright-field and dark-field images of the substrate glass at different object distances using a camera, and the clearest bright-field and dark-field images are used to make true value labels, which are then compared with the focusing results of the training samples to complete the network training. The processing process of each CIM module is: First, the convolutional channel attention mechanism CAM is used to obtain the channel self-attention vector and channel cross-attention vector corresponding to the input bright field and dark field feature maps respectively; Then, the channel self-attention vector corresponding to the bright field is multiplied by the bright field feature map channel by channel to obtain , the channel self-attention vector corresponding to the dark field is multiplied by the dark field feature map channel by channel to obtain , the channel-wise attention vector corresponding to the dark field is multiplied by the bright field feature map to obtain , the channel-wise multiplication of the attention vector corresponding to the bright field and the dark field feature map is obtained ; Next, for , , , Both use the convolutional spatial attention mechanism SAM to obtain the spatial self-attention matrix and and the spatial cross-attention matrix , ; Finally, , , , One-to-one correspondence with , , , Multiply element by element to get , , , ,and then and Add together to get , and Add together to get ,vector , It is the output of CIM module.
2. The method according to claim 1, characterized in that: The convolutional channel attention mechanism CAM is composed of a maximum pooling layer along the spatial dimension, an average pooling layer along the spatial dimension, a multi-layer perceptron MLP, and a Sigmoid function. The channel self-attention vector and the channel cross-attention vector obtained in the bright field are equal; the channel self-attention vector and the channel cross-attention vector obtained in the dark field are equal; The convolutional spatial attention mechanism SAM is composed of a maximum pooling layer along the channel dimension, an average pooling layer along the channel dimension, a convolutional layer, and a Sigmoid function.
3. The method according to claim 1, characterized in that: After obtaining a dark field image of the substrate glass captured by a camera to obtain a dark field image sequence, the method further includes performing gradient guidance on the dark field image sequence, and then inputting the gradient-guided dark field image sequence into a feature map extraction network to obtain a dark field feature map; The gradient guidance operation is as follows: First, for the dark field image sequence, the Laplacian operator is used to calculate the second-order gradient channel by channel, and the gradient below the gradient threshold is set to 0, and finally the gradient map sequence is obtained; Next, the variance of each gradient map is calculated in turn to obtain the variance vector; Then, a two-layer fully connected coding network is designed, and the variance vector is encoded. The encoding formula is as follows: ， ， In the formula, and is the first layer network training parameter, and is the training parameter of the second layer network, is the first layer output, is the output vector of the encoding network, V is the variance vector; Finally, the output vector The elements in are multiplied by the dark field image sequence to obtain the gradient-guided dark field image sequence.
4. The method according to claim 1, characterized in that: After acquiring the bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence, and before extracting the image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map, the method further includes: after acquiring the bright field image sequence and the dark field image sequence taken by the camera, using Gamma transformation to enhance the image of the bright field image sequence, and using a median filtering algorithm to sequentially process the images of the dark field image sequence.
5. The method according to claim 1, characterized in that: The camera uses a cross imaging method to shoot bright field and dark field images of the substrate glass at different object distances to construct training samples. The even number of times the object distance is adjusted is selected to shoot bright field images, and the odd number of times the object distance is adjusted is selected to shoot dark field images; or the odd number of times the object distance is adjusted is selected to shoot bright field images, and the even number of times the object distance is adjusted is selected to shoot dark field images; Then, the clearest bright field image and dark field image found in the training sample are taken as adjacent images, and the serial numbers of the clearest bright field image and dark field image are recorded. and , then use the serial number and Determine the sequence number of each bright field image and The distance, the serial number J of each dark field image and Distance: ， ， In the formula, is the bright field image serial number I and The distance is the dark field image sequence number J and The distance between the two distribution types is then converted into the distance vector , Converted into soft labels for training network model parameters, , is a distance vector formed by a bright field image sequence and a dark field image sequence; If the Laplace distribution is selected, the normalized exponential function and the distance vector , Constructed soft label and It is expressed as: ， ， In the formula, represents the normalized exponential function, is the true soft label of the bright field, is the true soft label of the dark field.
6. The method according to claim 5, characterized in that: When training the network, KL divergence is used as the training loss, ADAM is the optimizer, and Function processing and Get the vectors corresponding to the bright field and dark field , , so that the non-negative and one constraints are satisfied; the vector , The index corresponding to the maximum element value in corresponds to the bright field image and dark field image, which are the clearest bright field image and dark field image; The overall loss Loss is expressed as: ， ， ， In the formula, express , express , N is the number of vector elements, i is the vector index, , is the weight of bright field classification loss and dark field classification loss.
7. A system based on the method according to any one of claims 1 to 6, characterized in that: include: An image acquisition module is used to acquire the bright field and dark field images of the substrate glass taken by the camera to obtain a bright field image sequence and a dark field image sequence; A feature extraction module, used to extract image features of the bright field image sequence and the dark field image sequence to obtain a bright field feature map and a dark field feature map; A fusion module is used to construct a dual-branch feature cross-fusion network by using several serially connected CIM modules, and then connecting the global average pooling and the 1*1 convolution layer. The output of the previous CIM module is used as the input of the next CIM module. The CIM module is a 2-input 2-output spatial and channel attention cross-fusion module. Focus module, used to input the bright field feature map and dark field feature map into the double-branch feature cross-fusion network to obtain the network output vector corresponding to the bright field and dark field , , and use the network output vector , Determine the clearest bright field image and dark field image to achieve focus; Among them, the camera is used to take bright field and dark field images of the substrate glass at different object distances to construct training samples, and the clearest bright field images and dark field images are used to make true value labels, which are then compared with the focusing results of the training samples to complete network training.
8. An electronic terminal, characterized in that: include: one or more processors; a memory storing one or more computer programs; The processor calls the computer program to implement: The steps of the method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that: A computer program is stored, which is called by a processor to implement: The steps of the method according to any one of claims 1 to 6.