A Defect Detection Method for Mobile Phone Cameras Based on a Bilateral Feature Fusion Network

Abstract

The present invention discloses a method for defect detection of mobile phone cameras based on a bilateral feature fusion network. The method includes: obtaining mobile phone camera images and annotating the defect areas in each image to construct a data set; constructing a bilateral feature fusion network model, where the bilateral feature fusion network uses a lightweight network as the backbone network, and introduces a self-attention module and a bilateral feature fusion module. The self-attention module can emphasize the defect areas and improve the detection accuracy of minor defects, while the bilateral feature fusion module can enable two different levels of features to guide each other, thereby obtaining richer information; training the network model using the data set; applying the trained network to segment the defect areas of the mobile phone camera images. By introducing the self-attention module and the bilateral feature fusion module, the present invention can reduce the missed detection cases caused by minor defects and the false detection cases caused by complex image backgrounds, and improve the defect detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to a method for detecting defects in a mobile phone camera based on a bilateral feature fusion network. Background Art

[0002] Nowadays, mobile phones are very popular, and the camera effect of mobile phones, as an important performance index of modern smart phones, makes the mobile phone camera one of the important components of mobile phones. During the production and manufacturing of mobile phone cameras, the occurrence of defects is inevitable, such as scratches, dirt, hair and foreign objects, etc. These defects will affect the final camera effect. Moreover, the mobile phone camera is small in size, the glass area is colorless and transparent, and the area of some minor defects is small and the contrast with the background is low, making it quite difficult in quality inspection.

[0003] Traditional defect detection relies on technicians to identify defects manually. Such a method is inefficient, and it is also difficult to ensure the defect recognition effect of some materials. With the rise of machine vision, people have begun to apply machine vision technology to the field of defect detection. Defect detection methods based on machine vision can be divided into traditional machine vision detection methods and detection methods based on deep learning. Traditional detection methods manually extract effective feature information from images and use classifiers to classify and identify defects. Such detection methods rely on effective feature selection and design. However, the existence of more image noise, complex backgrounds of images, defects with low contrast, and weak texture defects will greatly reduce the detection effect of such methods. At the same time, the versatility of such methods is poor. One method can often only be used for defect detection of a certain characteristic material, resulting in a long development cycle of the solution when facing different product detections.

[0004] Compared with traditional machine vision methods, deep learning technology can automatically extract effective features by constructing a neural network, without the need for complex manual feature design, and can be applied to different scenarios. However, the existence of low-contrast images and images with complex backgrounds is still a great challenge to detection methods based on deep learning. During the production of mobile phone cameras, the existence of dust and minor scratches and the change of light intensity during image acquisition result in low contrast between defects and the background in the image, causing missed detection of defects. Moreover, there are some features in the black ink area coated with ink on the surrounding glass screen that are similar to the defect features of the central circular white light-transmitting glass area, which is likely to cause misdetection. Summary of the Invention

[0005] In view of the above problems, the object of the present invention is to provide a method for detecting defects in mobile phone cameras based on a bilateral feature fusion network. This network model can better obtain complementary information of different-level features and focus on the defect area, improving the detection accuracy of defects. At the same time, the network model has a faster detection speed and can better meet the actual industrial needs. A method for detecting defects in mobile phone cameras based on a bilateral feature fusion network specifically includes the following steps:

[0006] Step S1, collect images of mobile phone cameras, label the defect areas in the images, construct a mobile phone camera dataset, and divide it into a training set and a test set according to a certain proportion;

[0007] Step S2, establish a bilateral feature fusion segmentation network, which includes a backbone network, a self-attention module, and a bilateral feature fusion module.

[0008] Step S21, the backbone network uses a lightweight Resnet18 network. The features extracted from the third stage of Resnet18 are used as shallow features, and the features extracted from the last two stages are used as deep features. The self-attention module and the dual-branch feature fusion module are integrated into the basic network model to construct a new network model.

[0009] Step S22, the self-attention module takes the deep features as input. After the feature input, it will successively pass through a convolutional layer with a kernel size of 1×1, a multi-head self-attention layer, and a convolutional layer with a kernel size of 1×1. After each convolutional layer, Layer normalization and rectified linear unit (ReLU) are connected, and a residual calculation is performed at the end of the module as the final output. Self-attention is different from the convolutional layer. It is not restricted by the receptive field and is a better method for obtaining global and long-range semantic information, which can deepen the model's understanding of the entire feature map. At the same time, the self-attention mechanism can obtain the correlation between all feature points in the feature map, calculate the weighted information of one feature point and the other feature points, and while focusing on the key areas of segmentation, suppress the background information.

[0010] Step S23, the bilateral feature fusion module simultaneously uses the shallow feature information extracted by Resnet18 and the deep feature information output by the self-attention module for mutual guidance, better capturing the important complementary information of different-level features. After splicing different-level features, the weight vector of the features is calculated, and the features are re-weighted using the weight vector, and after a residual calculation, it is used as the final output of the module.

[0011] Step S3, use the image data of the training set to train the constructed bilateral feature fusion network, and obtain the trained network.

[0012] In step S31, the network uses a main loss function plus an auxiliary loss function to assist network convergence. The main loss function is used to supervise the output of the entire network, while the two auxiliary loss functions are used to supervise the output of deep semantic features. The overall network loss function is shown as follows:

[0013] L=l p +αl i

[0014] Where l p is the main loss function of the entire network, l i is the auxiliary loss function, L is the joint loss function, and the parameter α is used to balance the weights of the main loss and the auxiliary loss. The auxiliary loss function is only used during training.

[0015] In step S32, the cross entropy loss function is used in both the main loss function and the auxiliary loss function of the bilateral feature fusion network training process, and the calculation method is as follows:

[0016] l n = -ylogp-(1-y)log(1-p)

[0017] Where y represents the label of the sample, the positive class is 1 and the negative class is 0; p represents the probability that the sample is positive; n = {i, p}.

[0018] Step S4: Use the trained network to segment the mobile phone camera defect image data, locate each defect position with pixel-level accuracy through the model, and evaluate the network model performance based on F1-score, Precision, and Recall;

[0019]

[0020] Among them, TP, TN, FP and FN represent the number of defective areas classified correctly, the number of non-defective areas classified correctly, the number of defective areas missed detection and the number of non-defective areas falsely detected, respectively.

[0021] This paper constructs a bilateral feature fusion network to detect mobile phone camera defects. The network's self-attention module better captures global semantic information and focuses the model on key segmented areas. The bilateral feature fusion module fuses features extracted from different levels of the backbone convolutional neural network to better capture complementary information from features at different levels, thereby improving detection accuracy. Compared with existing methods, this network demonstrates optimal detection results on a mobile phone camera defect dataset in real industrial applications. It also achieves faster detection speed, better meeting practical industrial needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the invention of this application, the following drawings are provided:

[0023] Figure 1 The overall structure diagram of the network in this invention

[0024] Figure 2 This is the structure diagram of the bilateral feature fusion module in the present invention

[0025] Figure 3 This is the structure diagram of the self-attention module and the structure diagram of the multi-head self-attention layer in the present invention

[0026] Figure 4 Schematic diagram of the process of the present invention DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments.

[0028] This application uses a dataset of real mobile phone camera defect images, with the defect areas annotated by professionals from relevant companies. The white, translucent areas in the mobile phone camera images represent glass, referred to as the glass area; the surrounding black areas are actually ink applied to the glass, referred to as the ink area. The defects to be detected are those present in the glass area, including scratches, foreign matter, white spots, dust, dirt, and chipped edges. The dataset contains a total of 12,018 images.

[0029] The specific process of mobile phone camera defect detection includes the following steps:

[0030] S1, the dataset is divided into training set and test set in a ratio of 8:2, the image size is set to 128×128, and the defect areas in the image are annotated at the pixel level.

[0031] S2, build a bilateral feature fusion segmentation network based on pytorch (such as Figure 1 As shown in Figure 2, a self-attention module and a bilateral feature fusion module are introduced. The method includes:

[0032] In S21, the training image is input into the network. First, a lightweight Resnet18 network is used as the backbone network to downsample the image. The features extracted from the third stage of Resnet18 are used as shallow features, and the features extracted from the last two stages are used as deep features.

[0033] S22, where the self-attention module (such as Figure 3As shown, it takes the deep features as input. After the feature input, it will successively pass through a convolutional layer with a convolutional kernel size of 1×1, a multi-head self-attention layer, and a convolutional layer with a convolutional kernel size of 1×1. After each convolutional layer, Layer normalization and rectified linear unit (ReLU) will be connected, and a residual calculation will be performed at the end as the final output of the module.

[0034] S23. Subsequently, the bilateral feature fusion module (as shown in 2) simultaneously uses the shallow feature information extracted by Resnet18 and the deep feature information output by the self-attention module to guide each other using the sigmoid function. After splicing the features at different levels, the weight vector of the features is calculated, and the features are re-weighted using the weight vector. After a residual calculation, it is used as the final output of the module.

[0035] S3. Set parameters to train the bilateral feature fusion network. The training iteration times of the network are set to 30 epochs, the initial learning rate is set to 5e-4, the batch size is set to 16, and RMSprop is selected as the optimizer. And the main loss function plus the auxiliary loss function is used to assist the network convergence. Among them, the main loss function is used to supervise the output of the entire network, and the two auxiliary loss functions are used to supervise the output of the deep semantic features. The overall loss function of the network is shown in the following formula:

[0036] L = l p +αl i

[0037] In the formula, l p is the main loss function of the overall network, l i is the auxiliary loss function, L is the combined loss function, and the parameter α is used to balance the weights of the main loss and the auxiliary loss. In this embodiment, α is set to 0.7, and the auxiliary loss function is only used during training. The cross-entropy loss function is adopted in both the main loss function and the auxiliary loss function during the training process, and its calculation method is as follows: l n =-ylogp-(1 - y)log(1 - p)

[0038] In the formula, y represents the label of the sample, the positive class is 1, and the negative class is 0; p represents the probability that the sample is the positive class; n = {i, p}.

[0039] S4. Use the trained network to segment the defect image data of the mobile phone camera, locate each defect position through the model, and evaluate the performance of the network model according to F1-score, Precision, and Recall;

[0040]

[0041] Among them, TP, TN, FP, and FN represent the number of correctly classified defective areas, the number of correctly classified defect-free areas, the number of missed detections of defective areas, and the number of false detections of defect-free areas, respectively.

[0042] To further verify the detection performance of the network proposed in this paper, we selected classical segmentation networks such as FCN and U-Net for comparison of detection performance. Table 1 shows the results of the comparative experiments. Compared with other classical segmentation network models, the network used in this paper has significantly better detection effects on mobile phone camera defects due to the addition of the self-attention module and the bilateral feature fusion module.

[0043]

[0044] Table 1

[0045] The present invention provides a new segmentation network, the bilateral feature fusion network, for the task of mobile phone camera defect detection, and has the best performance on the mobile phone camera dataset in actual industrial production. In summary, the present invention has prominent exemplary characteristics and significant progress, and has good practical value and promotion value in the field of mobile phone camera defect detection.

[0046] The above embodiments are only the preferred embodiments of the present invention, and do not limit the protection scope of the present invention. Any changes made by using the design principle of the present invention and non-creative labor on this basis shall fall within the protection scope of the present invention.

Claims

1. A method for defect detection of mobile phone cameras based on a bilateral feature fusion network, characterized in that Design a feature fusion network to optimize the defect detection effect of mobile phone cameras. The method is as follows: S1. Construct a bilateral feature fusion network using a lightweight backbone network, a self-attention module, and a bilateral feature fusion module. The bilateral feature fusion network uses a lightweight Resnet18 network as the backbone network, takes the features extracted from the third stage of Resnet18 as shallow features, and the features extracted from the last two stages as deep features. The self-attention module takes deep features as input. After the feature input, it successively passes through a convolutional layer with a kernel size of 1×1, a multi-head self-attention layer, and a convolutional layer with a kernel size of 1×1. Each convolutional layer is followed by Layer normalization and rectified linear unit (ReLU), and a residual calculation is performed at the end of the module as the final output. The bilateral feature fusion module uses the shallow feature information extracted by Resnet18 and the deep feature information output by the self-attention module to guide each other using the sigmoid function, calculates the weight vector of the features after feature concatenation, re-weights the features using the weight vector, and performs a residual calculation once as the final output of the module. S2. Use image data to train the constructed bilateral feature fusion network and obtain the trained network. S3. Use the trained network to segment the defect image data of mobile phone cameras and evaluate the defect detection effect of the bilateral feature fusion network.

2. The method for defect detection of a mobile phone camera based on a bilateral feature fusion network according to claim 1, characterized in that, Step S2 also includes step S21: The network uses the main loss function plus the auxiliary loss function to train the network. The main loss function is used to supervise the output of the entire network, and the two auxiliary loss functions are used to supervise the output of the deep semantic features. The overall loss function of the network is shown in the following formula: L＝l p +αl i where l p is the main loss function of the overall network, l i is the auxiliary loss function, L is the combined loss function, and the parameter α is used to balance the weights of the main loss and the auxiliary loss. The auxiliary loss function is only used during training.

3. The method for defect detection of a mobile phone camera based on a bilateral feature fusion network according to claim 2, characterized in that, Step S2 also includes step S22: During the training process of the bilateral feature fusion network, both the main loss function and the auxiliary loss function use the cross-entropy loss function, and its calculation method is as follows: l n = -y log p - (1 - y) log(1 - p) In the formula, y represents the true label of the sample, which is 1 when the sample is a positive class and 0 when the sample is a negative class; p is the probability that the sample is a positive class.

Images