A method for detecting screen printing defects on mobile phone camera glass in a small sample environment

Abstract

This invention discloses a method for detecting silk-screen defects on mobile phone camera glass in a few-shot environment. A self-similarity-based background suppression network is designed. First, a masking operation is performed on the input support image to eliminate background information. A pre-trained backbone network is used to extract intermediate features of the query and support images. A multi-similarity generation module processes the intermediate features to generate a hyper-correlation tensor containing mutual similarity information and a hyper-autocorrelation tensor containing target class self-similarity information. The hyper-autocorrelation tensor is subjected to correlation masking, and after using the hyper-correlation tensor to generate a background activation map for the query image, the self-similarity of the background class in the query image is activated. The two resulting 4D hyper-correlation tensors are encoded, decoded, and fused to obtain a predicted mask for the query image. This method, pre-trained on non-industrial images and performing knowledge transfer in a few-shot environment, accurately detects silk-screen defects of varying shapes on mobile phone camera glass.

Description

Technical Field

[0001] The present invention relates to the fields of machine vision, silk screen defect detection technology, deep learning and small sample segmentation technology, and specifically to a method for detecting silk screen defects on mobile phone camera glass in a small sample environment. Background Art

[0002] Silkscreen defect detection is one of the main challenges faced in detecting defects in mobile phone camera back cover glass. During the silkscreen printing process, defects can easily appear in the printed area due to misalignment, excessive or insufficient distance between the screen and the glass cover, unstable ink pressure control, and excessive printing speeds. Common silkscreen defects include scratches, stains, ink overflow, ink deficiency, jagged edges, and white edges. Identifying these defects often relies on manual judgment, and existing defect detection methods are primarily based on template comparison and deep learning.

[0003] Template-matching methods typically require steps such as image registration, differencing, and morphological processing. While their advantage lies in their low sample requirements, they also require a sophisticated image registration algorithm. If registration errors or precision drift occur, subsequent image differencing and morphological processing will result in numerous false detections. Deep learning-based defect detection methods can effectively avoid false matching issues, but they require the collection of a large number of defect samples and the training of corresponding models. However, in actual production, defect samples are scarce, and obtaining sufficient training samples is difficult, resulting in suboptimal detection results.

[0004] In order to cooperate with the automatic detection and rejection of defective products in the glass cover screen printing production line, and to solve the problems of high cost of collecting defect label samples of mobile phone camera back cover glass and scarcity of defect samples in the actual production environment, it is difficult to obtain sufficient training samples, resulting in poor detection effect. At present, it is urgent to propose an accurate and reliable mobile phone camera glass screen printing defect detection method in a small sample environment. Summary of the Invention

[0005] The purpose of the present invention is to solve the above-mentioned defects in the prior art and provide a method for detecting screen printing defects on mobile phone camera glass in a small sample environment.

[0006] The purpose of the present invention can be achieved by taking the following technical solutions:

[0007] A method for detecting screen printing defects on mobile phone camera glass in a small sample environment, the defect detection method comprising the following steps:

[0008] S1. Use a specified type of public dataset to construct similar support query sample pairs, wherein the support query sample pairs include a support set image and a query set image. The support set image consists of a support image and a support mask corresponding to the support image. The query set image consists of a query image and a query mask corresponding to the query image. The support set image, the support mask corresponding to the support image, and the query set image in the support query sample pair are input into a background suppression network. The query mask corresponding to the query image is used to supervise and fine-tune the background suppression network.

[0009] S2. Build a background suppression network, which includes a feature extraction network, a multi-similar generation module, a background self-similar activation module, a 4D convolutional coding module, and a multi-similar fusion module. The feature extraction network takes the support query sample pair as input, performs an input mask operation on the input support image, eliminates background information, and extracts the intermediate layer features of the support image and the query image as support features and query features respectively; the multi-similar generation module calculates the support features of the support image and the query features of the query image through cosine similarity to obtain 4D super cross-correlation tension respectively. The background self-similar activation module obtains the background activation map by subjecting the 4D super cross-correlation tensor to correlation masking, averaging and dimensionality reduction, and normalization, and then performs masking operations on the background activation map and the 4D super autocorrelation tensor to obtain the 4D super autocorrelation tensor after background activation. The 4D convolutional coding module encodes the 4D super cross-correlation tensor and the 4D super autocorrelation tensor after background activation to obtain the cross-correlation tensor and autocorrelation tensor respectively. The multi-similar fusion module fuses the cross-correlation tensor and the autocorrelation tensor through decoding to obtain the predicted segmentation result of the query image.

[0010] S3. Use a public non-industrial dataset to train a background suppression network and fine-tune it using a mobile phone camera glass silk screen defect dataset.

[0011] S4. Input the camera image of the mobile phone to be tested into the trained and fine-tuned background suppression network to detect the silk-screen defects on the glass of the mobile phone camera.

[0012] Furthermore, the process of constructing the supporting query sample pairs in step S1 is as follows:

[0013] The specified type of public dataset (the specified type of public dataset uses PASCAL-5 i) are evenly divided into four groups by category, of which three groups serve as training sets and the other group serves as a test set. Both the training sample pairs in the training set and the test sample pairs in the test set contain support set images and query set images. The support masks corresponding to the support images serve as prior knowledge. During training, the query masks corresponding to the query images serve as supervisory information to train the parameters of the background suppression network. During testing, the query masks corresponding to the query images serve as a standard to evaluate the performance of the background suppression network. To alleviate overfitting caused by insufficient training data, meta-learning methods are currently commonly used to construct support query sample pairs for training models, also known as scenario training.

[0014] Furthermore, the working process of the background suppression network is as follows:

[0015] S2.1. Take a set {(I s ,M s ),I q}, where I s ∈R H×W×3 and M s ∈{0,1} are the support image and support binary segmentation mask respectively, I q ∈R H×W×3 is the query image, H and W represent the width and height of the image respectively, 3 represents the number of channels of the image, R represents the real number domain, and supports binary segmentation mask M s Expand to make M s The size is the same as the supporting image, and then the supporting image I s Perform mask operation to obtain the masked support image I′ s ∈R H×W×3 :

[0016]

[0017] Among them, ⊙ represents the Hadamard product, Is a M s The function of adjusting the size from H×W to H×W×3;

[0018] Traditional methods perform feature masking during the feature extraction stage to reduce the interference of background information, but the target class pixels have already integrated a large amount of background information. Compared with the prototype network, the semantic correspondence method pays more attention to the correspondence between points. Therefore, it directly masks the input support image to better eliminate background information interference.

[0019] S2.2, the masked support image I s ′ and query image I q , input into the feature extraction network to extract the intermediate layer features of the two, respectively as support features F s and query feature F q :

[0020] (F s ,F q )=ResNet(I′ s ,I q ), F s ,F q ∈R H×W×3

[0021] Among them, ResNet() represents the feature extraction network, F s 、F q Represent support features and query features respectively;

[0022] S2.3, will support feature F s And query feature F q Expand into and Then, it is input into the multi-similar generation module, and the similarity between each pixel in the tensor is calculated using cosine similarity to generate a 4D hyper-correlation tensor. and the 4D hyperautocorrelation tensor

[0023]

[0024]

[0025] Among them, p s and p q For and The spatial position of the feature, the operator “·” represents the 3D vector inner product operation at all positions of the feature, ‖ ‖ represents the l2 norm, and τ() is a C l and From R H×W →R H×W×3 function;

[0026] Compared with traditional prototype extraction, our method is more direct and effective because it can capture the self-similarity within the query image and better understand the spatial relationship of classes in the query image, further improving the performance and accuracy of the model.

[0027] S2.4. Use support mask M s Perform a second mask on the 4D hypercorrelation tensor Perform mask operation to obtain The background information is reduced again through the secondary mask, retaining the target class information. The formula is as follows:

[0028] Among them, the role of φ() is to adjust the mask M s Size is Hl ×W l , to fit the 4D cross-correlation tensor The size of is the 4D hyper-correlation tensor abbreviation of;

[0029] S2.5, the masked 4D hyper-correlation tensor Input background self-similar activation module to generate background activation map M b , then activate the 4D hyperautocorrelation tensor The high correlation area of ​​the upper background is obtained to obtain the activated 4D super autocorrelation volume

[0030]

[0031]

[0032] Among them, ξ() is the 4D hypercorrelation tensor after masking The function that averages the last two dimensions of , normalize() is the normalized function, is the 4D hyperautocorrelation tensor abbreviation of ; because the 4D hyper-correlation tensor after masking The value of the tensor belonging to the target class will be higher, so we use the above background self-similar activation operation to make the value of the tensor belonging to the background class higher, thereby achieving the purpose of activating the background class.

[0033] S2.6, the masked 4D hyper-correlation tensor and the 4D hyperautocorrelation tensor The cross-correlation tensor Z is obtained through the 4D convolutional coding module sq and the autocorrelation tensor Z qq For detailed implementation details of the 4D convolutional encoding module, please refer to the paper J.Min, D.Kang, and M.Cho, “Hypercorrelation squeeze for fewshot segmenation,” in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 6921–6932.Z. This is the first work to apply the hypercorrelation method used in image matching to few-shot segmentation. By using 4D high-dimensional convolutions to establish reliable correspondences between query and support images, a lightweight center-pivot 4D convolution is proposed, significantly reducing the number of parameters and computation.

[0034] Then, the cross-correlation tensor Z is transformed into sqand the autocorrelation tensor Z qq Fuse and predict dual channel mapping The multi-similar fusion module consists of a background suppression branch, a target class prediction branch, and three mask prediction blocks. By learning the fusion information of multiple branches, it can distinguish between background and target class pixels through training, achieving more accurate prediction.

[0035] During the training process, the three mask prediction blocks in the multi-similar fusion module generate three prediction masks respectively. By comparing with the real mask M q The corresponding three binary cross entropy losses Loss1, Loss2, and Loss3 are calculated together to optimize the network;

[0036] The binary cross entropy loss formula is as follows:

[0037]

[0038] in, and M q (x, y) represents the predicted mask and the given true mask of each mask prediction block, respectively, and (x, y) represents all pixel positions on the mask;

[0039] Finally, the overall loss Loss during the training phase final Assessment is done through:

[0040] Loss final =λ1Loss1+λ2Loss2+λ3Loss3

[0041] Among them, Loss1, Loss2, and Loss3 are the target class prediction branch, background suppression branch, and the binary cross entropy loss after fusing the above two branches, respectively. λ1, λ2, and λ3 are the weighted coefficients of Loss1, Loss2, and Loss3, respectively.

[0042] S2.7. Transfer the knowledge learned from the known classes in the training set to the unknown classes in the test set, output the predicted segmentation mask of the query image, and calculate the intersection over union (IoU) with the true segmentation mask to evaluate the detection performance. The IoU calculation formula is as follows:

[0043] IOU=TP / (TP+FP+FN)

[0044] Among them, TP represents the number of target pixels predicted correctly, FP represents the number of non-target pixels predicted as target pixels, and FN represents the number of target pixels predicted as non-target pixels.

[0045] Furthermore, the intersection-over-union (IoU) includes the mean intersection-over-union (mIoU) and the foreground-background intersection-over-union (FB-IoU). The mean intersection-over-union (mIoU) is the average of the intersection-over-union (IoU) values ​​of all target categories, while the foreground-background intersection-over-union (FB-IoU) is calculated by ignoring the category, treating all different target categories as the same foreground category, and areas other than the target category as the background category, and calculating the average of the foreground and background IoU values. In practical applications, mIoU is used as the primary evaluation metric because it has a stronger performance evaluation capability for unbalanced category distributions and small target segmentation. FB-IoU can provide a convenient comparison of results, making it easier to compare the results with previous methods.

[0046] Furthermore, in step S3, the background suppression network is trained using a public non-industrial dataset, and the fine-tuning process is performed using a mobile phone camera glass silk screen defect dataset as follows:

[0047] S3.1. Use meta-learning method to train the background suppression network: the public non-industrial dataset PASCAL-5, which contains 20 categories of natural images, is used to train the background suppression network. i , divided into a training set D containing the base class train and a test set D containing new classes test The training set is divided into 3 groups according to categories, and the test set is 1 group. The target classes of each training set and test set are different. The training set and test set are composed of multiple scenarios, each scenario contains a support set S and a query set Q. In the training phase, the model is inferred how to correctly segment the base class objects in the training dataset by learning labeled images and labels. In the testing phase, the model is usually required to infer the location of new class objects in the test dataset from a given small number of labeled images, and the model performance is evaluated by comparing the predicted mask output by the model with the densely labeled mask in the test dataset.

[0048] S3.2. Using the same setup, the background suppression network was fine-tuned using a dataset of silk-screen defects on mobile phone camera glass. During training, the model learned how to correctly segment and identify silk-screen defects in the training dataset by studying images and labels of silk-screen defects on mobile phone camera glass. During testing, the model was required to infer the locations of silk-screen defects in the test dataset from a small number of labeled images. The model performance was evaluated by comparing the predicted masks output by the model with the densely labeled masks in the test dataset.

[0049] Furthermore, the process of using the background suppression network to detect defects in the mobile phone camera glass silk-screen defect dataset in step S4 is as follows:

[0050] Select one of the images from the test set of the mobile phone camera glass silk screen defect dataset as the support image I s, the mask corresponding to the support set image is manually annotated and is denoted as M s , the remaining images are used as query set images. Select a query image I q , will I s 、M s and I q The input is fed into the background suppression network, which outputs the defect segmentation result of the query image. By using the self-similar information of the query image to suppress the background class, the network's ability to distinguish between the target class and the background class is greatly enhanced. Furthermore, the background information in the query image is fully utilized, significantly improving the detection accuracy.

[0051] The present invention has the following advantages and effects compared to the prior art:

[0052] (1) The present invention proposes a background suppression branch, which includes a multi-similar generation module and a multi-similar fusion module. By utilizing self-similarity to suppress the background class, the segmentation of the target class is made more accurate.

[0053] (2) The present invention proposes a multi-similarity generation module that can simultaneously generate 4D super cross-correlation tensors and 4D super autocorrelation tensors, which can better extract useful information from target classes and background classes.

[0054] (3) The present invention proposes a background self-similarity activation module, which uses the 4D super-correlation tensor generated by the multi-similarity generation module and the support mask to generate a background activation map, activates the high-correlation areas of the background in the 4D super-autocorrelation tensor, and helps the network better suppress the background class.

[0055] (4) The present invention proposes a multi-similar fusion module, which can better integrate the tensor information of the background suppression branch and the target prediction branch, guide the training through a customized loss function, and perform fusion prediction.

[0056] (5) The present invention achieves state-of-the-art performance in both the 1-bullet scenario and the 5-bullet scenario of the PASCAL-5i and COCO-20i datasets. The method of building a dual-branch network using mutual similarity and self-similarity has great potential to become a new baseline. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0058] Figure 1 This is a flow chart of a method for detecting screen printing defects on mobile phone camera glass in a small sample environment disclosed by the present invention;

[0059] Figure 2is a structural diagram of a multi-similar generation module in an embodiment of the present invention;

[0060] Figure 3 is a structural diagram of a background self-similar activation module in an embodiment of the present invention;

[0061] Figure 4 is a structural diagram of a multi-similar fusion module in an embodiment of the present invention;

[0062] Figure 5 This is a defect sample image from the mobile phone camera glass silk-screen defect dataset in an embodiment of the present invention;

[0063] Figure 6 3 is a schematic diagram of the results of the segmentation of the silk-screen defects on the mobile phone camera glass in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0065] Example 1

[0066] This embodiment discloses a method for detecting screen printing defects on mobile phone camera glass in a small sample environment. Figure 1 This is a flow chart of a method for detecting screen printing defects on mobile phone camera glass in a small sample environment disclosed in this embodiment. The following is a specific embodiment to illustrate the method. The specific steps are as follows:

[0067] S1. Use the public dataset PASCAL-5i to construct similar support query sample pairs. The support query sample pairs include a support set image and a query set image. The support set image consists of a support image and a support mask corresponding to the support image. The query set image consists of a query image and a query mask corresponding to the query image. The support set image, the support mask corresponding to the support image, and the query set image in the support query sample pair are input into the background class suppression network. The query mask corresponding to the query image is used to supervise and fine-tune the background class suppression network.

[0068] In step S1 of this embodiment, the public non-industrial dataset PASCAL-5, which contains 20 categories of natural images, is used to collect i’ Divided into a training set D containing 15 base classes train and a test set D containing 5 new classes testThe training set is divided into 3 groups according to the category, and the test set is 1 group, and their target classes are mutually exclusive. The model trained on the base class should be able to generalize to new classes with minimal labeled data. The training set and test set consist of multiple scenarios, each of which contains a support set S and a query set Q. Each set consists of an image-label pair, where the label indicates the location of the target class object in the image. Support set S = {S1, S2,,…S k} consists of k samples, each sample Contains a supporting image I s and its corresponding binary mask label M s , while a query image I q and the corresponding binary mask label M q A query set Q is formed. In the PASCAL test set, there are 1000 supported query sample pairs, which further tests the ability of this method to handle complex scenarios.

[0069] S2. Build a background suppression network, which includes a feature extraction network, a multi-similar generation module, a background self-similar activation module, a 4D convolutional coding module, and a multi-similar fusion module. The feature extraction network takes the support query sample pair as input, performs an input mask operation on the input support image, eliminates background information, and extracts the intermediate layer features of the support image and the query image as support features and query features respectively; the multi-similar generation module calculates the support features of the support image and the query features of the query image through cosine similarity to obtain 4D super cross-correlation tension respectively. The background self-similar activation module obtains the background activation map by subjecting the 4D super cross-correlation tensor to correlation masking, averaging and dimensionality reduction, and normalization, and then performs masking operations on the background activation map and the 4D super autocorrelation tensor to obtain the 4D super autocorrelation tensor after background activation. The 4D convolutional coding module encodes the 4D super cross-correlation tensor and the 4D super autocorrelation tensor after background activation to obtain the cross-correlation tensor and autocorrelation tensor respectively. The multi-similar fusion module fuses the cross-correlation tensor and the autocorrelation tensor through decoding to obtain the predicted segmentation result of the query image.

[0070] In step S2 of this embodiment, a set {(I s ,M s ),I q}, where I s ∈R H×W×3 and M s ∈{0,1} are the support image and support binary segmentation mask respectively, I q ∈R H×W×3 is the query image, H and W represent the width and height of the image respectively, 3 represents the number of channels of the image, and R represents the real number domain. In order to mask the background information contained in the support image, we first perform as Expand it to make it the same size as the supporting image, and then s Perform mask operation to obtain the masked support image I s ′∈R H×W×3 :

[0071] Among them, ⊙ represents the Hadamard product, Is a M s The function of adjusting the size from H×W to H×W×3.

[0072] The masked support image I obtained above s ′ and query image I q , input into the feature extraction network to extract the intermediate layer features of the two, respectively as support features F s and query feature F q :

[0073] (F s ,F q )=ResNet(I′ s ,I q ), F s ,F q ∈R H×W×3

[0074] Among them, ResNet() represents the feature extraction network, F s ,F q Represent support features and query features respectively; the present invention uses the ResNet50 network pre-trained by the ImageNet dataset as the feature extraction network.

[0075] like Figure 2 As shown, the multi-similar generation module converts the support features F obtained above into s And query feature F q Expand into and Use cosine similarity to calculate the similarity between each pixel in the tensor and generate a 4D hypercorrelation tensor and the 4D hyperautocorrelation tensor

[0076]

[0077]

[0078] Among them, p s and p q For and The spatial position of the feature, the symbol “·” represents the 3D vector inner product operation at all positions of the feature, ‖ ‖ represents the l2 norm, and τ() is a C l and From R H×W →R H×W×3 function;

[0079] Generate 4D hyper-correlation tensor through multi-similarity generation module and the 4D hyperautocorrelation tensor Then, use the support mask M s Perform a second masking operation on each layer of the 4D hypercorrelation tensor to obtain The formula is as follows:

[0080] Among them, the role of φ() is to adjust the mask M s Size is H l ×W l , to fit the 4D hyper-correlation tensor The size of is the 4D hyper-correlation tensor abbreviation of ;

[0081] like Figure 3 As shown, the background self-similar activation module converts the above masked 4D super cross-correlation tensor Input background self-similarity activation module to generate background self-similarity activation map Then, using the 4D super autocorrelation volume generated in the multi-similarity module It contains the target class and background information of the query image, and activates the 4D super autocorrelation volume The high correlation area in the middle background is the activated 4D super autocorrelation volume

[0082]

[0083]

[0084] Among them, ξ() is the 4D hypercorrelation tensor after masking The function that takes the average of the last two dimensions of the function, normalize() is a normalized function that will scale all values ​​to the range [0,1]. is the 4D hyperautocorrelation tensor abbreviation of;

[0085] The masked 4D hyper-correlation tensor obtained above is 4D super autocorrelation tensor generated by the multi-similarity modules of and After being squeezed into the cross-correlation tensor Z by the 4D encoding module proposed in HSNetsq and the autocorrelation tensor Z qq For details on the implementation of the 4D convolutional encoding module, see J.Min, D.Kang, and M.Cho, “Hypercorrelation squeeze for fewshot segmenation,” in 2021 IEEE / CVF International Conference on Computer Vision (ICCV), 2021, pp. 6921–6932.

[0086] like Figure 4 As shown, the multi-similar fusion module converts the cross-correlation tensor Z sq and the autocorrelation tensor Z qq Combined to better obtain the information visualization correspondence between the support image target class and the query image target class, and predict the dual channel mapping The two channel values ​​represent the probabilities of foreground and background. The multi-similar fusion module consists of a background suppression branch, an object class prediction branch, and three mask prediction blocks.

[0087] During the training process, Figure 4 As shown, the three mask prediction blocks generate three prediction masks By comparing with the real mask M q The corresponding three binary cross entropy losses Loss1, Loss2, and Loss3 are calculated together to optimize the network; the binary cross entropy loss formula is as follows:

[0088]

[0089] in, and M q (x, y) represents the predicted mask and the given true mask of each mask prediction block, respectively, and (x, y) represents all pixel positions on the mask;

[0090] Finally, the overall loss during the training phase can be evaluated as follows:

[0091] Loss final =λ1Loss1+λ2Loss2+λ3Loss3

[0092] Among them, Loss1, Loss2, and Loss3 are the target class prediction branch, background suppression branch, and the binary cross entropy loss after fusing the above two branches, respectively. λ1, λ2, and λ3 are the weighted coefficients of Loss1, Loss2, and Loss3, respectively.

[0093] S3. Use a public non-industrial dataset to train a background suppression network and fine-tune it using a mobile phone camera glass silk screen defect dataset.

[0094] In step S3 of this embodiment, the background suppression network is trained using a public non-industrial dataset, and the fine-tuning process is performed using a mobile phone camera glass silk-screen defect dataset as follows:

[0095] A meta-learning method, also known as context training, is used to train the background class suppression network. The PASCAL-5 dataset, a public non-industrial dataset of natural images containing 20 classes, is used to train the background class suppression network. i , divided into a training set D containing 15 base classes train and a test set D containing 5 new classes test The training set is divided into 3 groups according to the category, and the test set is 1 group, and their target classes are different. The model trained on the base class should be able to generalize to new classes with very little labeled data. The training set and test set are composed of multiple scenarios, each of which contains a support set S and a query set Q. Each set consists of an image-label pair, and the label indicates the location of the target class object in the image. Support set S = {S1, S2,,…S k} consists of k samples, each sample Contains a supporting image I s and its corresponding binary mask label M s , while a query image I q and the corresponding binary mask label M q The model is trained on these labeled images and labels to form a query set Q. During training, the model infers how to correctly segment the base class objects in the training dataset. During testing, the model is typically asked to infer the locations of new class objects in the test dataset from a small number of labeled images. The model performance is evaluated by comparing the predicted masks output by the model with the densely labeled masks in the test dataset.

[0096] Using the same setup, the background suppression network was fine-tuned using a dataset of silk-screen defects on mobile phone camera glass. During training, the model learned how to correctly segment and identify silk-screen defects in the training dataset by learning images and labels of silk-screen defects on mobile phone camera glass. During testing, the model was required to infer the locations of silk-screen defects in the test dataset from a small number of labeled images. The model's performance was evaluated by comparing the predicted masks output by the model with the densely labeled masks in the test dataset.

[0097] S4. Input the camera image of the mobile phone to be tested into the trained and fine-tuned background suppression network to detect the silk-screen defects on the glass of the mobile phone camera.

[0098] In step S4 of this embodiment, the dataset consists of 20 defect samples of screen printing defects on mobile phone camera glass. The defects of the samples are mainly scratches and ink deficiency. 10 samples are randomly selected as support set images, and the remaining images are used as the query set. Each image contains at most one mobile phone camera glass.

[0099] Select one of the images from the test set of the mobile phone camera glass silk screen defect dataset, such as Figure 5 As shown, as supporting image I s , the mask corresponding to the support set image is manually annotated and is denoted as M s , the remaining images are used as query set images. Select a query image I q , will I s , M s and I q Input to the background suppression network and output the defect segmentation result of the query image;

[0100] In this example, two few-shot learning-based methods were compared with the proposed method. All methods used the same training and test sets, as well as the same support and query sets. The two methods used for comparison were PFENet and HSNet, and the proposed method was denoted as SSBSNet.

[0101] like Figure 6 As shown in the figure, the prediction results of SSBSNet for 10 mobile phone camera glass silk-screen sample defects are shown. The first row of images are support set images, the second column of images are annotations of support set defects, the third row of images are query set images, the fourth row of images are annotations of query set defects, and the fifth row of images are predicted defects in the query set. The measurement results of different methods are shown in Table 1 below:

[0102] Table 1. Results of different methods applied to the segmentation of screen printing defects on mobile phone camera glass

[0103]

[0104] From the experimental results in Table 1, it can be seen that the method SSBSNet proposed in this paper achieves the highest mIOU and the highest FB-IoU, has better segmentation performance than other methods, and can complete the segmentation of glass silk-screen defects in mobile phone cameras.

[0105] Example 2

[0106] In this embodiment, the adjustment parameter λ1 of the target class prediction branch loss, the adjustment parameter λ2 of the background suppression branch loss, and the adjustment parameter λ3 of the trunk loss after fusing the two branches are adjusted to identify the impact of different parameter values ​​on the model segmentation performance. All parameter adjustments use the ResNet50 network as the feature extractor. The segmentation results are shown in Table 2 below:

[0107] Table 2. Single-sample parameter adjustment semantic segmentation results based on the PASCAL dataset

[0108] <![CDATA[λ1]]> <![CDATA[λ2]]> <![CDATA[λ3]]> mIOU FB-IoU 0.0 0.0 1.0 64.45％ 75.36％ 0.5 0.5 1.0 65.13％ 76.31％ 0.5 1.0 1.0 65.04％ 76.06％ 1.0 1.0 1.0 65.53％ 77.28％

[0109] In summary, when the adjustment parameters λ1, λ2, and λ3 are all 1.0, the mIOU and FB-IOU are higher than those at other parameter values, indicating that the introduced loss function contributes significantly to the update and improvement of the model training parameters. The adjustment parameter λ1 can better iteratively optimize the target class prediction branch parameters, the adjustment parameter λ2 can better back-propagate the background suppression branch parameters, and the adjustment parameter λ3 can update the weights of all parameters.

[0110] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A method for detecting screen printing defects on mobile phone camera glass in a small sample environment, characterized in that: The defect detection method comprises the following steps: S1. Use a specified type of public dataset to construct similar support query sample pairs, wherein the support query sample pairs include a support set image and a query set image. The support set image consists of a support image and a support mask corresponding to the support image. The query set image consists of a query image and a query mask corresponding to the query image. The support set image, the support mask corresponding to the support image, and the query set image in the support query sample pair are input into a background suppression network. The query mask corresponding to the query image is used to supervise and fine-tune the background suppression network. S2. Build a background suppression network, which includes a feature extraction network, a multi-similar generation module, a background self-similar activation module, a 4D convolutional coding module, and a multi-similar fusion module. The feature extraction network takes the support query sample pair as input, performs an input mask operation on the input support image, eliminates background information, and extracts the intermediate layer features of the support image and the query image as support features and query features respectively; the multi-similar generation module calculates the support features of the support image and the query features of the query image through cosine similarity to obtain 4D super cross-correlation respectively. tensor and 4D super autocorrelation tensor; the background self-similar activation module obtains the background activation map by subjecting the 4D super cross-correlation tensor to correlation masking, averaging and dimensionality reduction, and normalization, and then performs masking operations on the background activation map and the 4D super autocorrelation tensor to obtain the 4D super autocorrelation tensor after background activation; the 4D convolutional coding module encodes the 4D super cross-correlation tensor and the 4D super autocorrelation tensor after background activation to obtain the cross-correlation tensor and autocorrelation tensor respectively; the multi-similar fusion module fuses the cross-correlation tensor and the autocorrelation tensor through decoding to obtain the predicted segmentation result of the query image; S3. Use a public non-industrial dataset to train a background suppression network and fine-tune it using a mobile phone camera glass silk screen defect dataset. S4. Input the camera image of the mobile phone to be tested into the trained and fine-tuned background suppression network to detect the silk-screen defects on the glass of the mobile phone camera.

2. The method for detecting screen printing defects on mobile phone camera glass in a small sample environment according to claim 1, characterized in that: The process of constructing the supporting query sample pairs in step S1 is as follows: The specified type of public dataset is divided into 4 groups by category, 3 of which are used as training sets and the other 1 is used as a test set. The training sample pairs of the training set and the test sample pairs of the test set both contain support set images and query set images. The support masks corresponding to the support images are used as prior knowledge. During the training process, the query masks corresponding to the query images are used as supervisory information to train the parameters of the background suppression network. During the testing process, the query masks corresponding to the query images are used as a standard to evaluate the performance of the background suppression network.

3. The method for detecting screen printing defects on mobile phone camera glass in a small sample environment according to claim 1, characterized in that: The working process of the background suppression network is as follows: S2.

1. Take a set {(I s ,M s ),I q }, where I s ∈R H×W×3 and M s ∈{0,1} are the support image and support binary segmentation mask respectively, I q ∈R H×W×3 is the query image, H and W represent the width and height of the image respectively, 3 represents the number of channels of the image, R represents the real number domain, and supports binary segmentation mask M s Expand to make M s The size is the same as the supporting image, and then the supporting image I s Perform mask operation to obtain the masked support image I s ′∈R H×W×3 : Among them, ⊙ represents the Hadamard product, Is a M s The function of adjusting the size from H×W to H×W×3; S2.2, the masked support image I s ′ and query image I q , input into the feature extraction network to extract the intermediate layer features of the two, respectively as support features F s and query feature F q : (F s ,F q )=ResNet(I′ s ,I q ),F s ,F q ∈R H×W×3 Among them, ResNet() represents the feature extraction network, F s 、F q Represent support features and query features respectively; S2.3, will support feature F s And query feature F q Expand into and Then, it is input into the multi-similar generation module, and the similarity between each pixel in the tensor is calculated using cosine similarity to generate a 4D hyper-correlation tensor. and 4D hyperautocorrelation tensor Among them, p s and p q For and The spatial position of the feature, the operator "·" represents the 3D vector inner product operation at all positions of the feature, ‖‖ represents the l2 norm, and τ() is a C l and From R H×W →R H×W×3 function; S2.

4. Use support mask M s Perform a second mask on the 4D hypercorrelation tensor Perform mask operation to obtain The formula is as follows: in, The role is to adjust the mask M s Size is H l ×W l , to fit the 4D cross-correlation tensor The size of is the 4D hyper-correlation tensor abbreviation of; S2.5, the masked 4D hyper-correlation tensor Input background self-similar activation module to generate background activation map M b , then activate the 4D hyperautocorrelation tensor The high correlation area of ​​the upper background is obtained to obtain the activated 4D super autocorrelation volume Among them, ξ() is the 4D hypercorrelation tensor after masking The function that averages the last two dimensions of , normalize() is the normalized function, is the 4D hyperautocorrelation tensor abbreviation of; S2.6, the masked 4D hyper-correlation tensor and 4D hyperautocorrelation tensor The cross-correlation tensor Z is obtained through the 4D convolutional coding module sq and the autocorrelation tensor Z qq ; Then, the cross-correlation tensor Z is transformed into sq and the autocorrelation tensor Z qq Fuse and predict dual channel mapping Among them, the multi-similar fusion module consists of a background suppression branch, a target class prediction branch and three mask prediction blocks; S2.

7. Transfer the knowledge learned from the known classes in the training set to the unknown classes in the test set, output the predicted segmentation mask of the query image, and calculate the intersection over union (IoU) with the true segmentation mask to evaluate the detection performance. The IoU calculation formula is as follows: IOU=TP / (TP+FP+FN) Among them, TP represents the number of target pixels predicted correctly, FP represents the number of non-target pixels predicted as target pixels, and FN represents the number of target pixels predicted as non-target pixels.

4. The method for detecting screen printing defects on mobile phone camera glass in a small sample environment according to claim 3, characterized in that: During the training process, the three mask prediction blocks in the multi-similar fusion module generate three prediction masks respectively. By comparing with the real mask M q The corresponding three binary cross entropy losses Loss1, Loss2, and Loss3 are calculated together to optimize the network; the binary cross entropy loss formula is as follows: in, and M q (x, y) represents the predicted mask and the given true mask of each mask prediction block, respectively, and (x, y) represents all pixel positions on the mask; The overall loss during the training phase is Loss final Assessment is done through: Loss final =λ1Loss1+λ2Loss2+λ3Loss3 Among them, Loss1, Loss2, and Loss3 are the target class prediction branch, the background suppression branch, and the binary cross entropy loss after fusing the above two branches, respectively. λ1, λ2, and λ3 are the weighted coefficients of Loss1, Loss2, and Loss3, respectively.

5. The method for detecting screen printing defects on mobile phone camera glass in a small sample environment according to claim 3, characterized in that: The intersection-over-union (IoU) includes the mean intersection-over-union (mIoU) and the foreground-background intersection-over-union (FB-IoU), where the mean intersection-over-union (mIoU) is the average of the intersection-over-union (IoU) of all target categories, and the foreground-background intersection-over-union (FB-IoU) is to ignore the category, regard all different target categories as the same foreground category, and regard the area other than the target category as the background category, and calculate the average of the foreground and background IoU values.

6. The method for detecting screen printing defects on mobile phone camera glass in a small sample environment according to claim 1, characterized in that: The process of using the background suppression network to detect defects in the mobile phone camera glass silk screen defect dataset in step S4 is as follows: Select one of the images from the test set of the mobile phone camera glass silk screen defect dataset as the support image I s , the mask corresponding to the support set image is manually annotated and is denoted as M s , the remaining images are used as query set images; select a query image I q , will I s 、M s and I q Input to the background suppression network and output the defect segmentation result of the query image.

Images