A method for detecting LED chip solder ball PN connection defects in a few-sample environment

Abstract

The application discloses a kind of LED chip solder ball PN connection defect detection methods under few sample environment, this method first proposes the foreground-background iterative network based on few sample learning.In network, deep residual network extracts the features of each aspect of image and prior mask;Query prototype extraction module extracts query prototype and outputs support query sample to extract coarse prediction segmentation mask;Foreground prototype integration module integrates the foreground information of query image to obtain foreground prototype;Background prototype iteration module extracts the background information of image to facilitate to eliminate background area.Under the condition that only one annotated sample is used as supervision, the foreground-background iterative network can determine whether defects exist and locate them in grayscale images.The application can accurately detect different types of LED chip solder ball PN connection surface defects, with good robustness and compatibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and semantic segmentation, and particularly relates to a LED chip solder ball PN connection defect detection method in a few-sample environment. BACKGROUND

[0002] Defects inside or outside a product can reduce the performance of the product, and even threaten the life and property of an individual. Therefore, it is necessary to detect defects in the product to prevent any accidental loss.

[0003] In the past, the defect inspection of LED chip solder ball PN connection was completed by experienced inspectors. However, with the continuous evolution of industrial production lines, we have gradually shifted from a labor-intensive model to automation and intelligence. In the product monitoring process, most of the automatic detection algorithms based on machine vision have replaced manual naked-eye detection, greatly improving the detection efficiency. These detection algorithms can be divided into two categories: traditional methods and deep learning methods. In the case of not too complex problems, traditional methods can match specific types of defects by manually designing features and quickly give solutions. However, manually designed features can only be applied to specific objects and are difficult to generalize to other objects and tasks. Deep learning methods can handle more complex problems and do not require much human intervention. As a powerful tool, it can be applied to different objects without much modification. However, deep learning methods require sufficient training samples for learning, otherwise the training results are not ideal. On the one hand, data labeling requires a lot of manpower and time; on the other hand, in some scenarios such as medical and industrial fields, it is quite difficult to obtain sufficient samples. Therefore, it is particularly important to study networks that can be trained with only a few samples or networks that can be trained using other data sets. SUMMARY

[0004] The purpose of the present application is to solve the above-mentioned defects in the prior art and provide a LED chip solder ball PN connection defect detection method in a few-sample environment.

[0005] The purpose of the present application can be achieved by adopting the following technical solutions:

[0006] A LED chip solder ball PN connection defect detection method based on a few-sample environment, the detection method comprising the following steps:

[0007] S1, constructing a same kind of support query sample pair using a semantic segmentation public data set, the support query sample pair comprising a support set image, a support mask corresponding to the support set image, a query set image, and a query mask corresponding to the query set image, inputting the support set image, the support mask corresponding to the support set image, and the query set image in the support query sample pair into a foreground-background iterative network, and using the query mask as supervision and fine-tuning the foreground-background iterative network;

[0008] S2, building a foreground-background iterative network, the foreground-background iterative network comprising a deep residual network, a query prototype extraction module, a foreground prototype integration module, and a background prototype iteration module, wherein the deep residual network takes the support query sample pair as input to generate middle-layer fusion features and high-layer features corresponding to the support image and the query image, performs mask pooling operation on the middle-layer fusion features of the support image to obtain a support prototype, and performs prior mask operation on the high-layer features of the support image to obtain a prior mask; the query prototype extraction module uses the obtained middle-layer fusion features to extract a query prototype of the query image, and obtains a support coarse prediction segmentation mask and a query coarse prediction segmentation mask corresponding to the support query sample pair through the obtained high-layer features; the foreground prototype integration module obtains a foreground prototype by performing concatenation convolution kernel operation on the obtained prior mask, the support prototype, the query prototype, and the middle-layer fusion features of the query image; and the background prototype iteration module obtains a background prototype by performing reverse, bilinear interpolation, and mask pooling on the obtained query coarse segmentation mask, the middle-layer fusion features of the query image, the foreground prototype, and a decoder mask generated by a multi-layer decoder in the background prototype iteration module, splices the obtained background prototype and the foreground prototype, and inputs the spliced foreground prototype and background prototype into a prediction network to output a final segmentation prediction.

[0009] S3, detecting defects in an LED chip solder ball PN connection defect data set using the foreground-background iterative network.

[0010] Further, the process of constructing the support query sample pair in step S1 is as follows:

[0011] One semantic segmentation public data set is evenly divided into 4 groups according to categories. Among them, 3 groups are used as a training set, and the other 1 group is used as a test set. Each training sample and test sample includes a support set image and a query set image. The support set image is composed of a support image and a corresponding support mask, and the query set image is composed of a query image and a corresponding query mask. In the training process, the support mask corresponding to the support image is used as prior knowledge to train the parameters of the network, and the query mask corresponding to the query image is used as supervision information. In the test process, the query mask of the query image is used to evaluate the performance of the model.

[0012] Further, the working process of the foreground-background iterative network is as follows:

[0013] S2.1, in order to extract the features of the query image that can be matched, first take the support image I s , the query image I q and the corresponding support mask M s from a single support query sample pair in the training set, and the size of I s , I q is HxWx3, and then input the pre-trained deep residual network for feature extraction, and output the image features obtained after the pre-trained deep residual network is used for feature extraction, where H is the height of the support image, W is the width of the support image, and 3 is the number of channels of the support image:

[0014]

[0015] where Res() represents the pre-trained deep residual network, the pre-trained deep residual network uses ResNet50, which has been widely used in image classification and segmentation. The deep residual network is a deep learning algorithm that uses cross-layer linking to obtain residual errors. Specifically, the residual error is the difference between the output and the input of the convolutional layer, so the training target of the deep residual network is to make the residual error approach 0. respectively represent the support image features and the query image features of the i-th layer of the deep residual network, h represents the height of the features, w represents the width of the features, and c represents the number of channels of the features. Since the middle layer features are rich in semantic information of the image, the features of the 2nd and 3rd layers of the deep residual network are spliced and output through a 1x1 Conv(768, 256) convolution kernel with a step of 1 to obtain the middle layer fusion features of the support image and the middle layer fusion features of the query image Conv(input, output) represents the convolution kernel, input represents the input channel number, and output represents the output channel number. Then, the high-level features of the deep residual network are subjected to a prior mask operation to obtain a prior mask C mask , which helps to match the similar semantic information of the query image and the support image. The prior mask operation refers to the paper Z.T. Tian, H.S. Zhao, M. Shu, Z.C. Yang, R.Y. Li, and J.Y. Jia, “Prior guided feature enrichment network for few-shot segmentation”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 2, pp. 1050-1065, Feb. 2022. The size of the known support mask M s is adjusted to h x w using bilinear interpolation, and the support prototype V s:

[0016]

[0017]

[0018]

[0019]

[0020] wherein represents a 1x1 convolution kernel, represents a concatenation operation, represents a prior mask operation, and "⊙" represents a Hadamard product, is a mask pooling operation.

[0021] S2.2, the query prototype extraction module is composed of a strip pooling block, a residual block and a classification head. The strip pooling block is helpful to extract the scale feature information of the image, and has a good improvement for identifying target regions of different scales. The residual block is composed of two 3x3Conv(256, 256) convolution kernels. The classification head is composed of a 3x3Conv(256, 256) convolution kernel, a ReLU activation function and a 1x1Conv(256, 2) convolution kernel. The definition of the ReLU activation unit is ReLU(z) = max(0, z), where z is the input variable. The convolution kernel step is 1. Since the semantic information contained in the high-level features is more specific to the class than the intermediate-level features, it indicates that the former is more likely to have a positive impact on the model's prediction of specific classes. The support features obtained from the 4th layer of the deep residual network are input into the query prototype extraction module to obtain the support coarse prediction segmentation mask and the query feature input into the query prototype extraction module to obtain the support coarse prediction segmentation mask and the query coarse prediction segmentation mask

[0022]

[0023]

[0024] wherein represents a classification head, is the query prototype extraction module, and then the support mask M s and the query mask M q of the support image are used as supervision information to supervise the training and optimize the parameters of the query model extraction module, and the auxiliary loss function obtained is used to supervise the query prototype extraction module, so as to further optimize the matching accuracy of the query prototype, and the specific expression is:

[0025]

[0026]

[0027]

[0028] where BCE() is binary cross entropy;

[0029] the middle-level fusion features of the query image obtained in S2.1 are input into the query prototype extraction module to obtain a query prototype P q :

[0030]

[0031] S2.3, the middle-level fusion features of the query image obtained in S2.1 support prototype V s prior mask C mask and the query prototype P obtained in S2.2 q are sequentially subjected to a splicing operation, a 1x1Conv(512, 256) convolution kernel and a 1x1Conv(513, 256) convolution kernel operation to obtain a foreground prototype Q fg :

[0032]

[0033] where the bracketed is a 1x1Conv(512, 256) convolution kernel, and the one outside the bracket is a 1x1Conv(513, 256);

[0034] S2.4, the query rough prediction segmentation mask obtained in S2.1 is subjected to an inversion operation to obtain a query background rough prediction segmentation mask

[0035]

[0036] In order to accurately extract the background of the model, the iteratively calculate each layer of the query background mask:

[0037]

[0038] where decoder mask representing the t-1th layer output in the multi-layer decoder, the multi-layer decoder has 4 layers, each layer is composed of 2 1x1 convolution kernels and 5 3x3 convolution kernels; the specific implementation details of the multi-layer decoder refer to the paper Z.T. Tian, H.S. Zhao, M. Shu, Z.C. Yang, R.Y. Li, and J.Y. Jia, “Prior guided feature enrichment network for few-shot segmentation”, IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 2, pp. 1050-1065, Feb. 2022. The above obtained and The background prototype P is generated by superimposing the pooling operation:

[0039]

[0040] where P bg is the background prototype, and the extracted background prototype helps to segment the background area of the image, thereby better identifying the target area of the image. The foreground prototype Q fg and the background prototype P bg are input into the multi-layer decoder and the prediction network:

[0041]

[0042]

[0043] wherein and are the foreground prototype and the background prototype of the tth layer obtained by bilinear interpolation of Q fg and P bg , the prototypes of different scales in each layer are fused to further optimize the segmentation performance of the model, is the multi-layer decoder, and the prediction network includes and each layer of the multi-layer decoder represents the fusion of each layer, which is composed of 1 1x1 convolution kernel; composed of a 3x3 Conv(256, 256) convolution kernel, a ReLU activation function and a 1x1 Conv(256, 2) convolution kernel, Y final represents the predicted segmentation result of the query image, and the loss function loss of the entire training process includes the final predicted segmentation loss L final , the multi-layer decoder output L m and the auxiliary loss function L aux of S3.1:

[0044] loss = aL fina l + bL m + gL aux

[0045] L final = BCE(Y final , M q )

[0046] L final = BCE(Y out , M q )

[0047] Wherein a, b, g are adjustable parameters, respectively set to 1.0, 0.7, 1.0;

[0048] Further, the step S3 uses the foreground-background iterative network to detect defects in the LED chip solder ball PN connection defect data set, and the process is as follows:

[0049] S3.1, select a kind of LED chip solder ball PN connection defect data set;

[0050] S3.2, select one of the images as the support set image I S , the mask corresponding to the support set image is manually labeled, denoted as M S , and the remaining images are used as query set images;

[0051] S3.3, select a query set image I Q , input I S , M S and I Q to the foreground-background iterative network, and output the defect segmentation result of the query set image;

[0052] S3.4, repeat step S3.3, and output the segmentation results of all query set images;

[0053] S3.5, repeat steps S3.1-S3.4 N I times, and output the segmentation results of all categories of industrial images, wherein N I represents the number of categories of LED chip solder ball PN connection defects.

[0054] The present application has the following advantages and effects relative to the prior art:

[0055] (1) The present application is based on the public data set PASCAL, and through training on the public data set, the learned parameters in the training set can be migrated to the test set that does not appear in the training set, saving the labor and time of data labeling from the cost, completing the knowledge migration process in the few sample environment, and effectively solving the problem of degradation of deep learning generalization performance.

[0056] (2) The application proposes a query prototype guidance module to generate support prototypes, query prototypes and coarse segmentation masks, which can effectively guide the model extraction and segmentation of the foreground area of the query image, reduce the guidance deviation caused by a single support prototype, and thus more effectively segment the target area.

[0057] (3) The application proposes a foreground prototype integration module to integrate several prototypes and features into a foreground prototype to accurately identify the target area of the query image.

[0058] (4) The application proposes a background iteration prototype, which uses the query coarse prediction segmentation mask obtained by the foreground prototype extraction to generate the background prototype of the image, and iteratively optimizes the background iteration prototype, which can effectively exclude the background area of the image, thereby further optimizing the segmentation of the target area.

[0059] (5) The application integrates the foreground prototype, background prototype and query features into the decoder to generate a prediction mask, which can better segment the target area of the image. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and serve to explain the principles of the application, and do not limit the application. In the drawings:

[0061] Figure 1 is a flowchart of a semantic segmentation method in a few-shot environment disclosed by the application;

[0062] Figure 2 is a structural diagram of the query prototype extraction module in the embodiment of the application;

[0063] Figure 3 is a structural diagram of the foreground prototype integration module in the embodiment of the application;

[0064] Figure 4 is a structural diagram of the background prototype iteration module in the embodiment of the application. DETAILED DESCRIPTION

[0065] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0066] EMBODIMENT

[0067] The embodiment discloses a method for detecting PN connection defects of LED chip solder balls in a small sample environment.

[0068] S1, a same kind of support query sample pair is constructed by using a semantic segmentation public data set, the support query sample pair includes a support set image, a support mask corresponding to the support set image, a query set image, and a query mask corresponding to the query set image, the support set image, the support mask corresponding to the support set image, and the query set image in the support query sample pair are input into a foreground-background iterative network, and the query mask is used as supervision and fine-tuning of the foreground-background iterative network.

[0069] In step S1 of the embodiment, the semantic segmentation public data set is evenly divided into 4 groups according to categories, 3 groups are used as training sets, and the remaining 1 group is used as a test set, and a total of 4 different combinations of training sets and test sets can be established. For the PASCAL data set, the training set contains 15 categories, and the test set contains 5 categories. In the PASCAL test set, a total of 1000 support query sample pairs are used to further test the ability of the method in processing complex scenes.

[0070] S2, a foreground-background iterative network is built, the foreground-background iterative network includes a deep residual network, a query prototype extraction module, a foreground prototype integration module, and a background prototype iteration module, wherein the deep residual network takes the support query sample pair as input to generate middle layer fusion features and high layer features corresponding to the support image and the query image, the middle layer fusion features of the support image are subjected to mask pooling operation to obtain a support prototype, and the high layer features of the support image are subjected to prior mask operation to obtain a prior mask; the query prototype extraction module uses the obtained middle layer fusion features to extract a query prototype of the query image, and uses the obtained high layer features to output a support coarse prediction segmentation mask and a query coarse prediction segmentation mask corresponding to the support query sample pair; the foreground prototype integration module obtains a foreground prototype by performing a concatenation convolution kernel operation on the obtained prior mask, support prototype, query prototype, and middle layer fusion features of the query image; and the background prototype iteration module obtains a background prototype by performing inversion, bilinear interpolation, and mask pooling on the obtained query coarse segmentation mask, middle layer fusion features of the query image, foreground prototype, and decoder mask generated by a multi-layer decoder in the background prototype iteration module, and concatenates and inputs the obtained background prototype and foreground prototype into a prediction network to output a final segmentation prediction.

[0071] In step S2 of the embodiment, a support image I s , a query image I q , and a corresponding support mask M s are taken out from a single support query sample pair in the training set, and the size of each is HxWx3. s , I qAfter feeding the pre-trained deep residual network that performs feature extraction, the output obtains the image features, where H is the height of the support image, W is the width of the support image, and 3 is the number of channels of the support image.

[0072]

[0073] Where Res() represents the pre-trained deep residual network, and the pre-trained deep residual network uses ResNet50. Let h and w represent the support image features and query image features of each i-th layer of the deep residual network, respectively. h represents the height of the feature, w represents the width of the feature, and c represents the number of channels of the feature. The features from layers 2 and 3 of the deep residual network are concatenated and output through a 1×1 Conv(768,256) convolution kernel with a stride of 1 to obtain the mid-layer fused features of the support image. Mid-level fusion features of the query image Conv(input,output) represents the convolution kernel, input represents the number of input channels, and output represents the number of output channels. Then, the high-level features of the deep residual network are... The prior mask C is obtained through prior masking operations. mask The known support mask M is obtained by using bilinear interpolation. s The dimensions are adjusted to h×w, and the supporting prototype V of the supporting image is extracted using mask pooling operations. s :

[0074]

[0075]

[0076]

[0077]

[0078] in This represents a 1×1 convolution kernel. This indicates a splicing operation. This indicates a priori masking operations; "⊙" represents the Hadamard product. This is a mask pooling operation.

[0079] like Figure 2 As shown, the query prototype extraction module consists of a strip pooling block, a residual block, and a classification head. The classification head comprises a 3×3 Conv(256,256) convolutional kernel, a ReLU activation function, and a 1×1 Conv(256,2) convolutional kernel. The ReLU activation unit is defined as ReLU(z) = max(0,z), where z is the input variable, and the stride of each convolutional kernel is 1. Layer 4 of the deep residual network supports the features. and query features The input query prototype extraction module obtains a support coarse prediction segmentation mask and query coarse prediction segmentation mask

[0080]

[0081]

[0082] wherein represents a classification head, is a query prototype extraction module, and then a support mask M s and a query mask M q of the query image are obtained as supervision information to supervise the training of the parameters of the query model extraction module, and an auxiliary loss function is:

[0083]

[0084]

[0085]

[0086] wherein BCE is binary cross entropy;

[0087] The mid-level fusion features of the query image are further input into the query prototype extraction module to obtain a query prototype P q :

[0088]

[0089] As shown in Figure 3 , the foreground prototype integration module: the above-obtained mid-level fusion features of the query image support prototype V s , prior mask C mask and the query prototype P q obtained by S2.2 are subjected to a splicing operation, a 1×1Conv(512, 256) convolution kernel and a 1×1Conv(513, 256) operation to obtain a foreground prototype Q fg :

[0090]

[0091] wherein the bracketed is a 1×1Conv(512, 256) convolution kernel, and the non-bracketed is a 1×1Conv(513, 256).

[0092] As shown in Figure 4 ​The obtained query coarse prediction segmentation mask is shown as S2.1 The inverse operation is performed to obtain a query background coarse prediction segmentation mask

[0093]

[0094] To accurately extract the background of the model, we use the multi-layer decoder generated Each layer of the query background mask is calculated iteratively:

[0095]

[0096] wherein represents the decoder mask output of the t-1th layer in the multi-layer decoder, and the multi-layer decoder has four layers, each of which is composed of 2 1x1 convolution kernels and 5 3x3 convolution kernels. Subsequently, the obtained and The background prototype is generated by superimposing the pooling operation:

[0097]

[0098] wherein P bg is the background prototype. The foreground prototype Q fg obtained by S2.3 and the background prototype P bg are input into the multi-layer decoder and the prediction network:

[0099]

[0100]

[0101] wherein and are the foreground prototype and the background prototype of the tth layer obtained by bilinear interpolation of Q fg and P bg , respectively, is the multi-layer decoder, and the prediction network includes and Each layer of the multi-layer decoder is fused, which is composed of 1 1x1 convolution kernel; which is composed of a 3x3 Conv(256, 256) convolution kernel, a ReLU activation function, and a 1x1 Conv(256, 2) convolution kernel. Y final represents the prediction segmentation result of the query image. The loss function loss of the entire training process includes the final prediction segmentation loss L final , the multi-layer decoder output L m , and the auxiliary loss function L aux of S3.1:

[0102] loss = aL final + bL m + gL aux

[0103] L final = BCE(Y final , M q )

[0104] L final = BCE(Y out , M q )

[0105] where a, b, g are adjustable parameters, which are set to 1.0, 0.7, 1.0, respectively;

[0106] S3, using the foreground-background iterative network to detect defects in the LED chip solder ball PN connection defect dataset.

[0107] In step S3 of the embodiment, the dataset is the LED chip solder ball PN connection defect, the chip surface contains the solder ball PN connection defect, each sample contains one or several defects, and the total number of samples is 25. A sample is randomly selected as a support set image, and the remaining images are used as a query set. In the embodiment, three methods based on few-shot learning are compared with the method proposed in the application, and all methods use the same training set and test set, and support set and query set. The two methods used for comparison are SGOne and PFENet, and the method proposed in the application is denoted as FBINet. The gradient is updated by the SGD algorithm to complete the training of the model, wherein the learning rate is set to 0.0025, and the number of iterations is 200. In the testing process, the knowledge learned by the model in the training set is migrated to the LED chip solder ball PN connection defect dataset, the segmentation mask of the query image is predicted, and the performance of the model is evaluated by the average intersection over union mIoU and the foreground-background intersection over union FB-IoU.

[0108] Figure 1 is a framework diagram of an LED chip solder ball PN connection defect detection method based on few-shot segmentation in the embodiment. In the embodiment, the fine-tuning dataset is the public dataset PASCAL, and a total of 20 classes of images are evenly divided into 4 parts, 3 of which are used as the training set, and the remaining 1 is used as the test set; the dataset to be segmented and detected is the LED chip solder ball PN connection defect industrial dataset. The method proposed in the embodiment is compared with a plurality of few-shot semantic segmentation methods. The methods used for comparison are SGOne and PFENet, and the method proposed in the application is denoted as FBINet. All methods use the ResNet50 network as a feature extractor, and the segmentation results are shown in Table 1:

[0109] Table 1. Single sample semantic segmentation result table based on PASCAL dataset

[0110]

[0111] Wherein mIOU is the average intersection over union of the foreground class of all test samples, FB-IoU does not distinguish the foreground class in the LED chip dataset, and the foreground region is regarded as a separate class, and the background region not belonging to the foreground region is regarded as another class. Wherein IOU = TP / (TP+FP+FN), TP represents the number of target pixel points predicted correctly, FP represents the number of non-target pixel points predicted as target pixel points, and FN represents the number of target pixel points predicted as non-target pixel points. mIOU and FB-IoU represent the ratio relationship between the intersection and the union of the predicted mask and the real mask, reflecting the prediction accuracy of the model. From the results in Table 1, it can be seen that the method FBINet proposed in the present application achieves the highest mIOU and FB-IoU, that is, the optimal segmentation performance, and can complete the single sample LED chip solder ball PN connection defect segmentation task.

[0112] The above embodiments are preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations, simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and are all included in the protection scope of the present application.

Claims

1. A method for detecting a PN connection defect of a solder ball of an LED chip based on a small sample environment, characterized in that, The detection method comprises the following steps: S1, constructing a same-class support query sample pair using a semantic segmentation public data set, the support query sample pair comprising a support set image, a support mask corresponding to the support set image, a query set image, and a query mask corresponding to the query set image, inputting the support set image, the support mask corresponding to the support set image, and the query set image in the support query sample pair into a foreground-background iterative network, and using the query mask as supervision and fine-tuning of the foreground-background iterative network; S2, building the foreground-background iterative network, the foreground-background iterative network comprising a deep residual network, a query prototype extraction module, a foreground prototype integration module, and a background prototype iteration module, wherein the deep residual network takes the support query sample pair as input to generate middle-layer fusion features and high-layer features corresponding to the support image and the query image, performs mask pooling operation on the middle-layer fusion features of the support image to obtain a support prototype, and performs prior mask operation on the high-layer features of the support image to obtain a prior mask; the query prototype extraction module uses the obtained middle-layer fusion features to extract a query prototype of the query image, and obtains a support coarse prediction segmentation mask and a query coarse prediction segmentation mask corresponding to the support query sample pair through the obtained high-layer features; the foreground prototype integration module obtains a foreground prototype by performing concatenation convolution kernel operation on the obtained prior mask, support prototype, query prototype, and middle-layer fusion features of the query image; and the background prototype iteration module obtains a background prototype by performing reverse, bilinear interpolation, and mask pooling on the obtained query coarse prediction segmentation mask, middle-layer fusion features of the query image, foreground prototype, and decoder mask generated by a multi-layer decoder in the background prototype iteration module, and concatenates and inputs the obtained background prototype and foreground prototype into a prediction network to output a final segmentation prediction; S3, detecting defects in the LED chip solder ball PN connection defect data set using the foreground-background iterative network. 2.The LED chip solder ball PN connection defect detection method based on a few sample environment of claim 1, wherein, The process of constructing the support query sample pair in step S1 is as follows: The semantic segmentation public data set is evenly divided into 4 groups according to categories, 3 of which are used as a training set, and the other 1 is used as a test set, wherein the training sample pairs of the training set and the test sample pairs of the test set both comprise support set images and query set images, the support set image is composed of a support image and a corresponding support mask, and the query set image is composed of a query image and a corresponding query mask, the support mask corresponding to the support image is used as prior knowledge, the query mask corresponding to the query image is used as supervision information to train the parameters of the network during training, and the query mask of the query image is used as a standard to evaluate the performance of the model during testing. 3.The LED chip solder ball PN connection defect detection method based on a few sample environment of claim 1, wherein, The working process of the foreground-background iterative network is as follows: S2.1, take out a support image I of size HxWx3 from a single support query sample pair of the training set s , a query image I q , and a corresponding support mask M s , I s , I q After being input to the pre-trained deep residual network, the output obtains an image feature, where H is the height of the support image, W is the width of the support image, and 3 is the channel number of the support image: Where Res() represents the pre-trained deep residual network, and the pre-trained deep residual network uses ResNet50. Let h and w represent the support image features and query image features of the i-th layer of the deep residual network, respectively. h represents the height of the feature, w represents the width of the feature, and c represents the number of channels of the feature. The features of the 2nd and 3rd layers of the deep residual network are concatenated and output through a 1×1 Conv(768,256) convolution kernel with a stride of 1 to obtain the mid-layer fused features of the support image. Mid-level fusion features of the query image Conv(input,output) represents the convolution kernel, input represents the number of input channels, and output represents the number of output channels. Then, the high-level features of the deep residual network are... The prior mask C is obtained through prior masking operations. mask The known support mask M is obtained by using bilinear interpolation. s The dimensions are adjusted to h×w, and the supporting prototype V of the supporting image is extracted using mask pooling operations. s : wherein denotes a 1x1 convolution kernel, denotes a concatenation operation, denotes a prior mask operation, "⊙" denotes a Hadamard product, is a mask pooling operation; S2.2, the query prototype extraction module is composed of a strip pooling block, a residual block and a classification head, the residual block is composed of two 3x3 Conv(256, 256) convolution kernels, the classification head is composed of a 3x3 Conv(256, 256) convolution kernel, a ReLU activation function and a 1x1 Conv(256, 2) convolution kernel, the definition of the ReLU activation unit is ReLU(z) = max(0, z), z is an input variable, and the convolution kernel steps are all 1 and query features inputting the query prototype extraction module to obtain support coarse prediction segmentation masks and query coarse prediction segmentation masks wherein denotes a classification head, is a query prototype extraction module, which then utilizes a support mask M s and a query mask M q as supervisory information to supervise the training of the parameters of the query model extraction module, resulting in an auxiliary loss function is: wherein BCE() is binary cross entropy; The middle layer fusion features of the query image are obtained again The query prototype P is obtained by inputting into the query prototype extraction module q : S2.3, the middle layer fusion features of the query image obtained in S2.1 Support prototype V s Prior mask C mask and the query prototype P obtained in S2.2 q The foreground prototype Q is obtained through a splicing operation, a 1x1Conv(512, 256) convolution kernel and a 1x1Conv(513, 256) convolution kernel operation in sequence fg ; S2.4, obtaining a query coarse prediction segmentation mask from S2.1 performing an inverse operation to obtain a query background coarse prediction segmentation mask generated using a multi-layer decoder iteratively computing each layer query background mask: wherein represents the decoder mask of the t-1 layer output in the multi-layer decoder, the multi-layer decoder has 4 layers, each layer is composed of 2 1x1 convolution kernels and 5 3x3 convolution kernels; then the above obtained and generate the background prototype by superimposing the pooling operation: Where P bg As the background prototype, the foreground prototype Q obtained in S2.3 is used. fg and background prototype P bg Input to the multi-layer decoder and prediction network: wherein and is Q fg and P bg foreground prototype and background prototype of the t-th layer obtained through bilinear interpolation, is a multi-layer decoder, the prediction network comprises and indicates fusion of each layer in the multi-layer decoder, and is composed of a 1x1 convolution kernel; is composed of a 3x3 Conv(256, 256) convolution kernel, a ReLU activation function and a 1x1 Conv(256, 2) convolution kernel, Y final indicates a prediction segmentation result of the query image, and a loss function loss of the whole training process comprises a final prediction segmentation loss L final , a multi-layer decoder output L m and an auxiliary loss function L aux of S3.1: loss = aL final + βL m + γL aux L final = B CE(Y final , M q ) L aux = B CE(Y out , M q ) wherein α, β, and γ are adjustable parameters, and are set to 1.0, 0.7, and 1.0 respectively.

4. The LED chip solder ball PN connection defect detection method based on a few sample environment according to claim 1, characterized in that, The process of step S3 of detecting defects in the LED chip solder ball PN connection defect data set using the foreground-background iterative network is as follows: S3.1, selecting a type of LED chip solder ball PN connection defect data set; S3.2, select one of the images as the support set image I S The mask corresponding to the support set image is manually labeled, denoted as M S The remaining images are used as query set images; S3.3, select a query set image I Q , I S , M S and I Q are input to the pre-background iterative network, and the defect segmentation result of the query set image is output; S3.4, repeating step S3.3 to output segmentation results of all query set images; S3.5, repeat steps S3.1-S3.4 for N times I The industrial image segmentation results of all categories are output, wherein N I The number of categories representing the LED chip solder ball PN connection defects.

Images