MURA defect segmentation method, system and device and storage medium

Abstract

The invention discloses an MURA defect segmentation method, system and device and a storage medium, which are used for reducing dependence on labels and improving MURA defect segmentation precision. The method comprises the following steps: acquiring sample image data; training a classification network and a segmentation network according to the sample image data; obtaining a to-be-segmented image, and inputting the to-be-segmented image into the classification network to obtain an initial CAM graph Performing weighted fusion on all the feature layers to obtain a binarized fine CAM graph, and inputting the binarized fine CAM graph into a segmentation network to obtain a target CAM graph; determining a low-confidence region in the target CAM graph; segmenting the low-confidence region to obtain a refined region, randomly copying the refined region to a normal region, generating a plurality of prediction results, analyzing consistency, and outputting an MURA defect state of the refined region; when the MURA defect state is that the MURA defect exists, changing the refined region into a high-confidence region; and performing MURA defect segmentation on the to-be-segmented image by taking the high-confidence region as a segmentation label.

Description

Technical Field

[0001] The present application relates to the field of screen detection, and in particular to a MURA defect segmentation method, system, device and storage medium. Background Art

[0002] In the field of display manufacturing, the MURA detection process is an important process step. MURA defects usually refer to color difference, uneven brightness, flaws, uneven lighting and other problems that may exist in the display panel. These problems may affect the display effect of the panel and thus affect the quality of the product.

[0003] In display panels, MURA defects usually appear as fuzzy and irregular areas, which makes it difficult for traditional annotation methods to obtain high-quality segmentation labels. In the prior art, MURA detection is mainly performed through threshold-based segmentation methods and region growing and watershed algorithms. The threshold-based segmentation method mainly extracts defective areas by setting thresholds such as brightness, contrast, and color. It is simple but has poor effects on complex display panel problems (such as uneven brightness or minor defects). The region growing and watershed algorithm divides regions according to the pixel similarity in the image, which is suitable for the detection of some continuous regional defects, but it is prone to mis-segmentation in the presence of noise or complex backgrounds. Both of the above methods have the problems of unbalanced data sets and difficult annotation. In the display panel defect segmentation task, the annotation of the data set is usually very difficult and resource-intensive. The defects may be local, small, and of various types. This may cause subjective differences among annotators during the annotation process. In addition, the types and severity of defects vary greatly, resulting in an imbalanced data set, which in turn affects the training effect of the model. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a MURA defect segmentation method, system, device and storage medium, which are used to reduce the dependence on labels and improve the MURA defect segmentation accuracy.

[0005] The technical solution provided in this application is described below: The first aspect of the present application provides a MURA defect segmentation method, comprising: Acquire sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; Training a classification network and a segmentation network according to the sample image data, wherein the classification network is used to classify whether an image contains a MURA defect, and the segmentation network is used to segment the MURA defect area; Acquire an image to be segmented, and input the image to be segmented into the classification network to obtain an initial CAM image; Extracting all feature layers of the initial CAM image, and performing weighted fusion on all feature layers to obtain a binary refined CAM image; Inputting the binarized fine CAM image into the segmentation network to obtain a target CAM image; Determining a low confidence region in the target CAM map; Segmenting the low confidence region to obtain a refined region; The refined area is randomly copied to the normal area, and multiple prediction results are generated by MC-DropOut technology; Analyzing the consistency of the multiple prediction results and outputting the MURA defect status of the refined area; When the MURA defect state is that a MURA defect exists, changing the refined area to a high confidence area; The high confidence region is used as a segmentation label to perform MURA defect segmentation on the image to be segmented.

[0006] Optionally, the training of a classification network according to the sample image data includes: Initialize an initial classification network according to preset training parameters, and define a loss function of the initial classification network as a cross entropy loss function; Inputting the sample image data into the initial classification network one by one to obtain a MURA defect prediction probability, wherein the MURA defect prediction probability is used to predict whether the sample image data has a MURA defect; The classification loss value between the MURA defect prediction probability and the true label is calculated by the cross entropy loss function, and the true label is obtained by the sample image data. The cross entropy loss function is as follows: ; Among them, y i c is the classification label of the i-th image of the label classification sample, p i c is the MURA defect probability value predicted by the classification network for the i-th image in the label classification sample, and N represents the number of images in the label classification sample; The classification loss value is back-propagated to optimize the initial classification network, and the next sample image data is input into the initial classification network until all image inputs included in the sample image data are trained to obtain a classification network.

[0007] Optionally, extracting all feature layers of the initial CAM image and performing weighted fusion on all feature layers to obtain a binary refined CAM image includes: Extracting all feature layers of the initial CAM image; Determine a high-level feature map and a low-level feature map from all the feature layers; The high-level feature map and the low-level feature map are weightedly fused by the following formula to obtain a binary fine CAM map: (x)； Where L is the total number of all feature layers, (x) is the activation value of the cth channel when the number of feature layers is 1, is the number of feature layers, 1, and the weight of the cth channel. is the weight coefficient of the layer.

[0008] Optionally, training a segmentation network according to the sample image data includes: An initial segmentation network is constructed using a U-Net network as a backbone network, and a loss function and a defect area weighting coefficient of the initial segmentation network are defined; Re-labeling the MURA defect segmentation area of ​​the sample image data by using an imprecise label to obtain pseudo-label image data, wherein the labeled portion of the pseudo-label image is a foreground area and the unlabeled portion is a background area; The initial segmentation network is trained by the pseudo-label image data and the sample image data, and the loss of the MURA defect segmentation area is calculated for the pseudo-label image data and the sample image data by a weakly supervised loss function to obtain a first segmentation loss value, wherein the weakly supervised loss function is as follows: ； in, is the first segmentation loss value, is the weighting coefficient of the defect area, fg represents the foreground area, bg represents the background area, is the predicted value of the i-th image segmentation network of the pseudo-label image data and the sample image data; The constraint loss function is calculated by the initial segmentation network, and we get The second segmentation loss value, the constraint loss function is as follows: ; in, is the second segmentation loss value, is the first i The fine CAM map corresponding to the image data, is the pseudo-label image data and the sample image data i Prediction values ​​of the image segmentation network.

[0009] The initial segmentation network is updated by using the first segmentation loss value and the second segmentation loss value until all image data in the pseudo-label image data and the sample image data are trained to obtain a segmentation network.

[0010] Optionally, updating the initial segmentation network by using the first segmentation loss value and the second segmentation loss value includes: The total loss value is calculated by the first segmentation loss value and the second segmentation loss value, and the total loss value is calculated by the following formula: ; in, is the total loss value, , are the first segmentation loss value and the second segmentation loss value respectively The weight coefficient of The initial segmentation network is updated according to the total loss value.

[0011] Optionally, the segmentation network includes a foreground network and a background network, and a weighted coefficient of a defect area of ​​the foreground network is greater than a weighted coefficient of a defect area of ​​the background network.

[0012] Optionally, determining the low confidence area in the target CAM map includes: Acquire a first confidence of the target CAM image through a foreground network; Acquire a second confidence of the target CAM image through the background network; When the confidence states of the first confidence and the second confidence are different, determining the confidence state of the target CAM map as low confidence; When the confidence states of the first confidence and the second confidence are the same, the confidence state of the target CAM map is determined to be high confidence.

[0013] A second aspect of the present application provides a MURA defect segmentation system, including: A first acquisition unit is used to acquire sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; A training unit, used for training a classification network and a segmentation network according to the sample image data, wherein the classification network is used for classifying whether an image contains a MURA defect, and the segmentation network is used for segmenting the MURA defect area; A second acquisition unit is used to acquire the image to be segmented, and input the image to be segmented into the classification network to obtain an initial CAM image; A weighted fusion unit, used for extracting all feature layers of the initial CAM image, and performing weighted fusion on all feature layers to obtain a binary refined CAM image; An input unit, used for inputting the binarized fine CAM image into the segmentation network to obtain a target CAM image; A determination unit, used to determine a low confidence area in the target CAM map; A first segmentation unit, used to segment the low confidence region to obtain a refined region; A prediction unit, used for randomly copying the refined area to the normal area and generating multiple prediction results by using the MC-DropOut technology; An analysis unit, configured to analyze the consistency of the plurality of prediction results and output the MURA defect state of the refined area; a changing unit, configured to change the refined area to a high confidence area when the MURA defect state indicates that a MURA defect exists; The second segmentation unit is used to perform MURA defect segmentation on the image to be segmented by using the high confidence region as a segmentation label.

[0014] Optionally, the training unit is specifically used for: Initialize an initial classification network according to preset training parameters, and define a loss function of the initial classification network as a cross entropy loss function; Inputting the sample image data into the initial classification network one by one to obtain a MURA defect prediction probability, wherein the MURA defect prediction probability is used to predict whether the sample image data has a MURA defect; The classification loss value between the MURA defect prediction probability and the true label is calculated by the cross entropy loss function, and the true label is obtained by the sample image data. The cross entropy loss function is as follows: ； Among them, y i c is the classification label of the i-th image of the label classification sample, p i c is the MURA defect probability value predicted by the classification network for the i-th image in the label classification sample, and N represents the number of images in the label classification sample; The classification loss value is back-propagated to optimize the initial classification network, and the next sample image data is input into the initial classification network until all image inputs included in the sample image data are trained to obtain a classification network.

[0015] Optionally, the weighted fusion unit is specifically used for: Extracting all feature layers of the initial CAM image; Determine a high-level feature map and a low-level feature map from all the feature layers; The high-level feature map and the low-level feature map are weightedly fused by the following formula to obtain a binary fine CAM map: (x)； Where L is the total number of all feature layers, (x) is the activation value of the cth channel when the number of feature layers is 1, is the number of feature layers, 1, and the weight of the cth channel. is the weight coefficient of the layer.

[0016] Optionally, the training unit is further used for: An initial segmentation network is constructed using a U-Net network as a backbone network, and a loss function and a defect area weighting coefficient of the initial segmentation network are defined; Re-labeling the MURA defect segmentation area of ​​the sample image data by using an imprecise label to obtain pseudo-label image data, wherein the labeled portion of the pseudo-label image is a foreground area and the unlabeled portion is a background area; The initial segmentation network is trained by the pseudo-label image data and the sample image data, and the loss of the MURA defect segmentation area is calculated for the pseudo-label image data and the sample image data by a weakly supervised loss function to obtain a first segmentation loss value, wherein the weakly supervised loss function is as follows: ; in, is the first segmentation loss value, is the weighting coefficient of the defect area, fg represents the foreground area, bg represents the background area, is the predicted value of the i-th image segmentation network of the pseudo-label image data and the sample image data; The constraint loss function is calculated by the initial segmentation network, and we get The second segmentation loss value, the constraint loss function is as follows: ； in, is the second segmentation loss value, is a fine CAM image corresponding to the pseudo-label image data and the i-th image data of the sample image data, is the predicted value of the i-th image segmentation network of the pseudo-label image data and the sample image data.

[0017] The initial segmentation network is updated by using the first segmentation loss value and the second segmentation loss value until all image data in the pseudo-label image data and the sample image data are trained to obtain a segmentation network.

[0018] Optionally, the training unit is further used for: The total loss value is calculated by the first segmentation loss value and the second segmentation loss value, and the total loss value is calculated by the following formula: ； in, is the total loss value, , are the first segmentation loss value and the second segmentation loss value respectively The weight coefficient of The initial segmentation network is updated according to the total loss value.

[0019] Optionally, the segmentation network includes a foreground network and a background network, and a weighted coefficient of a defect area of ​​the foreground network is greater than a weighted coefficient of a defect area of ​​the background network.

[0020] Optionally, the training unit is specifically used for: Acquire a first confidence of the target CAM image through a foreground network; Acquire a second confidence of the target CAM image through the background network; When the confidence states of the first confidence and the second confidence are different, determining the confidence state of the target CAM map as low confidence; When the confidence states of the first confidence and the second confidence are the same, the confidence state of the target CAM map is determined to be high confidence.

[0021] A third aspect of the present application provides a MURA defect segmentation device, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and any optional method in the first aspect.

[0022] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional method in the first aspect.

[0023] It can be seen from the above technical solutions that this application has the following advantages: The region containing MURA defects in the image to be segmented is roughly classified through the classification network to obtain the initial CAM image of the image to be segmented. The feature layer of the initial CAM image is weighted fused to obtain a binary fine CAM image that distinguishes the MURA defect region from the normal region by binarization. After determining the binary fine CAM image, the binary fine CAM image is input into the segmentation network to segment the binary fine CAM image again. The obtained target CAM image contains the probability that each pixel block in the binary fine CAM image belongs to the MURA defect region. The low confidence region is determined according to the probability, and the low confidence region is segmented. The obtained refined regions are judged one by one by the MC-DropOut technology, and the refined regions that pass the judgment are determined as high confidence regions. After all the refined regions are judged, the high confidence regions are used as segmentation labels for MURA defect segmentation of the image to be segmented. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A schematic diagram of an embodiment of the MURA defect segmentation method in this application; Figure 2a A schematic diagram of an embodiment flow chart of the first stage of the MURA defect segmentation method in the present application; Figure 2b A flowchart of an embodiment of the second stage of the MURA defect segmentation method in the present application; Figure 3 A flowchart of an embodiment of a training method for a classification network in the MURA defect segmentation method of the present application; Figure 4 A flow chart of an embodiment of a training method for a segmentation network in the MURA defect segmentation method of the present application; Figure 5 A schematic diagram of the structure of an embodiment of the MURA defect segmentation system in this application; Figure 6 It is a schematic diagram of the structure of an embodiment of the MURA defect segmentation device in the present application. DETAILED DESCRIPTION

[0026] It should be noted that the MURA defect segmentation method provided in this application can be applied to a terminal, a system, or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as an example for the execution subject.

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] See also Figure 1 The present application first provides an embodiment of a MURA defect segmentation method, which includes: S101, obtaining sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; The sample image data is sample data used to train classification networks and segmentation networks. The sample image data contains sufficient image data, and these image data contain pixel-level labeled MURA defect segmentation. The pixel-level labeled MURA defect segmentation is reflected in the image data as a segmentation label, that is, the MURA defect edge in the image data is reflected in the label state, and the MURA defect segmentation accuracy in the image data is pixel-level segmentation.

[0029] Because MURA defects usually appear as blurred and irregular areas, MURA defect annotations in sample image data are generated through manual intervention to improve the accuracy of MURA defect labels in sample image data.

[0030] S102, training a classification network and a segmentation network according to the sample image data, wherein the classification network is used to classify whether the image contains MURA defects, and the segmentation network is used to segment the MURA defect area; The classification network is used to classify whether the image data has MURA defects. In actual situations, the sample image data is obtained from the production quality inspection of one or more batches of display screens and the historical quality inspection records of the corresponding products of the relevant production lines. The sample image data contains sufficient training samples to train the classification network and the segmentation network.

[0031] The model networks generated by training with sample image data are a classification network and a segmentation network, wherein the classification network is used to distinguish whether the image contains MURA defects, and the segmentation network is used to segment the MURA defect area in the image, that is, to generate MURA defect labels.

[0032] S103, obtaining an image to be segmented, and inputting the image to be segmented into a classification network to obtain an initial CAM image; After the image to be segmented is input into the classification network, if the image to be segmented is an image containing MURA defects, the classification network classifies the image to be segmented by determining whether the image to be segmented has MURA defects, and in the process of obtaining the existence status of MURA defects, the classification network can determine the area where MURA defects may exist in the image to be segmented. At this time, the area will be marked as the label that the classification network determines that the image to be segmented belongs to an image containing MURA defects. Therefore, after the image to be segmented is input into the classification network, an initial CAM image will be generated at its output. The initial CAM image is marked with a label that may be a MURA defect area. This label is a rough label, so the image to be segmented cannot be segmented for MURA defects through this label.

[0033] S104, extracting all feature layers of the initial CAM image, and performing weighted fusion on all feature layers to obtain a binary refined CAM image; After determining the initial CAM map, the terminal will refine the features in the CAM map according to the information of each feature layer of the initial CAM map.

[0034] Specifically, the feature map of each layer of the initial CAM map contains information at different levels. The low-level feature map captures details and edge information, while the high-level feature map contains more abstract semantic information. Through the weighted fusion of multi-layer feature maps, we can obtain a more detailed CAM map. Finally, a binary fine CAM map is generated. The binary fine CAM map will provide the probability of whether each pixel is a defective area. These binary fine CAM maps binarize the probability of defective areas by designing thresholds and are used as pseudo labels in subsequent segmentation tasks. That is, when the probability that the area is a MURA defect is greater than the design threshold, the probability is recorded as 1, otherwise it is recorded as 0. The MURA defect area of ​​the initial CAM map is binarized by this recording method to obtain a binary fine CAM map.

[0035] The information carried in the binary fine CAM image includes all areas believed to have MURA defects. By refining the features, the binary fine CAM image only contains relevant features of MURA defects, and other features (such as the edge of the display screen) are discarded to improve the efficiency of acquiring features of MURA defect areas in the binary fine CAM image.

[0036] S105, inputting the binary refined CAM image into the segmentation network to obtain the target CAM image; According to the description of the above steps, it can be seen that the MURA defect area in the binary refined CAM image is imperfect, so it is necessary to further refine the MURA defect area in the binary refined CAM image through a segmentation network.

[0037] Specifically, because the aforementioned steps do not perfectly divide the MURA defect area, the MURA defect area label of the binary refined CAM image obtained by the current terminal is defined as an imprecise label. In this embodiment, the imprecise label is regarded as a weakly supervised label through weak supervision loss, so as to judge the foreground area (i.e., the MURA defect area) and the background area (normal area) by changing the weight of the segmentation network. In this embodiment, since the foreground area is a defect area, giving it a larger weight helps the network to better focus on the MURA defect, while the background area is given a smaller weight to avoid overfitting the background.

[0038] The image output by the segmentation network is the target CAM map. Compared with the binary fine CAM map, the target CAM map calculates the probability of MURA defect areas for all pixels included in the label positions that are considered to be MURA defect areas through multiple judgments of the foreground area and the background area. Therefore, in the target CAM map, the MURA defect area label is non-binarized, and the binary parameters are replaced by probability values.

[0039] S106, determining a low confidence area in the target CAM map; After obtaining the target CAM image, it is necessary to evaluate the confidence that each pixel in the image belongs to the MURA defect area. The terminal will determine the area with a probability value lower than the preset probability value through the probability value of the MURA defect area label. The process of determining the preset confidence area is to repeatedly obtain the target CAM image, that is, to obtain the probability value of the MURA area of ​​the image to be segmented multiple times by modifying the weight distribution of the segmentation network. When the average probability value in a MURA area is less than the preset probability value, it is determined that the area belongs to a low confidence area.

[0040] S107, segmenting the low confidence region to obtain a refined region; When the average probability value of a MURA defect area is less than the preset probability value, it means that it cannot be determined whether the area is a MURA defect area. At this time, the terminal needs to refine the area through superpixel technology. The superpixel technology divides the image into multiple smaller areas to improve the segmentation accuracy of the network in the detail area.

[0041] In this embodiment, the terminal segments the low confidence region by using super-pixel technology to obtain a refined region, and the refined region achieves the purpose of refining the region that may be a MURA defect by subdividing the low confidence region.

[0042] S108, randomly copying the refined area to the normal area, and generating multiple prediction results through the MC-DropOut technology; The refined regions are randomly copied to normal regions in the image to test the consistency of the model's predictions for these regions. Monte Carlo DropOut (MC-DropOut) technique is applied to generate multiple different prediction results by randomly dropping neurons during inference.

[0043] S109, analyzing the consistency of multiple prediction results, and outputting the MURA defect status of the refined area; Analyze multiple prediction results generated by MC-DropOut to check whether the refined region is consistently marked as a MURA defect region in all predictions. If the refined region is marked as a MURA defect region in all predictions, output the MURA defect status of the region as MURA defect exists, and execute step S110; otherwise, output that MURA defect does not exist.

[0044] S110, when the MURA defect state is that a MURA defect exists, changing the refined area to a high confidence area; If the refined region is determined to have a MURA defect, it is updated from a low confidence region to a high confidence region. The target CAM map is updated to update the label of the refined region from low confidence to high confidence to reflect the more accurate MURA defect segmentation result.

[0045] S111, performing MURA defect segmentation on the image to be segmented using high confidence regions as segmentation labels.

[0046] If the refined region is determined to have a MURA defect, it is updated from a low confidence region to a high confidence region. The target CAM map is updated to update the label of the refined region from low confidence to high confidence to reflect the more accurate MURA defect segmentation result.

[0047] The region containing MURA defects in the image to be segmented is roughly classified through the classification network to obtain the initial CAM image of the image to be segmented. The feature layer of the initial CAM image is weighted fused to obtain a binary fine CAM image that distinguishes the MURA defect region from the normal region by binarization. After determining the binary fine CAM image, the binary fine CAM image is input into the segmentation network to segment the binary fine CAM image again. The obtained target CAM image contains the probability that each pixel block in the binary fine CAM image belongs to the MURA defect region. The low confidence region is determined according to the probability, and the low confidence region is segmented. The obtained refined regions are judged one by one by the MC-DropOut technology, and the refined regions that pass the judgment are determined as high confidence regions. After all the refined regions are judged, the high confidence regions are used as segmentation labels for MURA defect segmentation of the image to be segmented.

[0048] See also Figure 2a and Figure 2b , the present application embodiment provides another embodiment of the MURA defect segmentation method, which includes: S201, obtaining sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; S202, training a classification network and a segmentation network according to the sample image data, wherein the classification network is used to classify whether the image contains MURA defects, and the segmentation network is used to segment the MURA defect area; S203, obtaining an image to be segmented, and inputting the image to be segmented into a classification network to obtain an initial CAM image; Steps S201 to S203 in this embodiment are similar to steps S101 to S103 in the aforementioned embodiment, and will not be described in detail here.

[0049] S204, extracting all feature layers of the initial CAM image; Feature Map is the output of the convolutional layer in a convolutional neural network (CNN). It represents a set of mappings obtained after the input image is processed by the convolutional layer. Each feature map captures different features or patterns of the input image, such as edges, textures, shapes, etc. Each element of the feature map corresponds to the feature response of a local area in the input image. The characteristics of feature maps include: Multi-channel: A convolutional layer usually contains multiple filters (or called convolution kernels), each of which generates a feature map. Therefore, the output of a convolutional layer is a set of multi-channel feature maps.

[0050] Dimensionality reduction: After processing through a pooling layer (such as max pooling or average pooling), the spatial size of the feature map is usually reduced, thereby reducing the number of parameters and computational complexity.

[0051] Hierarchical: In deep CNNs, as the network layers deepen, feature maps gradually transition from capturing low-level features (such as edges and textures) to capturing higher-level features (such as object parts).

[0052] The generation of the CAM map relies on the feature map output by the convolutional layer. The feature map provides local feature information of the image, and the CAM map is the weighted sum of these feature information, which is used to highlight the most important areas for classification decisions.

[0053] Specifically, the terminal extracts feature maps of all layers from the initial CAM map generated by the classification network. These feature maps contain different levels of information from low to high layers, and each layer provides information about different aspects of the image.

[0054] S205, determining a high-level feature map and a low-level feature map from all feature layers; According to the position of the feature layer in the network, the feature map is divided into high-level feature maps and low-level feature maps. Low-level feature maps usually contain more details and edge information about the image, while high-level feature maps contain more abstract semantic information.

[0055] S206, weighted fusion is performed on the high-level feature map and the low-level feature map by the following formula to obtain a binary fine CAM map: (x) ; Among them, L is the total number of all feature layers, (x) is the activation value of the cth channel when the number of feature layers is 1, is the number of feature layers, 1, and the weight of the cth channel. is the weight coefficient of the layer.

[0056] Binary Refined CAM (BR-CAM) is a processed class activation map that is used to more accurately locate and segment MURA defect areas in images.

[0057] The marking of MURA defects in the initial CAM image before weighted fusion directly covers the possible MURA defects, but the range of MURA defects is difficult to determine, so the marking will be classified as a coarse marking when distinguished by fineness. Binarize the information in the CAM image.

[0058] S207, inputting the binary refined CAM image into the segmentation network to obtain a target CAM image; The segmentation network includes a foreground network and a background network, and the weighted coefficient of the defect area of ​​the foreground network is greater than the weighted coefficient of the defect area of ​​the background network.

[0059] The binary refined CAM image is input into the segmentation network (which includes the foreground network and the background network) to further refine the MURA defect area. The segmentation network obtains multiple results by modifying the weights of the foreground (defect area) and the background (normal area) to improve the accuracy of defect segmentation, where the weighted coefficient of the defect area of ​​the foreground network is greater than the weighted coefficient of the defect area of ​​the background network.

[0060] S208, obtaining a first confidence of the target CAM image through the foreground network; The foreground network in the segmentation network is used to evaluate the confidence that each pixel in the target CAM image belongs to the MURA defect area, and the first confidence is obtained.

[0061] S209, obtaining a second confidence level of the target CAM image through the background network; The background network in the segmentation network is used to evaluate the confidence that each pixel in the target CAM image belongs to the MURA defect area to obtain the second confidence.

[0062] It should be noted that the change in MURA defect weights of the foreground network and the background network affects the fitting result of the segmentation network model. Therefore, the target CAM images output by different networks will have different confidence levels for MURA defects. The foreground network and the background network are distinguished, and the confidence levels obtained are the first confidence level and the second confidence level, respectively.

[0063] S210, when the confidence states of the first confidence and the second confidence are different, determining the confidence state of the target CAM map as low confidence; In actual situations, the terminal stores a preset confidence value, that is, taking the preset confidence as the boundary, if the first confidence and / or the second confidence is higher than the preset confidence, the first confidence and / or the second confidence is determined to be high confidence, otherwise it is determined to be low confidence.

[0064] In actual situations, the terminal determines the state corresponding to the confidence, i.e., the foreground confidence and the background confidence, through the coverage state of the coarse label. The foreground confidence is the area covered by the coarse label, i.e., the MURA defect area. Therefore, the first confidence and the second confidence obtained correspond to the confidence that the area is a MURA defect area. The background confidence is the area not covered by the coarse label, i.e., the normal area. Therefore, the first confidence and the second confidence obtained correspond to the confidence that the area is a normal area.

[0065] S211. When the confidence states of the first confidence and the second confidence are the same, determine the confidence state of the target CAM map as high confidence.

[0066] Similar to step S210, if the evaluation results of the foreground network and the background network for the same pixel are consistent, that is, both believe that the pixel belongs to the defective area or both believe that it belongs to the normal area, then the confidence state of the pixel is marked as high confidence.

[0067] S212, segmenting the low confidence region to obtain a refined region; S213, randomly copying the refined area to the normal area, and generating multiple prediction results through the MC-DropOut technology; S214, analyzing the consistency of multiple prediction results, and outputting the MURA defect status of the refined area; S215, when the MURA defect state is that a MURA defect exists, changing the refined area to a high confidence area; S216: Perform MURA defect segmentation on the image to be segmented using the high confidence region as the segmentation label.

[0068] Steps S212 to S216 in this embodiment are similar to steps S107 to S111 in the aforementioned embodiment, and will not be described in detail here.

[0069] In this embodiment, the terminal analyzes the same target CAM graph multiple times through multiple segmentation networks with different weights, thereby improving the accuracy of segmenting the target CAM graph.

[0070] Specifically, in step S102, the classification network and the segmentation network need to be trained. For the specific training process of the classification network, please refer to Figure 3 , Figure 3 An embodiment of a training method for a classification network in a MURA defect segmentation method is provided for an embodiment of the present application, and the embodiment includes: S301, initializing an initial classification network according to preset training parameters, and defining a loss function of the initial classification network as a cross entropy loss function; First, the terminal initializes an initial classification network according to preset training parameters (such as learning rate, batch size, network architecture, etc.), and defines the loss function of the initial classification network as the cross entropy loss function. The cross entropy loss function is a loss function for classification tasks, which measures the difference between the model's predicted probability distribution and the true label distribution.

[0071] S302, inputting sample image data into the initial classification network one by one to obtain a MURA defect prediction probability, where the MURA defect prediction probability is used to predict whether the sample image data has a MURA defect; The sample image data is input into the initial classification network one by one. These image data have been pixel-level labeled and are used to train the network to identify MURA defects. The network processes each input image and outputs the predicted probability of MURA defects. This probability value indicates the possibility of MURA defects in the image.

[0072] S303, calculating the classification loss value between the MURA defect prediction probability and the true label through the cross entropy loss function, the true label is obtained through the sample image data, and the cross entropy loss function is as follows: ; Among them, y i c is the classification label of the i-th image of the label classification sample, p i c is the MURA defect probability value predicted by the classification network for the i-th image in the label classification sample, and N represents the number of images in the label classification sample; The cross entropy loss function is used to calculate the difference between the predicted probability and the true label. This loss value reflects the prediction error of the model at the current state.

[0073] S304, back-propagating the classification loss value to optimize the initial classification network, and inputting the next sample image data into the initial classification network until all image inputs included in the sample image data have completed training, thereby obtaining a classification network.

[0074] During back propagation, the classification loss value is calculated and back propagated through the network to calculate the gradient of each parameter. The purpose of parameter optimization is to update the network parameters according to the gradient through the optimization algorithm to reduce the loss value and improve the prediction accuracy of the model.

[0075] Repeat the above process until all sample image data are input into the network for training. This process may require multiple iterations until the model performance meets the requirements or reaches the preset stopping condition.

[0076] This embodiment is used to train a classification network. The classification network extracts features layer by layer to determine whether an image contains a MURA defect area, and classifies the image according to the existence status of the MURA defect area.

[0077] Specifically, in step S102, the classification network and the segmentation network need to be trained. For the specific training process of the segmentation network, please refer to Figure 4 , Figure 4 An embodiment of the present application provides an embodiment of a training method for a segmentation network in a MURA defect segmentation method, the embodiment comprising: S401, constructing an initial segmentation network using the U-Net network as the backbone network, and defining the loss function and defect area weighting coefficient of the initial segmentation network; U-Net is a network architecture for image segmentation, which is particularly suitable for image segmentation tasks. Define the loss function and defect area weighting coefficient of the initial segmentation network. The loss function is used to measure the difference between the predicted result and the true label, while the weighting coefficient is used to adjust the importance of different areas (such as foreground area and background area) in the loss calculation.

[0078] S402, re-labeling the MURA defect segmentation area of ​​the sample image data by using an imprecise label to obtain pseudo-label image data, where the marked portion of the pseudo-label image is the foreground area and the unmarked portion is the background area; The MURA defect segmentation area of ​​the sample image data is re-labeled by using inaccurate labels to generate pseudo-label image data. In these pseudo-label images, the labeled area is the foreground area (defect area) and the unlabeled area is the background area.

[0079] In actual situations, the labels of the MURA defect segmentation areas carried in the sample image data are accurate strong labels. In order to improve the segmentation ability of the segmentation network for inaccurate weak labels, during training, weak label data similar to those marked by the classification network will be used for training, and the sample image data will be used as the segmentation target for training the initial segmentation network.

[0080] S403, training the initial segmentation network through the pseudo-label image data and the sample image data, and calculating the loss of the MURA defect segmentation area on the pseudo-label image data and the sample image data through the weak supervision loss function to obtain a first segmentation loss value, and the weak supervision loss function is as follows: ; in, is the first segmentation loss value, is the defect area weighting coefficient, fg represents the foreground area, bg represents the background area, is the predicted value of the segmentation network of the i-th image of the pseudo-label image data and the sample image data; The initial segmentation network is updated by the first segmentation loss value and the second segmentation loss value until all image data in the pseudo-label image data and the sample image data are trained to obtain a segmentation network.

[0081] The weakly supervised loss function is used to calculate the difference between the predicted value and the pseudo label, which takes into account the different weights of the foreground and background regions.

[0082] The initial segmentation network is trained using pseudo-label image data and sample image data. The loss value of the MURA defect segmentation area is calculated through a weakly supervised loss function, which takes into account the defect area weighting coefficient to better focus on the defect area.

[0083] S404, calculate the constraint loss function through the initial segmentation network, and obtain The second segmentation loss value, the constraint loss function is as follows: ; in, is the second segmentation loss value, is the fine CAM image corresponding to the i-th image data of the pseudo-label image data and the sample image data, is the predicted value of the segmentation network of the i-th image of the pseudo-label image data and the sample image data; The constraint loss function is calculated through the initial segmentation network to obtain the second segmentation loss value. This loss function is used to measure the consistency between the prediction results of the segmentation network and the refined CAM map to further guide the learning of the segmentation network.

[0084] S405, calculating the total loss value by using the first segmentation loss value and the second segmentation loss value, and the total loss value is calculated by the following formula: ; in, is the total loss value, , are the first segmentation loss and the second segmentation loss respectively. The weight coefficient of The total loss value is calculated by the first segmentation loss value and the second segmentation loss value. The total loss value is the weighted sum of the two loss values, where the weight coefficient is used to balance the influence of the two loss values.

[0085] S406: Update the initial segmentation network according to the total loss value to obtain a segmentation network.

[0086] The initial segmentation network is updated according to the total loss value, so that the terminal adjusts the segmentation network parameters according to the total loss value to minimize the total loss value, thereby improving the performance of the segmentation network.

[0087] By repeating the updating process from step S403 to step S406 until all image data have completed training, an optimized segmentation network is finally obtained.

[0088] This embodiment is a specific training process of a segmentation network, and the segmentation network can perform more accurate MURA defect area segmentation on image data containing only MURA defects with weak labels.

[0089] The MURA defect segmentation method in the embodiment of the present application is described in detail above, and the MURA defect segmentation system and device will be described in detail below.

[0090] See also Figure 5 The present application provides an embodiment of a MURA defect segmentation system, which includes: The first acquisition unit 501 is used to acquire sample image data, where the sample image data is image data after pixel-level annotation MURA defect segmentation; A training unit 502 is used to train a classification network and a segmentation network according to sample image data, wherein the classification network is used to classify whether an image contains a MURA defect, and the segmentation network is used to segment a MURA defect area; The second acquisition unit 503 is used to acquire the image to be segmented, and input the image to be segmented into the classification network to obtain an initial CAM image; The weighted fusion unit 504 is used to extract all feature layers of the initial CAM image and perform weighted fusion on all feature layers to obtain a binary refined CAM image; An input unit 505 is used to input the binary refined CAM image into the segmentation network to obtain a target CAM image; A determination unit 506, configured to determine a low confidence region in a target CAM map; A first segmentation unit 507 is used to segment the low confidence region to obtain a refined region; A prediction unit 508, used to randomly copy the refined area to the normal area and generate multiple prediction results through the MC-DropOut technology; An analysis unit 509 is used to analyze the consistency of multiple prediction results and output the MURA defect status of the refined area; A changing unit 510, configured to change the refined region to a high confidence region when the MURA defect state is that a MURA defect exists; The second segmentation unit 511 is used to perform MURA defect segmentation on the image to be segmented by using the high confidence region as the segmentation label.

[0091] In this embodiment, the training unit 502 is specifically used for: Initialize the initial classification network according to the preset training parameters, and define the loss function of the initial classification network as the cross entropy loss function; Input sample image data into the initial classification network one by one to obtain the MURA defect prediction probability, which is used to predict whether the sample image data has a MURA defect; The classification loss value between the MURA defect prediction probability and the true label is calculated by the cross entropy loss function. The true label is obtained through sample image data. The cross entropy loss function is as follows: ; Among them, y i c is the classification label of the i-th image of the label classification sample, p i c is the MURA defect probability value predicted by the classification network for the i-th image in the label classification sample, and N represents the number of images in the label classification sample; The classification loss value is back-propagated to optimize the initial classification network, and the next sample image data is input into the initial classification network until all image inputs contained in the sample image data are trained to obtain a classification network.

[0092] In this embodiment, the weighted fusion unit 504 is specifically used for: Extract all feature layers of the initial CAM image; Determine high-level feature maps and low-level feature maps from all feature layers; The high-level feature map and the low-level feature map are weightedly fused by the following formula to obtain a binary fine CAM map: (x) ; Among them, L is the total number of all feature layers, (x) is the activation value of the cth channel when the number of feature layers is 1, is the number of feature layers, 1, and the weight of the cth channel. is the weight coefficient of the layer.

[0093] In this embodiment, the training unit 502 is further configured to: Use the U-Net network as the backbone network to build the initial segmentation network, and define the loss function of the initial segmentation network and the defect area weighting coefficient; The MURA defect segmentation area of ​​the sample image data is re-labeled by using an inaccurate label to obtain pseudo-label image data, where the labeled area of ​​the pseudo-label image is the foreground area and the unlabeled area is the background area; The initial segmentation network is trained by using pseudo-label image data and sample image data, and the loss of the MURA defect segmentation area is calculated for the pseudo-label image data and sample image data using a weakly supervised loss function to obtain the first segmentation loss value. The weakly supervised loss function is as follows: ; in, is the first segmentation loss value, is the defect area weighting coefficient, fg represents the foreground area, bg represents the background area, is the predicted value of the segmentation network of the i-th image of the pseudo-label image data and the sample image data; The constraint loss function is calculated through the initial segmentation network, and we get The second segmentation loss value, the constraint loss function is as follows: ; in, is the second segmentation loss value, is the fine CAM image corresponding to the i-th image data of the pseudo-label image data and the sample image data, is the predicted value of the segmentation network for the i-th image of the pseudo-label image data and the sample image data.

[0094] The initial segmentation network is updated by the first segmentation loss value and the second segmentation loss value until all image data in the pseudo-label image data and the sample image data are trained to obtain a segmentation network.

[0095] In this embodiment, the training unit 502 is further configured to: The total loss value is calculated by the first segmentation loss value and the second segmentation loss value. The total loss value is calculated by the following formula: ; in, is the total loss value, , are the first segmentation loss and the second segmentation loss respectively. The weight coefficient of Update the initial segmentation network based on the total loss value.

[0096] In this embodiment, the segmented network includes a foreground network and a background network, and the weighted coefficient of the defect area of ​​the foreground network is greater than the weighted coefficient of the defect area of ​​the background network.

[0097] In this embodiment, the training unit 502 is specifically used for: Obtain the first confidence of the target CAM image through the foreground network; Obtaining a second confidence of the target CAM image through the background network; When the confidence states of the first confidence and the second confidence are different, determining the confidence state of the target CAM map as low confidence; When the confidence states of the first confidence and the second confidence are the same, the confidence state of the target CAM map is determined to be high confidence.

[0098] In this embodiment, the functions of each unit are the same as those described above. Figures 1 to 4 The steps in the illustrated embodiments correspond to each other and will not be described again here.

[0099] See also Figure 6 The present application embodiment provides another embodiment of a MURA defect segmentation device, including: Processor 601, memory 602, input and output unit 603, bus 604; The processor 601 is connected to the memory 602, the input and output unit 603 and the bus 604; Processor 601 specifically executes Figures 1 to 4 The operations corresponding to the steps in the method are not described in detail here.

[0100] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0101] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0102] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0103] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0104] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0105] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A MURA defect segmentation method, characterized in that: The method comprises: Acquire sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; Training a classification network and a segmentation network according to the sample image data, wherein the classification network is used to classify whether an image contains a MURA defect, and the segmentation network is used to segment the MURA defect area; Acquire an image to be segmented, and input the image to be segmented into the classification network to obtain an initial CAM image; Extracting all feature layers of the initial CAM image, and performing weighted fusion on all feature layers to obtain a binary refined CAM image; Inputting the binarized fine CAM image into the segmentation network to obtain a target CAM image; Determining a low confidence region in the target CAM map; Segmenting the low confidence region to obtain a refined region; The refined area is randomly copied to the normal area, and multiple prediction results are generated by MC-DropOut technology; Analyzing the consistency of the multiple prediction results and outputting the MURA defect status of the refined area; When the MURA defect state is that a MURA defect exists, changing the refined area to a high confidence area; The high confidence region is used as a segmentation label to perform MURA defect segmentation on the image to be segmented.

2. The MURA defect segmentation method according to claim 1, characterized in that: The step of training a classification network according to the sample image data comprises: Initialize an initial classification network according to preset training parameters, and define a loss function of the initial classification network as a cross entropy loss function; Inputting the sample image data into the initial classification network one by one to obtain a MURA defect prediction probability, wherein the MURA defect prediction probability is used to predict whether the sample image data has a MURA defect; The classification loss value between the MURA defect prediction probability and the true label is calculated by the cross entropy loss function, and the true label is obtained by the sample image data. The cross entropy loss function is as follows: ； Among them, y i c is the classification label of the i-th image of the label classification sample, p i c is the MURA defect probability value predicted by the classification network for the i-th image in the label classification sample, and N represents the number of images in the label classification sample; The classification loss value is back-propagated to optimize the initial classification network, and the next sample image data is input into the initial classification network until all image inputs included in the sample image data are trained to obtain a classification network.

3. The MURA defect segmentation method according to claim 1, characterized in that: The step of extracting all feature layers of the initial CAM image and performing weighted fusion on all feature layers to obtain a binary refined CAM image includes: Extracting all feature layers of the initial CAM image; Determine a high-level feature map and a low-level feature map from all the feature layers; The high-level feature map and the low-level feature map are weightedly fused by the following formula to obtain a binary fine CAM map: (x) ； Where L is the total number of all feature layers, (x) is the activation value of the cth channel when the number of feature layers is 1, is the number of feature layers, 1, and the weight of the cth channel. is the weight coefficient of the layer.

4. The MURA defect segmentation method according to claim 1, characterized in that: The step of training a segmentation network according to the sample image data comprises: An initial segmentation network is constructed using a U-Net network as a backbone network, and a loss function and a defect area weighting coefficient of the initial segmentation network are defined; Re-labeling the MURA defect segmentation area of ​​the sample image data by using an imprecise label to obtain pseudo-label image data, wherein the labeled portion of the pseudo-label image is a foreground area and the unlabeled portion is a background area; The initial segmentation network is trained by the pseudo-label image data and the sample image data, and the loss of the MURA defect segmentation area is calculated for the pseudo-label image data and the sample image data by a weakly supervised loss function to obtain a first segmentation loss value, wherein the weakly supervised loss function is as follows: ； in, is the first segmentation loss value, is the weighting coefficient of the defect area, fg represents the foreground area, bg represents the background area, is the predicted value of the i-th image segmentation network of the pseudo-label image data and the sample image data; The constraint loss function is calculated by the initial segmentation network, and we get The second segmentation loss value, the constraint loss function is as follows: ； in, is the second segmentation loss value, is a fine CAM image corresponding to the pseudo-label image data and the i-th image data of the sample image data, is the predicted value of the i-th image segmentation network of the pseudo-label image data and the sample image data; The initial segmentation network is updated by using the first segmentation loss value and the second segmentation loss value until all image data in the pseudo-label image data and the sample image data are trained to obtain a segmentation network.

5. The MURA defect segmentation method according to claim 4, characterized in that: The updating of the initial segmentation network by using the first segmentation loss value and the second segmentation loss value comprises: The total loss value is calculated by the first segmentation loss value and the second segmentation loss value, and the total loss value is calculated by the following formula: ； in, is the total loss value, , are the first segmentation loss value and the second segmentation loss value respectively The weight coefficient of The initial segmentation network is updated according to the total loss value.

6. The MURA defect segmentation method according to any one of claims 1 to 5, characterized in that: The segmentation network includes a foreground network and a background network, and a weighted coefficient of a defect area of ​​the foreground network is greater than a weighted coefficient of a defect area of ​​the background network.

7. The MURA defect segmentation method according to claim 6, characterized in that: Determining the low confidence area in the target CAM map includes: Acquire a first confidence of the target CAM image through a foreground network; Acquire a second confidence of the target CAM image through the background network; When the confidence states of the first confidence and the second confidence are different, determining the confidence state of the target CAM map as low confidence; When the confidence states of the first confidence and the second confidence are the same, the confidence state of the target CAM map is determined to be high confidence.

8. A MURA defect segmentation system, characterized in that: The MURA defect segmentation system comprises: A first acquisition unit is used to acquire sample image data, where the sample image data is image data that has completed pixel-level annotation MURA defect segmentation; A training unit, used for training a classification network and a segmentation network according to the sample image data, wherein the classification network is used for classifying whether an image contains a MURA defect, and the segmentation network is used for segmenting the MURA defect area; A second acquisition unit is used to acquire the image to be segmented, and input the image to be segmented into the classification network to obtain an initial CAM image; A weighted fusion unit, used for extracting all feature layers of the initial CAM image, and performing weighted fusion on all feature layers to obtain a binary refined CAM image; An input unit, used for inputting the binarized fine CAM image into the segmentation network to obtain a target CAM image; A determination unit, used to determine a low confidence area in the target CAM map; A first segmentation unit, used to segment the low confidence region to obtain a refined region; A prediction unit, used for randomly copying the refined area to a normal area and generating multiple prediction results by using the MC-DropOut technology; An analysis unit, configured to analyze the consistency of the plurality of prediction results and output the MURA defect state of the refined area; a changing unit, configured to change the refined region into a high confidence region when the MURA defect state indicates that a MURA defect exists; The second segmentation unit is used to perform MURA defect segmentation on the image to be segmented by using the high confidence region as a segmentation label.

9. A MURA defect segmentation device, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the method according to any one of claims 1 to 7 is performed.

Images