A screen defect detection method and system based on a cooperative neural network

Abstract

This application belongs to the field of intelligent image defect recognition, and more specifically, relates to a method and system for screen defect detection based on a collaborative neural network. It utilizes a trained feature relationship discrimination network to obtain target defect distribution feature maps corresponding to each defect type by fusing the target defect feature map and the target defect features. This allows for the preliminary acquisition of the distribution of various defects in the target feature map using a small number of samples of images of different defect types, improving the generalization ability of detection and recognition. By calculating the cosine similarity between each block of the target feature map and the defect-free feature map, an abnormal distribution feature map is obtained that accurately reflects the distribution of abnormal blocks in the target feature map. Furthermore, the overlap of anchor boxes in the abnormal distribution feature map and the target defect distribution feature map corresponding to each defect type can be used to accurately obtain the actual detection and recognition results of multiple defect types in the target feature map through a dual-channel joint judgment method.

Description

Technical Field

[0001] This application belongs to the field of intelligent image defect recognition, and more specifically, relates to a method and system for screen defect detection based on a collaborative neural network. Background Technology

[0002] In the manufacturing and surface treatment stages of LCD / LED displays, the finished screen surface is highly susceptible to various appearance defects due to multiple factors, including the cleanliness of the production environment, the precision of the operating process, the stability of equipment operation, and the surface characteristics of materials. These defects include scratches, cracks, bubbles, pits, edge chipping, discoloration, foreign matter adhesion, dirt, and slight module deformation. These defects directly affect the visual display effect and product yield. Therefore, high-precision, full-category defect detection of the screen surface is a core aspect of quality control in the display production process. However, in actual production scenarios, the occurrence of various appearance defects is a low-probability event, and the inducing conditions for different defects vary significantly. This results in a very small number of actual production cases for each type of defect. Defect image samples that can be collected, professionally annotated, and used for training target detection models are particularly scarce; typically, only 5-10 valid samples can be obtained for each type of defect. This severe shortage of samples prevents deep learning-based target detection networks from fully learning the feature representations of various defects, making it difficult to effectively identify the specific type of detected defect.

[0003] To enable object detection networks to detect complex types of defects using small sample data, most existing object detection network-based methods employ generative adversarial networks (GANs) to augment the small sample data. They also pre-train backbone networks on defect data from other products such as metal workpieces and ordinary glass panels, adapting them for display screen surface defect detection by freezing the backbone and fine-tuning only the detection head (transfer learning). However, defect images generated by GANs may introduce artifacts and distortion features, leading to unstable quality of the synthesized defect images. Furthermore, transfer learning can result in insufficient generalization ability of the object detection network to defect scenarios with significantly different defect forms, making it difficult to adapt to the actual defect detection needs of LCD / LED display industrial production. Summary of the Invention

[0004] In view of the above-mentioned defects in the existing technology, this application provides a method and system for screen defect detection based on collaborative neural networks. It aims to solve the problems of insufficient generalization ability and low detection accuracy of the existing methods for detecting and identifying complex surface defects of multiple types of displays caused by the combined effects of scarce defect samples, significant differences in defect features, and unstable quality of augmented and synthesized defect samples during the display production process.

[0005] In a first aspect, this application provides a screen defect detection method based on a collaborative neural network, characterized in that it includes: S1. Input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map. Then, group the defect images of each type in the defect image dataset according to the defect type and input them into the shared feature extraction network to obtain at least one set of target defect feature maps. Each set of target defect feature maps corresponds to one type of defect. S2. Perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type; S3. In the spatial dimension, the feature maps of each target defect corresponding to each defect type are spliced ​​to obtain at least one spatial feature defect sequence. The target feature map is then enhanced by each spatial feature defect sequence to obtain the target fusion defect feature map corresponding to each defect type. S4. Based on the target fusion defect feature map and target defect feature vector corresponding to each defect type, obtain multiple defect score maps corresponding to the target feature map. Each defect score map is used to characterize the occurrence probability of a defect type in each block of the target feature map. S5. Based on the defect score maps, determine the defect type of each block on the target feature map and generate a target defect distribution feature map; obtain the deviation score map based on the cosine similarity between each block of the target feature map and the defect-free feature map, and determine the abnormal blocks in the target feature map based on the deviation score map to generate an abnormal distribution feature map. S6. Based on the anomaly distribution feature map and the defect distribution feature map of each target, obtain the final defect detection result of the target feature map.

[0006] Among them, the feature extraction processes of the shared feature extraction network for each defect image, the product image to be detected, and the product image without defects are independent of each other. Using the same feature extraction network is to ensure that the image features of the obtained target defect feature map, target feature map, and defect-free feature map are in the same semantic space, providing a basis for the subsequent collaborative interaction of each feature map.

[0007] Furthermore, a global feature extraction and fusion operation is performed on each target defect feature map within each target defect feature map set, including: Average pooling is performed on each target defect feature map in the target defect feature map set from the spatial dimension to obtain at least one initial defect feature vector. Then, element-wise averaging is performed on each initial defect feature vector to obtain the target defect feature vector.

[0008] Furthermore, the shared feature extraction network is the DINOv2 self-supervised network.

[0009] Furthermore, the specific steps of S3 include: Expand each target defect feature map in the spatial dimension to obtain at least one feature sequence block of the same size, and perform feature splicing on each feature sequence block to obtain a spatial feature defect sequence. Expand the target feature map in the spatial dimension to obtain the target spatial feature sequence. Based on the cross-attention mechanism and spatial feature defect sequence, feature enhancement processing is performed on the target spatial feature sequence, and the dimension of the target spatial feature sequence after feature enhancement processing is restored to obtain the target fusion defect feature map. The target fusion defect feature map has the same size as the target feature map.

[0010] Furthermore, based on the target fused defect feature map and target defect feature vector corresponding to each defect type, multiple defect score maps corresponding to the target feature map are obtained, including: The target defect feature vector corresponding to the defect type is copied and expanded in the spatial dimension to obtain the target global defect feature map. The target global defect feature map and the target fused defect feature map have the same size. The target global defect feature map and the target fused defect feature map are input into the feature relationship discrimination network, and the defect occurrence probability of each block of the target fused defect feature map is obtained based on the feature relationship discrimination network. Based on the distribution of the probability of defect occurrence in each block of the target fusion defect feature map, a defect score map is obtained. The defect score map is used to characterize the probability of occurrence of each defect type in each block of the target feature map corresponding to the target fusion defect feature map.

[0011] Furthermore, based on each defect score map, the defect type of each block on the target feature map is determined, and a target defect distribution feature map is generated, including: Based on the defect score maps, the probability of each defect type occurring in each block of the target feature map is predicted by the classification head of the detection head network; The NMS algorithm is used to obtain the anchor boxes of each defect type on the target feature map through the regression head of the detection head network. The anchor boxes are then drawn on the target feature map to obtain the target defect distribution feature map.

[0012] Furthermore, based on the deviation score map, abnormal blocks in the target feature map are identified, and an abnormal distribution feature map is generated, including: Based on the deviation score map, the deviation score corresponding to each block on the target feature map is obtained, and the blocks with deviation scores greater than the preset value are marked as abnormal blocks. Connectivity analysis is performed on adjacent abnormal blocks to obtain at least one anchor frame composed of abnormal blocks in the same connected component. Each anchor frame is drawn on the target feature map to obtain the abnormal distribution feature map.

[0013] Furthermore, based on the anomaly distribution feature map and the distribution feature maps of each target defect, the final defect detection result of the target feature map is obtained, including: Each anchor frame in the abnormal distribution feature map that represents the distribution of abnormal blocks is taken as a reverse anchor frame, and the anchor frame in the target defect distribution feature map that represents the distribution of defect blocks is taken as a forward anchor frame. Based on the overlap of each forward anchor frame and reverse anchor frame, the final defect detection result of the target feature map is obtained.

[0014] Furthermore, based on the overlap of each forward and reverse anchor frame, the final defect detection result of the target feature map is obtained, including: Each forward anchor box and reverse anchor box is mapped to the same target feature map. The forward anchor box is labeled with the defect type and the average defect score of the defect type corresponding to each block in the forward anchor box. The reverse anchor box is labeled with the anomaly label and the average deviation score of each block in the reverse anchor box. Based on the NMS algorithm, each positive anchor frame corresponding to the same defect type is deduplicated, and each negative anchor frame is deduplicated to obtain at least one target positive anchor frame and one target negative anchor frame. When a target block in the target feature map is simultaneously covered by at least one positive anchor frame and one negative anchor frame, the defect type of the positive anchor frame with the highest average defect score covering the target block is taken as the defect type of the target block. Connectivity analysis is performed on target blocks of the same defect type. Target blocks within the same connected region are connected and their bounding boxes are labeled to obtain the final defect detection result of the target feature map.

[0015] Secondly, this application also provides a screen defect detection system based on a cooperative neural network for performing any of the methods in the first aspect, including: The feature map acquisition module is used to input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map. It also inputs the defect images of each type in the defect image dataset into the shared feature extraction network according to the defect type to obtain at least one set of target defect feature maps. The global feature extraction and fusion module is used to perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type. The spatial feature enhancement module is used to perform feature stitching processing on the target defect feature maps corresponding to each defect type in the spatial dimension to obtain at least one spatial feature defect sequence, and to perform feature enhancement processing on the target feature map through each spatial feature defect sequence to obtain the target fused defect feature map corresponding to each defect type. The defect score acquisition module is used to obtain multiple defect score maps corresponding to the target feature map based on the target fused defect feature map and the target defect feature vector corresponding to each defect type. The distribution feature acquisition module is used to determine the defect type of each block on the target feature map based on each defect score map, and generate the target defect distribution feature map; it also obtains the deviation score map based on the cosine similarity between each block of the target feature map and the defect-free feature map, and determines the abnormal blocks in the target feature map based on the deviation score map, generating the abnormal distribution feature map. The defect detection result acquisition module is used to obtain the final defect detection result of the target feature map based on the abnormal distribution feature map and the target defect distribution feature map.

[0016] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art: This application obtains target defect feature vectors corresponding to each defect type by performing global feature extraction and fusion operations on target defect feature maps of various defect images of the same defect type. Simultaneously, a cross-attention mechanism is used to combine the target defect feature maps of the same defect type to enhance the spatial features of the target feature map corresponding to the product to be inspected, generating target fused defect feature maps corresponding to each defect type with stable image quality. Then, a trained feature relationship discrimination network is used to obtain target defect distribution feature maps corresponding to each defect type based on the target fused defect feature maps and target defect features. This allows for the preliminary acquisition of the distribution of various defects in the target feature map using a small number of samples of different defect images, improving the generalization ability of detection and recognition. By calculating the cosine similarity between each block of the target feature map and the defect-free feature map, an abnormal distribution feature map is obtained that accurately reflects the distribution of abnormal blocks in the target feature map. Then, the overlap of anchor boxes in the abnormal distribution feature map and the target defect distribution feature map corresponding to each defect type is used to accurately obtain the actual detection and recognition results of multiple defect types in the target feature map through a dual-channel joint judgment method of "defect type discrimination + abnormal region verification". Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced one by one below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the screen defect detection method based on a collaborative neural network provided in this application embodiment.

[0019] Figure 2 This is another schematic flowchart of the screen defect detection method provided in the embodiments of this application.

[0020] Figure 3 This is a schematic diagram of the workflow of the feature relationship discrimination network provided in the embodiments of this application.

[0021] Figure 4 This is a schematic diagram of the structure of the screen defect detection system based on a collaborative neural network provided in an embodiment of this application. Detailed Implementation

[0022] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0023] In the following description, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The following description provides multiple embodiments of this application, which can be substituted or combined with each other. Therefore, this application can also be considered to include all possible combinations of the same and / or different embodiments described. Thus, if one embodiment includes features A, B, and C, and another embodiment includes features B and D, then this application should also be considered to include embodiments containing one or more other possible combinations of A, B, C, and D, even if such embodiments are not explicitly described in the following text.

[0024] The following description provides examples and does not limit the scope, applicability, or examples set forth in the claims. Changes may be made to the function and arrangement of the described elements without departing from the scope of this application. Various processes or components may be appropriately omitted, substituted, or added to the examples. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Furthermore, features described with respect to some examples may be combined into other examples.

[0025] Figure 1 This is a flowchart illustrating the screen defect detection method based on a collaborative neural network provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes at least the following steps: S1. Input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map. Then, group the defect images of each type in the defect image dataset according to the defect type and input them into the shared feature extraction network to obtain at least one set of target defect feature maps.

[0026] In this embodiment, the execution entity of the method can be the controller of a production line image recognition device. The shooting environment, location, and production process conditions of the product image to be detected and the defect-free product image are consistent. Therefore, the defect-free product image can be used to identify abnormal areas in the product image to be detected. Using the same shared feature extraction network for the product image to be detected and the defect-free product image ensures that the target feature map and the defect-free feature map obtained from both are in the same semantic space, providing a prerequisite for subsequent feature interaction. For example, by inputting K defect images of scratch defect types, 1 product image to be detected, and the defect-free product image into the shared feature extraction network, a set of target defect feature maps that can characterize "different scratch degrees on the screen" and a defect-free feature map that can characterize "what an undamaged screen looks like" can be obtained. In addition, the shared feature extraction network can be a neural network obtained by combining a simple backbone and fully connected layers. For example, the backbone can be a ResNet50 supervised model trained on the ImageNet training set, or a DINOv2 self-supervised model trained on the LVD-142M training set, but the types of shared feature extraction networks are not limited to these.

[0027] S2. Perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type.

[0028] In one possible implementation, a global feature extraction and fusion operation is performed on each target defect feature map within each target defect feature map set, including: Average pooling is performed on each target defect feature map in the target defect feature map set from the spatial dimension to obtain at least one initial defect feature vector. Then, element-wise averaging is performed on each initial defect feature vector to obtain the target defect feature vector.

[0029] In the embodiments of this application, such as Figure 2As shown, the processing of the target defect feature map set can be divided into path A and path B, with step S2 corresponding to the operation in path A. The purpose of obtaining the target defect feature vector is to extract and compress the global features of all target defect feature maps corresponding to the same type of defect. First, a global average pooling operation is performed on each target defect feature map in the spatial dimension, that is, the feature value at each position of the target defect feature map is averaged, and compressed into a target defect feature vector. For example, H×W×C target defect feature maps are compressed into a C-dimensional vector. This C-dimensional vector can retain the global features of the target defect feature map. Different C-dimensional vectors can retain different degrees of features of the same defect type (such as light scratches and deep scratches). Then, the C-dimensional vectors corresponding to all target defect feature maps of the same defect type are averaged element by element, and multiple C-dimensional vectors are aggregated into a C-dimensional target defect feature vector. The target defect feature vector can effectively reflect the global features of the corresponding defect type.

[0030] S3. In the spatial dimension, the feature maps of each target defect type are spliced ​​together to obtain at least one spatial feature defect sequence. The target feature map is then enhanced by each spatial feature defect sequence to obtain the target fusion defect feature map corresponding to each defect type.

[0031] In one possible implementation, step S3 specifically includes: Expand each target defect feature map in the spatial dimension to obtain at least one feature sequence block of the same size, and perform feature splicing on each feature sequence block to obtain a spatial feature defect sequence. Expand the target feature map in the spatial dimension to obtain the target spatial feature sequence. Based on the cross-attention mechanism and spatial feature defect sequence, feature enhancement processing is performed on the target spatial feature sequence, and the dimension of the target spatial feature sequence after feature enhancement processing is restored to obtain the target fusion defect feature map. The target fusion defect feature map has the same size as the target feature map.

[0032] In the embodiments of this application, such as Figure 2 As shown, step S3 mainly utilizes spatial detail information (such as local texture, edge morphology, and location distribution of defects) in the target defect feature map to enhance the feature representation of the target feature map corresponding to the image to be detected. Taking the target defect feature maps corresponding to scratch defect types as an example, by unfolding and splicing each target defect feature map in the spatial dimension into a spatial feature defect sequence, and then using the cross-attention mechanism and the spatial feature defect sequence to enhance the target spatial feature sequence of the target feature map, the feature values ​​of each scratched block in the target feature map can be increased, that is, visually presenting the scratched area as brighter and the normal area as darker.

[0033] S4. Based on the target fusion defect feature map and target defect feature vector corresponding to each defect type, obtain multiple defect score maps corresponding to the target feature map. Each defect score map is used to characterize the occurrence probability of a defect type in each block of the target feature map.

[0034] In one possible implementation, based on the target fused defect feature map and the target defect feature vector corresponding to each defect type, multiple defect score maps corresponding to the target feature map are obtained, including: The target defect feature vector corresponding to the defect type is copied and expanded in the spatial dimension to obtain the target global defect feature map. The target global defect feature map and the target fused defect feature map have the same size. The target global defect feature map and the target fused defect feature map are input into the feature relationship discrimination network, and the defect occurrence probability of each block of the target fused defect feature map is obtained based on the feature relationship discrimination network. Based on the distribution of the probability of defect occurrence in each block of the target fusion defect feature map, a defect score map is obtained. The defect score map is used to characterize the probability of occurrence of each defect type in each block of the target feature map corresponding to the target fusion defect feature map.

[0035] In the embodiments of this application, such as Figure 3 As shown, the operations in path B above can only characterize whether defects exist in each block of the target feature map, but cannot determine the specific type of defect in each block. Therefore, it is necessary to further determine the probability of different defects occurring in each block based on the target fused defect feature map and target defect feature vector corresponding to each defect type. This application uses a feature relationship discrimination network, which obtains the probability of each defect type occurring in different blocks based on the target defect feature vector corresponding to each defect type and the target fused defect feature map corresponding to the image of the product to be inspected.

[0036] like Figure 3 As shown, to facilitate the feature relationship discrimination network's differentiation between the two, the C-dimensional target defect feature vector needs to be copied and expanded spatially to have the same spatial size H×W×C as the target fused defect feature map, resulting in the target global defect feature map. Then, the target global defect feature map and the target fused defect feature map are concatenated along the channel dimension to form a fused feature map of size H×W×2C. The defect occurrence probability of each block in the target fused defect feature map can then be obtained through the feature matching results between the target fused defect feature map and the target global defect feature map.

[0037] In this application, the feature relation discrimination network is a self-built network, which consists of an Atrous Spatial Pyramid Pooling (ASPP) layer, a Squeeze-and-Excitation (SE) channel attention layer, a Convolutional Block Attention Module (CBAM) spatial attention layer, and a multi-head relation discrimination layer connected in sequence. The ASPP layer is used to extract multi-scale features from the input feature map. The SE channel attention layer is used to enhance the feature channels of important defects and suppress irrelevant background channels. The CBAM spatial attention layer is used to spatially weight the defect feature regions, focusing on the specific spatial location of the defects and suppressing image features of background regions.

[0038] like Figure 3 As shown, the multi-head relationship discrimination layer can use a head convolutional structure composed of multiple sets of convolutional layers (Conv) and activation layers (ReLU) to learn the feature matching logic of defect types from three independent dimensions: morphology, scale, and semantics. Then, the matching results of these dimensions are fused, and the probability of defect occurrence of each block of the fused defect feature map is output position by position in the H×W feature dimension using a convolution + sigmoid layer to obtain the defect score map.

[0039] S5. Based on each defect score map, determine the defect type of each block on the target feature map and generate a target defect distribution feature map; obtain the deviation score map based on the cosine similarity between each block of the target feature map and the defect-free feature map, and determine the abnormal blocks in the target feature map based on the deviation score map to generate an abnormal distribution feature map. In one possible implementation, based on each defect score map, the defect type of each block on the target feature map is determined, and a target defect distribution feature map is generated, including: Based on the defect score maps, the probability of each defect type occurring in each block of the target feature map is predicted by the classification head of the detection head network; The NMS algorithm is used to obtain the anchor boxes of each defect type on the target feature map through the regression head of the detection head network. The anchor boxes are then drawn on the target feature map to obtain the target defect distribution feature map.

[0040] In this embodiment, the detection head network can use the detection head portion of the existing YOLO series object detection networks. For a region of the target feature map corresponding to a defect type, the regression head of the detection head network, combined with the defect score map, will predict multiple highly overlapping anchor boxes (bounding boxes used to locate the defect region in the image). The NMS algorithm can determine this by ranking by confidence and using an IoU threshold (e.g., 0.5, which can be adjusted according to requirements), retaining only the boxes with the highest confidence and deleting highly overlapping low-confidence boxes, so that each defect corresponds to only one accurate predicted box / anchor box, avoiding redundancy and confusion in the detection results. This application uses cosine similarity to obtain the deviation score map instead of SSIM because SSIM is severely affected by factors such as light and temperature and humidity, which will cause large fluctuations in the deviation score of normal areas outside the defect area. However, this application aims to analyze geometric defects (scratches, stains, bubbles, dents, and chipped edges, etc.) in various areas of the display screen, so cosine similarity is more appropriate.

[0041] In one possible implementation, anomaly blocks in the target feature map are determined based on the deviation score map, and anomaly distribution feature map is generated, including: Based on the deviation score map, the deviation score corresponding to each block on the target feature map is obtained, and the blocks with deviation scores greater than the preset value are marked as abnormal blocks. Connectivity analysis is performed on adjacent abnormal blocks to obtain at least one anchor frame composed of abnormal blocks in the same connected component. Each anchor frame is drawn on the target feature map to obtain the abnormal distribution feature map.

[0042] In the embodiments of this application, such as Figure 2 As shown, the process of obtaining the anomaly distribution feature map is used to further determine whether the detection location of the preliminary defect detection result of the target defect distribution feature map is accurate, reducing misjudgments caused by insufficient defect feature map samples. This application uses a dual-channel joint judgment method of anomaly distribution feature map and defect score map, which can effectively utilize a small number of samples to accurately locate and identify the location and type of each defect area on the product to be inspected.

[0043] S6. Based on the anomaly distribution feature map and the defect distribution feature map of each target, obtain the final defect detection result of the target feature map.

[0044] In one possible implementation, the final defect detection result of the target feature map is obtained based on the anomaly distribution feature map and each defect score map, including: Each anchor frame in the abnormal distribution feature map that represents the distribution of abnormal blocks is taken as a reverse anchor frame, and the anchor frame in the target defect distribution feature map that represents the distribution of defect blocks is taken as a forward anchor frame. Each forward anchor box and reverse anchor box is mapped to the same target feature map. The forward anchor box is labeled with the defect type and the average defect score of the defect type corresponding to each block in the forward anchor box. The reverse anchor box is labeled with the anomaly label and the average deviation score of each block in the reverse anchor box. Based on the NMS algorithm, each positive anchor frame corresponding to the same defect type is deduplicated, and each negative anchor frame is deduplicated to obtain at least one target positive anchor frame and one target negative anchor frame. When a target block in the target feature map is simultaneously covered by at least one positive anchor frame and one negative anchor frame, the defect type of the positive anchor frame with the highest average defect score covering the target block is taken as the defect type of the target block. Connectivity analysis is performed on target blocks of the same defect type. Target blocks within the same connected region are connected and their bounding boxes are labeled to obtain the final defect detection result of the target feature map.

[0045] In this embodiment, any block in the target feature map may be covered by multiple positive anchor frames of different defect types. Therefore, when a block is covered by multiple positive anchor frames and at least one reverse anchor frame simultaneously, it indicates that the block has a defect. However, it is necessary to determine the defect type. Therefore, the defect type determination result of the target positive anchor frame with the highest average defect score is selected as the defect type of the block. In addition, the role of the reverse anchor frame is to determine whether a block may be abnormal. It can be used to further determine whether the selection of the positive anchor frame is biased. Therefore, whether a block has a defect needs to be covered by both the reverse and positive anchor frames.

[0046] In addition, since the types of images of various defect types collected may not be complete, and new defect types may appear on the display screen during the production process, when an area is not covered by a positive anchor frame but is covered by a negative anchor frame, the area can be automatically marked as "abnormal to be determined" and the image of the negative anchor frame corresponding to the area can be entered into the database to facilitate further judgment on the defect type or whether there is a defect.

[0047] Figure 4 This is a schematic diagram of the structure of the screen defect detection system based on a collaborative neural network provided in an embodiment of this application, as shown below. Figure 4 As shown, the system includes at least: The feature map acquisition module is used to input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map. It also inputs the defect images of each type in the defect image dataset into the shared feature extraction network according to the defect type to obtain at least one set of target defect feature maps. The global feature extraction and fusion module is used to perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type. The spatial feature enhancement module is used to perform feature stitching processing on the target defect feature maps corresponding to each defect type in the spatial dimension to obtain at least one spatial feature defect sequence, and to perform feature enhancement processing on the target feature map through each spatial feature defect sequence to obtain the target fused defect feature map corresponding to each defect type. The defect score acquisition module is used to obtain multiple defect score maps corresponding to the target feature map based on the target fused defect feature map and the target defect feature vector corresponding to each defect type. The distribution feature acquisition module is used to determine the defect type of each block on the target feature map based on each defect score map, and generate the target defect distribution feature map; it also obtains the deviation score map based on the cosine similarity between each block of the target feature map and the defect-free feature map, and determines the abnormal blocks in the target feature map based on the deviation score map, generating the abnormal distribution feature map. The defect detection result acquisition module is used to obtain the final defect detection result of the target feature map based on the abnormal distribution feature map and the target defect distribution feature map.

[0048] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0049] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0050] It is understood that the processor in the embodiments of this application can be a CPU (Central Processing Unit), or other general-purpose processors, DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0051] The method steps in this application embodiment can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, ROM (Read-only Memory), PROM (Programmable ROM), EPROM (Erasable PROM), EEPROM (Electrically Erasable EPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0052] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line DSL) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD (Solid State Disk)).

[0053] It is understood that the various numerical designations used in the embodiments of this application are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0054] Those skilled in the art will readily understand that the above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A screen defect detection method based on a collaborative neural network, characterized in that, include: S1. Input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map. Then, group the defect images of each type in the defect image dataset according to the defect type and input them into the shared feature extraction network to obtain at least one set of target defect feature maps. Each set of target defect feature maps corresponds to one type of defect. S2. Perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type; S3. In the spatial dimension, the feature maps of each target defect corresponding to each defect type are spliced ​​together to obtain at least one spatial feature defect sequence, and the target feature map is enhanced by each spatial feature defect sequence to obtain the target fusion defect feature map corresponding to each defect type. S4. Based on the target fusion defect feature map and target defect feature vector corresponding to each defect type, obtain multiple defect score maps corresponding to the target feature map. Each defect score map is used to characterize the occurrence probability of a defect type in each block of the target feature map. S5. Based on each of the defect score maps, determine the defect type of each block on the target feature map, and generate a target defect distribution feature map; Based on the cosine similarity between each block of the target feature map and the defect-free feature map, a deviation score map is obtained, and based on the deviation score map, abnormal blocks in the target feature map are determined to generate an abnormal distribution feature map. S6. Based on the abnormal distribution feature map and each of the target defect distribution feature maps, obtain the final defect detection result of the target feature map.

2. The screen defect detection method based on a collaborative neural network according to claim 1, characterized in that, Global feature extraction and fusion operations are performed on each target defect feature map within each of the aforementioned target defect feature map sets, including: Average pooling is performed on each target defect feature map in the target defect feature map set from the spatial dimension to obtain at least one initial defect feature vector, and element-wise averaging is performed on each initial defect feature vector to obtain the target defect feature vector.

3. The screen defect detection method based on a collaborative neural network according to claim 1, characterized in that, The shared feature extraction network is the DINOv2 self-supervised network.

4. The screen defect detection method based on a cooperative neural network according to claim 1, characterized in that, The specific steps of S3 include: The target defect feature maps are expanded in the spatial dimension to obtain at least one feature sequence block of the same size. The feature sequence blocks are then spliced ​​together to obtain a spatial feature defect sequence. The target feature maps are then expanded in the spatial dimension to obtain a target spatial feature sequence. Based on the cross-attention mechanism and the spatial feature defect sequence, feature enhancement processing is performed on the target spatial feature sequence, and the dimension of the target spatial feature sequence after feature enhancement processing is restored to obtain a target fusion defect feature map, wherein the target fusion defect feature map has the same size as the target feature map.

5. The screen defect detection method based on a cooperative neural network according to claim 1, characterized in that, The step of obtaining multiple defect score maps corresponding to the target feature map based on the target fused defect feature map and target defect feature vector corresponding to each defect type includes: The target defect feature vector corresponding to the defect type is copied and expanded in the spatial dimension to obtain a target global defect feature map, wherein the target global defect feature map and the target fused defect feature map have the same size; The target global defect feature map and the target fused defect feature map are input into a feature relationship discrimination network, and the defect occurrence probability of each block of the target fused defect feature map is obtained based on the feature relationship discrimination network. Based on the distribution of the probability of defect occurrence in each block of the target fusion defect feature map, a defect score map is obtained. The defect score map is used to characterize the probability of occurrence of the defect type in each block of the target feature map corresponding to the target fusion defect feature map.

6. The screen defect detection method based on a cooperative neural network according to claim 1, characterized in that, The step of determining the defect type of each block on the target feature map based on each defect score map and generating a target defect distribution feature map includes: Based on the defect score maps, the probability of each defect type occurring in each block of the target feature map is predicted by the classification head of the detection head network; The NMS algorithm is used to obtain the anchor boxes of each defect type on the target feature map through the regression head of the detection head network, and the anchor boxes are drawn on the target feature map to obtain the target defect distribution feature map.

7. The screen defect detection method based on a cooperative neural network according to claim 1, characterized in that, The step of determining abnormal blocks in the target feature map based on the deviation score map and generating an abnormal distribution feature map includes: Based on the deviation score map, the deviation score corresponding to each block on the target feature map is obtained, and the blocks with deviation scores greater than a preset value are marked as abnormal blocks. Connectivity analysis is performed on adjacent abnormal blocks to obtain at least one anchor frame composed of abnormal blocks in the same connected component. Each anchor frame is drawn on the target feature map to obtain an abnormal distribution feature map.

8. The screen defect detection method based on a cooperative neural network according to claim 1, characterized in that, The step of obtaining the final defect detection result of the target feature map based on the abnormal distribution feature map and each of the target defect distribution feature maps includes: Each anchor frame in the abnormal distribution feature map used to characterize the distribution of abnormal blocks is taken as a reverse anchor frame, and the anchor frame in the target defect distribution feature map used to characterize the distribution of defect blocks is taken as a forward anchor frame. Based on the overlap of each forward anchor frame and reverse anchor frame, the final defect detection result of the target feature map is obtained.

9. The screen defect detection method based on a cooperative neural network according to claim 8, characterized in that, The step of obtaining the final defect detection result of the target feature map based on the overlap of each of the forward and reverse anchor frames includes: Each of the forward anchor frames and the reverse anchor frames is mapped to the same target feature map. The forward anchor frames are labeled with the defect type and the average defect score of the defect type corresponding to each block in the forward anchor frame. The reverse anchor frames are labeled with the anomaly label and the average deviation score of each block in the reverse anchor frame. Based on the NMS algorithm, the positive anchor frames corresponding to the same defect type are deduplicated, and the reverse anchor frames are deduplicated to obtain at least one target positive anchor frame and one target reverse anchor frame. When a target block in the target feature map is simultaneously covered by at least one positive anchor frame and one negative anchor frame, the defect type of the positive anchor frame that covers the target block with the highest average defect score is taken as the defect type of the target block. Connectivity analysis is performed on the target blocks of the same defect type, and the target blocks within the same connected region are connected and labeled with bounding boxes to obtain the final defect detection result of the target feature map.

10. A screen defect detection system based on a cooperative neural network, used to perform the method as described in any one of claims 1-9, characterized in that, include: The feature map acquisition module is used to input the product image to be detected and the defect-free product image into the shared feature extraction network to obtain the target feature map and the defect-free feature map, and to input the defect images of each type in the defect image dataset into the shared feature extraction network according to the defect type to obtain at least one set of target defect feature maps; The global feature extraction and fusion module is used to perform global feature extraction and fusion operations on each target defect feature map in each target defect feature map set to obtain the target defect feature vector corresponding to each defect type. The spatial feature enhancement module is used to perform feature stitching processing on the target defect feature maps corresponding to each defect type in the spatial dimension to obtain at least one spatial feature defect sequence, and to perform feature enhancement processing on the target feature map through each spatial feature defect sequence to obtain the target fused defect feature map corresponding to each defect type. The defect score acquisition module is used to acquire multiple defect score maps corresponding to the target feature map based on the target fused defect feature map and the target defect feature vector corresponding to each defect type. The distribution feature acquisition module is used to determine the defect type of each block on the target feature map based on each defect score map, and generate a target defect distribution feature map; Based on the cosine similarity between each block of the target feature map and the defect-free feature map, a deviation score map is obtained, and based on the deviation score map, abnormal blocks in the target feature map are determined to generate an abnormal distribution feature map. The defect detection result acquisition module is used to acquire the final defect detection result of the target feature map based on the abnormal distribution feature map and each of the target defect distribution feature maps.

Images