Photoelectric chip anomaly detection method and device based on reconstruction and computing equipment

Abstract

The invention discloses a reconstruction-based photoelectric chip anomaly detection method and device and computing equipment, and the method comprises the steps: carrying out the preliminary anomaly detection of an original image of a photoelectric chip through an AnmallyDINO model, obtaining an anomaly thermodynamic diagram, extracting a candidate anomaly region from the anomaly thermodynamic diagram, and dividing the candidate anomaly region into a plurality of superpixel regions; reconstructing the original image by using a Dinomal model to obtain a reconstructed image, and carrying out pixel-by-pixel difference calculation on the reconstructed image and the original image to obtain a reference difference chart; performing mask processing on each super-pixel region to generate a corresponding super-pixel difference image, and performing difference calculation on the super-pixel difference image and the reference difference image to obtain a final difference image; calculating an average gray value of each super-pixel region in the corresponding final difference image as an abnormal score of the super-pixel region; and according to the abnormal scores of all the super-pixel regions, screening out a real abnormal super-pixel region, thereby obtaining a final abnormal region. On the basis, the anomaly detection accuracy and detection efficiency of the photoelectric chip can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of anomaly detection, and particularly relates to a reconstruction-based optoelectronic chip anomaly detection method, a reconstruction-based optoelectronic chip anomaly detection device and a computing device. BACKGROUND

[0002] As a core component of modern electronic and lighting technology, optoelectronic chips play a key role in a variety of devices. They are widely used in high-performance electronic devices, various lighting fields and the like due to their high efficiency, long service life and strong reliability. Although the production efficiency of optoelectronic chips is high, the number of daily quality detection is limited to only seven hundred to eight hundred chips, and the anomalies on the optoelectronic chips are usually distributed in different areas, and there are multiple anomalies appearing at the same time, which makes it difficult to identify all the anomalies on the optoelectronic chips at the same time.

[0003] In recent years, a variety of methods have been developed for optoelectronic chip anomaly detection. Supervised deep learning models often need a large number of accurately labeled defect samples for training, which is difficult to meet the requirements of optoelectronic chip quality inspection in actual production, where negative samples are scarce and morphologies are diverse. Therefore, unsupervised methods for anomaly detection show significant advantages. However, some previous unsupervised methods also have two major problems: 1) only reconstruction in the original pixel space, which is easily disturbed by complex background textures and is not sensitive to subtle anomalies, resulting in low detection accuracy; 2) lack of comparison at the semantic feature level, making it difficult to locate the abnormal areas with subtle semantic deviations from the normal mode.

[0004] Therefore, a reconstruction-based optoelectronic chip anomaly detection method is needed to solve the problems in the above technical solutions. SUMMARY

[0005] To this end, the present application provides a reconstruction-based optoelectronic chip anomaly detection method and device to solve or at least alleviate the above problems.

[0006] According to an aspect of the present application, a reconstruction-based optoelectronic chip anomaly detection method is provided, which is executed in a computing device and includes: obtaining an original image of an optoelectronic chip to be detected; performing preliminary anomaly detection on the original image of the optoelectronic chip by using an AnomalyDINO model to obtain an anomaly heat map; extracting a high-response region from the anomaly heat map as a candidate anomaly region; performing superpixel segmentation on the candidate anomaly region by using an SICLE superpixel segmentation algorithm to divide the candidate anomaly region into a plurality of superpixel regions; reconstructing the original image of the optoelectronic chip by using a Dinomaly model to obtain a reconstructed image of the optoelectronic chip, and performing pixel-by-pixel difference calculation on the original image of the optoelectronic chip and the reconstructed image of the optoelectronic chip to generate a reference difference map; for each superpixel region, performing mask processing on the superpixel region on the original image of the optoelectronic chip to obtain a superpixel mask image, and generating a superpixel difference map corresponding to the superpixel region based on the superpixel mask image; performing difference calculation on the superpixel difference map corresponding to the superpixel region and the reference difference map to generate a final difference map corresponding to the superpixel region; calculating the average gray value of each superpixel region in the corresponding final difference map as the anomaly score of the superpixel region; and filtering one or more real anomaly superpixel regions from all superpixel regions according to the anomaly scores of all superpixel regions, and obtaining a final anomaly region based on the one or more real anomaly superpixel regions.

[0007] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, performing preliminary anomaly detection on the original image of the optoelectronic chip by using the AnomalyDINO model to obtain an anomaly heat map includes: performing feature extraction on the original image of the optoelectronic chip by using the AnomalyDINO model to extract a feature region; and calculating the similarity between the feature region and a corresponding sample feature region in a memory bank to obtain the anomaly heat map.

[0008] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, extracting a high-response region from the anomaly heat map as a candidate anomaly region includes: performing normalization processing on the anomaly heat map to adjust the size of the anomaly heat map to be matched with the size of the original image of the optoelectronic chip to obtain a normalized heat map; performing binary segmentation on the normalized heat map based on a preset threshold to generate an initial mask image; and performing inflation and corrosion operations on the initial mask image to extract a high-response region as a candidate anomaly region.

[0009] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, the Dinomaly model comprises an encoder, a bottleneck layer, a decoder and a reconstruction CNN network coupled in sequence; the Dinomaly model is used to reconstruct the original image of the optoelectronic chip to obtain a reconstructed image of the optoelectronic chip, comprising: the encoder is used to extract features from the original image of the optoelectronic chip to obtain encoded features; the bottleneck layer is used to perform nonlinear transformation on the encoded features to obtain bottleneck features; the decoder is used to decode and reconstruct the bottleneck features based on a plurality of Transformer modules to obtain reconstructed features, and the last layer features are extracted from the reconstructed features and reshaped into a spatial feature map; and the reconstruction CNN network is used to restore the spatial feature map step by step to generate the reconstructed image of the optoelectronic chip.

[0010] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, the original image of the optoelectronic chip and the reconstructed image of the optoelectronic chip are subjected to pixel-by-pixel difference calculation to generate a reference difference map, comprising: for each original pixel in the original image of the optoelectronic chip, the absolute difference of the gray value between the original pixel and the corresponding reconstructed pixel in the reconstructed image of the optoelectronic chip is calculated, and the absolute difference of the gray value is normalized to a standard gray range to generate the reference difference map.

[0011] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, the superpixel difference map corresponding to the superpixel region is generated based on the superpixel mask image, comprising: the Dinomaly model is used to reconstruct the superpixel mask image to obtain a superpixel reconstructed image; and the superpixel mask image and the superpixel reconstructed image are subjected to pixel-by-pixel difference calculation to generate the superpixel difference map corresponding to the superpixel region.

[0012] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, the superpixel difference map corresponding to the superpixel region and the reference difference map are subjected to difference calculation to generate the final difference map corresponding to the superpixel region, comprising: the absolute difference of the gray value between the superpixel difference map and the reference difference map is calculated, and the absolute difference of the gray value is normalized to a standard gray range to generate the final difference map corresponding to the superpixel region.

[0013] Optionally, in the reconstruction-based optoelectronic chip anomaly detection method according to the present application, further comprising: statistically analyzing the distribution characteristics of the anomaly scores of all superpixel regions to obtain anomaly score statistical characteristics of all superpixel regions; wherein the higher the anomaly score of a superpixel region is, the greater the possibility that the superpixel region in the candidate abnormal region is misjudged as abnormal is.

[0014] Optionally, in the reconstruction-based photoelectric chip anomaly detection method according to the present application, the filtering of one or more real abnormal superpixel regions from all superpixel regions according to the anomaly scores of all superpixel regions comprises: determining a classification threshold according to the anomaly score statistical features of all superpixel regions; and determining one or more superpixel regions with anomaly scores lower than the classification threshold as one or more real abnormal superpixel regions.

[0015] According to an aspect of the present application, a reconstruction-based photoelectric chip anomaly detection device is provided, which is deployed in a computing device and is adapted to execute the method as described above, and the device comprises: An acquisition unit adapted to acquire a photoelectric chip original image to be detected; A preliminary detection unit adapted to perform preliminary anomaly detection on the photoelectric chip original image by using an AnomalyDINO model to obtain an anomaly heat map; An extraction unit adapted to extract a high-response region from the anomaly heat map as a candidate abnormal region; A superpixel segmentation unit adapted to perform superpixel segmentation on the candidate abnormal region by using an SICLE superpixel segmentation algorithm to divide the candidate abnormal region into a plurality of superpixel regions; A reconstruction difference unit adapted to perform reconstruction on the photoelectric chip original image by using a Dinomaly model to obtain a photoelectric chip reconstructed image, and perform pixel-by-pixel difference calculation on the photoelectric chip original image and the photoelectric chip reconstructed image to generate a reference difference map; A superpixel difference unit adapted to, for each superpixel region, perform mask processing on the superpixel region on the photoelectric chip original image to obtain a superpixel mask image, and generate a superpixel difference map corresponding to the superpixel region based on the superpixel mask image; A secondary difference unit adapted to perform difference calculation on the superpixel difference map corresponding to the superpixel region and the reference difference map to generate a final difference map corresponding to the superpixel region; An anomaly scoring unit adapted to calculate the average gray value of each superpixel region in the corresponding final difference map as the anomaly score of the superpixel region; A filtering unit adapted to filter one or more real abnormal superpixel regions from all superpixel regions according to the anomaly scores of all superpixel regions, and obtain a final abnormal region based on the one or more real abnormal superpixel regions.

[0016] According to an aspect of the present application, there is provided a computing device comprising: at least one processor; a memory storing program instructions configured to be executed by the at least one processor, the program instructions comprising instructions for performing the reconstruction-based optoelectronic chip anomaly detection method as described above.

[0017] According to an aspect of the present application, there is provided a computer program product comprising computer program instructions which, when executed by a processor, implement the method as described above.

[0018] According to an aspect of the present application, there is provided a readable storage medium storing program instructions which, when read and executed by a computing device, cause the computing device to perform the reconstruction-based optoelectronic chip anomaly detection method as described above.

[0019] According to the technical solution of the present application, a reconstruction-based optoelectronic chip anomaly detection method and device are provided. The AnomalyDINO model is used to perform preliminary anomaly detection on the original image of the optoelectronic chip, obtain an anomaly heat map, and extract a high-response region as a candidate anomaly region from the anomaly heat map. Then, the candidate anomaly region is divided into a plurality of superpixel regions. Next, the Dinomaly model is used to reconstruct the original image of the optoelectronic chip to obtain a reconstructed image of the optoelectronic chip, and perform pixel-by-pixel difference calculation on the original image of the optoelectronic chip and the reference difference map to obtain a reference difference map. Each superpixel region is subjected to mask processing to generate a corresponding superpixel difference map. Then, the superpixel difference map is subjected to difference calculation with the reference difference map to obtain a final difference map. The average gray value of each superpixel region in the corresponding final difference map is calculated as the anomaly score of the superpixel region. One or more real anomaly superpixel regions are selected according to the anomaly scores of all superpixel regions, and thus the final anomaly region is obtained. According to the technical solution of the present application, the AnomalyDINO model can quickly generate an anomaly heat map without training, thereby significantly improving the efficiency and generalization ability of preliminary anomaly detection on the optoelectronic chip. The SICLE superpixel segmentation algorithm is used to perform fine segmentation only on the suspected anomaly region, thereby avoiding unnecessary calculation in the normal region, improving the processing efficiency while ensuring the boundary accuracy. The Dinomaly model can effectively eliminate the misjudgment in the preliminary anomaly detection on the optoelectronic chip by using the reconstruction difference, thereby significantly reducing the misjudgment rate of anomaly detection and ensuring the anomaly detection accuracy in the case of a small number of negative samples. Based on this, the present application can significantly improve the anomaly detection accuracy and efficiency of the optoelectronic chip while ensuring a high recall rate, thereby meeting the requirements of industrial detection.

[0020] The above description is only a summary of the technical solutions of the present application. In order to enable the technical means of the present application to be more clearly understood, and to be implemented according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0021] To the accomplishment of the foregoing and related ends, certain illustrative aspects are described herein in connection with the following description and the annexed drawings. These aspects are indicative of various ways in which the principles disclosed herein can be practiced and all aspects and equivalents thereof are intended to be within the scope of the claimed subject matter. The foregoing and other objects, features, and advantages of the present application will become more fully apparent from the following detailed description, appended claims, and accompanying drawings in which like reference numerals identify like components and elements throughout the several views. The detailed description implements the best modes presently contemplated for carrying out the present application.

[0022] Figure 1 A schematic diagram of a computing device 100 is shown according to an embodiment of the present application; Figure 2 A flowchart of a reconstruction-based optoelectronic chip anomaly detection method 200 is shown according to an embodiment of the present application; Figure 3 A schematic diagram of an optoelectronic chip raw image is shown according to an embodiment of the present application; Figure 4 A schematic diagram of the principle of preliminary anomaly detection of an optoelectronic chip raw image using an AnomalyDINO model in an embodiment of the present application is shown; Figure 5 Exemplary schematic diagrams of an optoelectronic chip raw image and its corresponding anomaly heat map and candidate anomaly region in some embodiments of the present application are shown; Figure 6 Exemplary schematic diagrams of the superpixel segmentation effect of a candidate anomaly region in some embodiments of the present application are shown; Figure 7 A schematic diagram of the principle of reconstruction of an optoelectronic chip raw image using a Dinomaly model in some embodiments of the present application is shown; Figure 8 Exemplary schematic diagrams of an optoelectronic chip raw image and its corresponding optoelectronic chip reconstructed image and reference difference map in some embodiments of the present application are shown; Figure 9 Exemplary schematic diagrams of an optoelectronic chip raw image, superpixel mask image, optoelectronic chip reconstructed image, superpixel difference map, final difference map, screened anomaly region, and final anomaly region in some embodiments of the present application are shown; Figure 10A schematic diagram of a reconstruction-based optoelectronic chip anomaly detection device 1000 according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it is understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present application can be more thoroughly understood, and the scope of the present application can be accurately conveyed to those skilled in the art.

[0024] To solve the problems of existing optoelectronic chip anomaly detection schemes, the present application proposes a reconstruction-based optoelectronic chip anomaly detection method. The AnomalyDINO model is used to quickly generate an abnormal heat map without training, thereby significantly improving the efficiency and generalization ability of preliminary anomaly detection of optoelectronic chips. The SICLE superpixel segmentation algorithm is used to finely segment only the suspected abnormal area, avoiding unnecessary calculation in the normal area, improving the processing efficiency while ensuring the boundary accuracy. The Dinomaly model is used to effectively eliminate the misjudgment of preliminary anomaly detection of optoelectronic chips by reconstructing the difference, significantly reducing the misjudgment rate of anomaly detection, and ensuring the anomaly detection accuracy in the case of few negative samples. Based on this, the anomaly detection accuracy and detection efficiency of optoelectronic chips can be greatly improved while ensuring high recall rate.

[0025] To facilitate understanding of the technical solutions of the present application, the terms involved in the embodiments of the present application are first described below.

[0026] DINOv2: a self-supervised visual base model that can extract powerful visual features from images and is suitable for a variety of downstream tasks.

[0027] AnomalyDINO: a DINOv2 visual Transformer-based training-free anomaly detection method that realizes efficient abnormal region positioning through patch similarity calculation.

[0028] SICLE: an abbreviation of Superpixels through Iterative CLEarcutting, an object-aware superpixel segmentation method that supports multi-scale segmentation and object-guided segmentation strategies.

[0029] Dinomaly: a reconstruction-based anomaly detection method based on the Transformer architecture, which realizes image reconstruction through an encoder-decoder structure and detects anomalies using reconstruction differences.

[0030] Embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of a computing device 100 according to an embodiment of the present invention is shown. Figure 1 As shown, in a basic configuration, computing device 100 includes at least one processing unit 102 and system memory 104. According to one aspect, depending on the configuration and type of the computing device, the processing unit 102 may be implemented as a processor. System memory 104 includes, but is not limited to, volatile memory (e.g., random access memory), non-volatile memory (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 104 includes an operating system 105.

[0032] According to one aspect, operating system 105 is, for example, suitable for controlling the operation of computing device 100. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other applications, and are not limited to any particular application or system. Figure 1 The basic configuration is illustrated by the components within the dashed lines. According to one aspect, the computing device 100 has additional features or functions. For example, according to one aspect, the computing device 100 includes additional data storage devices (removable and / or non-removable), such as disks, optical discs, or magnetic tapes. This additional storage... Figure 1 The middle part is shown by removable storage device 109 and non-removable storage device 110.

[0033] As stated above, according to one aspect, program module 103 is stored in system memory 104. According to one aspect, program module 103 may include one or more applications. The present invention does not limit the type of application; for example, applications may include: email and contact applications, word processing applications, spreadsheet applications, database applications, slideshow applications, drawing or computer-aided applications, web browser applications, etc.

[0034] According to one aspect, program module 103 may include a plurality of program instructions adapted to execute the reconstructed optoelectronic chip anomaly detection method 200 of the present invention, such that computing device 100 is configured to execute the reconstructed optoelectronic chip anomaly detection method 200 of the present invention.

[0035] According to one aspect, program module 103 may include a reconstructed optoelectronic chip anomaly detection device 1000, which may be configured to perform the reconstructed optoelectronic chip anomaly detection method 200 of the present invention.

[0036] According to an aspect, examples can be practiced with electronic circuitry integrated on a single integrated circuit chip, with separate electronic elements interconnected off the chip, with a microprocessor, or with any other physical configuration. For example, examples can be implemented via a general-purpose computer, a special-purpose computer, a microprocessor, or a state machine. Examples can be implemented using any of a wide variety of Figure 1 microprocessors of one or more processors of a multi-processor core, micro-controllers, digital signal processors, dedicated circuits, and hardware required for supporting a wireless interface card to a communication network, and so on. Examples can also be implemented using any of a wide variety of alternative logical components, circuits, and / or physical structures.

[0037] According to an aspect, the computing device 100 can also have one or more input device(s) 112 such as a keyboard, a mouse, a pen, a microphone, a touch input device, etc. One or more output device(s) 114 such as a display, speakers, a printer, etc. can also be included. The aforementioned devices are examples and others can also be used. The computing device 100 can include one or more communication connections 116 allowing communications with other computing devices 118. Examples of suitable communication connections 116 include, but are not limited to: RF transmitter, receiver, and / or transceiver circuitry; universal serial bus (USB), parallel, and / or serial ports.

[0038] The term computer readable media as used herein includes computer storage media. Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, or program modules 103. The system memory 104, the removable storage device 109, and the non-removable storage device 110 are all computer storage media examples (i.e., memory storage.) Computer storage media can include Random Access Memory (RAM), Read-Only Memory (ROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information and which can be accessed by the computing device 100. According to an aspect, any such computer storage media can be part of the computing device 100. Computer storage media does not include a carrier wave or other propagated data signal.

[0039] According to an aspect, communication media typically embodies computer readable instructions, data structures, program modules 103, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. According to an aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0040] In embodiments according to the present application, the computing device 100 is configured to perform the reconstruction-based optoelectronic chip anomaly detection method 200 of the present application. The computing device 100 includes one or more processors, and one or more readable storage media storing program instructions which, when configured to be executed by the one or more processors, cause the computing device to perform the reconstruction-based optoelectronic chip anomaly detection method 200 in embodiments of the present application.

[0041] Figure 2 A flowchart of a reconstruction-based optoelectronic chip anomaly detection method 200 according to an embodiment of the present application is shown. The reconstruction-based optoelectronic chip anomaly detection method 200 can be performed in a computing device, such as the aforementioned computing device 100.

[0042] In embodiments of the present application, the computing device 100 for performing the reconstruction-based optoelectronic chip anomaly detection method 200 of the present application can be a terminal or a server.

[0043] AsFigure 2 As shown, the reconstructed optoelectronic chip anomaly detection method 200 includes the following steps 210-290.

[0044] Step 210, the computing device 100 can acquire an optoelectronic chip original image to be detected.

[0045] Figure 3 An exemplary schematic diagram of an optoelectronic chip original image is shown. As shown in the figure, Figure 3 As shown, the optoelectronic chip original image includes a light-emitting region, an epitaxial region, and an electrode lead region.

[0046] Step 220, the computing device 100 can use the AnomalyDINO model to perform preliminary anomaly detection on the optoelectronic chip original image to obtain an anomaly heat map.

[0047] Figure 4 An exemplary schematic diagram of using the AnomalyDINO model to perform preliminary anomaly detection on the optoelectronic chip original image is shown. As shown in the figure, Figure 4 As shown in step 220, the AnomalyDINO model can be used to extract features (DINOv2 feature extraction) from the optoelectronic chip original image to extract feature regions. It should be noted that the AnomalyDINO model is based on the DINOv2 visual Transformer architecture and can extract deep semantic features of an image without training. Further, the similarity between the feature region and the corresponding sample feature region in the memory (i.e., the sample feature region extracted from the optoelectronic chip sample image using the AnomalyDINO model) can be calculated to obtain an anomaly heat map. Based on the anomaly heat map obtained in this way, suspected abnormal regions (candidate abnormal regions) can be quickly located, providing an initial detection result with high recall rate for subsequent processing, and significantly improving the overall detection efficiency.

[0048] Since the AnomalyDINO model can quickly generate an anomaly heat map without training, it can significantly improve the efficiency and generalization ability of preliminary anomaly detection of optoelectronic chips.

[0049] Step 230, the computing device 100 can extract a high-response region from the anomaly heat map as a candidate abnormal region (i.e., a suspected abnormal region to be processed). Specifically, the computing device 100 can use an adaptive threshold segmentation algorithm to extract a high-response region from the anomaly heat map as a candidate abnormal region.

[0050] Figure 5 An exemplary schematic diagram of an optoelectronic chip original image, its corresponding anomaly heat map, and a candidate abnormal region is shown.

[0051] In some embodiments, in step 230, the abnormal heatmap can first be normalized to adjust its size to match the original image size of the optoelectronic chip, resulting in a normalized heatmap. Subsequently, the normalized heatmap can be binarized and segmented based on a preset threshold to generate an initial mask image. Then, dilation and erosion operations can be performed on the initial mask image to extract high-response regions as candidate abnormal regions. In one specific embodiment, the initial mask image can be dilated and eroded a specified number of times to connect adjacent abnormal regions and eliminate noise interference, ultimately forming complete candidate abnormal regions. Based on this, high-response regions in the heatmap can be effectively extracted, providing accurate target input for subsequent fine segmentation, while morphological processing can optimize the quality of region boundaries.

[0052] Step 240: The computing device 100 can use the SICLE superpixel segmentation algorithm to perform superpixel segmentation on the candidate anomaly region, so as to divide the candidate anomaly region into multiple superpixel regions.

[0053] Figure 6 An exemplary schematic diagram of the superpixel segmentation effect on candidate anomalous regions according to some embodiments of the present invention is shown. Figure 6 As shown, based on step 240, the candidate anomaly region can be divided into multiple superpixel regions with clear boundaries and uniform size. It should be noted that the SICLE superpixel segmentation algorithm fully considers the boundary information of objects in the image, ensuring that the segmentation result is highly consistent with the boundaries of real objects. This targeted segmentation strategy effectively avoids unnecessary segmentation calculations in normal regions, ensuring accurate division of the boundaries of anomaly regions while significantly improving overall processing efficiency.

[0054] It should also be noted that, according to the embodiments of the present invention, based on the preliminary anomaly detection, the SICLE superpixel segmentation algorithm is used to perform fine segmentation only on the candidate anomaly region (suspected anomaly region), avoiding unnecessary calculations in the normal region, and improving processing efficiency while ensuring boundary accuracy.

[0055] In step 250, the computing device 100 can reconstruct the original image of the photoelectric chip using a Dinomaly model to obtain a reconstructed image of the photoelectric chip. Then, a pixel-by-pixel difference calculation (pixel-by-pixel absolute difference calculation) can be performed between the original image of the photoelectric chip and the reconstructed image to generate a reference difference map. It should be understood that the reference difference map generated here is, based on the original image of the photoelectric chip, the reference difference map corresponding to the original image of the photoelectric chip generated using the reconstruction and difference methods in step 250.

[0056] In the embodiment of the present application, the Dinomaly model used in step 250 is obtained after training on the basis of a pre-trained Dinomaly model.

[0057] In some embodiments, before reconstructing the optoelectronic chip original image using the Dinomaly model, a normal sample data set can be constructed in advance, which can include a plurality of optoelectronic chip normal sample images. And based on each optoelectronic chip normal sample image in the normal sample data set, the pre-trained Dinomaly model is trained to learn the normal feature distribution of the optoelectronic chip surface.

[0058] It should be noted that in the process of reconstructing the optoelectronic chip original image using the Dinomaly model, the Dinomaly model can accurately restore the appearance features of the normal region in the optoelectronic chip original image, and for the abnormal region not seen in the training process, there will be obvious reconstruction error.

[0059] Figure 7 A schematic diagram showing the principle of reconstructing the optoelectronic chip original image using the Dinomaly model according to some embodiments of the present application is shown.

[0060] As shown in Figure 7 , the Dinomaly model is based on the encoder-decoder architecture of the Transformer. Specifically, the Dinomaly model includes an encoder, a bottleneck layer, a decoder, and a reconstruction CNN network coupled in sequence.

[0061] In step 250, the specific process of reconstructing the optoelectronic chip original image using the Dinomaly model is as follows: first, the optoelectronic chip original image can be input into the encoder of the Dinomaly model, and the encoder can extract features from the optoelectronic chip original image to convert the optoelectronic chip original image into a high-dimensional token feature representation, obtaining the encoded features. Subsequently, the encoded features are input into the bottleneck layer, and the bottleneck layer performs nonlinear transformation on the encoded features to obtain the bottleneck features. It should be noted that the bottleneck layer uses a multi-layer perceptron structure to expand and compress the features, which can enhance the expression ability of the features while keeping the feature dimension unchanged. Next, the bottleneck features are input into the decoder, and the decoder decodes and reconstructs the bottleneck features based on multiple Transformer modules to gradually recover the semantic information of the image, obtaining the reconstructed features. Then, the decoder can extract the last layer features from the reconstructed features and reshape the last layer features into a spatial feature map, and input the spatial feature map into the reconstruction CNN network. Finally, the abstract spatial feature map is gradually restored through a series of convolutional layers and up-sampling layers of the reconstruction CNN network, and finally the optoelectronic chip reconstructed image with the same size and format as the input optoelectronic chip original image is generated.

[0062] It should be understood that the structure of the pre-trained Dinomaly model is similar to that of the trained Dinomaly model. In the training process of the pre-trained Dinomaly model based on each optoelectronic chip normal sample image in the normal sample dataset, the pre-trained encoder in the pre-trained Dinomaly model can be used to extract features of the optoelectronic chip normal sample image, and normal sample coding features can be extracted. Then, the pre-trained bottleneck layer is used to perform nonlinear transformation on the normal sample coding features to obtain normal sample bottleneck features. Next, the pre-trained decoder is used to decode and reconstruct the normal sample bottleneck features to obtain normal sample reconstruction features, and the last layer features of the normal sample are extracted from the normal sample reconstruction features, the last layer features of the normal sample are reshaped into normal sample spatial feature maps, and the normal sample spatial feature maps are gradually restored by the pre-trained reconstruction CNN network to generate normal sample reconstruction images. It should be noted that a loose reconstruction loss is used in the training process, and the model focuses on learning the normal mode by feature grouping and ignoring easy reconstruction areas.

[0063] Figure 8 An exemplary schematic diagram of an optoelectronic chip original image, its corresponding optoelectronic chip reconstruction image and reference difference image is shown according to some embodiments of the present application.

[0064] For the sake of distinction, each pixel in the optoelectronic chip original image is referred to as an "original pixel", and each pixel in the optoelectronic chip reconstruction image is referred to as a "reconstruction pixel". In some embodiments, in step 250, the specific way of performing pixel-by-pixel difference calculation between the optoelectronic chip original image and the optoelectronic chip reconstruction image is as follows: for each original pixel in the optoelectronic chip original image, the absolute difference value of the gray value of the original pixel and the corresponding reconstruction pixel in the optoelectronic chip reconstruction image is calculated, and the absolute difference value of the gray value is normalized to a standard gray range to generate a reference difference image. The reference difference image provides important basic data for subsequent further fine analysis.

[0065] In step 260, the computing device 100 can perform mask processing on each superpixel region on the optoelectronic chip original image (the other regions in the optoelectronic chip original image remain unchanged), and the superpixel mask image can be obtained after the mask processing on the superpixel region. Further, the superpixel difference image corresponding to the superpixel region (i.e., the superpixel difference image corresponding to the superpixel mask image) can be generated based on the superpixel mask image by using the similar reconstruction and difference method in step 250.

[0066] Specifically, in step 260, the specific process of generating the superpixel difference map corresponding to the superpixel region based on the superpixel mask image is as follows: first, the superpixel mask image can be reconstructed by using the Dinomaly model to obtain a superpixel reconstruction image. The reconstruction principle is similar to that in step 250 described above, and will not be repeated here. Then, the superpixel mask image and the superpixel reconstruction image can be calculated by pixel-by-pixel difference (pixel-by-pixel absolute difference calculation) to generate the superpixel difference map corresponding to the superpixel region (i.e., the superpixel difference map corresponding to the superpixel mask image). By repeatedly performing the above process on the next superpixel region in turn, until all superpixel regions are processed, the superpixel difference map corresponding to each superpixel region can be finally obtained.

[0067] Step 270, the computing device 100 can perform difference calculation on the superpixel difference map corresponding to the superpixel region and the reference difference map for each superpixel region to generate the final difference map corresponding to the superpixel region.

[0068] In some embodiments, in step 270, the absolute difference of the gray value of the superpixel difference map corresponding to the superpixel region and the reference difference map can be calculated, and the absolute difference of the gray value is normalized to a standard gray range to generate the final difference map corresponding to the superpixel region.

[0069] Step 280, the computing device 100 can calculate the average gray value of each superpixel region in the corresponding final difference map as the anomaly score of the superpixel region.

[0070] It should be noted that in the embodiments of the present application, the higher the anomaly score of the superpixel region, the greater the possibility that the superpixel region in the candidate abnormal region is misjudged as abnormal in the preliminary anomaly detection process.

[0071] Step 290, the computing device 100 can filter one or more real abnormal superpixel regions from all superpixel regions according to the anomaly scores of all superpixel regions, and the final abnormal region can be obtained based on the one or more real abnormal superpixel regions.

[0072] In some embodiments, before step 290 is performed, statistical analysis can be performed on the distribution characteristics of the anomaly scores of all superpixel regions to obtain the anomaly score statistical characteristics of all superpixel regions. Then, in step 290, the classification threshold can be dynamically determined according to the anomaly score statistical characteristics of all superpixel regions, and one or more superpixel regions with an anomaly score lower than the classification threshold are determined as one or more real abnormal superpixel regions. Then, the final abnormal region can be obtained based on the one or more real abnormal superpixel regions. Based on this, the misjudgment of the AnomalyDINO model in the preliminary anomaly detection of the optoelectronic chip can be effectively eliminated, the misjudgment rate of the anomaly detection is significantly reduced, and the anomaly detection accuracy is ensured in the case of a small number of negative samples.

[0073] In some embodiments, after one or more real abnormal superpixel regions are screened out from all superpixel regions in step 290, the screening abnormal region can be directly obtained based on the one or more real abnormal superpixel regions. On this basis, the screening abnormal region can be refined to obtain the final abnormal region. Figure 9 Exemplary schematic diagrams of the optoelectronic chip original image, the superpixel mask image, the optoelectronic chip reconstructed image, the superpixel difference map, the final difference map, the screening abnormal region, and the final abnormal region according to some embodiments of the present application are shown.

[0074] Figure 10 A schematic diagram of a reconstruction-based optoelectronic chip anomaly detection device 1000 provided by an embodiment of the present application is shown. The reconstruction-based optoelectronic chip anomaly detection device 1000 can be deployed in a computing device 100, and the reconstruction-based optoelectronic chip anomaly detection device 1000 is configured to perform the reconstruction-based optoelectronic chip anomaly detection method 200 of the present application.

[0075] As shown in Figure 10 In an embodiment of the present application, the reconstruction-based optoelectronic chip anomaly detection device 1000 includes an acquisition unit 1100, a preliminary detection unit 1200, an extraction unit 1300, a superpixel segmentation unit 1400, a reconstructed difference unit 1500, a superpixel difference unit 1600, a secondary difference unit 1700, an anomaly score unit 1800, and a screening unit 1900, which are sequentially and communicatively connected.

[0076] The acquisition unit 1100 can acquire an original image of an optoelectronic chip to be detected.

[0077] The preliminary detection unit 1200 can use the AnomalyDINO model to perform preliminary anomaly detection on the original image of the optoelectronic chip to obtain an anomaly heat map.

[0078] The extraction unit 1300 can extract a high response region from the abnormal heat map as a candidate abnormal region.

[0079] The superpixel segmentation unit 1400 can perform superpixel segmentation on the candidate abnormal region by using a SICLE superpixel segmentation algorithm to divide the candidate abnormal region into a plurality of superpixel regions.

[0080] The reconstruction difference unit 1500 can perform reconstruction on the original photoelectric chip image by using a Dinomaly model to obtain a reconstructed photoelectric chip image, and perform pixel-by-pixel difference calculation on the original photoelectric chip image and the reconstructed photoelectric chip image to generate a reference difference map.

[0081] The superpixel difference unit 1600 can perform mask processing on the superpixel region on the original photoelectric chip image to obtain a superpixel mask image for each superpixel region, and generate a superpixel difference map corresponding to the superpixel region based on the superpixel mask image.

[0082] The secondary difference unit 1700 can perform difference calculation on the superpixel difference map corresponding to the superpixel region and the reference difference map to generate a final difference map corresponding to the superpixel region.

[0083] The anomaly scoring unit 1800 can calculate the average gray value of each superpixel region in the corresponding final difference map as the anomaly score of the superpixel region.

[0084] The screening unit 1900 can screen one or more real abnormal superpixel regions from all superpixel regions according to the anomaly scores of all superpixel regions, and obtain a final abnormal region based on the one or more real abnormal superpixel regions.

[0085] It should be pointed out that the acquisition unit 1100, the preliminary detection unit 1200, the extraction unit 1300, the superpixel segmentation unit 1400, the reconstruction difference unit 1500, the superpixel difference unit 1600, the secondary difference unit 1700, the anomaly scoring unit 1800, and the screening unit 1900 are respectively used to execute the aforementioned steps 210~290. Here, the specific execution logic of each unit can be referred to the description of steps 210~290 in the foregoing method 200, which will not be repeated here.

[0086] According to the anomaly detection method and device based on reconstruction of the photoelectric chip in the embodiment of the present application, the AnomalyDINO model is used to perform preliminary anomaly detection on the original image of the photoelectric chip, an abnormal heat map is obtained, and a high-response region is extracted from the abnormal heat map as a candidate abnormal region, and then the candidate abnormal region is divided into a plurality of super-pixel regions. Then, the Dinomaly model is used to reconstruct the original image of the photoelectric chip to obtain a reconstructed image of the photoelectric chip, and a reference difference map is obtained by performing pixel-by-pixel difference calculation on the original image of the photoelectric chip. Each super-pixel region is subjected to mask processing to generate a corresponding super-pixel difference map, and then the super-pixel difference map is subjected to difference calculation with the reference difference map to obtain a final difference map. The average gray value of each super-pixel region in the corresponding final difference map is calculated as the anomaly score of the super-pixel region, and one or more real abnormal super-pixel regions are selected according to the anomaly scores of all super-pixel regions, thereby obtaining a final abnormal region. According to the technical solution of the present application, the AnomalyDINO model can quickly generate an abnormal heat map without training, thereby significantly improving the efficiency and generalization ability of preliminary anomaly detection of the photoelectric chip. The SICLE super-pixel segmentation algorithm is used to perform fine segmentation only on the suspected abnormal region, thereby avoiding unnecessary calculation in the normal region, improving the processing efficiency while ensuring the boundary accuracy. The Dinomaly model can effectively eliminate the misjudgment in the preliminary anomaly detection of the photoelectric chip by using the reconstruction difference, thereby significantly reducing the misjudgment rate of anomaly detection and ensuring the anomaly detection accuracy in the case of few negative samples. Therefore, the present application can significantly improve the anomaly detection accuracy and efficiency of the photoelectric chip while ensuring high recall rate, thereby meeting the industrial detection requirements.

[0087] The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of both. Thus, the methods and apparatus of the present application, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard disks, USB flash drives, floppy diskettes, CD-ROMs, ROMs, or any other machine-readable storage medium wherein, when the program code is loaded into an internal memory of the machine such as a computer, the machine becomes an apparatus for practicing the present application.

[0088] Where a program code is executed on a programmable computer, the mobile terminal generally includes a processor, a processor-readable storage medium (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store program code; the processor is configured to execute instructions in the program code stored in the memory to perform the anomaly detection method based on reconstruction of the photoelectric chip of the present application.

[0089] By way of example, and not limitation, readable media can include volatile and non-volatile, removable and non-removable media implemented in a method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer readable media is accessible by a computer. However, computer readable media is not comprised of propagated signals per se. Combinations of the above should also be included within the scope of readable media.

[0090] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0091] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to not obscure the understanding of this description.

[0092] Similarly, it is to be understood that the mechanical details of the example embodiments of the application can also include any of the mechanical details of the example embodiments of the application described above, or combinations of such details.

[0093] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in a device as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined as a module or further divided into multiple sub-modules.

[0094] Unless otherwise stated, the use of ordinal adjectives such as "first", "second", "third", etc. to describe a common object is merely intended to distinguish that particular object from another common object whose name is similarly qualified with "first", "second", "third", etc. in the reference of the description. Therefore, the use of "first" object and "second" object subsequently after the initial use of "first" and "second" is merely intended to distinguish one particular object from another particular object and is not intended to imply that the one particular object precedes the other particular object in time, spatial arrangement, ordering, or the like.

Claims

1. A reconstruction-based method for detecting anomalies in an optoelectronic chip, executed in a computing device, comprising: Acquire the original image of the optoelectronic chip to be inspected; Using the AnomalyDINO model, preliminary anomaly detection is performed on the original image of the optoelectronic chip to obtain an anomaly thermal map; High-response regions are extracted from the anomaly heatmap as candidate anomaly regions; The candidate anomaly region is divided into multiple superpixel regions by using the SICLE superpixel segmentation algorithm. The original image of the optoelectronic chip is reconstructed using the Dinomaly model to obtain a reconstructed image of the optoelectronic chip. Then, pixel-by-pixel difference calculation is performed between the original image of the optoelectronic chip and the reconstructed image of the optoelectronic chip to generate a reference difference map. For each superpixel region, a masking process is performed on the original image of the photoelectric chip to obtain a superpixel mask image, and a superpixel difference map corresponding to the superpixel region is generated based on the superpixel mask image. The difference map corresponding to the superpixel region is calculated by performing a difference calculation with the reference difference map to generate the final difference map corresponding to the superpixel region. Calculate the average gray value of each superpixel region in the corresponding final difference map, and use it as the anomaly score of the superpixel region; Based on the anomaly scores of all superpixel regions, one or more real anomalous superpixel regions are selected from all superpixel regions, and the final anomalous region is obtained based on the one or more real anomalous superpixel regions.

2. The method as described in claim 1, wherein, Using the AnomalyDINO model, preliminary anomaly detection is performed on the original image of the photoelectric chip to obtain an anomaly heatmap, including: The AnomalyDINO model is used to extract features from the original image of the optoelectronic chip to extract feature regions. Calculate the similarity between the feature region and the corresponding sample feature region in the memory bank to obtain an anomaly heatmap.

3. The method as described in claim 1 or 2, wherein, High-response regions are extracted from the anomaly heatmap as candidate anomaly regions, including: The abnormal heatmap is normalized to adjust its size to match the original image size of the optoelectronic chip, thus obtaining a normalized heatmap. Based on a preset threshold, the normalized heatmap is binarized and segmented to generate an initial mask image; The initial mask image is subjected to dilation and erosion operations to extract high-response regions as candidate anomalous regions.

4. The method according to any one of claims 1-3, wherein, The Dinomaly model comprises an encoder, a bottleneck layer, a decoder, and a reconstructed CNN network, which are coupled sequentially. The original image of the optoelectronic chip is reconstructed using the Dinomaly model to obtain a reconstructed image of the optoelectronic chip, including: The encoder is used to extract features from the original image of the photoelectric chip to obtain coded features; The bottleneck features are obtained by performing a nonlinear transformation on the encoded features through the bottleneck layer. The bottleneck features are decoded and reconstructed using the decoder based on the multi-layer Transformer module to obtain reconstructed features. The last layer features are then extracted from the reconstructed features and reshaped into a spatial feature map. The spatial feature map is gradually restored using the reconstructed CNN network to generate a reconstructed image of the optoelectronic chip.

5. The method according to any one of claims 1-4, wherein, Perform pixel-by-pixel difference calculations between the original image of the photoelectric chip and the reconstructed image of the photoelectric chip to generate a reference difference map, including: For each original pixel in the original image of the photoelectric chip, the absolute difference in grayscale value between the original pixel and the corresponding reconstructed pixel in the reconstructed image of the photoelectric chip is calculated, and the absolute difference in grayscale value is normalized to a standard grayscale range to generate a reference difference map.

6. The method according to any one of claims 1-5, wherein, Generating a superpixel difference map corresponding to the superpixel region based on the superpixel mask image includes: The superpixel mask image is reconstructed using the Dinomaly model to obtain the superpixel reconstructed image; The superpixel mask image and the superpixel reconstructed image are subjected to pixel-by-pixel difference calculation to generate a superpixel difference map corresponding to the superpixel region.

7. The method according to any one of claims 1-6, wherein, The process of performing a difference calculation between the superpixel difference map corresponding to the superpixel region and the reference difference map to generate the final difference map corresponding to the superpixel region includes: Calculate the absolute difference of gray values ​​between the superpixel difference map and the reference difference map, and normalize the absolute difference of gray values ​​to the standard gray range to generate the final difference map corresponding to the superpixel region.

8. The method according to any one of claims 1-7, wherein, Also includes: Statistical analysis was performed on the distribution characteristics of abnormal scores in all superpixel regions to obtain the statistical features of abnormal scores in all superpixel regions. The higher the anomaly score of the superpixel region, the greater the likelihood that the superpixel region in the candidate anomaly region is misjudged as an anomaly.

9. The method according to any one of claims 1-8, wherein, Based on the anomaly scores of all superpixel regions, one or more truly anomalous superpixel regions are selected from all superpixel regions, including: The classification threshold is determined based on the statistical characteristics of the anomaly scores of all superpixel regions; One or more superpixel regions with an anomaly score lower than the classification threshold are identified as one or more true anomalous superpixel regions.

10. A reconstruction-based optoelectronic chip anomaly detection device, deployed in a computing device, suitable for performing the method as described in any one of claims 1-9, the device comprising: The acquisition unit is adapted to acquire the original image of the photoelectric chip to be detected; The preliminary detection unit is suitable for using the AnomalyDINO model to perform preliminary anomaly detection on the original image of the photoelectric chip and obtain an anomaly heatmap. The extraction unit is adapted to extract high-response regions from the abnormal heatmap as candidate abnormal regions. The superpixel segmentation unit is adapted to use the SICLE superpixel segmentation algorithm to perform superpixel segmentation on the candidate abnormal region, so as to divide the candidate abnormal region into multiple superpixel regions. The reconstruction difference unit is suitable for reconstructing the original image of the optoelectronic chip using the Dinomaly model to obtain the reconstructed image of the optoelectronic chip, and performing pixel-by-pixel difference calculation between the original image of the optoelectronic chip and the reconstructed image of the optoelectronic chip to generate a reference difference map. The superpixel difference unit is adapted to perform masking processing on the original image of the photoelectric chip for each superpixel region to obtain a superpixel mask image, and generate a superpixel difference map corresponding to the superpixel region based on the superpixel mask image. The second-order difference unit is adapted to perform difference calculation between the superpixel difference map corresponding to the superpixel region and the reference difference map to generate the final difference map corresponding to the superpixel region. An anomaly scoring unit is adapted to calculate the average gray value of each superpixel region in the corresponding final difference map, as the anomaly score of the superpixel region; The filtering unit is adapted to filter out one or more real abnormal superpixel regions from all superpixel regions based on the abnormality scores of all superpixel regions, and obtain the final abnormal region based on the one or more real abnormal superpixel regions.

11. A computing device, comprising: At least one processor; and A memory storing program instructions, wherein the program instructions are configured to be processed by the at least one processor, the program instructions including instructions for processing the method as described in any one of claims 1-9.

12. A computer program product comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method as described in any one of claims 1-9.

13. A readable storage medium storing program instructions that, when read and executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-9.

Images