A defect classification method, device and storage medium based on micro display device

By monitoring the CPU and GPU status in real time, using high-speed flying cameras and DPN classification network models to detect defects of microdisplay devices, the problem of low defect detection efficiency in the existing technology is solved, and the multi-objective simultaneous detection and full utilization of performance is achieved .

CN118691911BActive Publication Date: 2024-11-29SHENZHEN SEICHITECH TECHN CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411178893.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-29
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

The existing microdisplay device defect detection methods cannot achieve multi-objective simultaneous detection, resulting in dissatisfaction with CPU and GPU performance utilization and inefficient defect detection.

Method used

The high-speed fly-shoot camera collects the images to be detected by the micro-display device, monitors the operating status of the CPU and GPU in real time. When the preset conditions are met, the images are transferred to the detection thread and the re-check module for first inspection and re-checking, and the positioning module is used to use the positioning module. Classify defects with the DPN classification network model to achieve multi-objective simultaneous detection.

Benefits of technology

It improves the efficiency of defect detection of microdisplay devices, makes full use of CPU and GPU performance, achieves multi-objective simultaneous detection, and improves the performance of the entire defect classification system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118691911B_ABST
    Figure CN118691911B_ABST
Patent Text Reader

Abstract

The present application discloses a defect classification method, device and storage medium based on a micro-display device, which is used to improve the defect detection efficiency of the micro-display device. The present application includes: collecting images to be detected of the micro-display device; real-time monitoring of the operating status of the CPU central processing unit and the operating status of the GPU graphics processing unit; transmitting the images to be detected and template information to the detection thread; using the positioning module in combination with the template information to locate the minimum detection unit area on the image to be detected; performing a first defect inspection on the minimum detection unit area of ​​the image to be detected, and generating a first inspection analysis result; screening out images to be re-inspected from the image to be detected according to the first inspection analysis result; when the GPU operating state reaches a second preset condition, transmitting the images to be re-inspected to the re-inspection module of the GPU; re-inspecting the images to be re-inspected by the re-inspection module, and generating a re-inspection analysis result; and classifying and summarizing the defects of the micro-display device according to the first inspection analysis result and the re-inspection analysis result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present application relate to the field of micro-display device detection, and in particular, to a defect classification method, device and storage medium based on a micro-display device. Background Art

[0002] With the rapid development of industry and Internet technology, the industry's requirements for industrial product quality are becoming more refined and standardized. As far as the display panel industry is concerned, product quality directly affects its product competitiveness.

[0003] As micro-projectors and wearable devices gradually enter real production and life, the application prospects of micro-display devices are becoming more and more extensive. In particular, the application of micro-display devices in the fields of virtual reality (VR), augmented reality (AR), micro-projectors, etc. makes the new generation of active light-emitting devices have great application prospects.

[0004] Microdisplay devices can be regarded as miniaturized LEDs, which have the advantages of self-luminescence, low power consumption, independent pixel control and ultra-high resolution. On this basis, combined with nano-wiring technology, microdisplay devices can be made into flexible devices and applied to wearable devices. This has led to more and more research and development directions for microdisplay devices, but with this, the demand and requirements for microdisplay devices are also gradually increasing.

[0005] Nowadays, there are still certain technical difficulties in the preparation process of microdisplay devices. For example, when structural defects occur on the surface and inside of microdisplay devices, the performance and service life of the entire device are often seriously affected. The existing defect detection method for microdisplay devices is often a visual appearance inspection of microdisplay devices. This type of visual appearance inspection often requires manual re-judgment to classify the detected NG (defective) wafers, where the defect types include but are not limited to good products, damage, missing, scratches, cross-color stacking, etc. Each defect usually has its own unique characteristics, but there are also defect characteristics with similar appearance, or there are more than two defect characteristics on a wafer area. In order to speed up the efficiency of defect detection, the method of machine first inspection combined with manual re-inspection is often used. However, in real scenarios, in order to meet the tight beat requirements, high-speed flying cameras are generally used for image acquisition. This acquisition method will realize the capture image action for microdisplay devices in a very short time. This acquisition method has very high performance requirements for the entire defect classification system, but at present, neither manual re-inspection nor machine re-inspection can achieve multi-target simultaneous detection, and the performance of CPU and GPU cannot be fully utilized, resulting in a decrease in the efficiency of defect detection. Summary of the invention

[0006] The present application discloses a defect classification method, device and storage medium based on a micro display device, which are used to improve the defect detection efficiency of the micro display device.

[0007] The first aspect of the present application discloses a defect classification method based on a micro display device, comprising:

[0008] The image of the micro display device to be inspected is collected by a high-speed flying camera;

[0009] Real-time monitoring of the CPU central processing unit and GPU graphics processing unit operating status;

[0010] When the CPU operation state reaches the first preset condition, the image to be detected and the template information are transmitted to the detection thread, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0011] Use the positioning module combined with the template information to locate the minimum detection unit area on the image to be detected, where the minimum detection unit area is the effective area in the image to be detected;

[0012] Perform a first defect inspection on the smallest inspection unit area of ​​the image to be inspected and generate the first inspection analysis results;

[0013] Screening out images to be re-inspected from the images to be inspected according to the initial inspection analysis results;

[0014] When the GPU operating state reaches a second preset condition, the image to be re-inspected is transmitted to a re-inspection module of the GPU;

[0015] Re-inspect the image to be re-inspected through the re-inspection module and generate re-inspection analysis results;

[0016] The defects of the micro display devices are classified and summarized based on the initial inspection and analysis results and the re-inspection and analysis results.

[0017] Optionally, a defect initial inspection is performed on the minimum inspection unit area of ​​the image to be inspected, and initial inspection analysis results are generated, including:

[0018] Setting the target convolution kernel according to the wafer position information on the template information;

[0019] Set the non-detection area on the image to be detected to black;

[0020] Use the target convolution kernel to perform convolution emphasis on the minimum detection unit area on the image to be detected;

[0021] The minimum detection unit area after convolution emphasis processing is analyzed by the blob algorithm to generate defect feature information of the minimum detection unit area;

[0022] Perform defect comparison analysis based on defect feature information and generate first inspection analysis results.

[0023] Optionally, the image to be re-inspected is re-inspected by a re-inspection module to generate re-inspection analysis results, including:

[0024] Inputting the image to be reviewed into the input layer of the DPN classification network model, the DPN classification network model includes an input layer, an initial convolutional layer, at least two DPN blocks, a fully connected layer and an output layer, and the DPN block includes a feature extraction module and at least two DPU modules;

[0025] Extracting primary features from the image to be inspected through the initial convolutional layer of the DPN classification network model, and then inputting them into the feature extraction modules of at least two DPN blocks to extract secondary features;

[0026] The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU block to generate fused features;

[0027] The fused features extracted and fused by the DPU block are passed through the fully connected layer to quantize the fused features, and then classified through the output layer to generate the re-inspection analysis results.

[0028] Optionally, using a positioning module in combination with template information to locate a minimum detection unit area on the image to be detected includes:

[0029] Determine at least one rough positioning frame on the image to be detected according to the template information;

[0030] Determine the initial center coordinate position of the rough positioning frame area;

[0031] Generate an attention matrix for each pixel in the coarse positioning box of the image to be detected;

[0032] According to the attention matrix, the pixels whose attention is less than the preset attention threshold in each rough positioning frame are filtered out;

[0033] The offset center coordinate position is calculated by the pixel positions of the remaining pixels in the rough positioning frame;

[0034] Calculate the offset data according to the initial center coordinate position and the offset center coordinate position, the offset data including the offset distance and the offset direction;

[0035] When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixels are screened out, the offset center coordinate position is calculated, and the offset data is analyzed;

[0036] When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the coarse positioning frame on the current image to be detected.

[0037] Optionally, when the CPU operating state reaches a first preset condition, transmitting the image to be detected and the template information to the detection thread includes:

[0038] Determine the CPU idle detection thread according to the CPU running state;

[0039] When the number of idle detection threads of the CPU is greater than the preset number of first threads, the image to be detected and the template information are transmitted to the detection thread, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0040] When the GPU operation state reaches the second preset condition, the image to be re-inspected is transmitted to the re-inspection module of the GPU, including:

[0041] Determine the idle detection thread of the GPU according to the GPU operation status;

[0042] When the number of idle detection threads of the GPU is greater than the preset second thread number, and the number of images to be re-inspected reaches the preset re-inspection number, the images to be re-inspected are transmitted to the re-inspection module of the GPU.

[0043] Optionally, after collecting the image to be inspected of the micro display device by the high-speed flying camera and before monitoring the operating status of the CPU central processing unit and the GPU graphics processing unit in real time, the defect classification method further includes:

[0044] Perform Gaussian filtering to reduce noise on the image to be detected.

[0045] Optionally, a Gaussian filter is used to reduce noise on the image to be detected, including:

[0046] Obtain the image standard deviation, average grayscale and number of pixels of the image to be detected;

[0047] Calculate the center threshold based on the image standard deviation, average grayscale and number of pixels;

[0048] Determine the position of the pixel whose gray value is not less than the center threshold in the image to be detected;

[0049] Calculate the center coordinate position according to the position of the pixel whose gray value is not less than the center threshold;

[0050] Obtain the standard deviation of the spatial domain of the image to be detected;

[0051] Calculate the spatial domain Gaussian filter kernel according to the center coordinate position, the spatial domain standard deviation and the first preset Gaussian function;

[0052] Obtain the color domain standard deviation of the image to be detected;

[0053] Calculate the color domain Gaussian filter kernel according to the center coordinate position, the color domain standard deviation and the second preset Gaussian function;

[0054] Gaussian filtering is performed on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel.

[0055] The second aspect of the present application discloses a defect classification device based on a micro display device, comprising:

[0056] A collection unit, used for collecting the image to be detected of the micro display device through a high-speed flying camera;

[0057] A monitoring unit is used to monitor the operating status of the CPU central processing unit and the GPU graphics processing unit in real time;

[0058] A first transmission unit is used to transmit the image to be detected and the template information to the detection thread when the CPU operation state reaches the first preset condition, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0059] A positioning unit, used to use the positioning module in combination with template information to locate the minimum detection unit area on the image to be detected, where the minimum detection unit area is a valid area in the image to be detected;

[0060] A first generating unit is used to perform a first defect inspection on the minimum inspection unit area of ​​the image to be inspected and generate a first inspection analysis result;

[0061] A screening unit, generating images to be re-inspected from the images to be inspected according to the results of the first inspection analysis;

[0062] A second transmission unit, used for transmitting the image to be re-inspected to the re-inspection module of the GPU when the GPU operation state reaches a second preset condition;

[0063] The second generating unit is used to re-inspect the image to be re-inspected through the re-inspection module and generate a re-inspection analysis result;

[0064] The summary unit is used to classify and summarize the defects of the micro display device according to the initial inspection analysis results and the re-inspection analysis results.

[0065] Optionally, the first generating unit includes:

[0066] Setting the target convolution kernel according to the wafer position information on the template information;

[0067] Set the non-detection area on the image to be detected to black;

[0068] Use the target convolution kernel to perform convolution emphasis on the minimum detection unit area on the image to be detected;

[0069] The minimum detection unit area after convolution emphasis processing is analyzed by the blob algorithm to generate defect feature information of the minimum detection unit area;

[0070] Perform defect comparison analysis based on defect feature information and generate first inspection analysis results.

[0071] Optionally, the second generation unit includes:

[0072] Inputting the image to be reviewed into the input layer of the DPN classification network model, the DPN classification network model includes an input layer, an initial convolutional layer, at least two DPN blocks, a fully connected layer and an output layer, and the DPN block includes a feature extraction module and at least two DPU modules;

[0073] Extracting primary features from the image to be inspected through the initial convolutional layer of the DPN classification network model, and then inputting them into the feature extraction modules of at least two DPN blocks to extract secondary features;

[0074] The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU block to generate fused features;

[0075] The fused features extracted and fused by the DPU block are passed through the fully connected layer to quantize the fused features, and then classified through the output layer to generate the re-inspection analysis results.

[0076] Optional, positioning unit, including:

[0077] Determine at least one rough positioning frame on the image to be detected according to the template information;

[0078] Determine the initial center coordinate position of the rough positioning frame area;

[0079] Generate an attention matrix for each pixel in the coarse positioning box of the image to be detected;

[0080] According to the attention matrix, the pixels whose attention is less than the preset attention threshold in each rough positioning frame are filtered out;

[0081] The offset center coordinate position is calculated by the pixel positions of the remaining pixels in the rough positioning frame;

[0082] Calculate the offset data according to the initial center coordinate position and the offset center coordinate position, the offset data including the offset distance and the offset direction;

[0083] When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixels are screened out, the offset center coordinate position is calculated, and the offset data is analyzed;

[0084] When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the coarse positioning frame on the current image to be detected.

[0085] Optionally, the first transmission unit includes:

[0086] Determine the CPU idle detection thread according to the CPU running state;

[0087] When the number of idle detection threads of the CPU is greater than the preset number of first threads, the image to be detected and the template information are transmitted to the detection thread, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0088] Optionally, the first transmission unit includes:

[0089] Determine the idle detection thread of the GPU according to the GPU operation status;

[0090] When the number of idle detection threads of the GPU is greater than the preset second thread number, and the number of images to be re-inspected reaches the preset re-inspection number, the images to be re-inspected are transmitted to the re-inspection module of the GPU.

[0091] Optionally, after the acquisition unit and before the monitoring unit, the defect classification device further includes:

[0092] The Gaussian noise reduction unit is used to perform noise reduction processing on the image to be detected by Gaussian filtering.

[0093] Optional, Gaussian noise reduction unit, including:

[0094] Obtain the image standard deviation, average grayscale and number of pixels of the image to be detected;

[0095] Calculate the center threshold based on the image standard deviation, average grayscale and number of pixels;

[0096] Determine the position of the pixel whose gray value is not less than the center threshold in the image to be detected;

[0097] Calculate the center coordinate position according to the position of the pixel whose gray value is not less than the center threshold;

[0098] Obtain the standard deviation of the spatial domain of the image to be detected;

[0099] Calculate the spatial domain Gaussian filter kernel according to the center coordinate position, the spatial domain standard deviation and the first preset Gaussian function;

[0100] Obtain the color domain standard deviation of the image to be detected;

[0101] Calculate the color domain Gaussian filter kernel according to the center coordinate position, the color domain standard deviation and the second preset Gaussian function;

[0102] Gaussian filtering is performed on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel.

[0103] The third aspect of the present application provides a defect classification device based on a micro display device, comprising:

[0104] Processor, memory, input-output unit, and bus;

[0105] The processor is connected to the memory, the input and output unit, and the bus;

[0106] The memory stores a program, and the processor calls the program to execute the first aspect and any optional defect classification method of the first aspect.

[0107] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program performs the first aspect and any optional defect classification method of the first aspect.

[0108] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0109] In the present application, firstly, the image to be detected of the micro display device is collected by a high-speed flying camera. Then the operating state of the CPU central processing unit and the operating state of the GPU graphics processing unit are monitored in real time. When the CPU operating state reaches the first preset condition, the image to be detected and the template information are transmitted to the detection thread, wherein the template information is the template feature information extracted from the defect-free image of the micro display device. The positioning module is used to locate the minimum detection unit area on the image to be detected in combination with the template information, and the minimum detection unit area is the effective area in the image to be detected, that is, the effective detection area is determined, and the calculation amount of the first inspection system and the re-inspection system is reduced. The minimum detection unit area of ​​the image to be detected is first inspected for defects, and the first inspection analysis result is generated. Then, according to the first inspection analysis result, the image to be re-inspected is screened out from the image to be detected, wherein the image to be re-inspected is the image of the minimum detection unit area where the first inspection defect feature does not reach the judgment standard, or the image corresponding to the minimum detection unit area where there are at least two first inspection feature defects. When the GPU operating state reaches the second preset condition, the image to be re-inspected is transmitted to the re-inspection module of the GPU. The image to be re-inspected is re-inspected by the re-inspection module to generate a re-inspection analysis result. After the image to be re-inspected is determined, the image to be re-inspected is re-inspected by the re-inspection module to generate a re-inspection analysis result. Finally, the defects of the micro display device are classified and summarized according to the first inspection analysis result and the re-inspection analysis result.

[0110] By monitoring the operating status of the CPU and GPU, when the CPU operating status shows that the current CPU can distribute enough detection threads for the first inspection, multiple images to be detected and the corresponding template information are transmitted to different detection threads respectively, so that multiple images to be detected can be detected simultaneously in different detection threads. After the first inspection of the images to be detected, a part of the images that have been detected and met the standards are screened out, and the remaining images to be re-inspected are subjected to the subsequent re-inspection process. When the GPU operating status and the re-inspection start conditions are met, the images to be re-inspected are transmitted to the GPU's re-inspection module for re-inspection.

[0111] In the above process, multiple images to be inspected can be inspected simultaneously in different inspection threads, and the initial inspection and re-inspection of different images to be inspected can also be run at the same time. The CPU runs the initial inspection and the GPU runs the re-inspection. While achieving simultaneous detection of multiple targets, the performance of the CPU and GPU can be fully utilized, further improving the efficiency of the entire defect classification system. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0113] Figure 1 A schematic diagram of an embodiment of a defect classification method based on a micro display device of the present application;

[0114] Figure 2 This is a schematic diagram of an embodiment of the first stage of the defect classification method based on a micro display device of the present application;

[0115] Figure 3 A schematic diagram of an embodiment of the second stage of the defect classification method based on a micro display device of the present application;

[0116] Figure 4 This is a schematic diagram of an embodiment of the third stage of the defect classification method based on a micro display device of the present application;

[0117] Figure 5 This is a schematic diagram of an embodiment of the fourth stage of the defect classification method based on a micro display device of the present application;

[0118] Figure 6 This is a schematic diagram of an embodiment of a defect classification device based on a micro display device of the present application;

[0119] Figure 7 This is a schematic diagram of another embodiment of a defect classification device based on a micro display device according to the present application. DETAILED DESCRIPTION

[0120] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0121] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0122] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0123] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0124] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0125] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0126] In the field of micro-display device manufacturing, defect detection of micro-display devices is often a visual appearance inspection of micro-display devices. This type of visual appearance inspection often requires manual re-judgment to classify the detected NG (defective) wafers, where the defect types include but are not limited to good products, damage, missing, scratches, color-crossing and stacked wafers. Each defect usually has its own unique characteristics, but there are also defect characteristics with similar appearances, or there are more than two defect characteristics on a wafer area. In order to speed up the efficiency of defect detection, the method of machine first inspection combined with manual re-inspection is often used. However, in real scenarios, in order to meet the tight beat requirements, high-speed flying cameras are generally used for image acquisition. This acquisition method will realize the image capture action for micro-display devices in a very short time. This acquisition method has very high performance requirements for the entire defect classification system, but currently neither manual re-inspection nor machine re-inspection can achieve multi-target simultaneous detection, and the performance of the CPU and GPU cannot be fully utilized, resulting in a decrease in the efficiency of defect detection.

[0127] Based on this, the present application discloses a defect classification method, device and storage medium based on a micro display device, which are used to improve the defect detection efficiency of the micro display device.

[0128] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0129] The method of the present application can be applied to a server, a device, a terminal or other devices with logic processing capabilities, and the present application does not limit this. For the convenience of description, the following description is made by taking the execution subject as an example of a terminal.

[0130] See also Figure 1 The present application provides an embodiment of a defect classification method based on a micro display device, comprising:

[0131] 101. Collect the image to be inspected of the micro display device by a high-speed flying camera;

[0132] In this embodiment, the high-speed acquisition of the micro display device using the flying camera is mainly completed by the image acquisition module.

[0133] The image acquisition module mainly includes necessary devices such as the loading platform, crimping device, flying camera, point light source, displacement component and lens. Before acquisition, the parameters of the entire image acquisition module need to be set, such as exposure time, displacement speed, acquisition interval and lens aperture focal length, etc. Different working modes will be set for different micro-display devices, and the parameters of different working modes will be different.

[0134] Specifically, after adjusting the conventional parameters, the terminal fixes the aperture focal length of the lens under the condition that the high-speed flying camera system working mode is met, and then adjusts the position relationship between the loading platform and the flying camera, and fixes the point light source on one side of the flying camera and the lens, so that the degree of illumination of the point light source on the micro display device is as similar as possible during each acquisition process, and then saves the working parameters of the flying camera itself.

[0135] After completing the above adjustments, the terminal collects the image to be inspected of the micro-display device through a high-speed flying camera.

[0136] 102. Real-time monitoring of the operating status of the CPU central processing unit and the GPU graphics processing unit;

[0137] 103. When the CPU operation state reaches the first preset condition, the image to be detected and the template information are transmitted to the detection thread, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0138] In this embodiment, by monitoring the current operating status of the CPU and GPU, and sending the image to be detected to the system first inspection module through different detection threads for first inspection according to the current operating status of the CPU, re-inspection will be started later according to the current status of the GPU.

[0139] By monitoring the status of the CPU and GPU, the problems existing in the prior art, such as single monitoring function, incomplete monitoring, and poor real-time performance, can be solved. In this embodiment, the above problems are mainly solved by a performance monitoring module, which mainly includes a data acquisition unit, a data processing unit, and a data display unit.

[0140] The data acquisition unit is used to collect the CPU and GPU performance indicators such as usage rate, temperature, frequency, power consumption, etc. in real time. The data acquisition unit includes a sensor interface, an operating system interface, and an application program interface. The data acquisition unit is used to obtain CPU and GPU performance data from hardware sensors, operating system kernels, and applications. The data acquisition unit obtains data in a variety of ways.

[0141] In this embodiment, the terminal uses the System.Diagnostics namespace to obtain the CPU usage. Then the terminal uses the OpenHardwareMonitor library to obtain the temperature, frequency, and power consumption information of the CPU and GPU. The terminal uses the NVAPI library to obtain the usage rate and video memory usage of the NVIDIA GPU.

[0142] In this embodiment, the data processing unit is used to analyze and process the collected data and generate a comprehensive performance report. Specifically, the data processing unit can perform analysis operations such as data filtering, data storage, and data analysis. Data filtering is mainly used to clean and format the collected data. Data storage is used to store historical data, and data analysis is used to calculate comprehensive performance indicators such as the average utilization rate, maximum temperature, and frequency fluctuation of the CPU and GPU.

[0143] In this embodiment, the data display unit is used to display the CPU and GPU performance status in real time in the form of text, and the data display unit includes a recording module and an alarm module. The recording module is used to display real-time and historical CPU and GPU performance data in the form of text, and the alarm module is used to send an alarm message to the system when the CPU and GPU performance exceeds a preset threshold.

[0144] In this embodiment, the terminal obtains template information of the micro display device and collects images of normal and defect-free micro display devices before detection. The terminal needs to extract wafer feature information from the images and generate template information to prepare for subsequent detection.

[0145] 104. Using a positioning module in combination with template information, positioning a minimum detection unit area on the image to be detected, where the minimum detection unit area is a valid area in the image to be detected;

[0146] In this embodiment, in order to reduce the amount of calculation, the terminal first needs to use the positioning module to locate the position area of ​​all wafers on the image to be detected according to the template information. The wafer area is the minimum detection unit area, and other areas are set as background areas.

[0147] 105. Perform a first defect inspection on the minimum inspection unit area of ​​the image to be inspected, and generate a first inspection analysis result;

[0148] The terminal can perform a preliminary defect inspection on the minimum detection unit area of ​​the image to be inspected by a traditional method, or by a convolutional neural network, or by a variety of image recognition methods, which are not limited here. Through the preliminary inspection, the information of the defect characteristics on the minimum detection unit area can be determined, and a preliminary judgment can be made on the defect information.

[0149] Specifically, during the detection process, the terminal can compare the defect feature information with the label of the specific feature, and calculate the possibility that the defect feature belongs to the current specific feature through comparison. If no matching defect feature is found after comparing all specific features, it means that the minimum detection unit area is defect-free.

[0150] According to the above comparison method, the terminal can perform the initial defect classification on the wafer area (minimum detection unit area) according to the characteristics (area, grayscale ratio, aspect ratio, etc.) of the corresponding defect area. Determine the minimum detection unit area without defect characteristics, the minimum detection unit area with obvious defect characteristics, the minimum detection unit area with unclear defect characteristics, and the minimum detection unit area with multiple defects.

[0151] 106. Selecting images to be re-inspected from the images to be inspected according to the initial inspection analysis results;

[0152] The terminal determines the image to be re-inspected based on the analysis result of the first inspection. Specifically, after step 105, the terminal can determine the minimum detection unit area without defect features, the minimum detection unit area with obvious defect features, the minimum detection unit area with unclear defect features, and the minimum detection unit area with possible multiple defects, and select the image corresponding to the minimum detection unit area with unclear defect features and the image corresponding to the minimum detection unit area with possible multiple defects, and determine them as the images that need to be re-inspected.

[0153] It should be noted that the image to be re-inspected here may be a complete image to be detected, or may be an image to be re-inspected by cutting out a corresponding minimum detection area from the image to be detected.

[0154] 107. When the GPU operation state reaches a second preset condition, the image to be re-inspected is transmitted to a re-inspection module of the GPU;

[0155] As the CPU's detection thread continuously generates initial inspection analysis results, the detection thread will receive new images to be inspected for initial inspection according to the situation until all images to be inspected are inspected. At the same time, the terminal will generate more and more images to be re-inspected based on the initial inspection analysis structure. When the number of images to be re-inspected accumulates to a certain level, it can be determined whether the current GPU operating status meets the second preset condition.

[0156] In this embodiment, while the terminal uses the CPU to perform the first system inspection, it determines whether the current GPU operating state meets the second preset condition and whether the preset conditions for entering the system re-judgment are met. If they are met, the terminal inputs the image to be re-inspected into the system re-judgment module to perform GPU artificial intelligence network model reasoning. If not, the existing image to be re-inspected is saved and the above process is repeated until the conditions are met.

[0157] 108. Re-inspect the image to be re-inspected by the re-inspection module, and generate a re-inspection analysis result;

[0158] 109. Classify and summarize the defects of micro display devices based on the results of the initial inspection and re-inspection analysis.

[0159] The terminal classifies and summarizes the defects of the micro-display device according to the first inspection analysis results and the re-inspection analysis results, mainly integrating the first inspection analysis results and the re-inspection analysis results, and saving and outputting them to the corresponding storage area or display area. The terminal generates a data list (containing all feature information of each wafer) according to the first inspection analysis results and the image ID, obtains the image ID and wafer index of all re-inspection categories, and then obtains the re-inspection analysis results, and replaces the re-inspection analysis results into the data list according to the image ID and wafer index corresponding to the re-inspection category, and finally outputs the data list data.

[0160] In this embodiment, first, the image to be detected of the micro display device is collected by a high-speed flying camera. Then the operating state of the CPU central processing unit and the operating state of the GPU graphics processing unit are monitored in real time. According to the CPU operating state, the image to be detected and the template information are transmitted to the detection thread, wherein the template information is the template feature information extracted from the defect-free image of the micro display device. The positioning module is used to locate the minimum detection unit area on the image to be detected in combination with the template information, and the minimum detection unit area is the effective area in the image to be detected, that is, the effective detection area is determined, and the calculation amount of the first inspection system and the re-inspection system is reduced. The minimum detection unit area of ​​the image to be detected is first inspected for defects, and the first inspection analysis result is generated. Then, according to the first inspection analysis result, the image to be re-inspected is screened out from the image to be detected, wherein the image to be re-inspected is an image of the minimum detection unit area whose first inspection defect feature does not reach the judgment standard, or an image corresponding to the minimum detection unit area with at least two first inspection feature defects. The image to be re-inspected is transmitted to the re-inspection module of the GPU according to the GPU operating state. After the image to be re-inspected is determined, the image to be re-inspected is re-inspected by the re-inspection module to generate a re-inspection analysis result. Finally, the defects of micro display devices are classified and summarized based on the initial inspection and re-inspection analysis results.

[0161] By monitoring the operating status of the CPU and GPU, when the CPU operating status shows that the current CPU can distribute enough detection threads for the first inspection, multiple images to be detected and the corresponding template information are transmitted to different detection threads respectively, so that multiple images to be detected can be detected simultaneously in different detection threads. After the first inspection of the images to be detected, a part of the images that have been detected and met the standards are screened out, and the remaining images to be re-inspected are subjected to the subsequent re-inspection process. When the GPU operating status and the re-inspection start conditions are met, the images to be re-inspected are transmitted to the GPU's re-inspection module for re-inspection.

[0162] In the above process, multiple images to be inspected can be inspected simultaneously in different inspection threads, and the initial inspection and re-inspection of different images to be inspected can also be run at the same time. The CPU runs the initial inspection and the GPU runs the re-inspection. While achieving simultaneous detection of multiple targets, the performance of the CPU and GPU can be fully utilized, further improving the efficiency of the entire defect classification system.

[0163] See also Figure 2 , Figure 3 , Figure 4 and Figure 5 The present application provides an embodiment of a defect classification method based on a micro display device, comprising:

[0164] like Figure 2 As shown, 201, collecting the image to be detected of the micro display device by a high-speed flying camera;

[0165] Step 201 in this embodiment is similar to step 101 in the above embodiment and will not be described again here.

[0166] 202. Obtain the image standard deviation, average grayscale and number of pixels of the image to be detected;

[0167] 203. Calculate the center threshold according to the standard deviation, average grayscale and number of pixels of the image;

[0168] 204. Determine the position of a pixel whose gray value is not less than a central threshold in the image to be detected;

[0169] 205. Calculate the center coordinate position according to the position of the pixel whose gray value is not less than the center threshold;

[0170] 206. Obtaining a standard deviation of the spatial domain of the image to be detected;

[0171] 207. Calculate a spatial domain Gaussian filter kernel according to the center coordinate position, the spatial domain standard deviation, and a first preset Gaussian function;

[0172] 208. Obtaining the color domain standard deviation of the image to be detected;

[0173] 209. Calculate a color domain Gaussian filter kernel according to the center coordinate position, the color domain standard deviation, and a second preset Gaussian function;

[0174] 210. Performing Gaussian filtering on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel;

[0175] Images captured by industrial-grade flying cameras will have various noises. To ensure image quality, the following noise reduction processing must be performed first. In this embodiment, Gaussian filtering is mainly used, and mean filtering can be used depending on the situation. If mean filtering is used, the specific processing method of mean filtering can be to use a 3×3 mean filtering window to perform mean filtering on the image.

[0176] In this embodiment, the core Gaussian filtering processing method may be a traditional method, for example, using a Gaussian function as a filter, performing weighted averaging on the signal to smooth the signal and reduce noise, and filtering the image to be detected.

[0177] The Gaussian filter can be a conventional Gaussian filter, but in the image of the micro-display device, since a high-speed flying camera is used for rapid acquisition, the structure is different from the traditional image acquisition method and has higher requirements. The noise generated during the flying process reduces the effect of the aforementioned traditional Gaussian filter noise reduction method. Specifically, during the flying process, the shooting time is short, and the shooting camera and the micro-display device are displaced from each other, which is prone to generate new noise during the displacement.

[0178] This embodiment proposes a Gaussian filtering method, which is specifically used to process this type of fly-by noise, and has a significant noise reduction effect. First, the image standard deviation S, average grayscale M, and number of pixels N of the image to be detected are obtained, and then the center threshold D is calculated based on the image standard deviation S, average grayscale M, and number of pixels N. The image standard deviation calculation formula is as follows:

[0179]

[0180] The center threshold D is calculated as follows:

[0181]

[0182] in, is the variance adjustment coefficient, which is a positive number. The variance adjustment coefficient is determined according to the size of the average grayscale M. First, the average grayscale of the normal and defect-free micro-display device image collected historically is used as the reference average grayscale, and then the difference between the average grayscale of the image to be detected and the reference average grayscale is compared. The variance adjustment coefficient can be generated in proportion to the size of the difference. It should be noted that the variance adjustment coefficient can also be set manually by comparing the difference between the average grayscale and the reference average grayscale. Usually, the larger the difference, the larger the variance adjustment coefficient, and vice versa. If the difference between the average grayscale of the image to be detected and the reference average grayscale is within an allowed preset range, the variance adjustment coefficient is set to 1.

[0183] After the terminal calculates the center threshold, the terminal determines the positions of pixels in the image to be detected whose grayscale values ​​are not less than the center threshold. These pixels are the pixels of the wafer area. The terminal calculates the center coordinate position based on the positions of pixels that are not less than the center threshold. Specifically, it is obtained by performing center point fitting calculation on the position data of these pixels, or by performing a centroid algorithm on the area to find the center coordinate position of these pixels. There can be multiple fitting methods.

[0184] After the center coordinate position is determined, Gaussian filtering is performed on the image to be detected in the spatial domain and the color domain.

[0185] Specifically, first obtain the spatial domain standard deviation of the image to be detected , and then according to the center coordinate position , spatial domain standard deviation and the first preset Gaussian function Calculate the Gaussian filter kernel in the spatial domain. The formula is as follows:

[0186]

[0187] in, is the pixel position on the image to be detected whose grayscale value is not less than the center threshold.

[0188] Get the color domain standard deviation of the image to be detected , according to the center coordinate position , color domain standard deviation and the second preset Gaussian function Calculate the color domain Gaussian filter kernel. The formula is as follows:

[0189]

[0190] in, is the pixel position on the image to be detected whose grayscale value is not less than the center threshold. is the gray value of the center coordinate position, is the gray value of the pixel position whose gray value is not less than the center threshold, is the Euclidean distance.

[0191] Finally, Gaussian filtering is performed on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel. Specifically, the function of the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel is multiplied to obtain the target Gaussian filter kernel:

[0192]

[0193] Then pass The image to be detected is subjected to convolution processing. This filtering and denoising method generates a target Gaussian filter kernel for the flying camera system by analyzing the color domain and spatial domain of the image and combining it with a designed Gaussian function, thereby improving the denoising effect of the image of the flying camera system.

[0194] After step 210 , step 211 is executed.

[0195] like Figure 3 As shown, 211, real-time monitoring of the operating status of the CPU central processing unit and the GPU graphics processing unit;

[0196] Step 212 in this embodiment is similar to step 102 in the above embodiment and will not be described again here.

[0197] 212. Determine the idle detection thread of the CPU according to the CPU running state;

[0198] 213. When the number of idle detection threads of the CPU is greater than the preset first thread number, the image to be detected and the template information are transmitted to the detection thread, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0199] The first preset condition includes comprehensive performance indicators such as CPU usage, average usage, maximum temperature, frequency fluctuation, etc. The first inspection can only be performed when this preset condition is met. The terminal determines the detection threads that the CPU can be used to allocate based on the CPU operating status and the aforementioned comprehensive performance indicators. When the number of detection threads that the CPU can be used to allocate is greater than the preset number of first threads, the image to be detected and the template information are transmitted to the detection thread. The template information is the template feature information extracted from the defect-free image of the micro display device. The remaining threads wait for the conditions to be met before being distributed.

[0200] 214. Determine at least one rough positioning frame on the image to be detected according to the template information;

[0201] 215. Determine the initial center coordinate position of the rough positioning frame area;

[0202] 216. Generate an attention matrix for each pixel in the coarse positioning box of the image to be detected;

[0203] 217. Filter out the pixel points whose attention is less than a preset attention threshold in each rough positioning frame according to the attention matrix;

[0204] 218. Calculate the offset center coordinate position by using the pixel positions of the remaining pixels in the rough positioning frame;

[0205] 219. Calculate offset data according to the initial center coordinate position and the offset center coordinate position, the offset data including the offset distance and the offset direction, and determine whether the offset data meets the preset distance condition; if not, execute step 220; if yes, execute step 221;

[0206] 220. When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixel points are screened out, the offset center coordinate position is calculated, and the offset data is analyzed; that is, after adjusting the position of the coarse positioning frame on the image to be detected according to the offset data in step 220, return to step 215 and repeat step 215 and steps after 215;

[0207] 221. When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the rough positioning frame on the current image to be detected;

[0208] There are multiple calibration frames set on the template information. The terminal first determines at least one coarse positioning frame on the image to be detected based on the calibration frame on the template information. At the beginning, the coarse positioning frame may completely frame the minimum detection unit area (wafer area) on the image to be detected, and there may also be offsets in different directions. At this time, it is necessary to determine the initial center coordinate position of the coarse positioning frame area, and calculate the initial center position through the coordinate positions of the four corners of the coarse positioning frame on the image to be detected.

[0209] Next, the Dropout module is used to generate an attention matrix for each pixel in the coarse positioning frame of the image to be detected. The Dropout module includes BatchNorm-2*2DefConv-ReLU and BatchNorm-2*2DefConv-SigMiod. First, the coarse positioning frame area in the image to be detected is input into BatchNorm-2*2DefConv-ReLU for processing, and then its output is input into BatchNorm-2*2DefConv-SigMiod to generate an attention matrix of the same size as the coarse positioning frame area.

[0210] After the attention matrix is ​​generated, the terminal removes the pixels in each coarse positioning frame whose attention is less than the preset attention threshold according to the attention matrix, leaving only the pixels that meet the requirements. Most of the pixels with the required attention value belong to the wafer area (the minimum detection unit area), so it is necessary to calculate the offset center coordinate position through the pixel position of the remaining pixels in the coarse positioning frame (the centroid algorithm can be used). After calculating the offset center coordinate position, the terminal calculates the offset data according to the initial center coordinate position and the offset center coordinate position. The offset data includes the offset distance and the offset direction, that is, the error of the initial center coordinate position from the actual wafer area is determined according to the offset center coordinate position, and the initial center coordinate position is relocated.

[0211] When the offset data does not meet the preset conditions, the terminal adjusts the position of the coarse positioning frame on the image to be detected according to the offset data, and re-determines the initial center coordinate position of the new coarse positioning frame, calculates the attention matrix, screens out pixels, calculates the offset center coordinate position, and analyzes the offset data. When the distance between the two is not within the preset range, it is necessary to adjust the initial center point coordinate position through the offset data and redraw the coarse positioning frame for detection.

[0212] When the offset data meets the preset conditions, the terminal determines the minimum detection unit area and the background area based on the coarse positioning frame on the current image to be detected. Specifically, when the distance between the two is within the preset range, it means that the current coarse positioning frame and the actual wafer area position are basically consistent, and subsequent detection can be carried out.

[0213] After step 221 , step 222 is executed.

[0214] like Figure 4 As shown, 222, setting a target convolution kernel according to the wafer position information on the template information;

[0215] 223. Setting the non-detection area on the image to be detected to black;

[0216] 224. Use the target convolution kernel to perform convolution emphasis processing on the minimum detection unit area on the image to be detected;

[0217] 225. Analyze the minimum detection unit area after convolution emphasis processing through blob algorithm to generate defect feature information of the minimum detection unit area;

[0218] 226. Perform defect comparison analysis based on defect feature information and generate first inspection analysis results;

[0219] In this embodiment, since the wafer spacing is the same and periodic, the terminal sets the target convolution kernel according to the wafer position information on the template information. The setting of the convolution kernel mainly includes determining the size of the detection convolution kernel and related parameters of the convolution kernel.

[0220] The terminal sets the background area on the image to be detected to black, so that in the subsequent first detection process, the detection of irrelevant background areas is reduced, thereby reducing the amount of calculation.

[0221] The terminal uses the target convolution kernel to perform convolution emphasis processing on the minimum detection unit area on the image to be inspected to obtain the corresponding economic features. Then the terminal uses the blob algorithm to analyze and extract the features of the defective area after convolution emphasis, and calculates the defect feature information of the defective area.

[0222] The terminal classifies the defects on the wafer area according to the defect feature information of the defect area (defect area information, grayscale ratio of the defect area, aspect ratio of the defect area, etc.) and generates the first inspection analysis results. The wafer area with unclear defect features and multiple defect feature types is marked as a re-inspection category.

[0223] Based on the wafer position coordinates, the terminal cuts out the entire wafer area that needs to be re-inspected and generates an image to be re-inspected in preparation for subsequent re-inspection.

[0224] In this embodiment, the terminal can save the classified categories, count the number of images to be re-inspected and the number of re-inspected categories, and repeatedly execute the above-mentioned first inspection module and process to reduce the number of re-inspections as much as possible.

[0225] After step 226, step 227 is executed.

[0226] like Figure 5 As shown, 227, selecting the image to be re-inspected from the image to be inspected according to the first inspection analysis result;

[0227] Step 227 in this embodiment is similar to step 106 in the above embodiment and will not be described again here.

[0228] 228. Determine an idle detection thread of the GPU according to the GPU operation status;

[0229] 229. When the number of idle detection threads of the GPU is greater than the preset number of second threads, and the number of images to be rechecked reaches the preset number of rechecks, the images to be rechecked are transmitted to the recheck module of the GPU;

[0230] The second preset condition includes comprehensive performance indicators such as GPU temperature, frequency, power consumption information, average usage rate, maximum temperature, frequency fluctuation, usage rate and video memory usage.

[0231] In this embodiment, while using the CPU to perform the first system inspection, the terminal determines whether the current GPU operating state meets the conditions of the comprehensive performance index and whether it meets the preset conditions for entering the system re-judgment (for example, whether 5000 wafer small images are met). If satisfied, the terminal inputs the image to be re-inspected into the system re-judgment module to perform GPU artificial intelligence network model reasoning. If not satisfied, the existing image to be re-inspected is saved and the above process is repeated until the conditions are met.

[0232] 230. Input the image to be re-inspected into the input layer of the DPN classification network model, the DPN classification network model includes an input layer, an initial convolution layer, at least two DPN blocks, a fully connected layer and an output layer, and the DPN block includes a feature extraction module and at least two DPU modules;

[0233] 231. Extracting a primary feature from the image to be inspected through the initial convolutional layer of the DPN classification network model, and then inputting the feature extraction modules of at least two DPN blocks to extract secondary features;

[0234] 232. The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU block to generate fused features;

[0235] 233. The fusion features extracted and fused by the DPU block are passed through the fully connected layer to quantize the fusion features, and then classified through the output layer to generate the re-inspection analysis results;

[0236] In this embodiment, the terminal uses the DPN classification network model for re-inspection. DPN (Dual Path Network) is a deep neural network architecture that combines the advantages of ResNet (Residual Network) and DenseNet (Dense Convolutional Network) and is applied to the image defect classification and recognition of the micro display device in this embodiment.

[0237] The core of the DPN classification network model constructed in this embodiment is to construct a residual path and a dense path. Through the parallel residual path and dense path, more efficient feature extraction and transmission are achieved, thereby improving the performance and stability of the network.

[0238] Specifically, the architecture of the DPN classification network model mainly includes an input layer, an initial convolutional layer, multiple DPN blocks, a fully connected layer, and an output layer.

[0239] The input layer is mainly used to receive input image data.

[0240] The initial convolution layer is mainly used to perform initial convolution operations to extract low-level features.

[0241] Each DPN block consists of multiple Dual Path Units (DPUs). Among them, Dual Path Unit is the core component of DPN. Each DPU includes two parallel paths, the residual path and the dense path. The residual path is used to retain the input features, avoid the gradient vanishing problem through the skip connection, and pass the input features directly to the output. The parallel dense path can connect the output of each layer with the output of all previous layers to form a dense connection, which promotes the multiple use of features. The functions of this DPN block will be described in detail later.

[0242] The fully connected layer is used to vectorize the features extracted by the convolutional layer for classification and can cooperate with the output layer.

[0243] The output layer is used to output the classification results of defect features through an activation function (such as Softmax).

[0244] The architecture of the DPN classification network model in this embodiment can mainly perform feature extraction, feature fusion, and classification decision-making.

[0245] When the input image to be detected passes through the initial convolutional layer, it can extract features once, and then enter multiple DPN blocks to further extract secondary features. Feature extraction is one of the functions of the DPN block.

[0246] In each DPN block, the DPU simultaneously processes and transfers features through residual connections and dense connections to achieve efficient feature fusion.

[0247] The features extracted and fused by the DPN block pass through the fully connected layer and are finally classified through the output layer.

[0248] The DPN classification network model can perform efficient feature extraction and transmission. Specifically, by retaining the input features through the residual path and transmitting them directly to the output, it can avoid the problem of gradient vanishing and explosion, and can support deeper network structures. At the same time, the dense path promotes the multiple use of features, improving the efficiency of feature transmission and network performance.

[0249] Secondly, in this embodiment, deeper network training can be achieved by combining the residual connection and dense connection of DPN, so that a deeper network can be trained, thereby improving the defect classification accuracy.

[0250] In addition, the DPN classification network model can utilize features multiple times, and the densely connected paths enable the output of each layer to be directly connected to the subsequent layers, which enhances the multiple utilization and sharing of features and improves the expressiveness of the network model.

[0251] The DPN classification network model can reduce parameters and computational efficiency. Although the depth of the network increases, the efficient feature transfer and sharing mechanism of the path enables DPN to reduce the number of parameters and overall computational complexity while maintaining high performance.

[0252] Before training, a data set is collected, and wafer defect training images of different defect types on micro-display devices are collected. Pre-training is performed based on the collected wafer defect training images to obtain a trained DPN classification network model.

[0253] 234. Classify and summarize the defects of micro display devices based on the results of the initial inspection and re-inspection analysis.

[0254] Step 234 in this embodiment is similar to step 109 in the above embodiment and will not be described again here.

[0255] In this embodiment, firstly, the image to be detected of the micro display device is collected by a high-speed flying camera.

[0256] Obtain the image standard deviation, average grayscale and number of pixels of the image to be detected. Calculate the center threshold based on the image standard deviation, average grayscale and number of pixels. Determine the pixel positions whose grayscale values ​​in the image to be detected are not less than the center threshold. Calculate the center coordinate position based on the pixel positions whose grayscale values ​​are not less than the center threshold. Obtain the spatial domain standard deviation of the image to be detected. Calculate the spatial domain Gaussian filter kernel based on the center coordinate position, the spatial domain standard deviation and the first preset Gaussian function. Obtain the color domain standard deviation of the image to be detected. Calculate the color domain Gaussian filter kernel based on the center coordinate position, the color domain standard deviation and the second preset Gaussian function. Perform Gaussian filtering on the image to be detected based on the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel.

[0257] Then the running status of the CPU central processing unit and the running status of the GPU graphics processing unit are monitored in real time. The idle detection thread of the CPU is determined according to the running status of the CPU. When the number of idle detection threads of the CPU is greater than the preset first number of threads, the image to be detected and the template information are transmitted to the detection thread, and the template information is the template feature information extracted from the defect-free image of the micro display device.

[0258] At least one coarse positioning frame on the image to be detected is determined according to the template information. The initial center coordinate position of the coarse positioning frame area is determined. An attention matrix is ​​generated for the pixels in each coarse positioning frame of the image to be detected. The pixels in each coarse positioning frame whose attention is less than a preset attention threshold are screened out according to the attention matrix. The offset center coordinate position is calculated by the pixel position of the remaining pixels in the coarse positioning frame. The offset data is calculated according to the initial center coordinate position and the offset center coordinate position, and the offset data includes an offset distance and an offset direction. When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixels are screened out, the offset center coordinate position is calculated, and the offset data is analyzed. When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the coarse positioning frame on the current image to be detected.

[0259] The target convolution kernel is set according to the wafer position information on the template information. The non-detection area on the image to be detected is set to black. The minimum detection unit area on the image to be detected is subjected to convolution emphasis processing using the target convolution kernel. The minimum detection unit area after convolution emphasis processing is analyzed by the blob algorithm to generate defect feature information of the minimum detection unit area. Defect comparison analysis is performed according to the defect feature information to generate the first inspection analysis result. Then, the image to be re-inspected is screened out from the image to be detected according to the first inspection analysis result, wherein the image to be re-inspected is an image of the minimum detection unit area whose first inspection defect feature does not meet the judgment standard or an image corresponding to the minimum detection unit area with at least two first inspection feature defects. The idle detection thread of the GPU is determined according to the GPU running state. When the number of idle detection threads of the GPU is greater than the preset second thread number, and the number of images to be re-inspected reaches the preset re-inspection number, the image to be re-inspected is transmitted to the re-inspection module of the GPU. The image to be re-inspected is input into the input layer of the DPN classification network model, and the DPN classification network model includes an input layer, an initial convolution layer, at least two DPN blocks, a fully connected layer and an output layer, and the DPN block includes a feature extraction module and at least two DPU modules. The initial convolutional layer of the DPN classification network model is used to extract the primary features of the image to be re-inspected, and then the features are input into the feature extraction modules of at least two DPN blocks to extract the secondary features. The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU block to generate fused features. The fused features extracted and fused by the DPU block are passed through the fully connected layer to vectorize the fused features, and then classified through the output layer to generate the re-inspection analysis results.

[0260] Finally, the defects of micro display devices are classified and summarized based on the initial inspection and re-inspection analysis results.

[0261] By monitoring the operating status of the CPU and GPU, when the CPU operating status shows that the current CPU can distribute enough detection threads for the first inspection, multiple images to be detected and the corresponding template information are transmitted to different detection threads respectively, so that multiple images to be detected can be detected simultaneously in different detection threads. After the first inspection of the images to be detected, a part of the images that have been detected and met the standards are screened out, and the remaining images to be re-inspected are subjected to the subsequent re-inspection process. When the GPU operating status and the re-inspection start conditions are met, the images to be re-inspected are transmitted to the GPU's re-inspection module for re-inspection.

[0262] In the above process, multiple images to be inspected can be inspected simultaneously in different inspection threads, and the initial inspection and re-inspection of different images to be inspected can also be run at the same time. The CPU runs the initial inspection and the GPU runs the re-inspection. While achieving simultaneous detection of multiple targets, the performance of the CPU and GPU can be fully utilized, further improving the efficiency of the entire defect classification system.

[0263] Secondly, in this embodiment, the DPN classification network model can efficiently extract and transfer features. Specifically, the input features are retained through the residual path and directly transferred to the output, avoiding the gradient vanishing and explosion problems, and supporting a deeper network structure. At the same time, the dense path promotes the multiple use of features, improving the feature transfer efficiency and network performance.

[0264] Secondly, in this embodiment, through the combination of residual connection and dense connection of DPN, a deeper network can be trained, thereby improving the accuracy of defect classification. In addition, the DPN classification network model can reuse features multiple times, and the dense connection path enables the output of each layer to be directly connected to the subsequent layer, which enhances the multiple utilization and sharing of features and improves the expression ability of the model. The DPN classification network model can reduce parameters and computational efficiency, and the depth of the network is increased. Through the efficient feature transfer and sharing mechanism of the path, DPN can reduce the number of parameters and overall computational complexity while maintaining high performance.

[0265] Secondly, the color domain and spatial domain filter kernels are constructed by the designed Gaussian kernel function, which can better remove the noise of the fly-by images of micro display devices.

[0266] See also Figure 6 The present application provides an embodiment of a defect classification device based on a micro display device, comprising:

[0267] The acquisition unit 601 is used to acquire the image to be detected of the micro display device through a high-speed flying camera;

[0268] A Gaussian noise reduction unit 602, used for performing noise reduction processing by Gaussian filtering on the image to be detected;

[0269] Optionally, the Gaussian noise reduction unit 602 includes:

[0270] Obtain the image standard deviation, average grayscale and number of pixels of the image to be detected;

[0271] Calculate the center threshold based on the image standard deviation, average grayscale and number of pixels;

[0272] Determine the position of the pixel whose gray value is not less than the center threshold in the image to be detected;

[0273] Calculate the center coordinate position according to the position of the pixel whose gray value is not less than the center threshold;

[0274] Obtain the standard deviation of the spatial domain of the image to be detected;

[0275] Calculate the spatial domain Gaussian filter kernel according to the center coordinate position, the spatial domain standard deviation and the first preset Gaussian function;

[0276] Obtain the color domain standard deviation of the image to be detected;

[0277] Calculate the color domain Gaussian filter kernel according to the center coordinate position, the color domain standard deviation and the second preset Gaussian function;

[0278] Gaussian filtering is performed on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel.

[0279] The monitoring unit 603 is used to monitor the operating status of the CPU central processing unit and the GPU graphics processing unit in real time;

[0280] The first transmission unit 604 is used to transmit the image to be detected and the template information to the detection thread when the CPU operation state reaches the first preset condition, where the template information is the template feature information extracted from the defect-free image of the micro display device;

[0281] Optionally, the first transmission unit 604 includes:

[0282] Determine the CPU idle detection thread according to the CPU running state;

[0283] When the number of idle detection threads of the CPU is greater than the preset first thread number, the image to be detected and template information are transmitted to the detection thread, where the template information is template feature information extracted from a defect-free image of the micro display device.

[0284] A positioning unit 605 is used to use the positioning module in combination with the template information to locate the minimum detection unit area on the image to be detected, where the minimum detection unit area is a valid area in the image to be detected;

[0285] Optionally, the positioning unit 605 includes:

[0286] Determine at least one rough positioning frame on the image to be detected according to the template information;

[0287] Determine the initial center coordinate position of the rough positioning frame area;

[0288] Generate an attention matrix for each pixel in the coarse positioning box of the image to be detected;

[0289] According to the attention matrix, the pixels whose attention is less than the preset attention threshold in each rough positioning frame are filtered out;

[0290] The offset center coordinate position is calculated by the pixel positions of the remaining pixels in the rough positioning frame;

[0291] Calculate the offset data according to the initial center coordinate position and the offset center coordinate position, the offset data including the offset distance and the offset direction;

[0292] When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixels are screened out, the offset center coordinate position is calculated, and the offset data is analyzed;

[0293] When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the coarse positioning frame on the current image to be detected.

[0294] The first generating unit 606 is used to perform a first defect inspection on the minimum inspection unit area of ​​the image to be inspected and generate a first inspection analysis result;

[0295] Optionally, the first generating unit 606 includes:

[0296] Setting the target convolution kernel according to the wafer position information on the template information;

[0297] Set the non-detection area on the image to be detected to black;

[0298] Use the target convolution kernel to perform convolution emphasis on the minimum detection unit area on the image to be detected;

[0299] The minimum detection unit area after convolution emphasis processing is analyzed by the blob algorithm to generate defect feature information of the minimum detection unit area;

[0300] Perform defect comparison analysis based on defect feature information and generate first inspection analysis results.

[0301] A screening unit 607 is used to screen out images to be re-inspected from the images to be inspected according to the first inspection analysis result;

[0302] A second transmission unit 608, configured to transmit the image to be re-inspected to a re-inspection module of the GPU when the GPU operation state reaches a second preset condition;

[0303] Optionally, the second transmission unit 608 includes:

[0304] Determine the idle detection thread of the GPU according to the GPU operation status;

[0305] When the number of idle detection threads of the GPU is greater than the preset second thread number, and the number of images to be re-inspected reaches the preset re-inspection number, the images to be re-inspected are transmitted to the re-inspection module of the GPU.

[0306] The second generating unit 609 is used to re-inspect the image to be re-inspected through the re-inspection module and generate a re-inspection analysis result;

[0307] Optionally, the second generating unit 609 includes:

[0308] Inputting the image to be reviewed into the input layer of the DPN classification network model, the DPN classification network model includes an input layer, an initial convolutional layer, at least two DPN blocks, a fully connected layer and an output layer, and the DPN block includes a feature extraction module and at least two DPU modules;

[0309] Extracting primary features from the image to be inspected through the initial convolutional layer of the DPN classification network model, and then inputting them into the feature extraction modules of at least two DPN blocks to extract secondary features;

[0310] The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU block to generate fused features;

[0311] The fused features extracted and fused by the DPU block are passed through the fully connected layer to quantize the fused features, and then classified through the output layer to generate the re-inspection analysis results.

[0312] The summarizing unit 610 is used to classify and summarize the defects of the micro display device according to the initial inspection analysis result and the re-inspection analysis result.

[0313] See also Figure 7 , the present application provides a defect classification device based on a micro display device, comprising:

[0314] Processor 701 , memory 702 , input / output unit 703 , and bus 704 .

[0315] The processor 701 is connected to the memory 702 , the input and output unit 703 , and the bus 704 .

[0316] The memory 702 stores a program, and the processor 701 calls the program to execute the following steps: Figure 1 , Figure 2and Figure 3 , Figure 4 and Figure 5 Defect classification method in .

[0317] The present application provides a computer-readable storage medium, on which a program is stored, and when the program is executed on a computer, the program performs the following steps: Figure 1 , Figure 2 and Figure 3 , Figure 4 and Figure 5 Defect classification method in .

[0318] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0319] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0320] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0321] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0322] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A defect classification method based on a micro display device, characterized in that: include: The image of the micro display device to be inspected is collected by a high-speed flying camera; Real-time monitoring of the CPU central processing unit and GPU graphics processing unit operating status; When the CPU operation state reaches a first preset condition, the image to be detected and the template information are transmitted to the detection thread, wherein the template information is template feature information extracted from the defect-free image of the micro display device; Using a positioning module in combination with the template information to locate a minimum detection unit area on the image to be detected, the minimum detection unit area being a valid area in the image to be detected; Performing a first defect inspection on the minimum inspection unit area of ​​the image to be inspected, and generating a first inspection analysis result; Screening out images to be re-inspected from the images to be inspected according to the first inspection analysis result; When the GPU operating state reaches a second preset condition, transmitting the image to be re-inspected to a re-inspection module of the GPU; Re-inspecting the image to be re-inspected by the re-inspection module to generate a re-inspection analysis result; The defects of the micro display device are classified and summarized according to the first inspection analysis result and the re-inspection analysis result.

2. The defect classification method according to claim 1, characterized in that: Performing a defect initial inspection on the minimum inspection unit area of ​​the image to be inspected and generating an initial inspection analysis result includes: Setting a target convolution kernel according to wafer position information on the template information; Setting the non-detection area on the image to be detected to black; Using the target convolution kernel to perform convolution emphasis processing on the minimum detection unit area on the image to be detected; Analyze the minimum detection unit area after convolution emphasis processing by using a blob algorithm to generate defect feature information of the minimum detection unit area; A defect comparison analysis is performed based on the defect feature information to generate a first inspection analysis result.

3. The defect classification method according to claim 1, characterized in that: Re-inspecting the image to be re-inspected by the re-inspection module to generate a re-inspection analysis result, including: Inputting the image to be reviewed into the input layer of a DPN classification network model, wherein the DPN classification network model comprises an input layer, an initial convolutional layer, at least two DPN blocks, a fully connected layer and an output layer, wherein the DPN block comprises a feature extraction module and at least two DPU modules; Extracting primary features from the image to be inspected through the initial convolutional layer of the DPN classification network model, and then inputting them into the feature extraction modules of at least two DPN blocks to extract secondary features; The secondary features are processed and transmitted simultaneously through the residual connection and dense connection of the DPU module to generate fused features; The fused features extracted and fused by the DPU module are passed through the fully connected layer to quantize the fused features, and then classified through the output layer to generate a re-inspection analysis result.

4. The defect classification method according to any one of claims 1 to 3, characterized in that: Using a positioning module in combination with the template information to locate the minimum detection unit area on the image to be detected includes: Determine at least one rough positioning frame on the image to be detected according to the template information; Determine the initial center coordinate position of the rough positioning frame area; Generate an attention matrix for each pixel point in the coarse positioning frame of the image to be detected; Screening out pixel points in each of the rough positioning frames whose attention is less than a preset attention threshold according to the attention matrix; Calculate the offset center coordinate position by using the pixel positions of the remaining pixels in the rough positioning frame; Calculate offset data according to the initial center coordinate position and the offset center coordinate position, wherein the offset data includes an offset distance and an offset direction; When the offset data does not meet the preset distance condition, the position of the coarse positioning frame on the image to be detected is adjusted according to the offset data, and the initial center coordinate position of the new coarse positioning frame is re-determined, the attention matrix is ​​calculated, the pixel points are screened out, the offset center coordinate position is calculated, and the offset data is analyzed; When the offset data meets the preset distance condition, the minimum detection unit area is determined according to the coarse positioning frame on the current image to be detected.

5. The defect classification method according to claim 1, characterized in that: When the CPU operation state reaches the first preset condition, the image to be detected and the template information are transmitted to the detection thread, including: Determine an idle detection thread of the CPU according to the CPU running state; When the number of idle detection threads of the CPU is greater than the preset first thread number, the image to be detected and the template information are transmitted to the detection thread, wherein the template information is the template feature information extracted from the defect-free image of the micro display device; When the GPU operation state reaches the second preset condition, transmitting the image to be re-inspected to the re-inspection module of the GPU, including: Determine an idle detection thread of the GPU according to the GPU running state; When the number of idle detection threads of the GPU is greater than the preset second thread number, and the number of the images to be rechecked reaches the preset recheck number, the images to be rechecked are transmitted to the recheck module of the GPU.

6. The defect classification method according to any one of claims 1 to 3, characterized in that: After collecting the image to be inspected of the micro display device by a high-speed flying camera and before real-time monitoring of the operating status of the CPU central processing unit and the GPU graphics processing unit, the defect classification method further includes: The image to be detected is subjected to noise reduction processing using Gaussian filtering.

7. The defect classification method according to claim 6, characterized in that: The denoising process of the image to be detected is performed by Gaussian filtering, comprising: Obtaining the image standard deviation, average grayscale and number of pixels of the image to be detected; Calculate the center threshold according to the standard deviation, average grayscale and number of pixels of the image; Determine the position of the pixel whose gray value in the image to be detected is not less than the central threshold; Calculate the center coordinate position according to the position of the pixel point whose gray value is not less than the center threshold; Obtaining the standard deviation of the image to be detected in the spatial domain; Calculate a spatial domain Gaussian filter kernel according to the central coordinate position, the spatial domain standard deviation and a first preset Gaussian function; Obtaining the color domain standard deviation of the image to be detected; Calculate a color domain Gaussian filter kernel according to the center coordinate position, the color domain standard deviation, and a second preset Gaussian function; Gaussian filtering is performed on the image to be detected according to the spatial domain Gaussian filter kernel and the color domain Gaussian filter kernel.

8. A defect classification device based on a micro display device, characterized in that: include: A collection unit, used for collecting the image to be detected of the micro display device through a high-speed flying camera; The monitoring unit is used to monitor the operating status of the CPU central processing unit and the GPU graphics processing unit in real time; A first transmission unit, configured to transmit the image to be detected and template information to a detection thread when the CPU operation state reaches a first preset condition, wherein the template information is template feature information extracted from a defect-free image of the micro display device; A positioning unit, used for positioning a minimum detection unit area on the image to be detected by using a positioning module in combination with the template information, wherein the minimum detection unit area is a valid area in the image to be detected; A first generating unit, configured to perform a first defect inspection on the minimum inspection unit area of ​​the image to be inspected, and generate a first inspection analysis result; A screening unit, screening out images to be re-inspected from the images to be inspected according to the first inspection analysis result; A second transmission unit, configured to transmit the image to be re-inspected to a re-inspection module of the GPU when the GPU operation state reaches a second preset condition; A second generating unit, configured to re-inspect the image to be re-inspected through the re-inspection module and generate a re-inspection analysis result; A summarizing unit is used to classify and summarize the defects of the micro display device according to the first inspection analysis result and the re-inspection analysis result.

9. A defect classification device based on a micro display device, characterized in that: include: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the defect classification method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the defect classification method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Mask defect detection method and device, electronic equipment and storage medium

    CN114881990A

  • Image augmentation method and apparatus, computer device, and storage medium

    WO2021139177A1