Mask defect detection method based on GPU (Graphics Processing Unit) acceleration and related equipment

Abstract

The invention discloses a GPU (Graphics Processing Unit) acceleration-based mask defect detection method and related equipment, and the method comprises the following steps: establishing a hardware link between a camera and a GPU through a PCIE (Peripheral Component Interface Express) bus architecture, and constructing a target memory environment of the camera matched with the GPU; image transmission is carried out on multi-channel image data, collected by the camera, of the to-be-detected mask plate based on the hardware link and the target memory environment, the multi-channel image data is transmitted to the GPU, and image transmission follows an asynchronous transmission mechanism; gPU data processing is carried out on the multi-channel image data through a GPU based on a CUDA flow technology, and a defect binary image is obtained; gPU data processing and image transmission are executed in parallel; carrying out data downloading on the defect binary image based on an asynchronous memory copying mechanism of a CUDA (Compute Unified Device Architecture), and downloading the defect binary image to a CPU (Central Processing Unit) connected with the GPU; data downloading and GPU data processing are executed in parallel; and performing defect feature extraction on the defect binary image through the CPU based on Blob analysis to obtain a defect identification result. Therefore, rapid and efficient mask defect detection can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of detection and recognition, and in particular to a mask defect detection method based on GPU acceleration and related equipment. BACKGROUND

[0002] A mask (photomask) is a key component in semiconductor manufacturing and display panel production, and its quality directly affects the yield of the final product. With the continuous shrinking of the process node (currently it has entered the era of 3nm and below), mask defect detection technology is facing unprecedented challenges. Traditional manual microscope detection methods cannot meet the requirements of modern industry for detection speed, accuracy and consistency, and automated optical inspection (AOI, Automated Optical Inspection) technology has emerged as the times require.

[0003] Common photoelectric detection techniques can include bright field detection: using vertical illumination, suitable for detecting surface contamination and larger size defects; dark field detection: using oblique illumination, more sensitive to small particles and edge defects. Phase shift detection technology uses phase shift mask characteristics to enhance defect contrast, especially suitable for sub-resolution assist feature (SRAF, Sub-Resolution Assist Feature) detection. Electron beam detection technology, scanning electron microscope (SEM, scanning electron microscope) detection, ultra-high resolution (up to 1nm or less), slow speed, usually used for review and defect classification. Multi-beam electron beam detection, parallel multi-electron beam scanning, improves throughput. Computational detection techniques, such as reference comparison method: die-to-database, die-to-die, require high-precision image registration algorithms. That is, the existing mask defect detection techniques in semiconductor manufacturing all have different technical bottlenecks, such as CPU (Central Processing Unit, Central Processing Unit) serial processing limitations, traditional algorithms (such as FFT, morphological operations, registration) are inefficient on CPU; I / O delay problem: high-resolution image (>1 billion pixels) transmission becomes a performance bottleneck.

[0004] Therefore, how to improve the performance of mask defect detection in semiconductor manufacturing is a problem to be solved. SUMMARY

[0005] To solve the above problems, embodiments of the present application provide a mask defect detection method and system based on GPU acceleration, an electronic device, a computer readable storage medium, and a computer program product.

[0006] In a first aspect, to solve the above technical problems, the present application provides a mask defect detection method based on GPU acceleration, comprising: establish a hardware link between the camera and the GPU through a PCIE bus architecture, and build a target memory environment of the camera that is adapted to the GPU; perform image transmission on multi-channel image data of a test mask collected by the camera based on the hardware link and the target memory environment, and transmit the multi-channel image data to the GPU, wherein the image transmission follows an asynchronous transmission mechanism; perform GPU data processing on the multi-channel image data based on a CUDA stream technology through the GPU to obtain a defect binary image, wherein the GPU data processing is performed in parallel with the image transmission; perform data download on the defect binary image based on a CUDA asynchronous memory copy mechanism and download the defect binary image to a CPU coupled to the GPU, wherein the data download is performed in parallel with the GPU data processing; perform defect feature extraction on the defect binary image based on Blob analysis through the CPU to obtain a defect recognition result.

[0007] The technical scheme provided in the embodiments of the present application has the beneficial effects that: a hardware link is established between a camera and a GPU through a PCIE bus architecture, and a target memory environment of the camera that is adapted to the GPU is built; image transmission is performed on multi-channel image data of a test mask collected by the camera based on the hardware link and the target memory environment, and the multi-channel image data is transmitted to the GPU, wherein the image transmission follows an asynchronous transmission mechanism; GPU data processing is performed on the multi-channel image data based on a CUDA stream technology through the GPU to obtain a defect binary image, wherein the GPU data processing is performed in parallel with the image transmission; data download is performed on the defect binary image based on a CUDA asynchronous memory copy mechanism and the defect binary image is downloaded to a CPU coupled to the GPU, wherein the data download is performed in parallel with the GPU data processing; and defect feature extraction is performed on the defect binary image based on Blob analysis through the CPU to obtain a defect recognition result. In this way, the present application can bypass the traditional data transmission path of the CPU and the system memory, build a high-speed dedicated channel between the camera device, the GPU display memory and the CPU, significantly reduce the image data transmission delay, and improve the utilization rate of GPU computing resources; an independent asynchronous computing stream is built through the CUDA stream technology to realize efficient parallel computing of image transmission, image processing and defect detection, and millisecond-level parallel processing of multi-channel images can be realized, which improves the efficiency compared with single-stream serial processing; and finally, Blob analysis is used to realize accurate extraction of defect features, thereby realizing fast and efficient mask defect detection.

[0008] Further, the building of the target memory environment of the camera that is adapted to the GPU comprises: configuring an address space of the camera as a CUDA accessible memory that matches the GPU; setting a memory type of the camera as page-locked memory; setting the access engine for the camera to be a direct memory access engine; forming a target memory environment based on the CUDA accessible memory, the page-locked memory, and the direct memory access engine.

[0009] Further, the GPU data processing on the multi-channel image data based on the CUDA stream technology to obtain a defect binary image comprises: creating a plurality of independent computing streams based on the CUDA stream technology, each computing stream of the plurality of independent computing streams corresponding to one channel of image data of the multi-channel image data; performing a preset defect detection task on the corresponding one channel of image data through each computing stream to obtain a detection result; the plurality of independent computing streams perform the defect detection task in parallel, and the defect detection task comprises flat field correction, filtering and noise reduction, positioning registration, and out-of-tolerance detection; performing binaryzation processing on the detection result through a CUDA kernel function to obtain a binaryzation result; merging the binaryzation result corresponding to the detection result of each computing stream based on a parallel reduction operation to obtain a defect binary image of the multi-channel image data.

[0010] Further, the defect feature extraction on the defect binary image based on the Blob analysis by the CPU to obtain a defect recognition result comprises: uniformly dividing the defect binary image to obtain a plurality of image blocks; performing defect feature extraction on the plurality of image blocks based on a multi-thread parallelized connected component analysis algorithm to obtain a defect recognition result.

[0011] Further, the defect feature extraction on the plurality of image blocks based on the multi-thread parallelized connected component analysis algorithm to obtain a defect recognition result comprises: identifying connected regions in the plurality of image blocks through 8-neighborhood or 4-neighborhood labeling methods; obtaining geometric features of the connected regions and threshold conditions corresponding to the geometric features; comparing the geometric features with preset threshold values corresponding to the threshold conditions, and screening geometric features satisfying the threshold conditions from the geometric features as target geometric features; taking the connected regions corresponding to the target geometric features as target defect regions, and extracting defect information of the target defect regions to obtain a defect recognition result based on the defect information.

[0012] Further, the comparison of the geometric features with the preset threshold values corresponding to the threshold conditions to screen geometric features satisfying the threshold conditions from the geometric features as target geometric features comprises: The geometry features are compared with preset threshold values corresponding to the threshold conditions by using vector calculation optimization technology to obtain comparison results, which represent whether the threshold conditions are met or not met; Based on the comparison results, geometry features meeting the threshold conditions are selected from the geometry features as target geometry features; The preset threshold values are dynamically optimized by historical defect identification results.

[0013] Further, the method further comprises: The defect identification results are stored in a structured data format to obtain format data; The format data is transmitted to a downstream system in a multi-thread mode through a message queue or an RPC interface; The defect identification results are superimposed on the multi-channel image data through a visual debugging interface to perform algorithm verification and parameter tuning.

[0014] In a second aspect, the present application provides a GPU-accelerated mask defect detection system, comprising a link construction unit configured to establish a hardware link between a camera and a GPU through a PCIE bus architecture, and to construct a target memory environment of the camera compatible with the GPU; A camera image transmission unit is configured to perform image transmission of multi-channel image data of a mask to be tested collected by the camera to the GPU based on the hardware link and the target memory environment, and the image transmission follows an asynchronous transmission mechanism; An image processing unit is configured to perform GPU data processing on the multi-channel image data based on a CUDA stream technology through the GPU to obtain a defect binary image; the GPU data processing is performed in parallel with the image transmission; A defect binary image transmission unit is configured to perform data download of the defect binary image based on a CUDA asynchronous memory copy mechanism to a CPU coupled with the GPU; the data download is performed in parallel with the GPU data processing; An analysis unit is configured to perform defect feature extraction on the defect binary image based on Blob analysis through the CPU to obtain a defect identification result.

[0015] In a third aspect, the present application further provides an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the GPU-accelerated mask defect detection method as described above.

[0016] In a fourth aspect, the present application also provides a computer readable storage medium having computer readable instructions stored thereon, which, when executed by a processor of a computer, cause the computer to perform the GPU-accelerated mask defect detection method as described above.

[0017] In a fifth aspect, the present application also provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device performs the GPU-accelerated mask defect detection method provided in various optional embodiments described above.

[0018] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application. It is readily apparent to one of ordinary skill in the art that the accompanying drawings are merely some embodiments of the present application, and other drawings can be obtained from the accompanying drawings without any creative effort. In the drawings: Figure 1 is a flow chart of a GPU-accelerated mask defect detection method according to an exemplary embodiment of the present application; Figure 2 is a schematic diagram of the connection between a camera, a GPU and a CPU according to an embodiment of the present application; Figure 3 is a schematic diagram of the parallel processing of GPU data, image transmission and data download according to an embodiment of the present application; Figure 4 is a schematic diagram of obtaining a defect binary image as a GPU output result based on a CUDA stream technology according to an embodiment of the present application; Figure 5 is a flow chart of transmitting the defect binary image output by the GPU to the CPU for processing according to an exemplary embodiment of the present application; Figure 6 is a block diagram of a GPU-accelerated mask defect detection system according to an exemplary embodiment of the present application; Figure 7 is a structural schematic diagram of a computer system of an electronic device suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0020] The exemplary embodiments will be described in detail below with reference to the drawings. In the following description, the same numbers are used to denote the same elements throughout the several views. The embodiments described in the following exemplary embodiments are not meant to represent all implementations consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0021] The block diagrams shown in the drawings are merely functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the drawings are merely exemplary illustrations, and do not necessarily include all contents and operations / steps, nor are they necessarily executed in the order described. For example, some operations / steps can be further divided, and some operations / steps can be combined or partially combined, so the actual execution order can be changed depending on the actual situation.

[0023] In the present application, "a plurality of" means two or more. The association relationship of "and / or" describes the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after it.

[0024] In order to solve the problem of low performance caused by various bottlenecks in the mask defect detection technology in semiconductor manufacturing, the embodiments of the present application propose a GPU accelerated mask defect detection method and system, electronic equipment, and computer readable storage medium, which mainly relate to the GPU accelerated mask defect detection technology included in the detection and recognition technology. The embodiments will be described in detail below.

[0025] First, please refer to Figure 1 , Figure 1 is a flowchart of a GPU accelerated mask defect detection method according to an exemplary embodiment of the present application. The method can be specifically executed by a server, which can be an independent server, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms, etc. Basic cloud computing services are not limited here.

[0026] As Figure 1As shown, in an exemplary embodiment, the GPU acceleration-based mask defect detection method can include steps S101 to S105, which are described in detail as follows: Step S101, a hardware link between the camera and the GPU is established through a PCIE bus architecture, and a target memory environment of the camera compatible with the GPU is constructed.

[0027] Step S102, based on the hardware link and the target memory environment, the multi-channel image data of the mask to be tested collected by the camera is transmitted to the GPU for image transmission, and the image transmission follows an asynchronous transmission mechanism.

[0028] Step S103, the multi-channel image data is processed by the GPU based on the CUDA stream technology to obtain a defect binary image; the GPU data processing is performed in parallel with the image transmission.

[0029] Step S104, the defect binary image is downloaded based on the asynchronous memory copy mechanism of CUDA to the CPU connected with the GPU; the data download is performed in parallel with the GPU data processing.

[0030] Step S105, the defect binary image is analyzed by the CPU based on the Blob to extract the defect features and obtain the defect recognition result.

[0031] In this embodiment, the PCIE (Peripheral Component Interconnect Express) bus architecture is used to realize the physical direct connection between the camera device and the GPU (Graphics Processing Unit, graphics processor) computing card, and the PCIE 4.0 / 5.0 high-speed interface is also used to provide a theoretical bandwidth of 16 GB / s or more, to ensure low latency and high throughput of data transmission. Moreover, for the camera, a target memory environment of the camera compatible with the GPU is constructed to realize direct copying of image data to the GPU display memory space.

[0032] When transmitting the multi-channel image data of the mask to be tested collected by the camera, the asynchronous transmission mechanism is followed, and the CUDA (Compute Unified Device Architecture, computing unified device architecture) stream technology is used for GPU data processing of the multi-channel image data. Through the CUDA event (CUDA Event) and the asynchronous transmission mechanism, the precise scheduling of data transmission and computing tasks can be realized.

[0033] GPU data processing and image transmission are performed in parallel. In this way, image acquisition and GPU data processing tasks are parallelized by using an asynchronous transmission mechanism and a CUDA stream (CUDAStream) technology, so as to avoid performance bottlenecks caused by CPU intervention. Moreover, the application layer control program developed based on the CUDA programming model can be used to complete the GPU data processing and image transmission, so as to realize software cooperative control.

[0034] Then, the defect binary image is downloaded based on a CUDA-based asynchronous memory copy mechanism (CUDAMemcpyAsync) and is transmitted to the CPU system memory. The data download and the GPU data processing are performed in parallel. By setting appropriate CUDA stream priority and event synchronization (CUDAEvent), it is ensured that the image data can be quickly transferred to the CPU connected with the GPU in a zero-copy or pinned memory (page-locked memory) mode after the GPU calculation is completed, so as to reduce data transmission delay. Meanwhile, the high bandwidth characteristics of the PCIE bus are used to realize efficient transmission of GB-level image data.

[0035] In addition, the GPU and the CPU can also be connected through the PCIE bus. For example, a PCIE Gen4x16 channel between the GPU and the CPU is constructed based on the PCIE bus, as shown in Figure 2 . Figure 2 is a connection diagram among the camera, the GPU and the CPU in an embodiment of the present application. In the diagram, the PCIE bus among the camera, the GPU and the CPU is Gen4x16.

[0036] Please refer to Figure 3 , Figure 3 is a diagram of parallel GPU data processing, image transmission and data download in an embodiment of the present application. As shown in Figure 3 , H2D (Host to Device) represents the process of transmitting the multi-channel image data of the camera-acquired test mask to the GPU and its memory in the embodiments provided in the present application. The processing flow is the process of performing GPU data processing on the multi-channel image data by the GPU to obtain a defect binary image. D2H (Device to Host) represents the process of downloading the defect binary image from the GPU to the CPU in the embodiments provided in the present application.

[0037] Based on the CUDA stream (CUDAStream) technology, stream1 uploads data to the GPU memory while stream2 processes data, and stream2 uploads data to the GPU memory while stream3 processes data. In this way, data uploading or downloading and data processing are performed in parallel among multiple streams, so as to achieve the effect of efficiency optimization.

[0038] From the above, in the method provided in the embodiment, on the one hand, the traditional data transmission path of the CPU and the system memory is bypassed through the PCIE bus, a high-speed special channel is constructed among the camera device, the GPU display memory and the CPU, the image data transmission delay can be significantly reduced, and the GPU computing resource utilization rate can be improved; on the other hand, an independent asynchronous computing stream is constructed through the CUDA stream technology, efficient parallel computing of image transmission, image processing and defect detection is realized, millisecond-level parallel processing of multiple images can be realized, compared with single-stream serial processing, the efficiency is improved; and the Blob analysis is used to realize accurate extraction of defect features. Therefore, the application realizes a fast and efficient mask defect detection method from three aspects of data transmission link architecture, multi-thread parallel computing and defect feature recognition accuracy.

[0039] In an example embodiment of the application, the specific steps of constructing the target memory environment of the camera compatible with the GPU can include: configuring the address space of the camera as CUDA accessible memory compatible with the GPU; setting the memory type of the camera as page-locked memory; setting the access engine for the camera as a direct memory access engine; forming the target memory environment based on the CUDA accessible memory, the page-locked memory and the direct memory access engine.

[0040] The memory environment of the camera includes address space, memory type and access engine. In the embodiment, the address space of the camera is configured as CUDA accessible memory (CUDA-Aware Memory) compatible with the GPU, so that the images collected by the camera can be directly mapped to the address space recognizable by the GPU; the direct memory access (DMA, Direct Memory Access) engine is set; the memory type of the camera is set as page-locked memory (Page-Locked Memory), which is physical memory that cannot be swapped out.

[0041] Preferably, in the operating system kernel mode driver and the CUDA runtime environment, the NVIDIA GVAPI or OpenCL extension interface is enabled, the access permission of the camera device and the GPU display memory is configured, and the zero-copy data transmission link is established. By registering the camera memory pool as CUDA accessible memory, the memory page lock and the direct memory access (DMA) engine are set, and the direct copy transmission of image data from the camera sensor to the GPU display memory is realized.

[0042] In this way, through the above embodiment, the application can bypass the CPU memory copy link to realize the direct copy transmission of image data from the camera sensor to the GPU display memory.

[0043] In an example embodiment of the present application, the specific steps of obtaining the defect binary image by GPU data processing on the multi-channel image data based on the CUDA stream technology can include: creating a plurality of independent computing streams based on the CUDA stream technology, each computing stream in the plurality of independent computing streams corresponding to one channel of image data; performing a preset defect detection task on the corresponding one channel of image data through each computing stream to obtain a detection result; the plurality of independent computing streams perform the defect detection task in parallel, and the defect detection task includes flat field correction, filter denoising, positioning registration, and out-of-tolerance detection; performing binaryzation processing on the detection result through a CUDA kernel function to obtain a binaryzation result; merging the binaryzation result corresponding to the detection result of each computing stream based on a parallel reduction operation to obtain a defect binary image of the multi-channel image data.

[0044] In this embodiment, the CUDA multi-stream architecture is designed to create N independent computing streams (CUDA Stream) in the CUDA programming model, and each stream corresponds to one channel of image data processing. Please refer to Figure 4 , Figure 4 In an embodiment of the present application, the schematic diagram of obtaining the defect binary image as the GPU output result based on the CUDA stream technology is shown in FIG. 1.

[0045] As shown in Figure 4 , by using the asynchronous characteristics of the CUDA stream, the flat field correction, filter denoising, positioning registration, and out-of-tolerance detection tasks of the multi-channel image data are allocated to different streams and performed in parallel, and the efficient use of computing resources is realized by using the SIMT (Single Instruction Multiple Threads) architecture of the GPU. In each computing stream, the detection result is binaryzated by using the CUDA kernel function to obtain the binaryzation result output by each computing stream, and then the asynchronous result merging is performed on all the binaryzation results based on the parallel reduction operation to obtain the defect binary image of the multi-channel image data as the output result. At the same time, the data consistency of each processing stage is ensured by using the stream synchronization mechanism to avoid read-write conflicts.

[0046] In another embodiment, the defect detection task includes image preprocessing and positioning registration and out-of-tolerance detection, and the image and assistant include flat field correction and filter denoising.

[0047] Image preprocessing: flat field correction, calling CUDA kernel function for each image, eliminating non-uniformity of camera sensor by pixel-by-pixel multiplication and division of reference flat field image (or pre-calculated gain / offset coefficient). Using shared memory (Shared Memory) to cache flat field data, reducing global memory access delay. Filter denoising, using CUDA parallelized median filter or Gaussian filter algorithm, processing image pixel neighborhood through thread block cooperation mode. For large size filter kernel (such as 5x5 or above), use separable filter strategy to decompose two-dimensional filter into one-dimensional calculation in horizontal and vertical directions, reduce computational complexity.

[0048] Positioning registration and out-of-tolerance detection: positioning registration, based on CUDA accelerated template matching method or feature point matching algorithm such as SIFT (Scale-Invariant Feature Transform), ORB (Oriented FAST and Rotated BRIEF), feature extraction and matching between the image to be detected and the standard template image. Utilize CUDA's grid (Grid)-thread block (Block)-thread (Thread) hierarchical architecture to process image block regions in parallel, and achieve fast transmission of feature data through asynchronous memory copy. Out-of-tolerance detection, construct parallel threshold comparison CUDA kernel function, compare preprocessed image pixel values with preset tolerance range pixel by pixel. Use atomic operation (Atomic Operation) to aggregate out-of-tolerance pixel statistical information, and combine parallel prefix sum algorithm to quickly locate the defect area.

[0049] Thus, through the above embodiments, the application realizes millisecond-level parallel processing of multiple images, which improves the efficiency compared with single-stream serial processing, and can improve the result accuracy through defect detection tasks including flat field correction, filter denoising, positioning registration and out-of-tolerance detection.

[0050] In an example embodiment of the application, the specific steps of obtaining a defect recognition result by CPU based on Blob analysis of the defect binary image can include: uniformly dividing the defect binary image to obtain a plurality of image blocks; performing defect feature extraction on the plurality of image blocks based on a multi-thread parallelized connected component analysis algorithm to obtain a defect recognition result.

[0051] In another example embodiment, the specific steps of performing defect feature extraction on the plurality of image blocks based on a multi-thread parallelized connected component analysis algorithm to obtain a defect recognition result can include: identifying connected regions in the plurality of image blocks through 8-neighborhood or 4-neighborhood labeling method; Obtaining geometric features of the connected region and a threshold condition corresponding to the geometric features; Comparing the geometric features with a preset threshold corresponding to the threshold condition, and screening geometric features satisfying the threshold condition from the geometric features as target geometric features; Taking the connected region corresponding to the target geometric features as a target defect region, and extracting defect information of the target defect region to obtain a defect recognition result based on the defect information.

[0052] In this embodiment, a multi-thread parallel connected component analysis (CCA) algorithm is used at the CPU end, a binary image is divided into multiple image blocks for parallel processing based on an OpenMP or TBB (Threading Building Blocks) parallel framework. The connected regions (Blobs) in the image are identified through 8-neighborhood or 4-neighborhood labeling method, the geometric features (area, perimeter, centroid, aspect ratio, etc.) of each Blob are calculated, and are stored in a data structure (such as a linked list or an array). In addition, traditional machine learning algorithms such as XGBoost can be used to classify the defects of the target defect region.

[0053] In this way, through the above embodiments, the CPU multi-thread parallel computing and efficient data transmission can realize the Blob analysis task of processing hundreds of images per second, and meet the demand for rapid positioning and classification of defects.

[0054] In another exemplary embodiment, the specific steps of comparing the geometric features with a preset threshold corresponding to the threshold condition, and screening geometric features satisfying the threshold condition from the geometric features as target geometric features can include: The geometric features are compared with a preset threshold corresponding to the threshold condition by using a vector calculation optimization technology to obtain a comparison result, and the comparison result represents whether the threshold condition is satisfied or not satisfied; Based on the comparison result, geometric features satisfying the threshold condition are screened from the geometric features as target geometric features; The preset threshold is dynamically optimized by historical defect recognition results.

[0055] In this embodiment, the geometric features of the connected regions are compared with the preset threshold value through the preset configurable defect screening rule engine. The area, circularity, position coordinates and other parameters of each connected region are quickly screened through parallelized conditional judgment logic. Vector calculation optimization techniques such as SSE (Streaming SIMD Extensions) instruction set and AVX (Advanced Vector Extensions) instruction set are used to speed up numerical comparison operations, thereby improving the screening efficiency. Meanwhile, dynamic threshold adjustment is supported, and the threshold parameters are automatically optimized through historical data statistical analysis.

[0056] Then, the geometric features satisfying the threshold condition are screened from the geometric features based on the comparison result, as target geometric features.

[0057] In this way, the accurate extraction of defect features is realized through parallel identification of various geometric features according to the above-mentioned embodiments.

[0058] In an example embodiment provided by the present application, after outputting the defect recognition result, a post-processing step is further included, which can specifically include: storing the defect recognition result into a structured data format to obtain format data; transmitting the format data to a downstream system in a multi-threaded manner through a message queue or an RPC interface; superimposing the defect recognition result on multi-channel image data through a visual debugging interface to perform algorithm verification and parameter tuning.

[0059] In this embodiment, the subsequent output of the defect recognition result is divided into two categories: data transmission to a downstream system and visualization of the final result. The output to the downstream mainly includes serializing the Blob defect line to generate JSON or XML format and transmitting the defect information to the downstream through a message queue, and the whole process is realized through multi-threading. The visualization operation includes marking the defect contour to the ROI region corresponding to the original image and displaying it with different colors, which facilitates users to check the detection situation.

[0060] Please refer to Figure 5 , Figure 5 is a flowchart of transmitting the defect binary image output by the GPU to the CPU for processing in an example embodiment of the present application. As shown in Figure 5 , the binary image on the GPU side is transmitted in an asynchronous manner, and the data download and GPU data processing are parallelized. The binary image is transmitted to the CPU side through the PCIE Gen4x16 channel in the form of lock page memory.

[0061] The CPU adopts parallel Blob analysis, and parallelization is performed through OPMM or TBB. Specifically, the defect binary image is divided into 12 blocks, 12 threads are started, and the 12 blocks of images are processed in parallel. The processing process is to perform 8-neighbor connected component search, calculate the area, perimeter, centroid position, length and width of each connected component, and screen through the preset threshold of each attribute. The screened connected components are put into the Blob list.

[0062] The results are output subsequently, and are divided into two categories: data transmission to a downstream system and visualization of the final results. The output to the downstream mainly includes serializing the Blob defect line to generate a JSON or XML format, and transmitting the defect information to the downstream through a message queue. The whole process is implemented through multi-threading. The visualization operation includes marking the defect contour to the ROI region corresponding to the original image, and displaying it with different colors, which facilitates users to check the detection situation.

[0063] Figure 6 is a block diagram of a GPU-accelerated mask defect detection system 600 according to an example embodiment of the present application. As shown in Figure 6 , the system includes: a link construction unit 601 configured to establish a hardware link between a camera and a GPU through a PCIE bus architecture, and to construct a target memory environment of the camera compatible with the GPU; a camera image transmission unit 602 configured to perform image transmission of multi-channel image data of a mask to be tested collected by the camera to the GPU based on the hardware link and the target memory environment, the image transmission following an asynchronous transmission mechanism; an image processing unit 603 configured to perform GPU data processing of the multi-channel image data based on a CUDA stream technology through the GPU to obtain a defect binary image; the GPU data processing is performed in parallel with the image transmission; a defect binary image transmission unit 604 configured to perform data download of the defect binary image based on an asynchronous memory copy mechanism of CUDA to a CPU coupled with the GPU; the data download is performed in parallel with the GPU data processing; an analysis unit 605 configured to perform defect feature extraction of the defect binary image based on Blob analysis through the CPU to obtain a defect recognition result.

[0064] The system applies the GPU acceleration-based mask defect detection method provided in the application, the link construction unit 601 establishes a hardware link between the camera and the GPU through a PCIE bus architecture, and constructs a target memory environment of the camera that is adapted to the GPU; the camera image transmission unit 602 performs image transmission on the multi-channel image data of the mask to be tested collected by the camera based on the hardware link and the target memory environment, and transmits the image data to the GPU, and the image transmission follows an asynchronous transmission mechanism; the image processing unit 603 performs GPU data processing on the multi-channel image data based on the CUDA stream technology through the GPU, and obtains a defect binary image; the GPU data processing and the image transmission are executed in parallel; the defect binary image transmission unit 604 downloads the defect binary image based on the asynchronous memory copy mechanism of CUDA, and downloads the defect binary image to the CPU connected with the GPU; the data download and the GPU data processing are executed in parallel; the analysis unit 605 extracts defect features from the defect binary image based on Blob analysis through the CPU, and obtains a defect recognition result.

[0065] In this way, the application can bypass the traditional data transmission path of the CPU and the system memory, construct a high-speed special channel between the camera device, the GPU video memory and the CPU, significantly reduce the image data transmission delay, and improve the utilization rate of GPU computing resources; and an independent asynchronous computing stream is constructed through the CUDA stream technology, efficient parallel computing of image transmission, image processing and defect detection is realized, millisecond-level parallel processing of multi-channel images is realized, and the efficiency is improved compared with single-stream serial processing; finally, the Blob analysis is adopted to realize accurate extraction of defect features, thereby realizing a fast and efficient mask defect detection method.

[0066] In another exemplary embodiment, the link construction unit 601 is further configured to configure an address space of the camera as a CUDA accessible memory matched with the GPU; set the memory type of the camera as page-locked memory; set the access engine for the camera as a direct memory access engine; and form the target memory environment based on the CUDA accessible memory, the page-locked memory and the direct memory access engine.

[0067] In another exemplary embodiment, the image processing unit 603 is further configured to create a plurality of independent computing streams based on the CUDA stream technology, each computing stream of the plurality of independent computing streams corresponding to one channel of image data; execute a preset defect detection task on the corresponding one channel of image data through each computing stream to obtain a detection result; the plurality of independent computing streams execute the defect detection task in parallel, and the defect detection task includes flat field correction, filter denoising, positioning registration and out-of-tolerance detection; perform binaryzation processing on the detection result through a CUDA kernel function to obtain a binaryzation result; and merge the binaryzation result corresponding to the detection result of each computing stream based on a parallel reduction operation to obtain a defect binary image of the multi-channel image data.

[0068] In another example embodiment, the analysis unit 605 is further configured to uniformly divide the defect binary image to obtain a plurality of image blocks; and perform defect feature extraction on the plurality of image blocks based on a multi-thread parallelization connected domain analysis algorithm to obtain a defect recognition result.

[0069] In another example embodiment, the analysis unit 605 is further configured to identify a connected region in each of the plurality of image blocks by using 8-neighbor or 4-neighbor labeling method; obtain a geometric feature of the connected region and a threshold condition corresponding to the geometric feature; compare the geometric feature with a preset threshold corresponding to the threshold condition, and filter out a geometric feature satisfying the threshold condition from the geometric feature as a target geometric feature; identify a target defect region corresponding to the target geometric feature, and extract defect information of the target defect region to obtain the defect recognition result based on the defect information.

[0070] In another example embodiment, the analysis unit 605 is further configured to compare the geometric feature with the preset threshold corresponding to the threshold condition by using a vector calculation optimization technique to obtain a comparison result, the comparison result indicating whether the geometric feature satisfies the threshold condition or not; filter out the geometric feature satisfying the threshold condition from the geometric feature as the target geometric feature based on the comparison result; and wherein the preset threshold is dynamically optimized based on historical defect recognition results.

[0071] In another example embodiment, the system further comprises: The result output unit is configured to store the defect recognition result into a structured data format to obtain format data; transmit the format data to a downstream system based on a multi-thread mode through a message queue or an RPC interface; and superimpose the defect recognition result on the multi-channel image data through a visual debugging interface to perform algorithm verification and parameter tuning.

[0072] It should be noted that the GPU-accelerated mask defect detection system provided in the above embodiments and the GPU-accelerated mask defect detection method provided in the above embodiments belong to the same concept, and the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be repeated here. In actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0073] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a storage device configured to store one or more programs, when the one or more programs are executed by the one or more processors, the electronic device implements the GPU-accelerated mask defect detection method provided in each of the above embodiments.

[0074] Figure 7 A structural diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application is shown. It should be noted that, Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and usage range of the embodiments of the present application.

[0075] As Figure 7 shown, the computer system 700 includes a central processing unit (CPU) 701 which can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 702 or loaded from a storage section 708 into a random access memory (RAM) 703, such as performing the methods in the above embodiments. Various programs and data required for system operation are also stored in the RAM 703. The CPU 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0076] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; the storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as necessary. A removable recording medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 710 as necessary, so that a computer program read therefrom is installed into the storage section 708 as necessary.

[0077] In particular, the processes described above with reference to the flow charts can be implemented as computer software programs in accordance with the embodiments of the present application. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing computer programs for executing the methods shown in the flow charts. In such embodiments, the computer program can be downloaded and installed from a network by the communication section 709, and / or installed from the detachable medium 711. When the computer program is executed by the central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0078] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagating in the baseband or as part of a carrier wave propagating in the baseband, in which the computer readable computer program is carried. Such a propagating data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit the program for use by or in connection with an instruction execution system, apparatus or device. The computer program contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, or the like, or any suitable combination of the above.

[0079] The flowcharts and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowcharts, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0080] The units described in the embodiments of the present application can be implemented by software, or by hardware, or by a combination of software and hardware. The units described may

[0081] Another aspect of the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the GPU-accelerated mask defect detection method as described above. The computer readable storage medium can be included in the electronic device as described in the embodiments above, or can exist separately and not be assembled into the electronic device.

[0082] Another aspect of the present application provides a computer program product or computer program, which comprises computer instructions. The computer instructions are stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the GPU-accelerated mask defect detection method provided in the embodiments above.

[0083] The above only describes the preferred embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement or improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A GPU-accelerated mask defect detection method, characterized in that, The method includes: A hardware link between the camera and the GPU is established through the PCIe bus architecture, and a target memory environment for the camera that is compatible with the GPU is constructed. Based on the hardware link and the target memory environment, the multi-channel image data of the mask under test acquired by the camera is transmitted to the GPU, and the image transmission follows an asynchronous transmission mechanism. The GPU performs GPU data processing on the multi-channel image data based on CUDA streaming technology to obtain a defective binary image; the GPU data processing and the image transmission are executed in parallel. The CUDA-based asynchronous memory copy mechanism downloads the defective binary image to a CPU connected to the GPU; the data download and GPU data processing are executed in parallel. The CPU extracts defect features from the binary image of the defect based on Blob analysis to obtain the defect identification result.

2. The method according to claim 1, characterized in that, The construction of the target memory environment for the camera adapted to the GPU includes: Configure the camera's address space to CUDA-accessible memory that matches the GPU; Set the camera's memory type to page-locked memory; Set the access engine for the camera to the direct memory access engine; The target memory environment is formed based on the CUDA-accessible memory, the page-locked memory, and the direct memory access engine.

3. The method according to claim 1, characterized in that, The step of processing the multi-channel image data using the GPU based on CUDA streaming technology to obtain a defect binary image includes: Multiple independent computation streams are created based on CUDA streaming technology, and each of the multiple independent computation streams corresponds to one image in the multiple image data streams. Each computational stream performs a preset defect detection task on a corresponding image data stream to obtain a detection result; the multiple independent computational streams execute the defect detection task in parallel, and the defect detection task includes flat field correction, filtering and noise reduction, localization and registration, and out-of-tolerance detection; The detection results are binarized using a CUDA kernel function to obtain a binarized result. The binarized results corresponding to the detection results of each computation stream are merged based on the parallel reduction operation to obtain the defect binary image of the multi-channel image data.

4. The method according to claim 1, characterized in that, The step of extracting defect features from the binary defect image using Blob analysis by the CPU to obtain defect identification results includes: The defect binary image is uniformly divided to obtain multiple image blocks; Based on a multi-threaded parallelized connected component analysis algorithm, defect features are extracted from the multiple image blocks to obtain defect identification results.

5. The method according to claim 4, characterized in that, The connected component analysis algorithm based on multi-threading parallelization extracts defect features from the multiple image patches to obtain defect identification results, including: Connected regions in the multiple image blocks are identified using 8-neighbor or 4-neighbor labeling methods; Obtain the geometric features of the connected region and the threshold conditions corresponding to the geometric features; The geometric features are compared with a preset threshold corresponding to the threshold condition, and geometric features that meet the threshold condition are selected from the geometric features as target geometric features; The connected regions corresponding to the target geometric features are taken as the target defect regions, and the defect information of the target defect regions is extracted. Based on the defect information, the defect identification result is obtained.

6. The method according to claim 5, characterized in that, The step of comparing the geometric features with a preset threshold corresponding to the threshold condition, and selecting geometric features that satisfy the threshold condition from the geometric features as target geometric features, includes: The geometric features are compared with the preset threshold corresponding to the threshold condition using vector calculation optimization technology to obtain a comparison result, which indicates whether the threshold condition is met or not. Based on the comparison results, geometric features that meet the threshold conditions are selected from the geometric features and used as target geometric features; The preset threshold is dynamically optimized based on historical defect identification results.

7. The method according to claim 1, characterized in that, The method further includes: The defect identification results are stored in a structured data format to obtain formatted data; The data in the specified format is transmitted to the downstream system via a message queue or RPC interface using a multi-threaded approach. The defect identification results are superimposed onto the multi-channel image data through a visual debugging interface for algorithm verification and parameter optimization.

8. A GPU-accelerated mask defect detection system, characterized in that, include: The link building unit is used to establish a hardware link between the camera and the GPU through the PCIe bus architecture, and to build a target memory environment for the camera that is compatible with the GPU. The camera image transmission unit is used to transmit multiple image data of the test mask acquired by the camera to the GPU based on the hardware link and the target memory environment. The image transmission follows an asynchronous transmission mechanism. An image processing unit is used to perform GPU data processing on the multi-channel image data using the GPU based on CUDA streaming technology to obtain a defective binary image; the GPU data processing and the image transmission are executed in parallel. The defect binary image transmission unit is used to download the defect binary image to a CPU connected to the GPU based on the CUDA asynchronous memory copy mechanism; the data download and GPU data processing are executed in parallel. The analysis unit is used to extract defect features from the binary image of the defect using the CPU based on Blob analysis, and obtain the defect identification result.

9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the GPU-accelerated mask defect detection method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the GPU-accelerated mask defect detection method according to any one of claims 1 to 7.

Images