Screen defect online detection method and system based on multi-modal fusion mechanism

Abstract

The embodiment of the application provides a screen defect online detection method and system based on a multi-modal fusion mechanism, and belongs to the technical field of image processing and identification. The method steps comprise: scanning and preprocessing a to-be-detected film layer through a two-dimensional bright field imaging system, obtaining a two-dimensional gray image, determining and screening a candidate defect region, adding the candidate defect region to a three-dimensional scanning queue, performing three-dimensional measurement and dark field imaging processing through a white light interference three-dimensional measurement device, performing multi-feature fusion analysis in combination with a multi-modal detection mode, and determining whether the candidate defect region is an in-film defect or an out-of-film defect according to a fusion score. The embodiment of the application provides two aspects of advantages of high-speed screening and high-precision detection, reduces a false positive rate, reduces false defects caused by illumination, reflection and texture, and can distinguish in-film defects and out-of-film defects based on 3D height data, accurately judges a defect level, and greatly surpasses the detection level of related technologies of separate 2D and separate 3D.

Description

Technical Field

[0001] This application relates to the technical field of image processing and recognition, specifically to an online screen defect detection method and system based on a multimodal fusion mechanism. Background Technology

[0002] In the manufacturing process of optical film materials or display panels, thin films are usually formed into multi-layer structures through coating, lamination, or pressing processes. Due to factors such as tiny particles, bubbles, or material inhomogeneity in the production environment, defects at different levels can easily form in the film structure. Based on the location of the defects, they can usually be divided into two categories: intra-film defects and extra-film defects.

[0003] Intra-membrane defects refer to defects located inside the membrane layer or between the membrane layer and the substrate, such as internal bubbles, inclusions, or localized non-uniform structures of the material; extra-membrane defects are usually located on the surface of the membrane layer, such as dust particles, contaminants, or attached foreign matter.

[0004] With the rapid development of display technologies (such as OLED, LCD, and Mini-LED), the requirements for defect detection in screen manufacturing are becoming increasingly stringent. This is especially true for micron-level defects such as tiny particles, pits, scratches, black spots, and bright spots. High-throughput detection and high-precision height measurement are needed to determine whether the defect is located on the outer surface of the film or within the internal structure. First, rapid and efficient detection of common micron-level defects such as surface particles, scratches, pits, black spots, and bright spots is required (typically using bright-field imaging). Second, the ability to identify defects located within the film layer (such as internal foreign objects, bubbles, and delamination) is also necessary, as these defects are difficult to detect under conventional bright-field imaging. Furthermore, for critical defects affecting subsequent processes, quantitative height measurement capabilities are required to assess their impact on product reliability.

[0005] Currently, common defect detection methods in related technologies include:

[0006] 1. 2D vision camera inspection (e.g., Micro 2D camera): Advantages: High speed, high resolution, suitable for large-area rapid scanning.

[0007] Disadvantages: It cannot effectively measure depth information, cannot determine whether a defect is an internal or external surface defect, and has a high false alarm rate.

[0008] 2. 3D measurement (e.g., white light interferometer): Advantages: High depth measurement accuracy, capable of determining the height, depth, and layer location of defects.

[0009] Disadvantages: slow scanning speed, inability to scan the entire panel point by point, and high cost.

[0010] Based on the above methods of defect detection using 2D vision cameras and 3D measurement, the following problems and shortcomings exist in the relevant technologies: 1. Inability to balance speed and accuracy: Using 3D detection alone is too slow, while using 2D detection alone cannot provide height information.

[0011] 2. High risk of false alarms and false negatives: 2D can only be judged based on grayscale / texture, which is easily affected by lighting, surface reflection and process texture.

[0012] 3. Inability to distinguish between in-membrane and out-of-membrane defects: Simple 2D inspection cannot identify the hierarchical structure of defects, which directly affects process judgment and scrap decision.

[0013] 4. Lack of intelligent scheduling mechanism in defect screening process: In the existing system, 2D and 3D detection are usually performed independently, and no screening mechanism based on 2D features has been established to improve the efficiency of 3D detection.

[0014] In summary, the existing technologies can no longer meet people's requirements. Developing a comprehensive detection method that can simultaneously meet the needs of conventional surface defect detection, intrafilm defect identification, and high-quantification has become an urgent technical problem to be solved in this field. Summary of the Invention

[0015] The main objective of this application is to provide an online screen defect detection method and system based on a multimodal fusion mechanism to solve the problems existing in related technologies.

[0016] The embodiments of this application are implemented using the following technical solutions: According to one aspect of the embodiments of this application, an online screen defect detection method based on a multimodal fusion mechanism is provided, comprising: scanning the film layer to be detected using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image; preprocessing the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image; performing defect detection on the two-dimensional detection image to determine a first candidate defect region; filtering and scoring the first candidate defect region based on its two-dimensional visual features; determining a second candidate defect region when the score is higher than a preset threshold; adding the second candidate defect region to a three-dimensional scanning queue; performing three-dimensional measurement on the three-dimensional scanning queue using a white light interferometry three-dimensional measurement device; determining the scattering features of the second candidate defect region in the three-dimensional scanning queue through dark field imaging; performing multi-feature fusion analysis in conjunction with a multimodal detection method; obtaining a fusion score by calculating a comprehensive score; and determining whether the second candidate defect region is an intra-film defect or an extra-film defect based on the fusion score.

[0017] According to at least one specific embodiment of the present application, the two-dimensional bright field imaging system includes a bright field industrial camera and a uniform illumination source; the preprocessing of the two-dimensional grayscale image further includes: after obtaining the two-dimensional grayscale image, performing illumination homogenization correction and contrast enhancement processing on the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image.

[0018] According to at least one specific embodiment of the present application, the process of determining the second candidate defect region further includes: identifying gray-level change regions by calculating the gray-level changes between adjacent pixels in the image, determining adjacent pixels with the same or similar pixel values ​​in the first candidate defect region as the second candidate defect region by using a connected component analysis algorithm, and obtaining the geometric features of the second candidate defect region.

[0019] According to at least one specific embodiment of the present application, the second candidate defect region is added to a three-dimensional scanning queue, and a three-dimensional measurement of the three-dimensional scanning queue is performed using a white light interferometric three-dimensional measurement device. The method further includes: performing a three-dimensional scan of the three-dimensional scanning queue using the white light interferometric three-dimensional measurement device, and filtering the candidate defect region based on two-dimensional visual features, including: defect area, grayscale contrast, texture perturbation, and shape features; when filtering the candidate defect region based on the two-dimensional visual features, the filtering criteria include: defect area, grayscale contrast, texture perturbation, and shape features.

[0020] According to at least one specific embodiment of the present application, an axial scan of the second candidate defect region is performed using a white light interferometric three-dimensional measurement device, interference signals at different height positions are recorded, a three-dimensional height distribution map of the current scanned region is constructed, and the height value of each pixel object in the current scanned region is determined.

[0021] According to at least one specific embodiment of the present application, the step of determining the scattering characteristics of the second candidate defect region in the three-dimensional scanning queue by dark field imaging further includes: applying light irradiated at an inclined angle to the second candidate defect region, detecting whether the surface of the second candidate defect region generates scattered light, and if a ring-shaped scattering structure is detected in the dark field image, it is initially determined to be an intra-film defect, or: if no ring-shaped scattering structure is detected but a local bright spot is detected, it is initially determined to be an extra-film defect.

[0022] According to at least one specific embodiment of the present application, the step of determining the scattering characteristics of the second candidate defect region in the three-dimensional scanning queue through dark-field imaging, and performing multi-feature fusion analysis in combination with multi-modal detection, obtaining a fusion score by calculating a comprehensive score, and determining whether the second candidate defect region is an intra-membrane defect or an extra-membrane defect based on the fusion score, further includes: performing membrane deformation analysis on the second candidate defect region, determining the defect location in the three-dimensional height distribution map, and constructing an analysis region centered on the defect location; extracting all height data points in the analysis region to form a corresponding local height matrix, and determining whether the current membrane layer has bulging deformation by performing height distribution trend analysis on the local height matrix; if it is determined that the current membrane layer has bulging deformation, then searching outward from the defect location, determining the boundary of the bulging deformation when the height value drops to the background height, calculating the radius distance from the boundary to the center of the defect location, and judging whether the current membrane layer has undergone structural deformation based on the comparison result of the radius distance and the size of the defect location itself.

[0023] According to at least one specific embodiment of the present application, in the process of extracting all height data points in the analysis area, the height data points are smoothed using a neighborhood averaging method.

[0024] According to at least one specific embodiment of the present application, the multimodal detection method further includes: weighting the results of film deformation analysis, the characteristics of scattering ring structure and two-dimensional visual features, comprehensively judging the defect type through a fusion scoring mechanism to obtain a comprehensive judgment index, and finally determining whether it is an intra-film defect or an extra-film defect based on the comprehensive judgment index.

[0025] According to another aspect of the embodiments of this application, an online screen defect detection system based on a multimodal fusion mechanism is provided to implement the online screen defect detection method based on the multimodal fusion mechanism, comprising: a two-dimensional detection image preprocessing module, which scans the film layer to be detected using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image, preprocesses the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image; a candidate region generation module, which performs defect detection on the two-dimensional detection image to determine a first candidate defect region; a three-dimensional scanning queue generation module, which filters and scores the first candidate defect region based on its two-dimensional visual features, determines a second candidate defect region when the score is higher than a preset threshold, adds the second candidate defect region to the three-dimensional scanning queue, and performs three-dimensional measurement on the three-dimensional scanning queue using a white light interferometric three-dimensional measurement device; and a dark field imaging and multimodal detection judgment module, which determines the scattering features of the second candidate defect region in the three-dimensional scanning queue through dark field imaging, performs multi-feature fusion analysis in conjunction with the multimodal detection method, obtains a fusion score by calculating a comprehensive score, and determines whether the second candidate defect region is an intra-film defect or an extra-film defect based on the fusion score.

[0026] The beneficial technical effects of the embodiments of this application are: The technical solution provided in this application combines the advantages of high-speed screening and high-precision detection. 3D re-inspection can eliminate the results of 2D false detection, reduce the false alarm rate, and reduce false defects caused by illumination, reflection, and texture. It can also distinguish between intra-film and extra-film defects based on 3D height data, accurately determine the defect level, and significantly surpass the detection level of 2D and 3D alone in related technologies. It achieves functions that are difficult to achieve with 2D and 3D detection alone. Combining the advantages of 2D texture detection and 3D height detection, it makes the classification of intra-film and extra-film defects more accurate, greatly improves the defect judgment accuracy, and is suitable for quality control in high-end screen production.

[0027] This application embodiment achieves resource optimization and automatic scheduling by extracting grayscale abrupt change regions in two-dimensional detection images. By combining 2D rapid screening with 3D precise re-inspection, the number of 3D scans is significantly reduced. Only suspected defect areas in the 2D detection results are re-inspected in 3D, achieving high throughput of overall online detection data. This significantly reduces the workload of the white light interferometer and improves detection efficiency by 3–20 times.

[0028] The method and system provided in this application have strong adaptability and scalability. They are compatible with the visual architecture and motion platform of production lines in related technologies, and can also be extended to Mini-LED, glass substrate and other fields, with a wide range of application scenarios and markets. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the specific implementation methods or related technologies of this application, the accompanying drawings used in the description of the specific implementation methods or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some implementation methods of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of an online screen defect detection method based on a multimodal fusion mechanism.

[0031] Figure 2 This is a schematic diagram of the structure of defects inside and outside the screen film.

[0032] Figure 3 This is a flowchart illustrating the implementation of the embodiments of this application in a specific application scenario.

[0033] Figure 3A This is a flowchart of steps 1 to 6 in a specific application scenario.

[0034] Figure 3B This is a flowchart of steps 7 to 8 in a specific application scenario.

[0035] Figure 4 This is an architecture diagram of an online screen defect detection system based on a multimodal fusion mechanism.

[0036] Figure 5 This is a hardware framework diagram of an embodiment of this application in a specific application scenario. Detailed Implementation

[0037] To enable those skilled in the art to better understand the embodiments of this application, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the implementation methods of the embodiments of this application, and not all of the implementation methods. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the protection scope of the embodiments of this application.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of the present application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0039] like Figure 1 The screen defect online detection method based on multimodal fusion mechanism shown includes: Step S1: Scan the film layer to be detected using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image. Preprocess the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image.

[0040] Step S2: Perform defect detection on the two-dimensional detection image to determine the first candidate defect region. In this step, defect detection includes: grayscale change and connected region extraction. Due to the presence of defects, the grayscale of the image usually changes locally, such as sudden brightness changes, texture disturbances, or edge changes, etc. Therefore, defect detection is required for the two-dimensional detection image.

[0041] Step S3: Based on the two-dimensional visual features of the first candidate defect region, the region is screened and scored. When the score is higher than a preset threshold, the second candidate defect region is determined and added to the three-dimensional scanning queue. The three-dimensional scanning queue is then measured using a white light interferometric three-dimensional measurement device.

[0042] Step S4: The scattering characteristics of the second candidate defect region in the three-dimensional scanning queue are determined by dark field imaging, and multi-feature fusion analysis is performed by combining multi-modal detection method. A fusion score is obtained by calculating a comprehensive score, and the second candidate defect region is determined to be an intra-membrane defect or an extra-membrane defect based on the fusion score.

[0043] In the technical solutions provided in steps S1 to S4, firstly, a two-dimensional bright-field imaging system is used to perform a global scan and preprocessing of the film to be inspected, obtaining a high-quality two-dimensional inspection image. Then, using grayscale change detection and connected region extraction techniques, the first candidate defect region is quickly located from the two-dimensional inspection image. Based on the two-dimensional visual characteristics of the first candidate defect region, it is screened and scored. Regions with scores higher than a preset threshold are identified as second candidate defect regions and added to the three-dimensional scanning queue. High-precision three-dimensional measurement is then performed on them by a white light interferometry three-dimensional measurement device. Finally, multi-feature fusion analysis is performed by combining the scattering characteristics obtained from dark-field imaging and multi-modal detection data. By calculating a comprehensive score, it is determined whether the second candidate defect region belongs to an intra-film defect or an extra-film defect.

[0044] In the technical solutions provided in steps S1 to S4, firstly, a two-dimensional bright-field imaging system is used to perform a global scan and preprocessing of the film to be inspected, obtaining a high-quality two-dimensional inspection image. Then, using grayscale change detection and connected region extraction techniques, the first candidate defect region is quickly located from the two-dimensional inspection image. Based on the two-dimensional visual characteristics of the first candidate defect region, it is screened and scored. Regions with scores higher than a preset threshold are identified as second candidate defect regions and added to the three-dimensional scanning queue. High-precision three-dimensional measurement is then performed on them by a white light interferometry three-dimensional measurement device. Finally, multi-feature fusion analysis is performed by combining the scattering characteristics obtained from dark-field imaging and multi-modal detection data. By calculating a comprehensive score, it is determined whether the second candidate defect region belongs to an intra-film defect or an extra-film defect.

[0045] Step S1 provides a preprocessed two-dimensional inspection image. Step S2 has a fast two-dimensional defect detection speed and a large coverage area, and can complete a global scan of the entire product in a short time. However, it cannot provide height information of the defect area. Step S3 introduces a screening mechanism based on two-dimensional visual features, which sends only candidate defect areas with scores higher than a preset threshold into the three-dimensional scanning queue. This avoids the three-dimensional measurement device scanning all areas point by point in Step S4, and concentrates the computational resources of three-dimensional measurement on the defect areas that are most likely to affect product quality, thus balancing detection speed and measurement accuracy.

[0046] Since step S2 relies solely on grayscale changes for defect detection, it is easily affected by interference factors such as uneven lighting, surface reflection, or process textures, resulting in false detections. Therefore, step S3 uses two-dimensional visual features (such as defect area, grayscale contrast, texture disturbance, shape features, etc.) to perform secondary screening on the candidate regions output by step S2, eliminating false defects that clearly do not conform to the defect features, thereby reducing invalid targets entering the subsequent detection process. Step S4 further uses dark field scattering features and multimodal fusion analysis to further determine the remaining candidate regions.

[0047] In summary, steps S1 to S4 form a collaborative relationship of coarse inspection, screening, and fine judgment. The two-dimensional inspection in step S2 can only determine the presence of defects, not their layer location. While the screening mechanism in step S3 reduces the number of targets to be inspected, it also fails to provide layer information. Step S4 introduces dark-field imaging to obtain scattering characteristics and combines this with multi-modal inspection data for multi-feature fusion analysis. A comprehensive score is calculated to determine whether the defect is located inside or on the membrane layer. Through multiple steps of layer-by-layer inspection and filtering, the false alarm rate of the final output is minimized, ensuring reliable differentiation between internal and external defects and providing a reliable basis for subsequent process decisions and handling (such as rework or scrapping).

[0048] Preferably, in step S1, the two-dimensional bright field imaging system includes a bright field industrial camera and a uniform illumination source; the preprocessing of the two-dimensional grayscale image further includes: after obtaining the two-dimensional grayscale image, performing illumination homogenization correction and contrast enhancement processing on the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image.

[0049] During the production of display panels or optical films, various defects may exist on the product surface, such as dust particles, bubbles, and scratches. To achieve high-speed inspection, this step first requires scanning and imaging the entire product using a two-dimensional vision system.

[0050] The system scans the object to be inspected using a bright-field industrial camera and a uniformly illuminated light source, acquiring a two-dimensional grayscale image. However, in actual production environments, factors such as light source brightness distribution, lens vignetting, and environmental reflections can lead to uneven image brightness, thus affecting the stability of defect detection. Therefore, preprocessing of the acquired image is necessary in this step.

[0051] Preprocessing mainly includes the following aspects: (1) Illumination homogenization processing: The system first acquires an image of a defect-free sample as a background reference image, and uses this reference image to perform illumination correction on the detection image. Illumination homogenization processing can effectively eliminate the influence of uneven illumination on the detection results, making the brightness distribution of the entire image more stable.

[0052] (2) Image noise reduction: Since random noise is inevitably generated during the high-speed acquisition process of industrial cameras, it is necessary to perform noise reduction on the images. In the process of image noise reduction, spatial filtering is used to smooth the images to reduce the impact of noise on subsequent defect detection, while preserving the edge information of the defects as much as possible.

[0053] (3) Contrast enhancement: For some minor defects, the grayscale changes may be weak. Therefore, the system can improve the grayscale difference between the defect area and the background through image contrast enhancement algorithms, thereby improving the detection sensitivity.

[0054] Through the above preprocessing steps, a detection image with uniform illumination, low noise, and high contrast can be obtained, providing a reliable data foundation for subsequent defect detection.

[0055] Preferably, in step S2, defect detection is performed on the two-dimensional detection image to determine the first candidate defect region, which further includes: After image preprocessing, it is necessary to automatically detect potential defect areas in the image. Since defects usually cause local changes in image grayscale, such as abrupt changes in brightness, texture perturbations, or edge changes, the system can identify defects by analyzing local grayscale changes in the image.

[0056] Specific implementation methods include: (1) Gray-scale change detection: The system identifies areas with large gray-scale changes by calculating the gray-scale changes between adjacent pixels in the image. Areas with large gray-scale changes often correspond to defect edges or structurally abnormal areas.

[0057] (2) Connected component extraction: When a gray-scale change region is detected, the system merges adjacent abnormal pixels into a complete defect region through a connected component analysis algorithm, and calculates the geometric features of the region, such as area, width and position.

[0058] (3) Candidate defect generation: Through the above steps, the system can obtain a series of candidate defect regions, each of which contains the location coordinates and size information of the defect. These candidate defects will be used as input for subsequent analysis.

[0059] The advantage of this two-dimensional inspection step is its fast inspection speed and wide coverage, which can complete the scanning of the entire product in a short time.

[0060] After completing bright-field image acquisition and preprocessing (S1), potential defect areas need to be automatically detected from the image. This step uses bright-field illumination for two-dimensional defect detection. Its main task is to detect common surface defects. Bright-field illumination refers to light illuminating the surface to be detected in a vertical or near-vertical direction, with reflected light returning along the original path to the imaging system. Under bright-field conditions, flat areas of the surface exhibit uniform brightness, while common surface defects (such as scratches, particles, pits, contamination, etc.) disrupt the surface flatness, causing changes in local reflected light intensity, thus creating a clear contrast between light and dark in the image.

[0061] The specific implementation method is as follows: (1) Gray-scale change detection: The system identifies regions with significant grayscale variations by calculating the grayscale changes between adjacent pixels in an image. These regions often correspond to defect edges or areas of structural anomalies, such as linear grayscale abrupt changes caused by scratches, local bright spots caused by particles, and dark areas caused by pits.

[0062] (2) Connected component extraction: Once a grayscale variation region is detected, the system uses a connected component analysis algorithm to merge adjacent abnormal pixels into a complete defect region and calculates the geometric features of the region, such as area, aspect ratio, circumscribed rectangle size, and centroid coordinates.

[0063] (3) Enhanced contrast: For minor defects, adaptive histogram equalization is used to improve the grayscale difference between the defect area and the background, thereby enhancing detection sensitivity.

[0064] (4) Defect classification: The basis for preliminary classification of candidate defects is shown in Table 1. After completing the extraction of connected regions and obtaining the geometric features of each defect region, the system performs preliminary classification of candidate defects based on the shape, grayscale characteristics and size parameters of the defects.

[0065] Table 1. Candidate Defect Type Classification Table

[0066] Preferably, in the process of determining the second candidate defect region in step S3, the method further includes: identifying gray-level change regions by calculating the gray-level changes between adjacent pixels in the image, determining the adjacent pixels with the same or similar pixel values ​​in the first candidate defect region as the second candidate defect region by using a connected component analysis algorithm, and obtaining the geometric features of the second candidate defect region.

[0067] In step S3, the gray-scale abrupt change region refers to an area with a large change in gray-scale. This region often corresponds to defect edges or structural anomalies. A second candidate defect region is determined by identifying pixels with the same or similar values. The geometric features of the second candidate defect region include area, width, and location. Pixels with the same or similar values ​​can be defined as those where the absolute difference in gray-scale values ​​between adjacent pixels is less than or equal to a preset gray-scale difference threshold. This gray-scale difference threshold can be preset based on the background gray-scale fluctuation range of the film layer to be detected, or dynamically calculated based on the gray-scale standard deviation of the content of the first candidate defect region. The purpose of determining the second candidate defect region in step S3 is that although the two-dimensional detection provided in step S2 can quickly locate defects, it cannot provide information on the height of the defects, making it difficult to determine whether the defects are located inside or outside the film layer.

[0068] To further obtain three-dimensional information about the defects, a white light interferometric three-dimensional measurement device was introduced in subsequent steps. However, the white light interferometric three-dimensional scanning speed is relatively slow, and if all defects are measured in three dimensions, the detection efficiency will be severely reduced. Therefore, in step S3, the first candidate regions obtained from the two-dimensional detection need to be screened, and only the defects most likely to affect product quality are selected to enter the three-dimensional re-inspection process. Thus, step S3 needs to identify and determine the second candidate defect regions.

[0069] To obtain the second candidate region, the screening criteria for the first candidate region mainly include the following aspects: Defect area: Larger defects are usually more serious.

[0070] Gray-scale contrast: The higher the contrast, the more obvious the defects.

[0071] Texture perturbation: The degree of texture change in the defective area.

[0072] Shape characteristics: Irregular shapes may indicate abnormal structures.

[0073] The system calculates a comprehensive score for each defect by comprehensively evaluating the above characteristics. When the score is higher than a preset threshold, a second candidate defect region is identified and enters the 3D scanning queue.

[0074] In addition, in order to ensure the overall detection efficiency of the system, the screening threshold can be dynamically adjusted according to the current task load of the 3D scanning equipment in step S3. When there are too many scanning tasks, the system automatically increases the threshold, thereby reducing the number of 3D scans and improving scanning efficiency.

[0075] Preferably, in step S3, the second candidate defect region is added to the three-dimensional scanning queue, and the three-dimensional scanning queue is measured using a white light interferometric three-dimensional measurement device, further including: Step S31: Perform a three-dimensional scan of the three-dimensional scanning queue using a white light interferometric three-dimensional measurement device, and screen candidate defect regions based on two-dimensional visual features, including: defect area, grayscale contrast, texture perturbation, and shape features.

[0076] Step S32: When screening candidate defect regions based on two-dimensional visual features, the screening criteria include: defect area, grayscale contrast, texture perturbation, and shape features.

[0077] As an example, in step S31, the second candidate defect region is axially scanned using a white light interferometric 3D measurement device. Interference signals at different height positions are recorded, a 3D height distribution map of the currently scanned region is constructed, and the height value of each pixel object in the currently scanned region is determined. The purpose of constructing the 3D height distribution map is to obtain the height and shape of the defect, as well as the height variations of the surrounding area, by analyzing the 3D height map.

[0078] The purpose of using a white-light interferometry (BNI) device for local scanning is to further acquire the three-dimensional structural information of the defect region. The BNI device reconstructs the three-dimensional height distribution of the region by performing a small axial scan of the detection area and recording interference signals at different height positions. Since the two-dimensional detection has already determined the location of the defect, the BNI only needs to scan a small area near the defect, significantly reducing scanning time. After scanning, the system obtains a three-dimensional height map of the region, where each pixel corresponds to a height value. By analyzing the three-dimensional height map, the height and shape of the defect region, as well as the height variations of the surrounding area, can be obtained.

[0079] Preferably, in step S4, determining the scattering characteristics of the second candidate defect region in the three-dimensional scanning queue through dark-field imaging further includes: A light source at an oblique angle is applied to the second candidate defect region, and the surface of the region is examined for scattered light. If a ring-shaped scattering structure is detected in the dark-field image, it is preliminarily identified as an intra-film defect; conversely, if no ring-shaped scattering structure is detected but local bright spots are observed, it is preliminarily identified as an extra-film defect. Dark-field imaging technology is used to further identify intra-film defects. Dark-field illumination is a special illumination method that provides a very important criterion for determining intra-film defects. The light from dark-field illumination is applied to the detection surface at an oblique angle, so that flat areas produce almost no reflection, while only locations with changes in surface structure or internal structural anomalies produce scattered light.

[0080] In actual testing, it was found that defects inside the film usually form obvious ring-shaped scattering structures in dark-field images, while defects outside the film (particles) usually only appear as local bright spots and do not form complete scattering rings. The reason is that defects inside the film will change the propagation path of light inside the film layer, thereby forming a ring-shaped scattering structure. This feature is often difficult to observe in bright-field images and can only be revealed with the help of dark-field illumination.

[0081] Therefore, in step S3, the presence of a scattering ring is detected by analyzing the grayscale distribution around the defect under dark field illumination. If a distinct ring-shaped bright area can be detected within a certain range around the defect area, the defect is likely located inside the film layer, and the current defect can be preliminarily determined to be an intrafilm defect. The method is as follows: Global Defect Detection in Dark Field Images (Preset Period Difference Method): Based on the preset row and column periods, each pixel is differentiated from the pixels at the adjacent period positions on the left, right, top, and bottom to obtain an abnormal response map. Then, candidate defect regions are extracted through threshold segmentation and connected component analysis.

[0082] Local region extraction: For each candidate defect, the region of interest (e.g., 201×201 pixels) is extracted with the centroid as the center for subsequent scattering structure analysis.

[0083] Gray-scale distribution analysis: Calculate the mean and variance of gray-scale values ​​at different radii within the region of interest to identify whether there are annular bright areas that are significantly different from the surrounding background.

[0084] Ring structure determination: If a continuous or nearly continuous ring-shaped bright area is detected within a certain range around the defect, the defect is determined to have the dark field scattering characteristics of an intrafilm defect and is considered an intrafilm defect; otherwise, it is determined to be a surface or pseudo-defect (such defects can be covered by bright field detection and do not require further processing).

[0085] Dark-field scattering characteristics provide crucial criteria for identifying intrafilm defects. Bright-field and dark-field methods work in tandem—bright-field rapidly detects surface defects, while dark-field directly identifies intrafilm defects. The results from both methods can be fused and cross-validated to achieve comprehensive coverage of various defects. Intrafilm defects detected by the dark-field method will serve as the triggering condition for subsequent white-light interferometry measurements.

[0086] Preferably, in step S4, the step of determining the scattering characteristics of the second candidate defect region in the three-dimensional scanning queue through dark-field imaging, and performing multi-feature fusion analysis in conjunction with multi-modal detection, obtaining a fusion score by calculating a comprehensive score, and determining whether the second candidate defect region is an intra-membrane defect or an extra-membrane defect based on the fusion score, further includes: Step S41: Perform film deformation analysis on the second candidate defect region, determine the defect location in the three-dimensional height distribution map, and construct an analysis region centered on the defect location. The range of the analysis region is usually slightly larger than the defect size.

[0087] Step S42: Extract all height data points from the analysis area to form a corresponding local height matrix. Analyze the height distribution trend of the local height matrix to determine whether bulging deformation exists in the current membrane layer. For example, during the extraction of all height data points from the analysis area, a neighborhood averaging method is used to smooth the height data points. In this step, if bulging deformation exists in the membrane layer, the height curve typically exhibits a continuous decreasing trend from the center.

[0088] Step S43: If it is determined that there is a bulge deformation in the current membrane layer, then search outward from the defect location as the center. When the height value drops to the background height, determine the boundary of the bulge deformation, calculate the radius distance from the boundary to the center of the defect location, and determine whether the current membrane layer has undergone structural deformation based on the comparison result of the radius distance and the size of the defect location itself.

[0089] In the optimized technical solutions provided in steps S41 to S43, the three-dimensional height data obtained by the white light interferometry system is analyzed to identify the local deformation of the film surface and further determine whether the defect is located inside the film.

[0090] In the white light interferometry three-dimensional height measurement process in step S4, the preceding bright-field detection located conventional surface defects, and the dark-field detection further identified candidate defects with intrafilm defect scattering characteristics. To obtain accurate three-dimensional structural information of these defects and assess their potential impact on subsequent processes, this invention employs a white light interferometer for local scanning.

[0091] The white light interferometer reconstructs the three-dimensional height distribution of the detection area by performing a tiny axial scan (Z-axis scan) and recording the interference signals at different height positions.

[0092] Since two-dimensional detection has already located the defect, and dark-field detection further filters out suspected intrafilm defects, the white-light interferometer only needs to scan a small area near the defect, greatly reducing scanning time. The specific process is as follows: 1. Scanning area positioning: Based on the coordinates of the center of the defect in the membrane, set a local scanning area centered on those coordinates (e.g., 100μm × 100μm).

[0093] 2. Axial scanning and interference signal acquisition: Drive the white light interference objective to perform step-by-step scanning in the vertical direction, and simultaneously acquire the interference image sequence corresponding to each height position.

[0094] 3. 3D Height Reconstruction: Envelope demodulation or phase analysis is performed on the interference signal of each pixel to extract the height position corresponding to the peak value of the interference signal, and a complete 3D height map of the scanned area is reconstructed. Each pixel corresponds to a height value.

[0095] 4. Defect Height Parameter Extraction: By analyzing this height map, the following information can be obtained: Maximum and average height of the defect; The three-dimensional morphology and outline of the defect; Height variation curves of the area surrounding the defect (especially the annular area).

[0096] 5. Impact assessment on subsequent processes: The criteria for assessing test defects are shown in Table 2. The core purpose of assessing test defect height is to determine whether the defect will adversely affect subsequent processes. Specific assessment criteria are as follows: Table 2. Test Defect Assessment Basis Table

[0097] like Figure 2 As shown, during the film manufacturing process, when foreign objects are present inside the film layer, the film layer will be pushed up by internal particles or bubbles, causing localized bulging deformation on the film surface. However, when the defect is located outside the film layer, such as surface dust or attached particles, the film layer itself will not undergo significant deformation, only exhibiting localized height protrusions. Therefore, by analyzing the height variation trend of the film layer surface, it is possible to effectively distinguish between internal and external defects.

[0098] Glossary of terms in the attached image: The abbreviations in the attached diagram are standard professional abbreviations in the display panel (LCD / OLED) manufacturing industry, among which: OC (OverCoating) means: overcoating, outer coating, planarization layer, located above the RGB color resist layer (the top layer in the diagram). Its main functions include: protecting the underlying R / G / B color resist layer from physical or chemical damage; providing a highly flat surface for subsequent fabrication of ITO electrodes or encapsulation; and serving as part of the encapsulation protective layer in OLEDs. The material of OC is usually a transparent photosensitive resin or an organic / inorganic composite film.

[0099] R / G / B stands for Red / Green / Blue, which means red / green / blue. In Chinese, it means RGB color resist layer. In different application scenarios, it can represent color photoresist units or light-emitting units. The RGB layer is located below the OC layer and above the Glass substrate. The main function of the RGB layer is to realize color display. For example, in LCD, it can be the color resist of the color filter, and in OLED, it is the red, green and blue light-emitting material.

[0100] BM (Black Matrix) is a black matrix located above the glass substrate, between the RGB color resist layers, and below the edges. It is the black square between the R / G / B pixels and the glass substrate in the diagram. It is used to block light leakage between R / G / B pixels, improve display contrast, prevent optical crosstalk between adjacent color pixels, and prevent backlight light from leaking from the pixel gaps. Its manufacturing material is usually a black photosensitive resin containing carbon black.

[0101] From a structural perspective, intramembrane defects typically have the following characteristics: First, the defects are located inside the membrane layer, and therefore are surrounded by a transparent membrane material. When light enters the membrane layer, it is refracted and scattered at the interface between the internal particles and the membrane material.

[0102] Secondly, as the membrane is lifted up by the internal particles, a certain range of continuous deformation will form on the surface of the membrane, and this deformation usually shows a height distribution that gradually weakens from the center to the outside.

[0103] Third, internal defects have a certain spatial expansion characteristic in scattering light, so under dark field lighting conditions, obvious scattering ring structures often form around the defects.

[0104] From a physical perspective, extramembrane defects typically have the following characteristics: First, since the particles are directly attached to the surface of the film, their height variation is usually limited to the particles themselves and does not form a large area of ​​continuous deformation.

[0105] Secondly, since the particles are not encased in a transparent film, the scattering behavior of light on their surface is different from that of the particles inside the film. Therefore, no obvious ring-shaped scattering structure is usually formed in dark-field imaging.

[0106] Third, the morphology of extracellular particles is usually more random, and their size and shape differ to some extent from typical intracellular defects.

[0107] In summary, it can be seen that due to the different levels of defects, their characteristics in optical imaging and structural morphology also vary significantly.

[0108] In two-dimensional visual inspection using related technologies, intra-film defects and extra-film particles often appear similarly in images, typically as localized bright or dark areas. In such cases, relying solely on bright-field images makes it difficult to determine the layer location of the defect. While dark-field scattering detection can identify intra-film defects to some extent, in actual production environments, some surface particles may also produce scattering phenomena, leading to misjudgments. Therefore, relying solely on scattering features still has certain limitations.

[0109] To address the aforementioned issues, steps S41 to S43 introduce three-dimensional height information to analyze the membrane structure surrounding the defect area, thereby identifying whether membrane bulging deformation caused by internal foreign matter exists, thus providing a more reliable structural basis for defect classification.

[0110] Membrane deformation mechanism: When a foreign object is present inside the membrane, it provides local support, causing the membrane to bulge outwards. The membrane surface typically exhibits the following characteristics at this time: (1) The height of the central area is the highest: The center of the defect usually corresponds to the location of the foreign object inside, so the height of this area will be significantly higher than that of the surrounding area.

[0111] (2) The height gradually decreases outward: When expanding outward from the defect center, the film deformation will gradually weaken, and the height value will show a continuous downward trend.

[0112] (3) The deformation range is larger than the defect size: Due to the flexibility of the film, the deformation generated by the internal particles will diffuse to the surrounding area. Therefore, the deformation area is usually larger than the size of the defect itself, which can be approximated as a smooth bulge structure. Its height distribution can be expressed as:

[0113] in: H represents the maximum height of the film bulge, (xc,yc) represents the location of the defect center, and σ represents the deformation diffusion range.

[0114] In contrast, extracellular particles typically only exhibit localized high protrusions, while the surrounding area retains its original planar structure and does not form significant continuous deformation.

[0115] Steps S41 to S43 perform region analysis on the three-dimensional height map obtained by the white light interferometry system. The specific implementation process includes the following stages.

[0116] (1) Defect area location: First, based on the defect coordinate information obtained in the previous steps, determine the corresponding defect location in the three-dimensional height map, and construct the analysis area with this location as the center. The range of the analysis area is usually slightly larger than the defect size to ensure that the height change information around the defect can be fully obtained.

[0117] (2) Height data extraction: After determining the analysis area, the system extracts all height data points within that area and forms a local height matrix for subsequent analysis. In this process, to improve data stability, the height data can be simply smoothed, for example, by using the neighborhood averaging method to reduce the impact of measurement noise.

[0118] (3) Height variation trend analysis: After obtaining the regional height data, the system further analyzes the height distribution trend, which includes: calculating the maximum height value within the region; calculating the average height of the region; and analyzing the height variation curve from the defect center outward. If the membrane layer has bulging deformation, the height curve usually shows a continuous variation trend of gradually decreasing from the center.

[0119] (4) Deformation range assessment: After determining the height change trend, the system further calculates the spatial range of the deformation area. The specific method is as follows: Starting from the defect center, search outwards. When the height value drops to near the background height, determine the deformation boundary and calculate the distance (deformation radius) from the deformation boundary to the center of the region. If the deformation radius is significantly larger than the size of the defect itself, it can be considered that the membrane has undergone structural deformation.

[0120] Based on the above-mentioned film deformation determination logic, and combined with the height analysis results, the following determination rules can be established: Rule 1: There is a clear and continuous change in height: If the height gradually decreases from the center of the defect outwards and forms a clear bulge structure, it indicates that the film layer has deformed and the defect may be an intrafilm defect.

[0121] Rule 2: Height variation is limited to a local area: If the height variation is concentrated only within the defect itself, and the surrounding membrane layer remains flat, then the defect is an extra-membrane particle.

[0122] Rule 3: Deformation range is significantly larger than defect size: When the deformation range is significantly larger than defect size, it indicates that the film layer is affected by the internal support structure, and the defect belongs to the intrafilm defect.

[0123] Based on the above judgment rules, it can be finally determined whether the current defect is an intramembrane defect.

[0124] Preferably, the multimodal detection method further includes: The results of membrane deformation analysis, the characteristics of the scattering ring structure, and the two-dimensional visual features are weighted and assigned. A fusion scoring mechanism is used to comprehensively determine the defect type, resulting in a comprehensive judgment index S. Based on this comprehensive judgment index, the defect is ultimately identified as either an intramembrane defect or an extramembrane defect. During the comprehensive judgment process, data normalization and outlier filtering are performed, converting various features into quantifiable scoring indices. These scoring indices represent the degree to which the corresponding features support the determination of intramembrane defects.

[0125] The multimodal detection process follows two-dimensional bright-field detection, dark-field scattering feature analysis, and three-dimensional height measurement via white light interferometry. At this point, the system has acquired various data describing the defect characteristics, including the defect's morphological information in the two-dimensional image, scattering characteristics under dark-field conditions, and three-dimensional height distribution information. Since different detection methods emphasize different aspects of the information acquired, relying solely on one type of data for judgment is susceptible to influences such as illumination variations, surface reflection, or measurement noise, leading to misjudgments. Therefore, this step employs a multi-feature fusion judgment mechanism to uniformly process and comprehensively analyze data from different detection modules, thereby improving the accuracy and stability of identifying intra-membrane and extra-membrane defects.

[0126] I. The following section uses the sources of multimodal data, feature extraction, and main functions as examples to illustrate multimodal detection methods. Table 3 shows the methods for performing multimodal detection: Table 3 Multimodal Detection Methods

[0127] The above data are complementary in a physical sense: two-dimensional visual data reflects the planar structure of the defect, dark field scattering data reflects the optical scattering characteristics, and three-dimensional height data reflects the changes in the film structure.

[0128] II. Feature Data Preprocessing: Table 4 shows the steps, content, and purpose of feature data preprocessing. The system first preprocesses feature data from different detection modules, including feature scale unification and outlier filtering. Since the defect area, scattering intensity, and height data exhibit significant differences in numerical range, the system needs to normalize various features to convert different types of data into a unified numerical range. This prevents a particular feature from excessively influencing the judgment results in subsequent calculations due to an excessively large numerical range. Simultaneously, to reduce interference from outlier data caused by noise or local reflections, the system performs stability checks on height and scattering data, filtering or correcting outliers that significantly deviate from the overall trend to ensure high reliability of the data used in the fusion calculation.

[0129] Table 4 Feature Data Preprocessing Table

[0130] The original range and normalization process for feature type data transformation are shown in Table 5: Table 5 Feature Type Data Conversion Table

[0131] By performing the feature preprocessing described above, different features can have similar influence weights in the fusion calculation.

[0132] III. Feature Weight Allocation Mechanism: The weighting principles for different features are shown in Table 6. After data preprocessing, the system assigns weights to various features. Different features have varying degrees of importance in defect type determination. For example, three-dimensional height information can directly reflect whether the film layer has deformed, thus having high reference value in determining intra-film defects; dark-field scattering ring features can reflect changes in the propagation path of light inside the film layer and can serve as an important auxiliary criterion for intra-film defects; while two-dimensional visual features are mainly used to describe the shape and size of defects, and their role in determining defect hierarchy is relatively small. Therefore, during the fusion calculation process, corresponding weights are assigned to different features to allow key features to play a greater role in comprehensive analysis.

[0133] Table 6 Weight Allocation Principles

[0134] This weighting mechanism allows the most reliable detection information to play a dominant role in the fusion decision. IV. Logic of Integrated Scoring Calculation: After completing the scattering feature analysis and film deformation analysis, in order to further improve the reliability of defect classification, multiple detection features were fused and analyzed, and the defect type was comprehensively determined through a fusion scoring mechanism.

[0135] Specifically, the system first performs unified processing on the feature data obtained from different detection steps, including data normalization and outlier filtering, to eliminate the impact of differences in numerical scales between different features. Then, it converts various features into quantifiable scoring indicators to represent the degree to which the feature supports the judgment of intramembrane defects. The features involved in the fusion calculation mainly include the following three categories: Scattering characteristic score: used to reflect whether there is a significant scattering ring structure in the defect region in a dark field image. When the defect is located inside the film layer, the internal particles cause light to scatter within the film layer, thus forming a significant bright ring structure around the defect. Therefore, the higher the scattering intensity, the greater the support for the defect inside the film.

[0136] Film deformation score: This score reflects the presence of film bulge structures in the three-dimensional height data obtained by white light interferometry. When the film is pushed up by an internal foreign object, the defect area typically exhibits a continuous deformation structure with a higher center height that gradually decreases outward. Therefore, the more pronounced the deformation, the higher the score.

[0137] Two-dimensional morphology scoring: used to describe the area, edge, and shape features of defects in bright-field images. Although two-dimensional morphology cannot directly reflect the depth of defects, it can help determine whether a defect conforms to the morphological characteristics of a typical intrafilm defect.

[0138] After obtaining the feature scores mentioned above, the system performs a weighted calculation based on the importance of each feature to obtain a comprehensive score:

[0139] Wherein: F scatter For scattering characteristic scoring, F deform For film deformation scoring, F shape The two-dimensional morphological score is given, and (w1, w2, w3) are the corresponding weight coefficients.

[0140] Since membrane deformation features can directly reflect changes in membrane structure, their weight is usually higher than that of other features.

[0141] By calculating a comprehensive score, various detection information can be uniformly transformed into a single evaluation index. When the comprehensive score exceeds a preset threshold, the system determines that the defect is an intramembrane defect; when the score is below the threshold, it is determined to be an extramembrane defect or surface particles.

[0142] By using the above-mentioned fusion scoring calculation method, dark field scattering characteristics and film deformation characteristics can form a complementary judgment mechanism, thereby improving the accuracy of identifying defects inside and outside the film and reducing misjudgments caused by a single detection method.

[0143] V. Defect Classification and Judgment Rules: After obtaining the weighted results of each feature, the system calculates a comprehensive judgment index S, which represents the probability that the defect belongs to an intramembrane defect. The system first compares this comprehensive judgment index with a preset threshold to complete the preliminary classification judgment of the defect.

[0144] Specifically, when the comprehensive judgment index S is higher than the preset threshold, it indicates that both the scattering characteristics and the film deformation characteristics are relatively obvious. At this time, the system judges the defect as an intra-film defect. When the comprehensive judgment index is in the middle range, it indicates that some characteristics exist but do not fully meet the typical intra-film defect characteristics. Such defects can be further judged as intra-film micro-defects or weak internal defects. When the comprehensive judgment index is lower than the threshold, it indicates that the defect area does not show obvious internal scattering structure and film deformation characteristics. At this time, the defect is usually judged as external particles or surface contaminants.

[0145] The membrane feature-assisted judgment rules are shown in Table 7. To improve the stability of classification, feature rules can also be combined for auxiliary judgment. For example: When a defect simultaneously exhibits obvious scattering ring characteristics and shows continuous film bulging deformation in a three-dimensional height map, it can be further confirmed that the defect is an intrafilm defect. When a defect exhibits certain scattering characteristics but the film deformation is not significant, it may be a small-sized micro-defect within the film. When the defect does not have obvious scattering ring characteristics and the surrounding film height remains flat, it is usually an external particle or surface contaminant. If a localized high protrusion is detected but no scattering ring structure is formed, it is usually a particle attached to the surface of the film.

[0146] By combining the above-mentioned comprehensive scoring judgment with feature rule-assisted judgment, the accuracy of intra- and extra-membrane defect classification can be further improved, and the risk of misjudgment caused by fluctuations in a single feature can be reduced.

[0147] Table 7. Membrane Feature-Based Judgment Rules

[0148] like Figure 3and combined Figure 3A and Figure 3B The detection process shown is implemented in a specific application scenario, and the specific implementation of the embodiments of this application is described in detail. Figure 3 The disclosed implementation methods can be summarized as follows: First, acquire bright field images of the membrane surface, perform two-dimensional image processing (grayscale normalization + filtering and noise reduction), detect defect regions (edge ​​detection + connected component analysis), and extract two-dimensional features (area + shape + grayscale). Does the system comprehensively determine whether it meets the characteristics of an intramembrane defect? ​​If the determination result is "no", it is determined to be a surface particle, the result is directly output, and this implementation method ends; if the determination result is "yes", then dark field image acquisition, scattering feature extraction, scattering intensity calculation, and ring structure detection are performed. Determine if a ring-shaped scattering structure exists (Is>Tscatter?). If the result is "yes", record the scattering ring characteristics. If the result is "no", determine "no obvious scattering characteristics". Perform 3D height measurement to obtain the storage matrix. Perform scheduling data processing: smoothing filtering + denoising. Calculate deformation parameters: maximum height Hmax and deformation range Rdef. Determine if the maximum height Hmax and deformation range Rdef satisfy: Hmax>Th? or Rdef>Tf? If the maximum height Hmax is greater than the threshold Th or the deformation range Rdef is greater than the radius threshold Tf, then determine if there is film deformation Deform-1 (True), i.e., set Deform to 1. Otherwise, determine if there is no bulging deformation of the film layer Deform-0 (no bulging deformation of the film layer), i.e., set Deform to 0. Then perform feature fusion calculation and calculate the comprehensive score: S = w1 Scatter + w2 Deform + W3 Shape. Determine if the comprehensive score S is greater than the score threshold Tscore, i.e.: S>Tscore? If the judgment result is "yes" (S>Tscore), it is determined to be an internal defect of the membrane. If the judgment result is "no", it is determined to be an external defect of the membrane. Then, it is judged: Is the score close to the threshold? If the judgment result is "yes", that is, the score is close to the threshold, it is marked as an uncertain defect and enters manual re-inspection. The defect type is confirmed by manual inspection, and the final inspection result is output. If the judgment result is "no", that is, the score is not close to the threshold, the final inspection result is directly output, and this implementation method ends.

[0149] Step 1, Image Acquisition and 2D Image Preprocessing: First, a bright-field image of the film surface to be inspected is obtained using an industrial camera to acquire a two-dimensional grayscale image of the film surface. Bright-field imaging can effectively reflect particles, contaminants, and local optical inhomogeneities on the film surface, thus providing basic data for subsequent defect detection.

[0150] Since raw acquired images are often affected by factors such as uneven lighting, sensor noise, and environmental interference, preprocessing is performed on the images after acquisition, including: Image grayscale normalization is performed to ensure that the image grayscale distribution is within a uniform range, thereby reducing the impact of illumination differences on the detection results.

[0151] Smoothing filtering algorithms are used to denoise the image, such as Gaussian filtering or median filtering, thereby reducing the interference of random noise on defect detection.

[0152] Through the above processing, a two-dimensional image with low noise and stable grayscale distribution can be obtained, providing a reliable data foundation for subsequent defect identification.

[0153] The main problem addressed in step 1 is to improve the stability and analyzability of defect detection images under complex lighting conditions and equipment noise.

[0154] Step 2, Two-dimensional defect area detection: After obtaining the preprocessed image, defect region detection is performed on the image to identify areas where abnormal structures may exist.

[0155] In the specific implementation process, edge detection is first performed on the image to extract regions with significant grayscale changes. Edge detection effectively highlights defect boundaries, such as particle boundaries or local anomalies in the film layer. Subsequently, connected component analysis algorithms are used to divide the detected edge regions into sub-regions, identifying continuous regions composed of adjacent pixels as independent candidate defect regions. Each connected region is recorded as a potential defect target. Through the above processing, multiple candidate defect regions can be extracted from the entire image, providing a foundation for subsequent feature analysis.

[0156] The specific implementation process is as follows: Gray-scale change detection: Extracting regions with significant gray-scale changes in an image using edge detection algorithms (such as the Sobel operator). These regions correspond to defect boundaries or abnormal structures.

[0157] Connected component extraction: An 8-adjacent connected component analysis algorithm is used to merge adjacent anomalous pixels into independent candidate defect regions, and the geometric features of each region are calculated. Area: The total number of pixels contained within the area. The width W and height H of the bounding rectangle Aspect ratio R = max(W,H) / min(W,H) Centroid coordinates (xc, yc): The average of all pixel coordinates within the region. The criteria for determining connected regions are shown in Table 8. Based on the extracted geometric features and grayscale characteristics, each connected region is classified according to the following rules: Table 8. Classification of Connectivity Component Determination Criteria

[0158] The technical problem addressed in step 2 is to automatically identify areas that may contain defects in complex backgrounds and reduce the processing scope of subsequent calculations.

[0159] Step 3, Two-dimensional feature extraction and rapid filtering: After identifying candidate defect regions, two-dimensional feature parameters are extracted for each defect region. The extracted features mainly include: area features, shape features, and gray-level distribution features of the defect region.

[0160] Among them, area features are used to describe the size of the defect area; shape features are used to describe the shape of the defect area, such as whether it presents a regular circle or an irregular structure; grayscale features are used to describe the optical difference between the defect area and the surrounding background.

[0161] Subsequently, the aforementioned features are quickly filtered based on preset thresholds. When the defect area is small and relatively regular in shape, the defect is usually dust particles or surface contaminants attached to the membrane surface. For this type of defect, it can be directly identified as an external particle, and the detection result can be output without further complex detection.

[0162] Step 3 enables rapid screening of obvious surface particles, thereby reducing the computational load of subsequent detection steps and improving overall detection efficiency.

[0163] Step 4, Dark-field image acquisition and scattering feature analysis: Bright-field imaging excels at detecting surface-opening defects, while dark-field imaging is specifically designed to identify intrafilm defects (such as foreign objects, bubbles, and delamination) that are difficult to detect with bright-field methods. For defect areas that are not quickly screened out, dark-field imaging is further used to obtain scattering images of the defect areas. Under dark-field illumination, light is incident on the film surface at a relatively large angle. When defects exist inside the film, the film structure will produce local scattering phenomena, thus forming obvious ring-shaped scattering structures in the image. Particles on the film surface typically do not produce obvious ring-shaped scattering structures.

[0164] Therefore, the scattering feature analysis of the defect region in the dark field image can be achieved by: calculating the scattering intensity of the defect region, analyzing the gray-level distribution structure of the pixels around the defect, and determining whether there is a ring-shaped scattering structure.

[0165] When a distinct scattering ring structure is detected, the defect is recorded as having scattering characteristics.

[0166] Step 4 allows us to distinguish between internal defects and surface particle defects in the film using the principle of optical scattering.

[0167] Step 5, 3D height measurement: To further distinguish between intra-film defects and extra-film particles, three-dimensional morphology measurements can be performed on the defect area. White light interferometry is used to obtain three-dimensional height information of the film surface, thereby obtaining height matrix data of the film surface.

[0168] This height matrix reflects minute morphological changes on the film surface. When defects exist within the film, the film structure is affected by the impact of these internal foreign objects, resulting in localized bulge deformation on the surface. Surface particles, on the other hand, typically manifest as localized protrusions but do not cause overall deformation of the surrounding film.

[0169] Step 5 involves acquiring three-dimensional height data to further analyze changes in the membrane structure.

[0170] Step 6, film deformation analysis: After obtaining the three-dimensional height data, the height data is further processed to determine whether the membrane layer has undergone structural deformation.

[0171] First, the height data is smoothed and filtered to reduce the impact of measurement noise on morphology analysis. Then, the height distribution of the defect region is analyzed, and the following parameters are calculated: the maximum height of the defect region and the range of film deformation.

[0172] When a continuous height variation structure appears around the defect area and forms a bulge-like shape, it can be determined that there is film deformation in that area.

[0173] The technical significance of step 6 lies in identifying intramembrane defects using the deformation characteristics of the membrane structure. Since intramembrane defects will push the membrane upwards, they will usually form a relatively smooth and continuous height variation structure, while extramembrane particles mainly exhibit local protrusion structures.

[0174] Step 6 can further improve the accuracy of intramembrane defect identification by analyzing deformation characteristics.

[0175] Step 7, Multi-feature fusion determination: After completing two-dimensional feature analysis, scattering feature detection, and three-dimensional morphology analysis, the above-mentioned multiple features are fused and analyzed to obtain the final defect classification result. In the specific implementation process, a comprehensive evaluation index is first constructed based on scattering features, film deformation features, and two-dimensional morphological features. Different features have different degrees of importance in the defect identification process, therefore, corresponding weights can be assigned to each feature.

[0176] Subsequently, a comprehensive score index is calculated based on the weighted results to indicate the probability that the defect is an intramembrane defect. When the comprehensive score is higher than a preset threshold, the defect is determined to be an intramembrane defect; when the score is lower than the threshold, it is determined to be an extramembrane defect.

[0177] Step 7, through multi-feature fusion analysis, can make full use of the information provided by different detection methods, thereby effectively improving the accuracy and stability of defect identification.

[0178] Step 8, Handling Uncertain Defects and Manual Re-inspection: In actual testing, some defects may have characteristics near the judgment threshold, making them difficult to reliably classify using automated algorithms. Therefore, in this embodiment, when the overall score approaches the judgment threshold, the defect is marked as an uncertain defect and enters the manual re-inspection process. Operators can further observe the defect using a high-magnification microscope to confirm its type.

[0179] Step 8, by setting up a manual re-inspection mechanism, can avoid misjudgments by the automatic detection system under complex conditions, thereby further improving the overall reliability of the detection system.

[0180] In summary, the specific implementation methods of steps 1 to 8 achieve the following: Rapid localization of defect areas is achieved using two-dimensional image detection. Dark-field scattering characteristics can be used to distinguish between internal defects and surface particles in a film. Three-dimensional morphology analysis is used to identify membrane structural deformation. Improve defect identification accuracy through multi-feature fusion; reduce the risk of misjudgment through manual re-inspection mechanism.

[0181] like Figure 4 The screen defect online detection system based on a multimodal fusion mechanism shown is used to implement the screen defect online detection method based on a multimodal fusion mechanism described in any specific embodiment of this application, including: The two-dimensional detection image preprocessing module scans the film layer to be detected using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image, and preprocesses the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image. The candidate region generation module performs defect detection on the two-dimensional detection image and determines the first candidate defect region. The three-dimensional scanning queue generation module filters and scores the first candidate defect region based on its two-dimensional visual features. When the score is higher than a preset threshold, the second candidate defect region is determined and added to the three-dimensional scanning queue. The three-dimensional scanning queue is then measured using a white light interferometric three-dimensional measurement device. The dark field imaging and multimodal detection and determination module determines the scattering characteristics of the second candidate defect region in the three-dimensional scanning queue through dark field imaging, and performs multi-feature fusion analysis in combination with multimodal detection. It obtains a fusion score by calculating a comprehensive score, and determines whether the second candidate defect region is an intra-membrane defect or an extra-membrane defect based on the fusion score.

[0182] The implementation methods of the system described above are merely illustrative. For example, the various functional modules, units, or subsystems within the system may or may not be physically separate, or they may or may not be physical units; that is, they may be located in the same place or distributed across multiple different systems and their subsystems or modules. Those skilled in the art can select some or all of the functional modules, units, or subsystems to achieve the objectives of the embodiments of this application according to actual needs. Those skilled in the art can understand and implement the above-described situations without any creative effort.

[0183] like Figure 5 The system hardware framework diagram shown is as follows. Figure 5 The hardware framework diagram shown is for Figure 4 The above describes the implementation method of the screen defect online detection system based on multimodal fusion mechanism in a specific application scenario.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the specification of the embodiments of this application.

[0185] In the description of the embodiments of this application, the reference to terms such as "an embodiment," "example," "specific example," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0186] Furthermore, the technical solutions of the various implementation methods in this application can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the embodiments of this application.

[0187] All features disclosed in the embodiments of this application, or all steps in the disclosed methods or processes, may be combined in any way, except for mutually exclusive features and / or steps. Any feature disclosed in the specification of the embodiments of this application, unless specifically stated otherwise, may be replaced by other equivalent or similar alternative features. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features. Throughout the specification, the same reference numerals indicate the same elements.

[0188] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification of embodiments (including the corresponding claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0189] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of this application, and are not intended to limit them. Although the embodiments of this application have been described in detail with reference to the foregoing specific embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing specific embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions provided by the specific embodiments of this application.

Claims

1. A screen defect online detection method based on a multimodal fusion mechanism, characterized in that, include: The film layer to be detected is scanned using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image. The two-dimensional grayscale image is then preprocessed to generate a preprocessed two-dimensional detection image. Defect detection is performed on the two-dimensional detection image to determine the first candidate defect region; Based on the two-dimensional visual features of the first candidate defect region, a screening and scoring process is performed. When the score is higher than a preset threshold, a second candidate defect region is determined and added to the three-dimensional scanning queue. The three-dimensional scanning queue is then subjected to three-dimensional measurement using a white light interferometric three-dimensional measurement device. Further steps include: A three-dimensional scanning queue is performed using a white light interferometric three-dimensional measurement device. Candidate defect regions are then screened based on two-dimensional visual features, including: defect area, grayscale contrast, texture perturbation, and shape features. Specifically: The second candidate defect region is axially scanned using a white light interferometric 3D measurement device. Interference signals at different height positions are recorded to construct a 3D height distribution map of the current scanned region and determine the height value of each pixel object in the current scanned region. When screening candidate defect regions based on two-dimensional visual features, the screening criteria include: defect area, grayscale contrast, texture perturbation, and shape features. The scattering characteristics of the second candidate defect region in the 3D scanning queue are determined by dark-field imaging, and multi-feature fusion analysis is performed by combining multi-modal detection. A fusion score is obtained by calculating a comprehensive score, and the second candidate defect region is determined to be an intra-membrane defect or an extra-membrane defect based on the fusion score. Further steps include: Light is applied to the second candidate defect region at an oblique angle, and the surface of the second candidate defect region is detected to detect whether scattered light is generated. If a ring-shaped scattering structure is detected in the dark field image, it is preliminarily determined to be an intra-film defect; or, if no ring-shaped scattering structure is detected but a local bright spot is detected, it is preliminarily determined to be an extra-film defect. A membrane deformation analysis was performed on the second candidate defect region to determine the defect location in the three-dimensional height distribution map, and an analysis region was constructed with the defect location as the center. All height data points are extracted in the analysis area to form a corresponding local height matrix. By performing height distribution trend analysis on the local height matrix, it is determined whether there is bulging deformation in the current membrane layer. If it is determined that there is a bulge deformation in the current membrane layer, then a search is performed outward from the defect location. When the height value drops to the background height, the boundary of the bulge deformation is determined. The radius distance from the boundary to the center of the defect location is calculated. Based on the comparison between the radius distance and the size of the defect location itself, it is determined whether the current membrane layer has undergone structural deformation.

2. The online screen defect detection method based on multimodal fusion mechanism according to claim 1, characterized in that, The two-dimensional bright field imaging system includes a bright field industrial camera and a uniform illumination source; The preprocessing of the two-dimensional grayscale image further includes: after obtaining the two-dimensional grayscale image, performing illumination uniformity correction and contrast enhancement processing on the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image.

3. The online screen defect detection method based on multimodal fusion mechanism according to claim 1, characterized in that, The process of determining the second candidate defect region further includes: identifying gray-level abrupt change regions by calculating the gray-level changes between adjacent pixels in the image; determining the adjacent pixels with the same or similar pixel values ​​in the first candidate defect region as the second candidate defect region by using a connected component analysis algorithm; and obtaining the geometric features of the second candidate defect region.

4. The online screen defect detection method based on multimodal fusion mechanism according to claim 1, characterized in that, During the extraction of all height data points in the analysis area, the height data points are smoothed using a neighborhood averaging method.

5. The online screen defect detection method based on multimodal fusion mechanism according to claim 1, characterized in that, The multimodal detection method further includes: The results of film deformation analysis, the characteristics of scattering ring structure and two-dimensional visual features are weighted and assigned. The defect type is comprehensively judged by a fusion scoring mechanism to obtain a comprehensive judgment index. Based on the comprehensive judgment index, the defect is finally determined to be an internal or external defect.

6. A screen defect online detection system based on a multimodal fusion mechanism, used to implement the screen defect online detection method based on a multimodal fusion mechanism as described in any one of claims 1 to 5, characterized in that, include: The two-dimensional detection image preprocessing module scans the film layer to be detected using a two-dimensional bright field imaging system to obtain a two-dimensional grayscale image, and preprocesses the two-dimensional grayscale image to generate a preprocessed two-dimensional detection image. The candidate region generation module performs defect detection on the two-dimensional detection image to determine the first candidate defect region; A 3D scanning queue generation module filters and scores the first candidate defect region based on its two-dimensional visual features. When the score is higher than a preset threshold, a second candidate defect region is determined and added to the 3D scanning queue. The 3D scanning queue is then measured using a white light interferometric 3D measurement device. The module further includes: A three-dimensional scanning queue is performed using a white light interferometric three-dimensional measurement device. Candidate defect regions are then screened based on two-dimensional visual features, including: defect area, grayscale contrast, texture perturbation, and shape features. Specifically: The second candidate defect region is axially scanned using a white light interferometric 3D measurement device. Interference signals at different height positions are recorded to construct a 3D height distribution map of the current scanned region and determine the height value of each pixel object in the current scanned region. When screening candidate defect regions based on two-dimensional visual features, the screening criteria include: defect area, grayscale contrast, texture perturbation, and shape features. The dark-field imaging and multimodal detection and determination module determines the scattering characteristics of the second candidate defect region in the 3D scanning queue through dark-field imaging, and performs multi-feature fusion analysis in conjunction with multimodal detection. A fusion score is obtained by calculating a comprehensive score, and the second candidate defect region is determined to be either an intra-membrane defect or an extra-membrane defect based on the fusion score. Further, it includes: Light is applied to the second candidate defect region at an oblique angle, and the surface of the second candidate defect region is detected to detect whether scattered light is generated. If a ring-shaped scattering structure is detected in the dark field image, it is preliminarily determined to be an intra-film defect; or, if no ring-shaped scattering structure is detected but a local bright spot is detected, it is preliminarily determined to be an extra-film defect. A film deformation analysis was performed on the second candidate defect region to determine the defect location in the three-dimensional height distribution map, and an analysis region was constructed with the defect location as the center. All height data points are extracted in the analysis area to form a corresponding local height matrix. By performing height distribution trend analysis on the local height matrix, it is determined whether there is bulging deformation in the current membrane layer. If it is determined that there is a bulge deformation in the current membrane layer, then a search is performed outward from the defect location. When the height value drops to the background height, the boundary of the bulge deformation is determined. The radius distance from the boundary to the center of the defect location is calculated. Based on the comparison between the radius distance and the size of the defect location itself, it is determined whether the current membrane layer has undergone structural deformation.

Images