A chip surface defect detection method, device, system and storage medium

Abstract

The present application relates to the technical field of biochip defect detection, and discloses a chip surface defect detection method, device, system and storage medium. The present application improves the distinguishability of surface defects by acquiring bright-field and dark-field images of a biochip. Then, based on the classification and prediction distribution of defect pixels, a confidence score is constructed. Through the first feature information and the second feature information, the evaluation standard of the confidence score is enriched from the angles of the "consistency difference" between the same defect pixels in different images and the "individual difference" between different pixels of the same defect prediction type, and the confidence score is further refined. When the score is less than a first threshold, the defect pixel is identified as low confidence, and the image corresponding to the position is resampled, further improving the image accuracy and image detail expression, making the defect detection process and the obtained results more accurate, and thus improving the accuracy of biochip surface defect detection.

Description

Technical Field

[0001] This invention relates to the field of biochip defect detection technology, and in particular to a chip surface defect detection method, device, system, and storage medium. Background Technology

[0002] Biochips, as precision devices that highly integrate micro- and nano-fabrication technologies with biomolecular detection technologies, are playing an increasingly important role in disease diagnosis, drug screening, and gene analysis. Typical examples include gene chips, microfluidic chips, protein chips, and organ-on-a-chip. A common characteristic of these chips is that their function is highly dependent on the integrity of their surface structure and the uniformity of their surface chemical modifications. Any surface defects at the micrometer or even nanometer scale can lead to the failure of subsequent detection experiments or errors in diagnosis.

[0003] In existing technologies, biochips widely utilize highly transparent polymer materials such as glass, PDMS (polydimethylsiloxane), and PMMA (polymethyl methacrylate). These materials have high light transmittance, making it difficult to effectively identify defects. Furthermore, surface defects are small in scale, requiring high-powered microscopes for clear resolution. However, high-powered microscopes have extremely small fields of view, making it difficult to simultaneously achieve high resolution and high throughput, thus affecting the accuracy of chip surface defect detection. Therefore, the accuracy of chip surface defect detection in existing technologies needs improvement. Summary of the Invention

[0004] This invention provides a method, apparatus, system, and storage medium for detecting chip surface defects, which improves the accuracy of chip surface defect detection.

[0005] The first aspect of this invention discloses a method for detecting defects on the surface of a chip, the method comprising: An initial image set is obtained, which includes multiple registered initial images containing a biochip, and the initial images include bright-field images and dark-field images; Based on the initial image set, a defect primitive and its corresponding classification prediction distribution are determined. The defect primitive includes the prediction range and image information of the surface defects of the biochip. The classification prediction distribution includes the prediction probability that the defect primitive belongs to each type of surface defect. The surface defect type with the highest prediction probability is determined as the defect prediction category; first feature information and second feature information are obtained, wherein the first feature information is used to represent the difference of defect primitives at the same location between different initial images, and the second feature information is used to represent the difference between defect primitives of the same defect prediction category; Based on the first feature information and the second feature information, a first confidence score is obtained; when it is determined that the first confidence score is less than a preset first threshold, the defect primitive is resampled to obtain a resampled image; surface defect detection is performed on the resampled image to obtain a defect detection result.

[0006] As an optional implementation, in the first aspect of the present invention, the method further includes: Based on the classification prediction distribution, a first confidence parameter is obtained, which measures the dispersion of the classification prediction distribution; based on the initial image group, a second confidence parameter is obtained, which measures the similarity between the corresponding classification prediction distributions of defect primitives in the bright field image and defect primitives in the dark field image; based on the defect primitives and the corresponding initial images, a third confidence parameter is obtained, which measures the degree of attribute difference between the defect primitives and neighboring images. The first confidence parameter, the second confidence parameter, and the third confidence parameter are weighted and fused to obtain the second confidence score; A comprehensive confidence score is obtained based on the first confidence score and the second confidence score. When it is determined that the comprehensive confidence score is less than a preset second threshold, the defect primitive is resampled to perform surface defect detection on the resampled image and obtain a defect detection result. The second threshold is different from the first threshold.

[0007] As an optional implementation, in the first aspect of the present invention, the method further includes: Identify the common elements in the defective elements, and find that the common elements and the defective elements have a path connection relationship; Identify similar primitives among the common primitives, where the similar primitives correspond to the same defect prediction category as the defect primitives; Based on the image area ratio of the same type of primitives and the comprehensive confidence score, confidence sampling parameters are obtained. These confidence sampling parameters are used to control the sampling density of the defect primitive resampling process.

[0008] As an optional implementation, in the first aspect of the present invention, the method further includes: Acquire sample detection data, which includes sample chip images containing surface defects of the biochip and functional detection results, wherein the functional detection results are used to represent the signal quality of the sample chip under functional testing; Based on the classification prediction distribution, a prediction classification standard is determined, which is used to constrain the determination conditions for the classification type of defect primitives; based on the sample chip image and the prediction classification standard, the sample defect primitives and the sample prediction distribution corresponding to the sample defect primitives are determined. Based on the defect primitive and the sample defect primitive, a first adjustment parameter is obtained. The first adjustment parameter is used to measure the similarity between the sample defect primitive and the defect primitive. Based on the classification prediction distribution and the sample prediction distribution, a second adjustment parameter is obtained. The second adjustment parameter is used to measure the similarity between the classification prediction distribution and the sample prediction distribution. Based on the first adjustment parameter and the second adjustment parameter, the sample defect primitives are classified to obtain the sample classification result; Based on the sample classification results and the functional detection results, a third adjustment parameter is obtained. The third adjustment parameter is used to measure the correlation between the category of surface defects of the biochip and the functional detection results. The third adjustment parameter is fused with the classification prediction distribution to obtain an influence score. The influence score is used to update the confidence sampling parameters to adjust the sampling density of the defect primitive resampling process.

[0009] As an optional implementation, in the first aspect of the present invention, the prediction range of surface defects of the biochip is obtained by the following method: Based on the initial image group, a first bounding parameter and a second bounding parameter are obtained. The first bounding parameter is used to measure the image richness of the bright field images in the initial image group, and the second bounding parameter is used to measure the geometric complexity of the dark field images in the initial image group. Based on the first bounding parameter, the size of the defect element is determined; based on the second bounding parameter, the outline of the defect element is determined. The predicted range of surface defects on the biochip is determined based on the size and contour of the defect primitives.

[0010] As an optional implementation, in the first aspect of the present invention, the method further includes: Based on the sample chip image, a structural partition map is constructed. The structural partition map includes multiple structural partitions, which are used to represent the degree of influence of the positional relationship of surface defects on the functional test results. The location score is obtained based on the corresponding position of the defect element on the structural zoning map; The confidence sampling parameters are updated based on the location scores to adjust the sampling density of the defect primitive resampling process.

[0011] As an optional implementation, in the first aspect of the present invention, when it is determined that the overall confidence score is less than a preset second threshold, the defect primitives are resampled to perform surface defect detection on the resampled image and obtain a defect detection result, including: When it is determined that the overall confidence score is less than a preset second threshold, the defect primitive is resampled based on the confidence sampling parameters to obtain the first sampling result; Based on preset random sampling parameters, the initial image group is randomly sampled to obtain a second sampling result; The transition sampling parameters are obtained based on the confidence sampling parameters and the random sampling parameters; When it is determined that the overall confidence score is greater than a preset second threshold and less than a preset third threshold, the defect primitive is resampled based on the transition sampling parameters to obtain a third sampling result; Surface defect detection is performed on the first sampling result, the second sampling result, and the third sampling result to obtain the defect detection result.

[0012] A second aspect of the present invention discloses a chip surface defect detection device, the device comprising: An image acquisition module is used to acquire an initial image set, which includes multiple registered initial images containing a biochip, and the initial images include bright-field images and dark-field images. The defect determination module is used to determine the defect primitive and the corresponding classification prediction distribution based on the initial image group. The defect primitive includes the prediction range and image information of the surface defects of the biochip. The classification prediction distribution includes the prediction probability that the defect primitive belongs to each type of surface defect. The confidence assessment module is used to determine the surface defect type with the highest prediction probability as the defect prediction category; and to acquire first feature information and second feature information, wherein the first feature information is used to represent the difference between defect primitives at the same location in different initial images, and the second feature information is used to represent the difference between defect primitives of the same defect prediction category. The detection and judgment module is used to obtain a first confidence score based on the first feature information and the second feature information; when it is determined that the first confidence score is less than a preset first threshold, the defect primitive is resampled to obtain a resampled image; and the resampled image is subjected to surface defect detection to obtain a defect detection result.

[0013] As an optional implementation, in a second aspect of the invention, the apparatus further includes: A semantic evaluation module is used to obtain a first confidence parameter based on the classification prediction distribution, the first confidence parameter being used to measure the dispersion of the classification prediction distribution; to obtain a second confidence parameter based on the initial image group, the second confidence parameter being used to measure the similarity between the corresponding classification prediction distributions of defect primitives on the bright field image and defect primitives on the dark field image; to obtain a third confidence parameter based on the defect primitives and their corresponding initial images, the third confidence parameter being used to measure the degree of attribute difference between the defect primitives and neighboring images; and to perform a weighted fusion of the first confidence parameter, the second confidence parameter, and the third confidence parameter to obtain a second confidence score. The comprehensive judgment module is used to obtain a comprehensive confidence score based on the first confidence score and the second confidence score; when it is determined that the comprehensive confidence score is less than a preset second threshold, the defect primitive is resampled to perform surface defect detection on the resampled image and obtain a defect detection result, wherein the second threshold is different from the first threshold.

[0014] As an optional implementation, in a second aspect of the invention, the apparatus further includes: The density control module is used to identify common source elements among the defective elements, wherein the common source elements and the defective elements have a path connection relationship; identify similar elements among the common source elements, wherein the similar elements correspond to the same defect prediction category as the defective elements; and obtain confidence sampling parameters based on the image area ratio of the similar elements and the comprehensive confidence score, wherein the confidence sampling parameters are used to control the sampling density of the defective element resampling process.

[0015] As an optional implementation, in a second aspect of the invention, the apparatus further includes: A sample prediction module is used to acquire sample detection data, which includes an image of a sample chip containing surface defects and functional detection results. The functional detection results are used to represent the signal quality of the sample chip under functional testing. Based on the classification prediction distribution, a prediction classification standard is determined, which is used to constrain the judgment conditions for classifying defect primitives. Based on the sample chip image and the prediction classification standard, a sample defect primitive and its corresponding sample prediction distribution are determined. A similarity adjustment module is used to obtain a first adjustment parameter based on the defect primitive and the sample defect primitive, wherein the first adjustment parameter is used to measure the similarity between the sample defect primitive and the defect primitive; to obtain a second adjustment parameter based on the classification prediction distribution and the sample prediction distribution, wherein the second adjustment parameter is used to measure the similarity between the classification prediction distribution and the sample prediction distribution; to classify the sample defect primitive based on the first adjustment parameter and the second adjustment parameter to obtain a sample classification result; and to obtain a third adjustment parameter based on the sample classification result and the functional detection result, wherein the third adjustment parameter is used to measure the correlation between the category of the biochip surface defect and the functional detection result. The density update module is used to fuse the third adjustment parameter with the classification prediction distribution to obtain an influence score. The influence score is used to update the confidence sampling parameter to adjust the sampling density of the defect primitive resampling process.

[0016] A third aspect of this invention discloses a chip surface defect detection system, the system comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the chip surface defect detection method according to any of the first aspects of the present invention.

[0017] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked by a processor, are used to execute some or all of the steps in the chip surface defect detection method described in any of the first aspects of the present invention.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention improves the identifiability of surface defects and increases the number of defect detection samples by acquiring bright-field and dark-field images of a biochip. Furthermore, based on the identified defect primitives, it accurately locates the defect prediction range and determines image information and classification prediction distribution. Then, based on this classification prediction distribution, a confidence score is constructed. By using first and second feature information, the evaluation criteria for the confidence score are enriched from the perspectives of "consistency differences" between the same defect primitive in different images and "individual differences" between different primitives of the same defect prediction type, further refining the confidence score and determining the range requiring resampling. When the confidence score is less than a first threshold, the three types of information corresponding to the defect primitive are considered low-confidence, and the corresponding image is resampled. This further improves the image accuracy and the detailed representation of surface defects in the image during surface defect detection, making the obtained defect detection results more accurate and thus improving the precision of biochip surface defect detection. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic flowchart of a chip surface defect detection method disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a chip surface defect detection device disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a chip surface defect detection system disclosed in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0023] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0024] Example 1 Please see Figure 1 , Figure 1 This is a schematic flowchart of a chip surface defect detection method disclosed in an embodiment of the present invention. Wherein, Figure 1 The described chip surface defect detection method can be applied to chip surface defect detection devices. For example... Figure 1 As shown, the chip surface defect detection method may include the following operations: Step 101: Obtain the initial image group.

[0025] In this embodiment of the invention, the initial image group includes multiple initial images containing biochips and which have been registered.

[0026] The initial images include bright-field and dark-field images. First, the bright-field image uses vertical illumination, which is more suitable for acquiring the overall morphology, structural outline, color information, and opacity defects of the biochip. Its purpose is to provide a global spatial reference for the dark-field image. The dark-field image uses oblique incidence illumination with a black background, exhibiting extremely high sensitivity to scatterers such as tiny particles, fine scratches, and microchannel edges. It can highlight surface undulations, edges, scratches, and other structural details, and significantly improves the contrast of defects such as cracks and chipping in the biochip. The acquisition methods for bright-field and dark-field images can include single-camera time-division acquisition, multi-camera synchronous acquisition, and color camera combined with multi-angle illumination; this embodiment of the invention does not limit these methods.

[0027] The combination of bright and dark field images can enhance the visibility of surface defects and the robustness of the surface defect detection process. It can also adaptively solve the problem of incompatibility between high resolution and high throughput in surface defect detection. For example, a dark field image can be used to quickly scan the entire image and extract defect primitives, and then a bright field image can be used for high-magnification verification. Through image registration and coordinate mapping, a unified large field of view can be achieved.

[0028] In this embodiment of the invention, the biochip can be a microarray chip (gene chip, protein chip) with immobilized probe arrays, a microfluidic chip with microchannel mesh, an organ chip, or a digital PCR chip using microwell isolation. This embodiment of the invention does not limit the scope of the invention.

[0029] In this embodiment of the invention, the initial image set may also include other images with different illumination angles (or polarization directions), such as side-lit images, back-lit images, coaxial images, and polarized images. Among these images: side-lit images can capture minute changes in the height direction of the biochip; back-lit images can display tiny bubbles or foreign objects in the biochip; coaxial images can display microstructures etched on the surface; and polarized images, such as differential interference contrast, can display nanoscale defects on the biochip. Different illumination angles can display different defects on the biochip. This embodiment of the invention does not limit the selection of illumination images.

[0030] In this embodiment of the invention, when acquiring the initial image set, the field-of-view bias of the bright and dark field dual imaging system needs to be considered. For example, single-camera time-division acquisition may cause pixel shifts in the image due to slight displacement of the stage; in multi-camera acquisition, translation, rotation, scaling, and perspective distortion may occur due to differences in camera mounting poses. Therefore, image registration can be a method to establish a one-to-one spatial mapping relationship between points corresponding to the same physical spatial location in multiple images. This spatial mapping relationship includes affine transformation, projection transformation, elastic transformation, optical flow field transformation, displacement field transformation, etc. Image registration can also be implemented through various registration models, such as self-supervised training models like Superpoint, D2-Net, SuperGlue, and LoFTR, which are not limited in this embodiment of the invention.

[0031] Step 102: Based on the initial image group, determine the defect primitive and the corresponding classification prediction distribution.

[0032] In this embodiment of the invention, a primitive can be a series of independent, operable basic constituent units into which an image is decomposed. These basic constituent units can serve as the smallest units of various algorithms / operations during image processing. Furthermore, each unit can include image attributes (position, color, shape, size), relative relationships (adjacent, contained, similar), etc., which are not limited in this embodiment. A defect primitive can be a basic element constituting an image or scene representing surface defects; that is, an object that can be processed independently after the image is decomposed. Specifically, the forms of defect primitives can include the following: 1) Regular grid cells: The image is divided into grids according to fixed spatial coordinates. Grid cells containing surface defect information are called defect primitives. Each grid cell can accommodate multiple pixels simultaneously, or each grid cell can match one pixel. Grid cells can be rectangular grids, triangular grids, hexagonal grids, parallelogram grids, or Poisson disk grids. This embodiment of the invention is not limited to these types.

[0033] 2) Irregular patches: These patches can be sub-regions containing surface defect information obtained by content-adaptive image processing and irregular shape division. The image of this sub-region can be obtained based on the similarity of pixel color, grayscale, or texture. For example, an algorithm (such as SLIC) can be used to group pixels with similar colors and consistent textures together to form irregular primitives with boundaries conforming to physical contours. Alternatively, the image of this sub-region can be obtained based on a perceptual model, such as by processing the image using a segmentation model to obtain the corresponding irregular patches. Furthermore, the defect primitives can also be compact vectors or superpixels generated by aggregating all pixel features within the patch; this is not limited in the embodiments of the present invention.

[0034] 3) Structured nodes can be a structured representation of feature vectors that retain location information and surface defect information by masking the specific shape of image sub-regions. This facilitates the indirect implementation of image processing through node relationship reasoning.

[0035] In this embodiment of the invention, the defect primitive includes the predicted range and image information of the surface defects of the biochip. The predicted range of the defect primitive can be obtained through the form of the defect primitive and its corresponding acquisition method described in the above embodiments. This predicted range can, based on semantics, encompass a series of continuous and complete pixels of the same type of defect information. The precision of the predicted range can be adjusted according to the needs of the defect detection task, defect characteristics, and the accuracy of the model. Furthermore, due to limitations in image resolution or model accuracy, the predicted range may differ from the actual range of surface defects; this embodiment of the invention does not impose such limitations. The image information of the defect primitive can be image attribute information within the predicted range, which may include color, shape, position, size, contrast, resolution, etc.

[0036] In this embodiment of the invention, the classification prediction distribution can include the predicted probability that the defect element belongs to each type of surface defect. The type of surface defect can be classified by manual annotation. For example, based on the morphology of the defect, it can be divided into concave defects, convex defects, mixed concave-convex defects, defects in areas without obvious undulations, and appearance defects. Specifically, surface defects include scratches, roughness, indentations, pits, protrusions, cracks, chemical coating defects, particulate matter (dust, bubbles, chemical residues) defects, design / operation process defects, etc. Furthermore, the type of surface defect can also be determined based on model-assisted judgment. The corresponding classification prediction distribution can be obtained through manual identification and definition, or through machine, algorithm, or model-assisted judgment and evaluation. Therefore, the classification prediction distribution is essentially a probability prediction distribution of the defect type corresponding to the defect element. The form of this probability prediction distribution can be: "Defect element A: the probability of defect category a is 53%, and the probability of defect category b is 12%." This embodiment of the invention does not limit this.

[0037] In this embodiment of the invention, determining the probability distribution of the defect primitive and the corresponding defect primitive is equivalent to the pre-detection process of surface defect detection. Preferably, the method of obtaining the classification prediction distribution is consistent with the method used in the subsequent defect detection process. This embodiment of the invention does not limit this.

[0038] Step 103: Determine the surface defect type with the highest prediction probability as the defect prediction category; obtain the first feature information and the second feature information.

[0039] In this embodiment of the invention, the first feature information can be used to represent the difference between defect primitives at the same location between different initial images. Different initial images include differences between different bright-field images, different dark-field images, and the relationship between bright-field and dark-field images. Because of the different initial images, even after registration, differences will arise due to variations in bright and dark-field images, shooting angles, shooting devices, and image quality, resulting in differences in the defect primitives obtained based on semantic analysis. Therefore, this difference can include differences in the coordinates, size, color, texture, and brightness of the corresponding defect primitives. The specific operation for obtaining the first feature information can employ change detection methods, such as calculating image differences or exponential differences to obtain differences between image attributes, or using convolutional networks such as FC-EF or AMDANet to implement the detection; similarity evaluation methods can also be used, such as calculating the structural similarity between two defect primitives to evaluate the difference. This embodiment of the invention does not limit this approach.

[0040] In this embodiment of the invention, determining the defect prediction category is for the purpose of defining the differences using the second feature information. The second feature information can be used to represent the differences between defect primitives and defect primitives of the same defect prediction category. Since the defect prediction category represents the surface defect type with the highest prediction probability, defect primitives of different prediction categories generally mean that the actual surface defects they represent are of different types. While defect primitives of the same prediction category represent the same type of surface defect, their specific representation in the image may differ, leading to differences in the determined defect primitives. These differences can include the shape, texture, color, detail, brightness, and size of the defect primitives. This embodiment of the invention does not limit this aspect.

[0041] In this embodiment of the invention, the defect prediction category is the surface defect type with the highest prediction probability. Optionally, defect primitives of the same defect prediction category can also be directly determined from the prediction distribution, treating defect primitives with similar prediction distributions as defect primitives of the same classification category. This allows for more detailed classification of defect primitives and further enriches the expression of the second feature information. Specifically, this step may also include: A similarity analysis is performed on the classification prediction distribution and the corresponding classification prediction distribution of the target defect element to obtain the distribution similarity parameter. When the distribution similarity parameter is determined to be greater than the similarity threshold, the defect element is determined to be a defect element of the same prediction category as the target defect element.

[0042] Based on this, there may be situations where defect element A and defect element B have high similarity, and defect element B and defect element C have high similarity, but the similarity between A and C is not significant. Therefore, this step can also be layered according to the similarity relationship to obtain a high similarity layer for each type of defect element. At this time, the difference between defect elements of the same defect prediction category can be the overall difference between the layers mentioned above. This embodiment of the present invention does not limit this.

[0043] Step 104: Obtain a first confidence score based on the first feature information and the second feature information; when it is determined that the first confidence score is less than a preset first threshold, resample the defect primitives to obtain a resampled image; perform surface defect detection on the resampled image to obtain the defect detection result.

[0044] In this embodiment of the invention, the first confidence score is used to determine whether the corresponding defect primitive needs to be resampled. It can be a confidence score obtained by comprehensively evaluating the first feature information and the second feature information. The purpose of constructing the first confidence score using the first feature information is to determine whether the image corresponding to the defect primitive needs to be resampled based on its selection performance in bright and dark field images. Since the defect primitive includes a prediction range, it can be a prediction based on image information. Even if the bright and dark field images present different content for the same defect, they both contain the geometric features and size of the defect, which are inherent properties unaffected by illumination. Therefore, in the chip surface defect detection process, the selection of defect primitives usually exhibits consistency in different bright and dark field images.

[0045] If the aforementioned consistency deviates, meaning the difference described by the first feature information is too large, it may indicate problems affecting image quality, such as different image resolutions or noise interference, requiring further image quality improvement. It may also mean that the location is susceptible to lighting effects, and is likely a defect boundary, a small defect, or a complex texture area, requiring focused refinement. Such defects may be invisible in bright-field images but are extremely sensitive in dark-field images, necessitating image resampling to improve the detail representation of these defects. Therefore, the purpose of resampling is to improve the image quality corresponding to uncertain defect primitives, or the detail representation of small defects, thereby improving the accuracy of the defect detection process.

[0046] The purpose of constructing the first confidence score using the second feature information is to determine whether there are difficult-to-identify (difficult-to-classify) items in the image based on the recognition span (predicted span) of the same type of defect primitives. This allows for resampling to eliminate interference caused by the same image quality issues as mentioned above, and to improve the detailed representation of difficult-to-identify items, making the defect detection process more accurate and detailed.

[0047] In this embodiment of the invention, the method for obtaining the first confidence score based on two types of feature information can be either a method of directly weighting and fusing the quantified differences, or a method of calculating the posterior probability using Bayes' theorem. Specifically, this method may include the following steps: using the classification prediction distribution as a prior condition, determining the likelihood probability through the differences. This likelihood probability can be the "reasonableness of explaining the data using the hypothesis." In this embodiment, the likelihood probability is the "reasonableness of explaining the surface defects using the classification prediction distribution," and this reasonableness can be measured and confirmed through the differences. Based on this, the obtained posterior probability is used as the first confidence score and is used for subsequent judgments.

[0048] In this embodiment of the invention, the judgment condition of setting a first threshold for the first confidence score aims to resample defect primitives with low confidence. Resampling is intended to improve image quality, such as reducing jagged edges, suppressing noise, enhancing details, smoothing magnification, and improving measurement accuracy. Specific resampling operations can include upsampling, downsampling, rotation, stretching, and perspective correction. Preferably, the resampling process uses upsampling, combined with interpolation algorithms or super-resolution algorithms (such as ESRGAN), to make the image content more conducive to subsequent surface detection analysis and improve the accuracy of defect detection results.

[0049] In this embodiment of the invention, the defect detection results may include qualitative results, defect type, defect location, defect size / quantity, severity level, etc. Surface defect detection methods may include threshold segmentation, edge detection, template matching (comparison with standard part location), and blob analysis. Alternatively, artificial learning models such as ResNet and VGG classification models, R-CNN and YOLO detection models, U-Net and Mask R-CNN segmentation models, and anomaly detection models may be used. This embodiment of the invention does not limit the specific methods used in these methods.

[0050] As can be seen, in this embodiment of the invention, firstly, by acquiring bright and dark field images of the biochip, the contrast of surface defects on the biochip is improved, highlighting the details of the surface defects and facilitating subsequent surface defect detection operations. Then, defect primitives are determined, and the surface defect type with the highest probability in the classification prediction distribution is identified as the representative category of that defect primitive, i.e., the defect prediction category. Based on this defect prediction category, different types of defect primitives can be initially distinguished. Furthermore, the "consistency difference" of defects in the bright and dark field images is derived through first feature information, and the "individual difference" of surface defects is derived from the different representations of the same type of defect primitives through second feature information. Subsequently, two evaluation criteria for enriching the confidence score are used to further refine the confidence score, thereby determining the range requiring resampling. When the confidence score is less than a first threshold, the defect primitive is deemed to need resampling. The resampled image further improves image accuracy and defect detail representation, making the defect detection results more accurate, thereby improving the precision of chip surface defect detection.

[0051] In an optional embodiment, the first confidence score is constructed based on the differences obtained from comparing different defect primitives, using the image dimension as the basis for the confidence score. Furthermore, the classification prediction distribution in the second feature information is only used for the preliminary classification of defect primitives. To enrich the semantic dimension considerations of the confidence score, this optional embodiment constructs a new confidence score based on the classification prediction distribution and the intensity of defect performance, further refining the first confidence score and thus making the purpose of resampling clearer. Therefore, the above method may further include: The first confidence parameter is obtained based on the classification prediction distribution; the second confidence parameter is obtained based on the initial image group; and the third confidence parameter is obtained based on the defect primitives and the corresponding initial images. The first confidence parameter, the second confidence parameter, and the third confidence parameter are weighted and fused to obtain the second confidence score; A comprehensive confidence score is obtained based on the first confidence score and the second confidence score. When the comprehensive confidence score is determined to be less than the preset second threshold, the defect primitives are resampled to perform surface defect detection on the resampled image and obtain the defect detection result. The second threshold is different from the first threshold.

[0052] In this optional embodiment, the first confidence parameter can be used to measure the dispersion of the classification prediction distribution. Specifically, the first confidence parameter may include calculating the variance, standard deviation, range, quantile interval (such as interquartile range, percentile range), mean absolute deviation, coefficient of variation, etc. of the prediction distribution. Preferably, an entropy estimation method can be used to determine the dispersion based on the obtained entropy value. Optionally, the first confidence parameter may include a combination of the prediction distribution and the image structure, and an overall confidence judgment can be made based on this. This optional embodiment is not limited. Furthermore, the more dispersed the above-mentioned classification prediction distribution, the more difficult it is to classify the defect primitive, thus requiring resampling to improve the accuracy of subsequent defect detection processes.

[0053] The second confidence parameter can be used to measure the similarity between the corresponding classification prediction distributions of defect primitives in the bright-field image and the defect primitives in the dark-field image. Specifically, the second confidence parameter can be KL divergence, which measures the information loss of one distribution relative to another; it can also be JS divergence, cross-entropy; or it can be a method that considers the structural relationship of the prediction distribution in the image, such as Wasserstein distance. Preferably, the second confidence parameter can be used to measure the similarity between the corresponding classification prediction distributions of defect primitives and defect primitives of the same defect prediction category in the bright-field and dark-field images. Since the second feature information compares differences from the perspective of defect primitives, it is incomplete for detecting the true attributes of surface defects. Therefore, the second confidence parameter, from the perspective of semantic analysis, further supplements the second feature information, i.e., the part not covered in the first confidence parameter. Furthermore, the addition of the constraint of defect primitives of the same defect prediction category can, in practice, avoid situations where thin lines in the bright-field image are considered scratches, while discontinuous bright spots in the dark-field image are judged as particles.

[0054] The third confidence parameter measures the degree of attribute difference between a defect primitive and its neighboring images. This attribute difference can include brightness / grayscale, contrast, color, texture, frequency, edges, gradients, etc. The purpose of setting the third confidence parameter is to determine whether the surface defect identified by the defect primitive is significant in the image. If the attribute difference is not significant, this parameter is affected, causing the second confidence score to decrease, thus determining that the image corresponding to the defect primitive needs to be resampled to increase image accuracy and detail.

[0055] In this optional embodiment, the method for weighted fusion of the three confidence parameters may include weighted averaging, adaptive weighted fusion (assigning weights based on the variance of each parameter), or fuzzy weighted fusion. This optional embodiment is not limited in this regard. Furthermore, the method for obtaining the comprehensive confidence score may also employ the weighted fusion method in this optional embodiment, or a combination method based on Bayes' theorem, or a conservative fusion method that directly uses the minimum of the two; this optional embodiment is not limited in this regard.

[0056] In this optional embodiment, the purpose of setting the second threshold is to comprehensively consider the two confidence scores to determine whether the defect primitive needs to be resampled. Furthermore, a third threshold can be set to restrict only the second confidence score; this optional embodiment is not limited in this regard. It should be noted that, similar to the first threshold, both thresholds essentially determine whether the confidence score is too low. If the confidence score is too low, the corresponding defect primitive is resampled to improve image precision and detail representation, thereby improving the accuracy of the defect detection process.

[0057] As can be seen, this optional embodiment can evaluate the classification prediction distribution and the performance intensity of defects through three confidence parameters, further construct a second confidence score, enrich the semantic expression of the confidence score, and the setting of the second threshold forms a step with the first threshold, expanding the candidate capacity of the resampling process, refining the selection conditions of defect primitives in resampling, and thus obtaining a bright and dark field image with higher precision and more accurate detail expression, thereby improving the accuracy of the subsequent defect detection process.

[0058] In another optional embodiment, for small defects with low pixel ratios and high detection difficulty, in order to improve their detection quality, the above method may further include: Identify common-origin primitives in the defect primitives; Identify similar primitives within a common origin; The confidence sampling parameters are obtained based on the image area ratio of similar primitives and the overall confidence score.

[0059] In this optional embodiment, since there may be intersections or occlusions between surface defects and core functional areas of the biochip, or between surface defects themselves, in the initial image, or the same continuous surface defect may be divided into multiple elements during the defect element determination stage, in order to distinguish small defects, the homologous elements of the defect elements are first identified and classified. These homologous elements are then screened. Subsequently, for defects with a very small area proportion, a lower confidence sampling parameter can be assigned, thereby resampling the corresponding image to improve the accuracy of subsequent defect detection.

[0060] In this optional embodiment, source elements can have path connections with defect elements, and elements of the same type and defect elements correspond to the same defect prediction category. Path connection describes a connection relationship where adjacent nodes can be connected by a path. In this optional embodiment, defect elements are considered nodes, and the connection between nodes depends on whether the pixels corresponding to the elements are connected. Based on the connection relationship between every two nodes, other defect elements that can be connected to the defect element by a path are called source elements. Same-type elements can be a further classification based on source elements, because path connection does not necessarily mean they represent the same type of defect. The order of the above classification of source and same-type elements can also be reversed, with classification first, then classification of source elements; this optional embodiment does not limit this.

[0061] In this optional embodiment, the confidence sampling parameter is used to control the sampling density of the defect primitive resampling process. The sampling density can be the number of sampling points per unit area, or it can directly correspond to the image resolution. A lower confidence sampling parameter means that the surface defect corresponding to the defect primitive is more likely to be a small defect. Since the overall confidence score also includes a confidence level related to the intensity of the defect's appearance in the image, the area proportion here can complement it. For small defects, a low area proportion is their most obvious characteristic. A low area proportion and a low overall confidence score result in a low confidence sampling parameter, which further improves the image accuracy of the image corresponding to the small defect during resampling. Simultaneously, the participation of the overall confidence score in this step cleverly avoids the small defect being an easily detectable category, thus preventing wasted resource allocation in the re-detection process. The method for obtaining the confidence sampling parameter can be the same as the method for obtaining the second confidence score and the overall confidence score in the above embodiments, and will not be elaborated upon in this optional embodiment.

[0062] As can be seen, this optional embodiment can screen out small defects through a two-step continuous confirmation process, and then further analyze the detection difficulty based on the area ratio and comprehensive confidence score of the small defects. In the resampling stage, more sampling points are allocated to the small defects to improve image quality, thereby improving the accuracy of the defect detection process.

[0063] In another optional embodiment, one purpose of chip surface defect detection is to detect whether surface defects will affect the expression of biological signals. Therefore, by analyzing the relationship between surface defects and signal expression in the sample biochip image, the influence of defect primitives on the biochip signal expression can be predicted, and the sampling density corresponding to the defect primitives can be adjusted according to this possible influence. The above method may also include: Acquire sample testing data, which includes images of sample chips containing surface defects and functional testing results; Based on the classification prediction distribution, the prediction classification criteria are determined; based on the sample chip image and the prediction classification criteria, the sample defect primitive and the sample prediction distribution corresponding to the sample defect primitive are determined. The first adjustment parameter is obtained based on the defect primitive and the sample defect primitive; The second adjustment parameter is obtained based on the classification prediction distribution and the sample prediction distribution; Based on the first adjustment parameter and the second adjustment parameter, the sample defect primitives are classified to obtain the sample classification results; Based on the sample classification results and functional test results, the third adjustment parameter is obtained; The third adjustment parameter is fused with the classification prediction distribution to obtain the influence score. The influence score is used to update the confidence sampling parameters to adjust the sampling density of the defect primitive resampling process.

[0064] In this optional embodiment, the sample detection data includes images of the sample chip containing surface defects and functional detection results. The functional detection results are used to represent the signal quality of the sample chip under functional testing. The sample detection data can be obtained from any number of biochip images that have undergone defect detection, and the defect detection process includes functional signal detection of the biochip. The functional detection results can be the results obtained from the functional signal detection stage, mainly used to determine the availability and efficiency of the biochip in actual working conditions. The sample detection data can be obtained from an online shared database or local historical data; this optional embodiment is not limited to this.

[0065] In this optional embodiment, the predicted classification criteria are used to constrain the determination conditions for the classification type of defect primitives; wherein, the predicted classification criteria may include the types of predicted classification categories, the basis for classification, classification intervals, etc. The purpose of determining the predicted classification criteria is to ensure that the information structure of the sample predicted distribution and the classification predicted distribution are the same, and that the same set of classification criteria is used, thereby facilitating the second adjustment parameter to measure the similarity between the two.

[0066] In this optional embodiment, the first adjustment parameter is used to measure the similarity between sample defect primitives, and the second adjustment parameter is used to measure the similarity between the classification prediction distribution and the sample prediction distribution. By comprehensively measuring the primitive similarity and the prediction distribution similarity, sample defect primitives that are strongly correlated with defect primitives in the sample can be further identified, and then classified to obtain the sample classification result.

[0067] The above steps are to establish a connection between defect primitives and sample defect primitives, facilitating the mapping of biochip functional detection results corresponding to sample defect primitives to the predicted influence of the defect primitives. Therefore, to obtain the correspondence between sample defect primitives and functional detection results, in an optional embodiment, a third adjustment parameter is used to measure the correlation between the category of biochip surface defects and the functional detection results. Based on this third adjustment parameter, the influence corresponding to the defect primitive is determined, and the sampling density is further adjusted accordingly.

[0068] In this optional embodiment, the correspondence / mapping relationship between sample defect primitives and defect primitives, between classification prediction distribution and sample prediction distribution, and between sample classification results and functional detection results can also be achieved through end-to-end models, relation learning transfer models such as RelationAdapter and ImageBrush, and structure mapping segmentation models such as LT-Net and RaSP. This can save the computation time of the progressive correspondence relationship and improve efficiency. Alternatively, model calculation can be used in some of the three adjustment parameters. This optional embodiment does not limit the scope of this.

[0069] As can be seen, this optional embodiment can achieve the correlation chain between "pigment-sample" and "sample-influence" by setting three adjustment parameters, thereby obtaining an influence score that reflects the "pigment-predicted influence". Based on this influence score, the confidence sampling parameters are updated to adjust the sampling density, and ultimately the resampling process allocates more sampling points to areas with high influence, thereby improving the accuracy of the defect detection process while improving image quality.

[0070] In another optional embodiment, the predicted range of the defect primitives directly reflects the level of detail of the defect primitives, and the level of detail of the defect primitives determines the precision of subsequent sampling work. Therefore, the predicted range of the defects on the surface of the biochip can be obtained in the following way: Based on the initial image group, the first bounding parameter and the second bounding parameter are obtained; Based on the first bounding parameter, the size of the defect element is determined, and based on the second bounding parameter, the outline of the defect element is determined. The predicted range of surface defects on the biochip is determined based on the size and contour of the defect primitives.

[0071] In this optional embodiment, the first bounding parameter is used to measure the image richness of the bright-field images in the initial image group, and the second bounding parameter is used to measure the geometric complexity of the dark-field images in the initial image group. The metrics for measuring image richness may include: color (color gamut breadth, color levels, contrast), texture (high-frequency information, scale variation), and content (number of defect categories, defect morphology), etc. The metrics for measuring geometric complexity may include: morphological complexity (fractal dimension, roundness, convex hull ratio), and spatial distribution complexity (nearest neighbor distance, clustering degree, number of defect skeleton bifurcations, branch length), etc.

[0072] In this optional embodiment, the purpose of setting two bounding parameters is to allow for different defect primitive prediction and picking strategies to be adopted based on different images. The prediction range is not the actual range of the surface defect, but can be the result inferred from semantics. The higher the image richness, the smaller the corresponding defect primitive picking size should be to adapt to more refined picking operations. Conversely, the lower the image richness, the larger the defect primitive picking scale should be, thereby saving the running resources of this step and improving the efficiency of the defect primitive determination process.

[0073] The second bounding parameter adaptively controls the smoothness of the prediction range contour; the higher the geometric complexity, the coarser the contour. The purpose of using this parameter to adjust the contour smoothness is to minimize inconsistencies in the prediction range due to varying smoothness across images with different geometric complexities. This inconsistency generates noise during prediction picking, which is difficult to avoid when each image is independently predicted and picked for defective primitives.

[0074] As can be seen, in this optional embodiment, the determination conditions of defect primitives are further refined by setting two bounding parameters, so that different prediction picking strategies can be adopted for different images. This allows the method corresponding to this optional embodiment to adaptively select more accurate and suitable defect primitives for different images, thereby further improving the accuracy of the defect detection process.

[0075] In another optional embodiment, during the "sample-influence" mapping process, the impact of surface defects on signal expression is a comprehensive result of multiple surface defects. Therefore, the process of this comprehensive analysis needs to be further refined to improve the accuracy of the mapping process. The above method may also include: A structural partition map is constructed based on sample chip images; The location score is obtained based on the corresponding position of the defect element on the structural zoning map; The confidence sampling parameters are updated based on the location score to adjust the sampling density of the defect primitive resampling process.

[0076] In this optional embodiment, the structural zoning map includes multiple structural partitions, which represent the degree of influence of the positional relationship of surface defects on the functional test results. The partitioning method / criteria can be based on distance from the core functional area, fluid path (considering the interference of defects on the fluid, which can be biological samples, buffer solutions, and various liquids involved in the realization of biochip functions), or physical structure. Preferably, the partitioning method can be a combination of the above three methods, ultimately constructing a complete regional risk map, with each partition corresponding to a different comprehensive risk level.

[0077] The purpose of setting location scores is to quantify the partitioning results. Alternatively, location scores can be obtained by skipping the partitioning process and directly calculating the score based on the location. The location score represents the importance of the defect element's location; conversely, a higher score indicates a lower confidence sampling parameter for the defect element, requiring more resampling of the image in that area to improve the accuracy of subsequent defect detection.

[0078] As can be seen, in this optional embodiment, by constructing a structural zoning map, a location score that reflects the positional importance of defect primitives is obtained. At the same time, the confidence sampling parameters are updated using this location score so that during the resampling process, more sampling points are allocated to the defect primitives in the region, thereby improving image accuracy and the detailed representation of defects, and thus improving the accuracy of the subsequent defect detection process and the defect detection results.

[0079] In another optional embodiment, resampling is one of the core aspects of the chip surface defect detection method. Building upon the refinement of the sampling points / density in the above embodiments, the resampling process is further deepened to improve its accuracy. Therefore, the process of resampling the defect primitives when the overall confidence score is determined to be less than a preset second threshold, and then performing surface defect detection on the resampled image to obtain the defect detection result, can include: When it is determined that the overall confidence score is less than the preset second threshold, the defect primitives are resampled based on the confidence sampling parameters to obtain the first sampling result; Based on preset random sampling parameters, the initial image group is randomly sampled to obtain the second sampling result; The transition sampling parameters are obtained based on the confidence sampling parameters and the random sampling parameters; When it is determined that the overall confidence score is greater than the preset second threshold and less than the preset third threshold, the defect primitive is resampled based on the transition sampling parameters to obtain the third sampling result; Surface defect detection is performed on the first, second, and third sampling results to obtain the defect detection results.

[0080] In this optional embodiment, the first sampling result is a new image obtained by resampling the image corresponding to defect primitives with excessively low confidence sampling parameters. This sampling result focuses on images corresponding to small defects, uncertain defects, and defects in critical regions. Therefore, to avoid over-concentration of sampling points and excessive reliance on these concentrated regions during defect detection, leading to overfitting, this optional embodiment introduces random sampling to obtain a second sampling result, thereby improving the generalization ability of the detection process. The random sampling parameter can be used to control the sampling density of the random sampling process, the proportion of random sampling, and the area of ​​random sampling. This optional embodiment is not limited.

[0081] In this optional embodiment, the transition sampling parameter can be used to control the resampling of images corresponding to defect primitives with a low overall confidence score but slightly higher confidence than the first sampling result. Setting the transition sampling parameter is to expand the sampling range, avoid missing high-value points, and simultaneously create a transition between the third sampling result and the first and second sampling results. Therefore, the third sampling result can also be regarded as a hierarchical representation of the resampling process. Based on this, further hierarchical divisions can be implemented, allocating different parameters according to different importance levels to control the sampling density. This optional embodiment is not limited to this.

[0082] As can be seen, this optional embodiment, by controlling the sampling density or sampling ratio of the random sampling process through random sampling parameters, can avoid the defect detection process's excessive reliance on the resampling process, thereby improving the generalization ability of the defect detection process. Simultaneously, by resampling the corresponding defect primitives' images through transition sampling parameters, some high-value points can be avoided from being missed, thus supplementing the point set. With the cooperation of these two parameters, the resampling process is further refined, thereby improving the accuracy of the defect detection process.

[0083] A subset of biochips produced on an actual factory production line were tested using some of the chip surface defect detection methods described in the above embodiments. The biochip images were compiled into a dataset, and the training and testing sets were divided in an 8:2 ratio. The detection results of U-Net and the surface defect detection method after training are shown below (including individual tests for various defects).

[0084] Table 1. Accuracy of this method and U-Net algorithm in the detection of various surface defects.

[0085] The experimental results in Table 1 demonstrate that the accuracy (correctness) of surface defect detection is higher than that of the U-Net method used in existing technologies, and it exhibits very good performance. The specific operation of this method is as follows: First, in the image registration stage: 1) Geometric Alignment: Bright-field and dark-field cameras are synchronously triggered to acquire bright-field RGB and dark-field RGB images within the same field of view, and numbered and bound simultaneously based on temporal synchronization. Geometric alignment is then performed on the bright-field and dark-field images to obtain the aligned image. Corresponding feature points are extracted from the bright-field and dark-field images using the SIFT (or ORB) feature operator. The coordinates in the bright-field image are then determined. Pixel coordinates of dark field image After matching, the homography matrix is ​​solved using a random sampling consensus algorithm. Perform pixel-level coordinate transformation in homogeneous coordinate form. The transformation formula is as follows:

[0086] The geometric alignment operation described above can ensure that the same pixel coordinates in the bright field image and the dark field image correspond to the same physical position on the surface of the biochip, avoiding cross-modal feature misalignment, and thus improving the accuracy of subsequent selection of defect primitives.

[0087] 2) Brightness / Color Normalization: Perform brightness / color normalization on the aligned bright and dark field images. The formula for channel-wise normalization of the aligned image can be expressed as:

[0088] In the above formula, For aligned bright-field or dark-field images, The image after channel normalization. This represents the pixel mean of the corresponding channel. Standard deviation, To prevent the minimum value where the denominator is 0, take Biochip images collected in industrial settings are easily affected by fluctuations in light sources and changes in camera gain, resulting in significant differences in brightness and color distribution between different batches of images.

[0089] 3) Noise suppression: Gaussian filtering (which can be replaced by median filtering) is used to suppress random noise and the grain texture of the chip substrate in the image, reducing the risk of false detection in the subsequent defect primitive selection process.

[0090] 4) Uniform Size: Based on the biochip's contour prior or a preset template, the ROI of the chip region is cropped or scaled. Then, bilinear interpolation is used to unify the size of all input images, ensuring the consistency of the network's input dimensions. Biochip images often contain a large number of background areas that are not meaningful for detection. Directly inputting them into the network will increase the computational load and distract the detection process from the defect areas.

[0091] 5) Information fusion: The processed bright field 3-channel RGB image and dark field 3-channel RGB image are concatenated into a 6-channel input tensor. By inputting the 6 channels together, the global semantic information of the bright field and the detailed texture information of the dark field can be fused at the same time, which greatly improves the feature separability of different types of defects.

[0092] Secondly, the aforementioned 6-channel input tensor is input into the segmentation network to obtain the defect primitives and their corresponding classification prediction distributions: 1) The segmentation network adopts an encoder-decoder structure based on an improved U-Net architecture, incorporating skip connections and multi-scale feature fusion capabilities. It can simultaneously preserve high-level semantic information of surface defects and low-level boundary details. At the encoder end, multi-scale features are extracted level by level through stacked convolutional and pooling layers, completing the aggregation of semantic information. At the decoder end, shallow features from corresponding layers at the encoder end are fused through level-by-level upsampling and skip connection structures, gradually restoring the spatial resolution of the feature maps. The network ultimately outputs logits (the original output score).

[0093] 2) Convert logits into category probabilities of defect primitives using softmax, as shown in the following formula:

[0094] In the above formula, For the location of the defect element, This represents the logits value for class c at that position. Let be the probability that this position belongs to class c, and the sum of the probabilities of all classes is 1.

[0095] 3) Based on the probability of defect primitive categories, obtain the classification prediction distribution, and then select the category with the highest probability at each pixel location using argmax as the prediction result for that location.

[0096] 4) The segmentation network employs two independent encoding architectures to extract features from the bright-field and dark-field images respectively. The two feature sets are then concatenated and attention-based fused at either the encoding or decoding end. This allows for better extraction of the unique features of both modalities, further improving the efficiency of cross-modal information utilization.

[0097] Then, construct the second confidence score: 1) Constructing semantic uncertainty terms (Corresponding to the first confidence parameter). Existing technologies, which recover segmentation results through low-resolution upsampling, are prone to blurred boundaries and loss of minor defects. The core reason is the lack of fine-tuning for regions where the model's prediction is uncertain. The model's prediction distribution entropy directly reflects the uncertainty of classification; the higher the entropy value, the less confident the model is in its classification result at that location, and the more fine-tuning is needed. Therefore, this method constructs a semantic uncertainty term based on the prediction distribution entropy, as shown in the following formula:

[0098] 2) Construct cross-modal consistency terms (Corresponding to the second confidence parameter). For the same defect in a biochip, the prediction results of bright-field and dark-field images should be consistent. If the predicted distributions of the two modalities at the same location differ significantly, it indicates that the location is easily affected by illumination, and is likely a defect boundary, a small defect, or a complex texture area, requiring further refinement. Setting this parameter can be used to measure the difference between the bright-field and dark-field predicted distributions. When the backbone network adopts a dual-branch architecture and outputs bright-field probability distributions separately... With dark field probability distribution In this case, a cross-modal consistency term can be constructed using JS divergence, as shown in the following formula:

[0099] as well as

[0100] In the above formula, The average distribution of the two distributions; The KL divergence measures the difference between two probability distributions. The larger the JS divergence, the greater the discrepancy between the predicted distributions of the light and dark fields, and the higher the refinement requirement at that location.

[0101] 3) Construct texture guide items (Corresponding to the third confidence parameter). The core characteristics of slender defects such as chipped edges and cracks are edge and gradient changes. Relying solely on semantic uncertainty can easily miss low-confidence but actually existing fine crack boundaries. Dark-field images are highly sensitive to surface edges and minute undulations, and the gradient magnitude of their grayscale images can directly reflect the intensity of the boundaries. Therefore, this method constructs a texture / boundary guiding term based on the gradient magnitude of the dark-field grayscale image, as shown in the following formula:

[0102] In the above formula, This is the grayscale image corresponding to the dark field image. For gradient operators, The value is the L2 norm, and the result is the gradient magnitude at that location. The higher the magnitude, the greater the probability that the location is a defect boundary or a texture detail area.

[0103] 4) Combine the above uncertainty scores to generate a candidate set. The three scores are weighted and combined to obtain the final uncertainty score for each pixel location, as shown in the following formula:

[0104] In the above formula, , , The weighting coefficients for each score can be adjusted according to the actual detection scenario; in this method, the values ​​are 0.5, 0.3, and 0.2 respectively. Finally, the K curve primitives with the highest scores are selected to generate a candidate set, providing a basis for subsequent sampling operations.

[0105] Secondly, pixel sampling is performed on the candidate location set, and points are randomly selected from the entire image to supplement the point set. Random points account for 20% of the total number of points, and the total number of points accounts for 0.1% of the total number of pixels. A class-adaptive point budget is introduced, and the class weight is limited according to the pixel proportion. The formula for the class weight is as follows.

[0106]

[0107] In the above formula, For category weights, This is the weight adjustment coefficient, set to 0.5. The corresponding number of refinement points are assigned to each category based on its category weight.

[0108] Next, bilinear interpolation sampling is performed on each point to be refined. Bilinear interpolation sampling can ensure the continuity of sampling characteristics and sub-pixel accuracy, avoiding the loss of boundary accuracy caused by quantization error.

[0109] To construct point-level fusion features, the resampled results are concatenated with the logits output from the encoder along the channel dimension to construct point-level fusion features that integrate fusion details and semantics.

[0110] The feature is input into a lightweight point-level prediction head using a multilayer perceptron (MLP) with shared weights, which further outputs refined point-level logits. These logits are then used to backfill and replace the original logits. This backfilling operation performs refined prediction only at critical locations with high uncertainty, significantly improving the segmentation accuracy of boundaries and small defects while adding only a minimal amount of computation, thus balancing detection accuracy and inference efficiency.

[0111] The refined segmentation logits are decoded and post-processed to output pixel-level segmentation results and defect geometric parameters. First, the refined segmentation logits are decoded and upsampled to the original image resolution. Then, through Softmax and argmax operations, the final pixel-level segmentation results are output.

[0112] Building upon this foundation, morphological opening and closing operations can be further performed on the segmentation results to remove isolated false-detection pixels and fill in tiny holes in defective regions. Then, connected component analysis is performed according to defect type, outputting geometric parameters such as the location, area, perimeter, length, circumscribed rectangle, and center point coordinates of each defective region, providing quantitative data support for biochip quality assessment and yield analysis. For slender defects such as cracks and fissures, skeleton extraction algorithms can further output key indicators such as the defect's skeleton length and main direction.

[0113] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a chip surface defect detection device disclosed in an embodiment of the present invention. Figure 2 As shown, the chip surface defect detection device may include: The image acquisition module 201 is used to acquire an initial image group, which includes multiple initial images containing biochips and registered. The initial images include bright field images and dark field images. The defect determination module 202 is used to determine the defect primitive and the corresponding classification prediction distribution based on the initial image group. The defect primitive includes the prediction range and image information of the surface defects of the biochip, and the classification prediction distribution includes the prediction probability that the defect primitive belongs to each type of surface defect. The confidence assessment module 203 is used to determine the surface defect type with the highest prediction probability as the defect prediction category; and to acquire first feature information and second feature information, wherein the first feature information is used to represent the difference between defect primitives at the same location in different initial images, and the second feature information is used to represent the difference between defect primitives of the same defect prediction category. The detection and judgment module 204 is used to obtain a first confidence score based on the first feature information and the second feature information; when it is determined that the first confidence score is less than a preset first threshold, the defect primitive is resampled to obtain a resampled image; and the resampled image is subjected to surface defect detection to obtain a defect detection result.

[0114] In an optional embodiment, the device may further include: The semantic evaluation module is used to obtain a first confidence parameter based on the classification prediction distribution, which measures the dispersion of the classification prediction distribution; a second confidence parameter based on the initial image group, which measures the similarity between the corresponding classification prediction distributions of defect primitives in the bright field image and the defect primitives in the dark field image; a third confidence parameter based on the defect primitives and their corresponding initial images, which measures the attribute difference between the defect primitives and their neighboring images; and a weighted fusion of the first, second, and third confidence parameters to obtain a second confidence score. The comprehensive judgment module is used to obtain a comprehensive confidence score based on the first confidence score and the second confidence score; when it is determined that the comprehensive confidence score is less than the preset second threshold, the defect primitive is resampled to perform surface defect detection on the resampled image and obtain the defect detection result, wherein the second threshold is different from the first threshold.

[0115] In another alternative embodiment, the device may further include: The density control module is used to identify common-origin primitives in the defect primitives, where there is a path connection between the common-origin primitives and the defect primitives; to identify similar primitives among the common-origin primitives, where the similar primitives correspond to the same defect prediction category as the defect primitives; and to obtain confidence sampling parameters based on the image area ratio and comprehensive confidence score of similar primitives. The confidence sampling parameters are used to control the sampling density of the defect primitive resampling process.

[0116] In yet another alternative embodiment, the device may further include: The sample prediction module is used to acquire sample detection data, which includes images of sample chips containing surface defects and functional detection results. The functional detection results are used to represent the signal quality of the sample chip under functional testing. Based on the classification prediction distribution, the module determines the prediction classification criteria, which are used to constrain the judgment conditions for classifying defect primitives. Based on the sample chip images and the prediction classification criteria, the module determines the sample defect primitives and the sample prediction distribution corresponding to those sample defect primitives. The similarity adjustment module is used to obtain a first adjustment parameter based on the defect primitives and sample defect primitives, which measures the similarity between the sample defect primitives and the defect primitives; to obtain a second adjustment parameter based on the classification prediction distribution and the sample prediction distribution, which measures the similarity between the classification prediction distribution and the sample prediction distribution; to classify the sample defect primitives based on the first and second adjustment parameters, and to obtain the sample classification result; and to obtain a third adjustment parameter based on the sample classification result and the functional detection result, which measures the correlation between the category of the biochip surface defect and the functional detection result. The density update module is used to fuse the third adjustment parameter with the classification prediction distribution to obtain the influence score. The influence score is used to update the confidence sampling parameter to adjust the sampling density of the defect primitive resampling process.

[0117] Example 3 Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a chip surface defect detection system disclosed in an embodiment of the present invention. Figure 3 As shown, the chip surface defect detection system may include: Memory 301 storing executable program code; Processor 302 coupled to memory 301; The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in any of the chip surface defect detection methods in Embodiment 1 of the present invention.

[0118] Example 4 This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in any of the chip surface defect detection methods disclosed in Embodiment 1 of this invention.

[0119] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0120] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0121] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting defects on the surface of a chip, characterized in that, The method includes: An initial image set is obtained, which includes multiple registered initial images containing a biochip, and the initial images include bright-field images and dark-field images; Based on the initial image set, a defect primitive and its corresponding classification prediction distribution are determined. The defect primitive includes the prediction range and image information of the surface defects of the biochip. The classification prediction distribution includes the prediction probability that the defect primitive belongs to each type of surface defect. The surface defect type with the highest prediction probability is determined as the defect prediction category; first feature information and second feature information are obtained, wherein the first feature information is used to represent the difference of defect primitives at the same location between different initial images, and the second feature information is used to represent the difference between defect primitives of the same defect prediction category; Based on the first feature information and the second feature information, a first confidence score is obtained; when it is determined that the first confidence score is less than a preset first threshold, the defect primitive is resampled to obtain a resampled image; surface defect detection is performed on the resampled image to obtain a defect detection result.

2. The chip surface defect detection method according to claim 1, characterized in that, The method further includes: Based on the classification prediction distribution, a first confidence parameter is obtained, which measures the dispersion of the classification prediction distribution; based on the initial image group, a second confidence parameter is obtained, which measures the similarity between the corresponding classification prediction distributions of defect primitives in the bright field image and defect primitives in the dark field image; based on the defect primitives and the corresponding initial images, a third confidence parameter is obtained, which measures the degree of attribute difference between the defect primitives and neighboring images. The first confidence parameter, the second confidence parameter, and the third confidence parameter are weighted and fused to obtain the second confidence score; A comprehensive confidence score is obtained based on the first confidence score and the second confidence score. When it is determined that the comprehensive confidence score is less than a preset second threshold, the defect primitive is resampled to perform surface defect detection on the resampled image and obtain a defect detection result. The second threshold is different from the first threshold.

3. The chip surface defect detection method according to claim 2, characterized in that, The method further includes: Identify the common elements in the defective elements, and find that the common elements and the defective elements have a path connection relationship; Identify similar primitives among the common primitives, where the similar primitives correspond to the same defect prediction category as the defect primitives; Based on the image area ratio of the same type of primitives and the comprehensive confidence score, confidence sampling parameters are obtained. These confidence sampling parameters are used to control the sampling density of the defect primitive resampling process.

4. The chip surface defect detection method according to claim 3, characterized in that, The method further includes: Acquire sample detection data, which includes sample chip images containing surface defects of the biochip and functional detection results, wherein the functional detection results are used to represent the signal quality of the sample chip under functional testing; Based on the classification prediction distribution, a prediction classification standard is determined, which is used to constrain the determination conditions for the classification type of defect primitives; based on the sample chip image and the prediction classification standard, the sample defect primitives and the sample prediction distribution corresponding to the sample defect primitives are determined. Based on the defect primitive and the sample defect primitive, a first adjustment parameter is obtained, which is used to measure the similarity between the sample defect primitive and the defect primitive. Based on the classification prediction distribution and the sample prediction distribution, a second adjustment parameter is obtained. The second adjustment parameter is used to measure the similarity between the classification prediction distribution and the sample prediction distribution. Based on the first adjustment parameter and the second adjustment parameter, the sample defect primitives are classified to obtain the sample classification result; Based on the sample classification results and the functional detection results, a third adjustment parameter is obtained. The third adjustment parameter is used to measure the correlation between the category of surface defects of the biochip and the functional detection results. The third adjustment parameter is fused with the classification prediction distribution to obtain an influence score. The influence score is used to update the confidence sampling parameters to adjust the sampling density of the defect primitive resampling process.

5. The chip surface defect detection method according to claim 4, characterized in that, The predicted range of surface defects on the biochip was obtained through the following method: Based on the initial image group, a first bounding parameter and a second bounding parameter are obtained. The first bounding parameter is used to measure the image richness of the bright field images in the initial image group, and the second bounding parameter is used to measure the geometric complexity of the dark field images in the initial image group. Based on the first bounding parameters, the size of the defect element is determined; Based on the second framing parameters, the outline of the defect primitive is determined; The predicted range of surface defects on the biochip is determined based on the size and contour of the defect primitives.

6. The chip surface defect detection method according to claim 5, characterized in that, The method further includes: Based on the sample chip image, a structural partition map is constructed. The structural partition map includes multiple structural partitions, which are used to represent the degree of influence of the positional relationship of surface defects on the functional test results. The location score is obtained based on the corresponding position of the defect element on the structural zoning map; The confidence sampling parameters are updated based on the location scores to adjust the sampling density of the defect primitive resampling process.

7. The chip surface defect detection method according to claim 6, characterized in that, When it is determined that the overall confidence score is less than a preset second threshold, the defect primitives are resampled to perform surface defect detection on the resampled image, thereby obtaining a defect detection result, including: When it is determined that the overall confidence score is less than a preset second threshold, the defect primitive is resampled based on the confidence sampling parameters to obtain the first sampling result; Based on preset random sampling parameters, the initial image group is randomly sampled to obtain a second sampling result; The transition sampling parameters are obtained based on the confidence sampling parameters and the random sampling parameters; When it is determined that the overall confidence score is greater than a preset second threshold and less than a preset third threshold, the defect primitive is resampled based on the transition sampling parameters to obtain a third sampling result; Surface defect detection is performed on the first sampling result, the second sampling result, and the third sampling result to obtain the defect detection result.

8. A chip surface defect detection device, characterized in that, The device includes: An image acquisition module is used to acquire an initial image set, which includes multiple registered initial images containing a biochip, and the initial images include bright-field images and dark-field images. The defect determination module is used to determine the defect primitive and the corresponding classification prediction distribution based on the initial image group. The defect primitive includes the prediction range and image information of the surface defects of the biochip. The classification prediction distribution includes the prediction probability that the defect primitive belongs to each type of surface defect. The confidence assessment module is used to determine the surface defect type with the highest prediction probability as the defect prediction category; and to acquire first feature information and second feature information, wherein the first feature information is used to represent the difference between defect primitives at the same location in different initial images, and the second feature information is used to represent the difference between defect primitives of the same defect prediction category. The detection and judgment module is used to obtain a first confidence score based on the first feature information and the second feature information; when it is determined that the first confidence score is less than a preset first threshold, the defect primitive is resampled to obtain a resampled image; and the resampled image is subjected to surface defect detection to obtain a defect detection result.

9. A chip surface defect detection system, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the chip surface defect detection method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked by a processor, execute the chip surface defect detection method as described in any one of claims 1-7.

Images