A method, device, equipment and storage medium for intelligently detecting a glass-based metalized sample

Abstract

The application provides a kind of glass-based metalized sample microscopic intelligent detection method, device, equipment and storage medium, by first obtaining the sample panoramic image obtained by microscope automatic scanning splicing;Then call image segmentation algorithm to extract glass base profile, and eliminate image tilt and distortion by geometric correction and perspective transformation;Subsequently, the effective coating area is obtained by cutting, and non-target interference is excluded;Then the effective coating area is analyzed by a deep learning network to generate quantitative indicators, and a standardized detection result is generated based on the quantitative indicators, wherein the deep learning network includes a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative indicators include coating coverage and abnormal proportion;Finally, based on these indicators, a standardized detection result is output, realizing the full-process automation from image acquisition to quality evaluation, thereby significantly improving the detection efficiency and reducing subjective bias.

Description

Technical Field

[0001] This invention relates to the field of glass-based metallization, and particularly to a microscopic intelligent detection method, apparatus, equipment, and storage medium for glass-based metallized samples. Background Technology

[0002] As electronic devices evolve towards higher performance and miniaturization, glass-based advanced packaging technology has found widespread application in polymer materials and electronic plating due to its excellent electrical properties, thermal stability, and optical transparency. Glass-based metallization, as a key process, involves forming a metal layer on the surface of a glass substrate and is widely used in chip packaging, microelectronic device manufacturing, and optical device manufacturing.

[0003] However, existing technologies still face significant challenges in practical industrial applications, particularly in terms of the level of intelligence in the characterization process, which severely restricts the large-scale promotion and application of the technology and the development of formulations.

[0004] In the characterization of coating effects, while traditional inspection methods such as scanning electron microscopy (SEM) and atomic force microscopy (AFM) can provide high-precision surface morphology and composition analysis, they suffer from low efficiency, high cost, and complex operation. Although visual characterization methods, which have emerged in recent years, have increased inspection throughput, the entire process from image acquisition and defect identification to data analysis still requires significant manual intervention. This not only limits inspection speed but also makes the results susceptible to bias due to subjective judgment. This highly manual characterization model has become a key bottleneck restricting the efficiency and reliability of product quality control.

[0005] In view of the above, this application is hereby submitted. Summary of the Invention

[0006] This invention discloses a microscopic intelligent detection method, device, equipment, and storage medium for glass-based metallized samples, aiming to solve the problem that the detection of glass-based metallized samples cannot efficiently extract the effective coating area and quantitatively evaluate the coating coverage and the proportion of anomalies.

[0007] The first embodiment of the present invention provides a microscopic intelligent detection method for glass-based metallized samples, comprising:

[0008] A panoramic image of the sample obtained by automatic scanning and stitching with a microscope is acquired. An image segmentation algorithm is called to process the panoramic image of the sample to extract the glass substrate contour. After geometric correction and perspective transformation of the glass substrate contour, the effective coating area is cropped.

[0009] The effective coating area is analyzed by a deep learning network to generate a quantitative index, and a standardized detection result is generated based on the quantitative index, wherein the deep learning network comprises a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative index comprises a coating coverage rate and an abnormality proportion.

[0010] Preferably, the image segmentation algorithm is called to process the sample panoramic image to extract a glass base profile, and after geometric correction and perspective transformation of the glass base profile, the effective coating area is obtained by cropping.

[0011] According to the degree of differentiation between the coating metal and the background in the sample scanning image, the BoxPrompt method using the SAM model or the threshold-based binary processing is selected to preliminarily extract the glass base area.

[0012] The contour detection algorithm is called to find all closed regions in the preliminarily extracted glass base area, and the glass base area is filtered based on a set area threshold to remove noise and select the glass base area.

[0013] The minimum circumscribed rectangle of the glass base area is calculated, four corner points are obtained and sorted in the order of top left, top right, bottom right and bottom left, and a perspective transformation matrix is calculated according to the order of the corner points and the size of the target image to correct the inclined glass base area to a horizontal rectangular area.

[0014] The corrected glass base area is cropped according to a predetermined proportion to remove the uncoated area and the edge overflow part, and only the glass base area image containing effective coating is output.

[0015] Preferably, the effective coating area is analyzed by the coarse-grained classification preliminary screening and the fine-grained double-branch network, the quantitative results of the coating coverage rate and the abnormality proportion are calculated, and the standardized detection result is generated based on the quantitative results.

[0016] The effective coating area is analyzed by the coarse-grained classification preliminary screening branch network of the deep learning network to determine the overall coating state of the texture class abnormality, and when the sample preliminarily passes based on the overall coating state, the abnormal area is identified by the fine-grained branch network of the deep learning network and a pixel-level segmentation mask is generated, wherein the abnormal area includes an uncoated area, a crack and a blister.

[0017] The coverage rate is calculated based on the pixel-level segmentation mask, and the proportion of each type of abnormal area in the whole image is calculated to generate a quantitative index, wherein the expression of the coverage rate is: coverage rate = 1-(uncoated area pixel number / total effective area pixel number).

[0018] Based on the coarse-grained evaluation result and the quantitative index, a standardized JSON file containing the classification result, the coverage rate and the abnormality proportion is generated.

[0019] Preferably, the judgment of the preliminary passing of the sample also includes:

[0020] Convert the effective plating area into a gray scale image, and construct a gray scale co-occurrence matrix based on the gray scale image in multiple directions and multiple pixel spacings, the multiple directions including 0 degrees, 45 degrees, 90 degrees and 135 degrees;

[0021] Calculate the energy eigenvalue of each gray scale co-occurrence matrix, which is the sum of squares of each element of the normalized gray scale co-occurrence matrix, and calculate the average value of the energy eigenvalue in all directions and distances as the overall uniformity index, wherein the overall uniformity index ranges from 0 to 1, and the closer the value is to 1, the more uniform the plating surface texture is.

[0022] Preferably, it also includes:

[0023] An instance segmentation model YOLOv8-seg is used to identify and segment the laser-engraved numbered area in the effective plating area, and a pixel-level mask is generated.

[0024] The FindContours function of OpenCV is used to operate on the pixel-level mask to obtain the boundary, and the boundingRect function is used to extract the minimum circumscribed rectangle and rotate it to the horizontal direction, thereby generating a numbered area image.

[0025] A target detection model YOLOv8 is used to identify the numbered area image to generate each digital character, and each digital character is arranged and combined in the spatial order from left to right to generate a complete sample number.

[0026] Preferably, the construction method of the double-branch deep learning network includes:

[0027] Some weight layers of the pre-trained model are frozen to maintain the basic feature extraction capability, and the segmentation branch and the classification branch share the bottom feature extractor but optimize their respective loss functions independently, wherein the segmentation branch is used to output a pixel-level abnormal area mask, and the classification branch is used to output an image-level quality discrimination result.

[0028] Preferably, when the plating metal is copper, a threshold-based binaryzation processing is adopted.

[0029] When the plating metal is nickel, a Box Prompt method of the SAM model is adopted for processing.

[0030] The second embodiment of the present application provides a microscopic intelligent detection device for glass-based metalized samples, which comprises:

[0031] An effective coating area clipping unit is configured to acquire a sample panoramic image obtained by automatic scanning and splicing of a microscope, call an image segmentation algorithm to process the sample panoramic image to extract a glass base contour, and clip an effective coating area after geometric correction and perspective transformation of the glass base contour.

[0032] A standardized detection result generation unit is configured to analyze the effective coating area by a deep learning network to generate a quantitative index, generate a standardized detection result based on the quantitative index, wherein the deep learning network includes a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative index includes a coating coverage rate and an abnormality proportion.

[0033] The third embodiment of the present application provides a microscopic intelligent detection equipment for a glass-based metallized sample, characterized by comprising a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement the microscopic intelligent detection method for the glass-based metallized sample according to any one of the above.

[0034] The fourth embodiment of the present application provides a computer readable storage medium, characterized by storing a computer program, and the computer program can be executed by the processor of the device where the computer readable storage medium is located to implement the microscopic intelligent detection method for the glass-based metallized sample according to any one of the above.

[0035] The microscopic intelligent detection method, device, equipment and storage medium for the glass-based metallized sample are provided, the sample panoramic image obtained by automatic scanning and splicing of a microscope is acquired first, then an image segmentation algorithm is called to extract a glass base contour, and geometric correction and perspective transformation are used to eliminate image tilt and distortion, then an effective coating area is clipped to exclude non-target interference, then a deep learning network is used to analyze the effective coating area to generate a quantitative index, a standardized detection result is generated based on the quantitative index, wherein the deep learning network includes a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative index includes a coating coverage rate and an abnormality proportion, and finally the standardized detection result is output based on these indexes, realizing full-process automation from image acquisition to quality evaluation, thereby significantly improving detection efficiency and reducing subjective bias. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 is a flowchart of the microscopic intelligent detection method for the glass-based metallized sample provided by the first embodiment of the present application;

[0037] Figure 2 is a schematic diagram of the image after scanning and splicing provided by the present application;

[0038] Figure 3 is a glass-based segmentation result schematic diagram provided by the present application based on a SAM (Segment Anything Model) model segmentation;

[0039] Figure 4 is a glass-based segmentation result schematic diagram provided by the present application based on threshold segmentation;

[0040] Figure 5 is a schematic diagram provided by the present application for changing a target region in an image into a horizontal rectangular region;

[0041] Figure 6 is a schematic diagram of an effective film plating region provided by the present application;

[0042] Figure 7 is a graph of coarse-grained classification preliminary screening failure provided by the present application;

[0043] Figure 8 is a sample schematic diagram provided by the present application;

[0044] Figure 9 is a gray level co-occurrence matrix-uniformity index schematic diagram of a sample graph provided by the present application;

[0045] Figure 10 is a schematic diagram of the contour boundary of a label recognition part provided by the present application;

[0046] Figure 11 is a schematic diagram of recognizing numbers in a mask region image provided by the present application;

[0047] Figure 12 is a schematic diagram of a slide rail module provided by the present application;

[0048] Figure 13 is a module schematic diagram of a microscopic intelligent detection device for a glass-based metalized sample provided by the second embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0050] The specific embodiments of the present application will be described in detail below with reference to the drawings.

[0051] The application discloses a micro intelligent detection method, device and equipment for a glass-based metallized sample and a storage medium, and aims to solve the problem that the glass-based metallized sample detection cannot efficiently extract an effective coating area and quantitatively evaluate coating coverage and an abnormal proportion.

[0052] Please refer to Figure 1 The first embodiment of the application provides a micro intelligent detection method for a glass-based metallized sample, which can be executed by a micro intelligent detection device (hereinafter referred to as a detection device or system) for a glass-based metallized sample, in particular, by one or more processors in the detection device, to at least implement the following steps:

[0053] S101, a sample panoramic image obtained by automatic scanning and splicing of a microscope is acquired, an image segmentation algorithm is called to process the sample panoramic image to extract a glass base contour, and after geometric correction and perspective transformation of the glass base contour, an effective coating area is obtained by cropping;

[0054] In the embodiment, the system first controls the microscope stage to start from the upper left corner of the glass-based sample and perform full-automatic scanning according to a preset scanning path, and the scanning process covers the complete detection range from the upper left corner to the lower right corner. A scanning area containing a safety margin is set during the scanning, and the area is slightly larger than the actual size of the substrate to compensate for the positioning deviation caused by mechanical vibration. After the scanning is completed, the system combines multiple local images into a complete sample panoramic image by using an image splicing algorithm, as shown in Figure 2 .

[0055] After obtaining the panoramic image, the system adaptively selects an image segmentation strategy according to the type of the plated metal. For a copper plated sample, since there is a significant difference between the color of copper and the background of the stage, the system adopts a threshold-based binarization processing method: first, the color image is converted into a grayscale image, and then a low threshold parameter is set to make the glass substrate part completely converted into a white area, and the stage background remains a black area. For a nickel plated sample, since the color of nickel is difficult to distinguish from the background, the system calls a pre-trained SAM (Segment Anything Model) model, adopts a Box Prompt method, and guides the model to output an accurate segmentation result of the glass base through a preset rectangular frame position, as shown in Figure 3 . In a possible implementation manner of the application, the glass substrate part can be completely converted into a white area by grayscale and binarization segmentation by lowering the threshold, as shown in Figure 4 .

[0056] After the initial segmentation, the system uses the contour detection function of OpenCV to process the binary image or segmentation mask to extract all closed contours in the image. To exclude noise points and small interference areas on the stage, the system sets an area threshold parameter to retain only the largest contour area, i.e., the glass substrate main area. Subsequently, the system calculates the minimum enclosing rectangle of the contour and extracts the four corner coordinates of the rectangle. By calculating the x, y coordinates of the corner points and, the four corner points are rearranged in the order of top left, top right, bottom right, and bottom left to ensure the accuracy of subsequent perspective transformation.

[0057] Based on the sorted corner point coordinates, the system calculates the actual width and height of the rectangle to determine the target size of the perspective transformation. The getPerspectiveTransform function of OpenCV is used to calculate the perspective transformation matrix, and then the warpPerspective function is applied to perform the perspective transformation to correct the glass-based image that may be tilted due to mechanical vibration to a standard horizontal rectangular area, as shown in Figure 5 .

[0058] Finally, considering the actual coating process, to avoid damage to the glass metal surface by the mechanical arm clamps, the effective coating area only accounts for about 50% of the total area of the glass substrate. The system automatically removes the uncoated edge area and the coating overflow caused by the edge effect according to the preset cropping ratio. The cropping operation is achieved by calculating the image center point and then expanding outward by a predetermined width and height. Finally, a standardized image containing only the effective coating area is output, as shown in Figure 6 , for use by the subsequent quality analysis module. It should be noted that the effective coating area is defined as the preset detection area on the glass substrate sample, which is the target coverage range of the coating process, and may contain both coated and uncoated parts inside. The system performs quality detection and analysis on all contents within the preset area.

[0059] S102, analyzing the effective coating area through a deep learning network to generate a quantitative indicator, and generating a standardized detection result based on the quantitative indicator, wherein the deep learning network includes a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative indicator includes a coating coverage rate and an abnormality proportion.

[0060] In this embodiment, the system uses a deep learning network preloaded in the memory, which includes a coarse-grained classification preliminary screening branch and a fine-grained branch. The network is trained based on a large number of labeled glass metal surface data sets and can quickly judge the overall quality of the sample, outputting a binary classification result of "pass" or "fail" and its corresponding confidence probability value. The coarse screening mechanism can quickly filter out obviously unqualified samples, such as Figure 7The invalid calculation of subsequent complex algorithms is avoided.

[0061] After the preliminary passing of the sample is determined by the coarse-grained classification based on the deep learning network, a fine-grained analysis is performed by the fine-grained branch based on the deep learning network. The network adopts an innovative double-branch architecture, introduces an image-level classification branch based on an improved segmentation network (for example, a classification branch is introduced in deeplabv3+), and the two branches share a bottom feature extractor but optimize independent loss functions. The segmentation branch focuses on accurate positioning at the pixel level and can identify and segment local anomalies such as uncoated areas, cracks, and bubbles, and outputs high-precision pixel-level segmentation masks. Different label values correspond to different types of anomalies, enabling the system to accurately distinguish between different types of defects. The classification branch evaluates the coating quality from a global perspective and is particularly good at identifying complex texture anomalies such as scratches and color differences, which are difficult to accurately label through pixel-level segmentation.

[0062] Based on the mask data output by the segmentation branch, the system performs quantitative analysis and calculation. First, the total number of pixels marked as the "uncoated" category is counted, and combined with the total number of pixels in the pre-determined effective area, the coating coverage rate is calculated by the formula "coverage rate = 1 - (uncoated area pixel number / total effective area pixel number)". The coating coverage rate directly reflects the completion of the metallization process. At the same time, the system counts the number of pixels of each type of anomaly such as cracks and bubbles, and calculates the area proportion of the whole image. For example, the crack proportion is equal to the number of crack area pixels divided by the total number of effective area pixels, and the bubble proportion is calculated in a similar manner. These quantitative indicators provide an objective basis for subsequent quality determination.

[0063] The system generates a comprehensive detection report by integrating the preliminary judgment result of coarse-grained classification, the overall quality evaluation of the classification branch, and the quantitative indicators calculated by the segmentation branch. The report is stored in a standardized JSON format and includes sample batch number, sample ID passed through the central control software, final classification result (comprehensive consideration of coarse and fine-grained judgments), coating coverage percentage, uncoated area proportion, crack proportion, bubble proportion, and other key information. The structured design of the JSON file facilitates subsequent database storage, statistical analysis, and visual display. The entire analysis process is automatically completed under the coordination of the microscope control software, and the detection results are displayed in real time on the right side of the GUI interface, and are automatically written into the quality database, realizing the traceability of detection data and closed-loop management of quality control.

[0064] In a possible implementation of the present application, when it is determined that the sample is preliminarily passed, quantitative evaluation of the uniformity of the coating is performed. Specifically, the system first converts the color image of the effective coating area into a gray-scale image and performs a gray-scale quantization process. Specifically, the original 256-level gray-scale value is compressed to 16 levels. It should be noted that while the image texture features are retained, the computational complexity of the subsequent gray-level co-occurrence matrix is significantly reduced, and the processing efficiency is improved. The quantization process is achieved by dividing the original gray-scale value by 16 and taking the integer part, ensuring uniform mapping of the gray-scale distribution.

[0065] Based on the quantized gray-scale image, the system constructs a gray-level co-occurrence matrix in four standard directions, including 0 degrees (horizontal direction), 45 degrees (right diagonal direction), 90 degrees (vertical direction), and 135 degrees (left diagonal direction). At the same time, in order to capture texture features of different scales, the system sets multiple pixel distance parameters in each direction, usually selecting 1, 3, 5, etc. Each element P(i,j) of the gray-level co-occurrence matrix represents the frequency of the co-occurrence of pixels with gray-scale value i and pixels with gray-scale value j under a specific direction and distance. During the construction process, the system traverses the entire image, and the sample image is as shown in Figure 8 The number of pixel pairs that satisfy the spatial relationship is counted, and then normalized to make the sum of the matrix elements equal to 1.

[0066] For each constructed gray-level co-occurrence matrix, the system calculates its energy eigenvalue, which is also called angular second moment or uniformity measure. The sample image is as shown in Figure 9 The energy eigenvalue is obtained by calculating the sum of the squares of all elements of the normalized gray-level co-occurrence matrix, which is mathematically expressed as the accumulation of the squares of each element P(i,j) in the matrix. The energy value reflects the uniformity of the image gray-scale distribution and the fineness of the texture. When the image texture is uniform and regular, the gray-level co-occurrence matrix will exhibit the characteristics of a few elements with large values, resulting in a high energy value; on the contrary, when the texture is complex and chaotic, the matrix element distribution is dispersed, and the energy value is low.

[0067] The system calculates the energy eigenvalues under all combinations of directions and distances, and then takes the arithmetic mean as the final overall uniformity index. For example, if 4 directions and 3 distances are used, there are 12 energy values participating in the average calculation. The value range of this uniformity index is strictly limited between 0 and 1, where 1 represents a completely uniform solid color surface, and 0 represents an extremely complex random texture. In practical applications, high-quality glass-based metal coatings usually exhibit a higher uniformity index, with a typical value above 0.7. When the uniformity index is lower than the preset threshold (such as 0.5), the system will increase the attention weight of texture abnormalities in the subsequent double-branch network analysis, ensuring that potential quality problems are not missed. The uniformity evaluation results are recorded in the final JSON detection report together with indicators such as coverage and abnormality proportion, providing comprehensive quality feedback information for process optimization.

[0068] In one possible implementation of the present application, it further includes: to realize the automatic traceability management of glass-based samples, the system integrates a number recognition function based on deep learning. In actual production, each glass substrate is engraved with a unique digital number on its right non-coated area by a laser engraving machine before coating. To ensure that the number is clear and visible, the laser engraving process is usually performed twice. These number information is crucial for subsequent quality tracking and process optimization.

[0069] The system first calls the YOLOv8-seg instance segmentation model to process the effective coating area image. This model is based on a single-stage target detection framework and has end-to-end training and inference capabilities, enabling fast and accurate positioning of the number area in complex backgrounds. During the data set preparation stage, to increase the diversity of numbered samples and quickly expand the training data, the technical personnel engraved multiple sets of digital labels in different areas of the same glass substrate, and generated pixel-level segmentation labels through the labeling tool. The trained YOLOv8-seg model not only can detect the approximate position of the number, but also can output accurate pixel-level segmentation masks, accurately depicting the contour boundaries of the number area (as shown in Figure 10 Compared with traditional threshold or edge detection techniques, this instance segmentation method can still maintain high segmentation accuracy in cases where the number boundary is blurred, the background is complex, or the label shape is irregular.

[0070] After obtaining the segmentation mask of the numbered region, the system calls the OpenCV image processing library for subsequent processing. First, the FindContours function is used to extract the contours of the binary mask image. This function can track the boundaries of the mask and generate a set of contour points. Then, the system applies the boundingRect function to the extracted contours to calculate the minimum bounding rectangle that contains the entire numbered region. Considering that the laser-engraved numbers may be slightly tilted due to the sample placement angle, the system further calculates the rotation angle of the rectangle and rotates the numbered region to the horizontal direction through an affine transformation, ensuring that the subsequent character recognition can achieve the best recognition effect. After correction, the system crops the standardized numbered region image from the original image according to the coordinates of the bounding rectangle.

[0071] For the cropped numbered region image, the system deploys the YOLOv8 target detection model for single-digit character recognition. This model is trained on a dataset containing 0-9 digit classes and can accurately detect and recognize each digit character in the image. The YOLOv8 model outputs the coordinates, class, and confidence information of each detection box (as shown in Figure 11 The system spatially sorts all recognized digits according to their x coordinates to ensure that the characters are arranged in the correct order from left to right. For example, if the detected digits and their x coordinates are: 8(x=10), 9(x=30), 1(x=50), 2(x=70), 0(x=90), 3(x=110), 4(x=130), 5(x=150), 6(x=170), 7(x=190), 3(x=210), 3(x=230), the system will arrange them in ascending order of x coordinates, and finally generate the complete sample number "891203456733".

[0072] The generated sample number is recorded in the system database together with the detection batch information and written as a key field in the final JSON detection report. When the numbering recognition module cannot normally recognize due to image quality problems, the system will automatically generate a temporary number according to the current timestamp to ensure the continuity of the detection process. It is worth noting that the recognition of the glass sample number is performed before calculating the coverage rate, because the left half of the glass sample image is the coated area, which needs to be calculated for coverage rate; the right part is engraved with a number, which needs to be calculated through a neural network.

[0073] In a possible implementation of the present application, in the construction process of the dual-branch deep learning network, the system first loads a pre-trained feature extractor as a shared bottom structure, which usually adopts a classic architecture such as ResNet or EfficientNet. In order to maintain the model's ability to extract general visual features, the system adopts a selective freezing strategy, setting the weights of the first few convolutional layers of the pre-trained model to an untrainable state. Specifically, for the ResNet-50 backbone network, the system freezes all parameters of the first three residual blocks, which have learned basic visual features such as edges, textures, and shapes, and directly apply them to the glass-based coated image to effectively extract low-level features. The high-level part of the network remains trainable, allowing the model to learn specific feature representations for metal coating defects.

[0074] On the basis of the shared feature extractor, the network differentiates into two independent task branches. The segmentation branch gradually restores the spatial resolution of the feature map based on the feature map output by the feature extractor through a series of upsampling layers and skip connection structures. This branch adopts a similar U-Net encoding-decoding architecture, and the decoder part restores the feature map size to the original input size through transposed convolution or bilinear interpolation, finally outputting a segmentation mask with the same size as the input image. Each pixel position of the mask corresponds to a class label, including uncoated areas, cracks, bubbles, and other states. The segmentation branch uses a combination of cross-entropy loss and Dice loss as the optimization objective, the former focuses on pixel-level classification accuracy, and the latter emphasizes the area overlap, and the combination of the two can effectively handle the class imbalance problem.

[0075] The classification branch starts from the high-level features of the shared feature extractor, compresses the spatial features into fixed-length feature vectors through global average pooling, and then connects the fully connected layer for binary classification. This branch is specifically used to identify overall defects that are difficult to accurately label through pixel-level segmentation, such as surface scratches, uneven color, and other texture abnormalities. Although such defects visually affect product quality, their boundaries are fuzzy and irregular in shape, making it difficult to generate accurate segmentation labels. The classification branch uses a binary cross-entropy loss function to output the probability value of "pass" or "fail", providing an important reference for overall quality judgment.

[0076] Two branches optimize their respective loss functions independently during training, but achieve feature reuse and mutual enhancement through a shared feature extractor. The system adopts a multi-task learning framework, and the total loss function is the weighted sum of the loss of the two branches, and the weight coefficient is dynamically adjusted according to the importance of the task and the convergence speed. During training, an end-to-end method is used, and the entire network is optimized at the same time, so that the shared features can support accurate pixel-level segmentation and provide effective global discrimination information. It should be noted that the double-branch collaborative mechanism fully utilizes the different granularity information of the labeled data, and obtains the local defect positioning and overall quality evaluation results in the same forward propagation process, which significantly improves the detection efficiency.

[0077] In a possible implementation of the present application, before acquiring the image, the system needs to be initialized, wherein the initialization can include mechanical arm zero point calibration, vacuum chuck pressure self-checking, microscope stage reset, and adjusting the illumination parameters of the microscope to the preset value, and establishing a UDP communication connection between the detection device and the microscope control software.

[0078] Based on the established communication connection, the detection device sends control instructions to the mechanical arm to drive the mechanical arm to pick up the glass-based sample prepared by metallization from the cooling table through the rotating bottom shaft clamp jaw, and position the sample with the metal layer surface upward on the five-layer transfer stage.

[0079] Please refer to Figure 12 The detection device transmits the sample batch information to the microscope control software through UDP communication, and controls the four-axis feeding and discharging mechanism to move to the positioning point of the transfer stage along the linear slide rail (the control of the four-axis feeding and discharging mechanism can specifically include: moving the X-axis to make the suction nozzle close to the transfer stage, rotating the shaft to adjust the metal sheet with the suction nozzle to align with the glass sheet position, moving the Z-axis to switch different layers of samples, adjusting the Y-axis to make the suction nozzle accurately adsorb to the center position of the glass base, and rotating the shaft again to transfer the glass base to the center of the stage), the vacuum chuck performs a single picking action, the feeding and discharging mechanism carries the glass-based sample to move stably above the microscope stage, and after accurate positioning through the Z-axis, the vacuum adsorption is released, and the sample is accurately dropped in the center area of the stage; after completing the sample placement, the microscope control software drives the stage to move the left upper corner of the glass base to the center of the field of view.

[0080] In a possible implementation of the present application, after the detection is completed, the stage is controlled to move towards the transfer stage, and a discharging signal is sent, and the feeding and discharging mechanism transfers the detected sample to the completed area of the transfer stage.

[0081] Please refer to Figure 13 The second embodiment of the present application provides a microscope intelligent detection device for glass-based metallized samples, comprising:

[0082] The effective coating area clipping unit 201 is configured to acquire a sample panoramic image obtained by automatic scanning and splicing of a microscope, call an image segmentation algorithm to process the sample panoramic image to extract a glass base contour, and clip an effective coating area after geometric correction and perspective transformation of the glass base contour.

[0083] The standardized detection result generation unit 202 is configured to analyze the effective coating area by a deep learning network to generate a quantitative index, generate a standardized detection result based on the quantitative index, wherein the deep learning network comprises a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the quantitative index comprises a coating coverage rate and an abnormality proportion.

[0084] The third embodiment of the present application provides a microscopic intelligent detection equipment for a glass-based metallized sample, characterized by comprising a memory and a processor, the memory stores a computer program, and the computer program can be executed by the processor to implement the microscopic intelligent detection method for the glass-based metallized sample according to any one of the above.

[0085] The fourth embodiment of the present application provides a computer readable storage medium, characterized by storing a computer program, and the computer program can be executed by a processor of a device where the computer readable storage medium is located to implement the microscopic intelligent detection method for the glass-based metallized sample according to any one of the above.

[0086] The present application provides a microscopic intelligent detection method, device, equipment and storage medium for a glass-based metallized sample, which acquires a sample panoramic image obtained by automatic scanning and splicing of a microscope, calls an image segmentation algorithm to extract a glass base contour, eliminates image tilt and distortion through geometric correction and perspective transformation, clips an effective coating area to exclude non-target interference, analyzes the effective coating area by a deep learning network to generate a quantitative index, generates a standardized detection result based on the quantitative index, wherein the deep learning network comprises a coarse-grained classification preliminary screening branch network and a fine-grained branch network, the quantitative index comprises a coating coverage rate and an abnormality proportion, and finally outputs a standardized detection result based on these indexes, realizes full-process automation from image acquisition to quality evaluation, and significantly improves detection efficiency and reduces subjective bias.

[0087] Exemplarily, the computer program in the third and fourth embodiments of the present application can be divided into one or more modules stored in the memory and executed by the processor to complete the present application. The one or more modules can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the implementation of the micro-intelligent detection device for a glass-based metallized sample. For example, the device in the second embodiment of the present application.

[0088] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like. The processor is the control center of the micro-intelligent detection method for a glass-based metallized sample, and is connected to all parts of the implementation of the micro-intelligent detection method for a glass-based metallized sample through various interfaces and lines.

[0089] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the micro-intelligent detection method for a glass-based metallized sample by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application program required for a function (such as a sound playing function, a text conversion function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, text message data, etc.), and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0090] The implemented modules, if implemented in the form of software function units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods of the present application can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0091] It should be noted that the above-described device embodiments are only schematic, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. In addition, the connection relationship between the modules in the device embodiment provided by the present application indicates that there is a communication connection between them, which can be realized as one or more communication buses or signal lines. Those skilled in the art can understand and implement it without creative labor.

[0092] The above is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed by the present application can be easily thought of by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for micro-intelligent detection of a glass-based metalized sample, characterized in that, The method comprises the following steps: obtaining a sample panoramic image obtained by automatic scanning and splicing of a microscope, calling an image segmentation algorithm to process the sample panoramic image to extract a glass base profile, and after geometric correction and perspective transformation of the glass base profile, the effective coating area is obtained by cropping, specifically: according to the distinction degree of the coating metal and the background in the sample panoramic image, the Box Prompt method of the SAM model or the threshold-based binary processing is selected to preliminarily extract the glass base area; calling a contour detection algorithm to find all closed regions in the preliminarily extracted glass base area, filtering noise based on a set area threshold, and screening out the glass base area; calculating the minimum circumscribed rectangle of the glass base area, obtaining four corner points and sorting them in the order of top left, top right, bottom right and bottom left, and calculating the perspective transformation matrix according to the corner point order and the size of the minimum circumscribed rectangle to correct the inclined glass base area to a horizontal rectangular area; cropping the corrected glass base area according to the preset proportion to remove the uncoated area and the edge overflow part, and outputting a standardized image containing only the effective coating area; analyzing the effective coating area through a deep learning network to generate quantitative indicators, and generating a standardized detection result based on the quantitative indicators, specifically: the deep learning network comprises a coarse-grained classification preliminary screening branch network and a fine-grained branch network, the coarse-grained classification preliminary screening branch network and the fine-grained branch network share a bottom feature extractor but independently optimize their respective loss functions; the effective coating area is evaluated by the coarse-grained classification preliminary screening branch network, and if the sample is preliminarily judged to be unqualified, an unqualified result is directly outputted; if the sample is preliminarily judged to be qualified, the fine-grained branch network is triggered to perform pixel-level segmentation on the effective coating area, identify abnormal areas and generate corresponding segmentation masks, calculate the coverage rate and the proportion of each type of abnormal area based on the segmentation masks, wherein the abnormal areas include uncoated areas, cracks and bubbles, and the expression of the coverage rate is: coverage rate = 1-(uncoated area pixel number / total effective area pixel number); based on the evaluation result of the coarse-grained classification preliminary screening branch network and the quantitative indicators, a standardized JSON file containing the final classification result, the coverage rate and the proportion of the abnormal area is generated.

2. The method for micro-intelligent detection of a glass-based metalized sample according to claim 1, characterized in that, The method further comprises the following steps when the sample is preliminarily judged to be qualified: convert the effective coating area into a grayscale image, and construct a gray level co-occurrence matrix based on the grayscale image in multiple directions and multiple pixel spacings, the multiple directions including 0 degrees, 45 degrees, 90 degrees and 135 degrees; calculate the energy eigenvalue of each gray level co-occurrence matrix, the energy eigenvalue being the sum of squares of each element of the normalized gray level co-occurrence matrix, and the average value of the energy eigenvalues in all directions and pixel spacings is calculated as the overall uniformity index, wherein the overall uniformity index has a value range of 0 to 1, and the closer the value is to 1, the more uniform the texture of the coating surface is.

3. The method for micro-intelligent detection of a glass-based metalized sample according to claim 1, characterized in that, The method further comprises the following steps: The YOLOv8-seg instance segmentation model is used to identify and segment the laser-engraved number area from the effective coating area, and a pixel-level mask is generated. The FindContours function of OpenCV is used to operate on the pixel-level mask to obtain the boundary, and the boundingRect function is used to extract the minimum circumscribed rectangle and rotate it to the horizontal direction, thereby generating a number area image. The YOLOv8 target detection model is used to identify the number area image to generate each digital character, and the digital characters are arranged and combined in the spatial order from left to right to generate a complete sample number.

4. The method for micro-intelligent detection of a glass-based metalized sample according to claim 1, wherein, The construction method of the deep learning network includes: On the basis of the shared bottom layer feature extractor, two independent task branches are differentiated, namely the coarse-grained classification preliminary screening branch network and the fine-grained branch network; a pre-trained model is loaded as the shared bottom layer feature extractor, and a selective freezing strategy is used to freeze part of the weight layers of the pre-trained model to maintain the basic feature extraction capability of the shared bottom layer feature extractor; the fine-grained branch network uses the combination of cross-entropy loss and Dice loss as the optimization target for outputting a pixel-level segmentation mask; the coarse-grained classification preliminary screening branch network uses a binary cross-entropy loss function for outputting an image-level quality discrimination result; the total loss function of the deep learning network is the weighted sum of the loss function of the coarse-grained classification preliminary screening branch network and the loss function of the fine-grained branch network, and the weight coefficient is dynamically adjusted according to the task importance and convergence speed.

5. The method according to claim 1, wherein when the plated metal is copper, the threshold-based binarization processing is used; and when the plated metal is nickel, the Box Prompt method of the SAM model is used for processing. The method comprises: The effective coating area cropping unit is used to obtain a sample panoramic image obtained by automatic scanning and splicing of a microscope, call an image segmentation algorithm to process the sample panoramic image to extract a glass base profile, and after geometric correction and perspective transformation of the glass base profile, crop an effective coating area, specifically: according to the distinction degree of the plated metal and the background in the sample panoramic image, select the Box Prompt method of the SAM model or the threshold-based binarization processing to preliminarily extract a glass base area; call a contour detection algorithm to find all closed areas in the preliminarily extracted glass base area, filter noise based on a set area threshold, and select a glass base area; calculate the minimum circumscribed rotating rectangle of the glass base area, obtain four corner points and sort them in the order of top-left, top-right, bottom-right and bottom-left, calculate a perspective transformation matrix according to the corner point order and the size of the minimum circumscribed rotating rectangle, and correct the inclined glass base area to a horizontal rectangular area; crop and remove the uncoated area and the edge overflow part from the corrected glass base area according to a preset proportion, and output a standardized image containing only the effective coating area.

6. A device for micro-intelligent detection of a glass-based metallized sample, characterized in that, ​ ​ The standardized detection result generation unit is configured to analyze the effective coating area by using a deep learning network to generate a quantitative index, and generate a standardized detection result based on the quantitative index. Specifically, the deep learning network includes a coarse-grained classification preliminary screening branch network and a fine-grained branch network, and the coarse-grained classification preliminary screening branch network and the fine-grained branch network share a bottom layer feature extractor but independently optimize respective loss functions. The coarse-grained classification preliminary screening branch network is used to perform overall quality evaluation on the effective coating area. If it is determined that the sample preliminarily fails, an unqualified result is directly output. If it is determined that the sample preliminarily passes, the fine-grained branch network is triggered to perform pixel-level segmentation on the effective coating area, identify abnormal areas, and generate a corresponding segmentation mask. Based on the segmentation mask, a coverage rate and a proportion of each type of abnormal area are calculated. The abnormal area includes an uncoated area, a crack, and a blister. The expression of the coverage rate is: coverage rate = 1-(uncoated area pixel number / total effective area pixel number). Based on the evaluation result of the coarse-grained classification preliminary screening branch network and the quantitative index, a standardized JSON file containing a final classification result, the coverage rate, and the proportion of the abnormal area is generated.

7. A micro-intelligent detection device for a glass-based metalized sample, characterized in that, A computer program is stored in a memory of a processor, and the computer program can be executed by the processor to implement the microscopic intelligent detection method of the glass-based metallized sample according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program is stored in a memory of a processor, and the computer program can be executed by the processor to implement the microscopic intelligent detection method of the glass-based metallized sample according to any one of claims 1 to 5.

Images