Panel defect aggregation analysis method and system based on multi-modal large model

Abstract

The invention provides a panel defect aggregation analysis method and system based on a multi-modal large model, and relates to the technical field of defect aggregation analysis, and the method comprises the following steps: obtaining defect point coordinate data of a glass panel, and converting the defect point coordinate data into a defect scatter diagram; distinguishing different defect types by combining shape features and spatial distribution features of defect aggregation, and designing cue words according to the different defect types; inputting the defect scatter diagram and the cue words into the multi-modal large model for conjoint analysis, and performing performance evaluation and cue word iterative optimization on the multi-modal large model; and applying the cue words subjected to iterative optimization and the multi-modal large model for performance evaluation to panel defect aggregation analysis to obtain an analysis result. The method solves the problems that a traditional method is insufficient in recognition capability on a complex aggregation mode and high in parameter dependence, and a machine learning or deep learning method needs a large amount of labeled data and complex parameter tuning.

Description

Technical Field

[0001] This invention relates to the field of defect clustering analysis technology, and more specifically, to a panel defect clustering analysis method and system based on a multimodal large model. Background Technology

[0002] In the production of flat panel display devices such as liquid crystal displays (LCDs) and organic light-emitting diodes (OLEDs), various microscopic defects inevitably occur on the panel surface due to factors such as the cleanliness of the process environment, the operating status of equipment, the consistency of raw materials, and the stability of process parameters. These defects include particulate contamination, scratches, bright spots, dark spots, and linear defects. These defects not only reduce product yield but may also reflect systemic anomalies in the production line. In particular, when multiple defects are spatially non-randomly distributed and form specific geometric patterns, it often indicates a persistent deviation in a critical process or a local equipment malfunction. If these defects are not identified and addressed in a timely manner, they can easily lead to batch defects and severely reduce production yield.

[0003] Currently, mainstream panel production lines are generally equipped with automated optical inspection systems to scan each panel across its entire surface and accurately record the location coordinates, size, and grayscale characteristics of each defect. However, the raw defect data output by these automated optical inspection systems is merely a discrete set of points, and its value has not yet been fully realized. The key to truly determining the efficiency of quality control lies in identifying statistically significant and physically meaningful spatial clustering patterns from a massive number of defect points, such as horizontal lines, vertical lines, diagonal bands, local clusters, and central area aggregations. These defect clustering phenomena are often external manifestations of potential process anomalies. For example, horizontal stripe-like clusters are often associated with damage to the conveyor rollers; localized dense clusters may originate from particle shedding inside the cavity or nozzle blockage. Therefore, quickly and accurately identifying and classifying spatial clustering patterns of defects is of great significance for tracing the root causes of anomalies, reducing downtime for troubleshooting, and improving online quality early warning capabilities.

[0004] The current methods for analyzing defect clusters in the panel industry mainly include the following categories: (1) Statistical methods, such as analysis of variance and regional segmentation statistics, which determine clustering by calculating the distribution characteristics of defect points in different regions; (2) Clustering algorithm methods, such as DBSCAN and hierarchical clustering, which group defect points into clusters by density or distance and extract the geometric features of the clusters for classification; (3) Traditional image processing methods, such as geometric calculations of minimum circumcircle, convex hull, and rectangular box expansion, which quantify the clustering pattern; (4) Deep learning methods, such as convolutional neural networks, which perform pattern recognition on defect distribution maps.

[0005] Although the above methods have achieved defect cluster detection and classification to a certain extent, they still face many challenges in industrial applications: (1) Strong threshold dependence: Traditional statistical methods and clustering algorithms mostly rely on manually set threshold parameters (e.g., density radius, minimum number of points, distance threshold, variance threshold, etc.). These parameters are poorly adaptable to different products, different production lines, and different defect types, requiring a lot of experience and repeated debugging, making it difficult to achieve generalization and automation; (2) Dependence on labeled data: Methods based on traditional machine learning or deep learning require a large number of manually labeled training samples. The quality of the model results depends on the quantity and quality of labeled data, and the labeling process is time-consuming and laborious; (3) Lack of semantic understanding: Most existing methods are based on pure mathematics or pixel-level features for judgment, lacking semantic understanding of clustering patterns. They cannot combine shape, location, distribution patterns, and other multi-dimensional features for comprehensive reasoning like human experts, making it difficult to handle complex and ambiguous boundary situations. Summary of the Invention

[0006] This invention provides a panel defect clustering analysis method and system based on a multimodal large model, which solves the problems of insufficient recognition ability of traditional methods for complex clustering patterns, strong parameter dependence, and the need for a large amount of labeled data and complex parameter tuning for machine learning or deep learning methods.

[0007] In a first aspect, embodiments of the present invention provide a panel defect clustering analysis method based on a multimodal large model, the method comprising the following steps:

[0008] Obtain the coordinate data of defect points in the glass panel and convert the defect point coordinate data into a defect scatter plot;

[0009] Different defect types are distinguished by combining the shape and spatial distribution characteristics of defect clusters, and prompt words are designed according to different defect types;

[0010] The defect scatter plot and prompt words are input into the multimodal large model for joint analysis, and the performance of the multimodal large model is evaluated and the prompt words are iteratively optimized.

[0011] The iteratively optimized prompts and the multimodal large model of performance evaluation were applied to panel defect clustering analysis to obtain the analysis results.

[0012] In the above embodiments, the present invention uses the two-dimensional scatter plot converted from defect point coordinate data as visual image input, and uses the image understanding and semantic reasoning capabilities of multimodal large models to perform end-to-end clustering pattern recognition, realizing a cross-modal analysis paradigm shift from numerical coordinates to semantic understanding, giving full play to the advantages of large models in visual pattern recognition and semantic understanding, without the need to design complex feature extraction algorithms or clustering rules.

[0013] As one of the optional implementations of this application, the process of converting defect point coordinate data into a defect scatter plot is as follows:

[0014] Preprocessing and data augmentation of defect point coordinate data;

[0015] For all sub-panels of the glass panel, the coordinate data of the defect points of the sub-panels are plotted as a two-dimensional defect scatter plot.

[0016] As some optional implementations of this application, the process for designing prompt words according to different defect types is as follows:

[0017] The design of preset prompts follows certain principles; these principles include role positioning, task objectives, analysis process, and output requirements.

[0018] For two-dimensional defect scatter plots, defect types are distinguished by combining the shape characteristics and spatial distribution characteristics of defect clusters, and prompt words are designed to guide the analysis of multimodal large models in accordance with the design principles of prompt words.

[0019] In the above embodiments, the present invention directly uses a general open-source multimodal large model, and guides the model to perform cluster analysis through designed prompt words. It can accurately identify various clustering patterns such as horizontal lines, vertical lines, diagonal lines, local, surrounding, and center without any labeled data, model training or fine-tuning process.

[0020] As some alternative embodiments of this application, the defect types include horizontal line clusters, vertical line clusters, diagonal line clusters, local clusters, perimeter clusters, and center clusters.

[0021] As one of the optional implementations of this application, the process of inputting defect scatter plots and cue words into a multimodal large model for joint analysis is as follows:

[0022] The defect scatter plot is encoded, and the prompt words and the encoded defect scatter plot are input into the multimodal large model;

[0023] By using a cross-modal feature alignment mechanism in a multimodal large model, visual and semantic features are fused together. This allows the model to understand the semantic information of prompt words while analyzing the shape and spatial distribution features of the defect scatter plot, thereby outputting structured analysis results.

[0024] As some optional implementations of this application, the process of fusing visual and semantic features through the cross-modal feature alignment mechanism of a multimodal large model is as follows:

[0025] Feature extraction is performed on the defect scatter plot to obtain visual feature vectors representing spatial distribution, aggregation morphology differences, and density differences; and semantic modeling is performed on the prompt words to form semantic feature vectors.

[0026] Visual feature vectors and semantic feature vectors are fused through cross-modal attention and feature alignment mechanisms to achieve semantic understanding of defect clustering patterns;

[0027] The semantic understanding output based on the defect pattern includes clustering determination results and clustering type prediction results.

[0028] In the above embodiments, the present invention utilizes the general visual understanding capability obtained by pre-training a multimodal large model on massive data, which can automatically capture the spatial distribution pattern of scatter plots without any threshold parameter setting. It naturally possesses the ability to generalize to different products and different aggregation patterns, significantly improving the universality and robustness of panel defect aggregation analysis.

[0029] The following are some optional implementations of this application for performance evaluation and prompt word iterative optimization of multimodal large models:

[0030] Construct an independent validation set, and calculate performance metrics based on the output of the multimodal large model using the validation set;

[0031] Based on the performance index calculation results, the prompt words are iteratively optimized, and the design principles of the prompt words are adjusted to enhance the ability to distinguish defect categories and control reasoning priorities.

[0032] In the above embodiments, this invention designs a closed-loop iterative process of "evaluation-misclassification analysis-prompt word optimization" on the validation set. By analyzing the misclassified samples and their causes on the validation set, the prompt word content is adjusted in a targeted manner (e.g., adding judgment logic, adjusting category priority, supplementing feature descriptions, etc.), thereby continuously improving model performance. This lightweight optimization method does not require modification of model parameters or retraining; performance improvement can be achieved solely through adjustments to natural language instructions. Compared with traditional methods involving parameter tuning or model retraining, it is more efficient, lower in cost, and more interpretable.

[0033] In a second aspect, the present invention provides a panel defect clustering analysis system based on a multimodal large model, the system comprising:

[0034] A defect scatter plot unit is used to acquire the coordinate data of defect points on the glass panel and convert the coordinate data of defect points into a defect scatter plot.

[0035] The prompt word design unit is used to distinguish different defect types by combining the shape characteristics and spatial distribution characteristics of defect clusters, and to design prompt words according to different defect types;

[0036] A multimodal large model unit is used to input defect scatter plots and prompt words into a multimodal large model for joint analysis, and to perform performance evaluation and prompt word iterative optimization on the multimodal large model.

[0037] The defect clustering analysis unit is used to apply the iteratively optimized prompts and the multimodal large model of performance evaluation to the panel defect clustering analysis to obtain the analysis results.

[0038] In a third aspect, the present invention provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the panel defect clustering analysis method based on a multimodal large model.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the panel defect clustering analysis method based on a multimodal large model.

[0040] The beneficial effects of this invention are as follows:

[0041] (1) A technical approach that converts defect point coordinate data into a scatter plot and combines it with a multimodal large model for cluster analysis: Unlike existing technologies that rely on traditional clustering algorithms, geometric calculations, or CNN models, this invention uses the two-dimensional scatter plot converted from defect point coordinate data as visual image input. It leverages the image understanding and semantic reasoning capabilities of a multimodal large model to perform end-to-end cluster pattern recognition, achieving a cross-modal analysis paradigm shift from numerical coordinates to semantic understanding. This method fully utilizes the advantages of large models in visual pattern recognition and semantic understanding, without requiring the design of complex feature extraction algorithms or clustering rules.

[0042] (2) Achieve true zero-sample clustering pattern recognition: This invention directly uses a general open-source multimodal large model (e.g., Qwen3-VL) and guides the model to perform clustering analysis through designed prompt words. It can achieve accurate recognition of various clustering patterns such as horizontal lines, vertical lines, diagonal lines, local, surrounding, and central areas without any labeled data, model training, or fine-tuning process.

[0043] (3) An iterative optimization mechanism based on prompt word engineering is proposed: This invention designs a closed-loop iterative process of "evaluation-misclassification analysis-prompt word optimization" on the validation set. By analyzing the misclassified samples and their causes on the validation set, the prompt word content is adjusted in a targeted manner (e.g., adding judgment logic, adjusting category priority, and supplementing feature descriptions), thereby continuously improving model performance. This lightweight optimization method does not require modification of model parameters or retraining; performance improvement can be achieved simply by adjusting natural language instructions. Compared with traditional methods of parameter tuning or model retraining, it is more efficient, lower in cost, and more interpretable.

[0044] (4) Overcoming the limitations of threshold dependence and generalization in traditional methods: Existing technologies generally rely on multiple manually set threshold parameters, which have poor adaptability to different products and scenarios, requiring extensive expert experience and repeated debugging. This invention utilizes the general visual understanding capabilities obtained by pre-training a multimodal large model on massive amounts of data, which can automatically capture the spatial distribution patterns of scatter plots without any threshold parameter settings. It naturally possesses the ability to generalize to different products and different aggregation patterns, significantly improving the versatility and robustness of the method. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the panel defect clustering analysis method described in the embodiments of the present invention;

[0047] Figure 2 These are schematic diagrams illustrating different morphologies of the defect scatter point distribution as described in embodiments of the present invention;

[0048] Figure 3 This is a schematic diagram of the binary confusion matrix after optimizing the prompt words according to an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the multi-class confusion matrix after optimizing the prompt words according to an embodiment of the present invention. Detailed Implementation

[0050] To better understand the above technical solutions, the technical solutions of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solutions of the present invention, rather than limitations thereof. In the absence of conflict, the technical features in the embodiments and embodiments can be combined with each other.

[0051] It should also be understood that, in order to simplify the description of the invention and thus aid in the understanding of at least one embodiment, multiple features may sometimes be grouped into a single embodiment, drawing, or description thereof in the foregoing description of the embodiments of the invention. However, this method of disclosure does not imply that the subject matter of the invention requires more features than those mentioned in the claims. In fact, the embodiments contain fewer features than all the features of the single embodiment disclosed above.

[0052] Example 1

[0053] This invention provides a panel defect clustering analysis method based on a multimodal large model. By converting defect point coordinate data into a two-dimensional scatter plot and using prompts designed according to different clustering patterns, it leverages the visual understanding and semantic reasoning capabilities of the multimodal large model to achieve automated and intelligent defect clustering analysis. This method eliminates the need for large amounts of labeled data and complex parameter tuning, and boasts advantages such as high generalization ability and strong interpretability. For details, please refer to [link / reference]. Figure 1 , Figure 1 This is a flowchart of the panel defect clustering analysis method, which includes the following steps:

[0054] (1) Obtain the coordinate data of the defect points on the glass panel.

[0055] In the panel industry, glass substrates undergo multiple processing stages to ultimately form multiple sub-panels. These sub-panels are then inspected by an automated optical inspection system to obtain coordinate data for multiple defect points. The data collected in this embodiment of the invention includes the position and defect point coordinate data of each sub-panel. An example of defect point coordinate data is shown in Table 1.

[0056] x y panel_id 79886.7 699024 panel_id_1 800736 669160 panel_id_1 848519 20643900 panel_id_1 … … … -742780 0393589 panel_id_120 -436793 337816 panel_id_120

[0057] Table 1 shows the coordinate data of multiple defect points on a glass substrate.

[0058] In Table 1, x and y represent the coordinates of the defect points, and panel_id is the unique identifier of the sub-panel. For each glass panel requiring defect clustering analysis, defect point coordinate data for all its sub-panels needs to be collected. To more fully validate the model's effectiveness and optimize the prompts, defect coordinate data for glass substrates with different defect clustering types, as well as defect coordinate data for normal glass substrates without defect clustering, need to be collected.

[0059] (2) Convert the defect point coordinate data into a defect scatter plot.

[0060] The coordinate data of defect points of all sub-panels of each glass substrate are collected and plotted into a two-dimensional defect scatter plot, forming an image input for multimodal model recognition. When there is no obvious clustering, the defect points of the glass substrate are relatively uniformly distributed; while the defect clustering of qualified glass substrates is mainly reflected in two aspects: shape and location. In terms of shape, it includes but is not limited to horizontal line clustering, vertical line clustering, and diagonal line clustering; in terms of location, it includes but is not limited to central clustering, local clustering, and perimeter clustering.

[0061] The embodiments of the present invention do not limit the drawing style, such as adjusting the size, color, and transparency of the scatter points; at the same time, the coordinate point data can be preprocessed or data augmented, including but not limited to removing outliers, normalization, or labeling different clusters by clustering methods and distinguishing different clusters with different colors.

[0062] (3) Design prompt words according to different defect types.

[0063] In this embodiment of the invention, the process of designing prompt words according to different defect types is as follows:

[0064] (3.1) The design principles of the preset prompt words are as follows: The design principles of the prompt words include, but are not limited to, role positioning, task objectives, analysis process and output requirements.

[0065] (3.11) Role positioning: Glass panel defect cluster analysis expert, skilled in identifying defect clustering patterns through defect scatter plots.

[0066] (3.12) Task objective: Perform cluster analysis on the input defect scatter plot to determine whether defect clusters exist. If they exist (i.e., the scatter plot is not uniformly distributed), the most obvious cluster type needs to be further identified.

[0067] (3.13) Analysis process: Observe the overall distribution of the defect scatter plot. If the scatter plot is evenly distributed and there is no obvious clustering, it is judged as qualified; otherwise, it is judged as unqualified. If there is obvious clustering, according to the defect clustering pattern description, determine the most obvious clustering pattern of the defects and give the reason for the judgment.

[0068] Specifically, the defect clustering pattern is explained as follows:

[0069] Horizontal clustering: The defect scatter plot is generally evenly distributed, but some scatter points cluster along the horizontal direction.

[0070] Vertical clustering: Point-like defects are arranged in a vertical direction to form a cluster.

[0071] Diagonal clustering: Point-like defects are arranged along a diagonal direction to form a cluster.

[0072] Local clustering: Point-like defects form localized small-area clusters.

[0073] Clustering around: Point-like defects form clusters around the perimeter.

[0074] Central aggregation: Point-like defects form a central aggregation.

[0075] (3.14) Output requirements: Only output results in JSON format, and do not contain any extra text or symbols.

[0076] Furthermore, the specific content of the prompt words in this embodiment of the invention is not limited, and can be flexibly adjusted according to the complexity and recognition difficulty of the actual task. For example, example images can be provided for different defect clustering patterns, and these examples can be encoded and embedded into the system prompt words, or used as part of the user input to achieve few-shot learning, thereby improving the model's discrimination ability in specific scenarios. Preferably, this embodiment of the invention adopts a zero-shot inference mode, which does not require any parameter fine-tuning of the multimodal large model, nor does it rely on the input of specific example images. It only guides the model to perform autonomous analysis and category judgment on the generated defect scatter plot by referring to system-level prompt words designed for different defect clustering patterns, realizing generalized and intelligent analysis without the need for training data annotation.

[0077] (3.2) For two-dimensional defect scatter plots, the defect types are distinguished by combining the shape characteristics and spatial distribution characteristics of defect clusters, and prompt words are designed to guide the analysis of multimodal large models in accordance with the design principles of prompt words.

[0078] (4) Input the defect scatter plot and prompt words into the multimodal large model for joint analysis.

[0079] The system prompts are used as system prompts, and the defect scatter plot is Base64 encoded and used as user input. Together, they are input into the multimodal large model for joint analysis.

[0080] The multimodal large model has an image encoding module and a text encoding module. It achieves the fusion of visual features and semantic information through a cross-modal feature alignment mechanism. The model can analyze the shape features and spatial distribution patterns of defect scatter plots while understanding the semantics of prompt words, thereby outputting structured analysis results.

[0081] In this embodiment of the invention, the process for joint analysis of a multimodal large model is as follows:

[0082] (4.1) Encode the defect scatter plot and input the prompt words and the encoded defect scatter plot into the multimodal large model.

[0083] (4.2) The fusion of visual and semantic features is achieved through the cross-modal feature alignment mechanism of the multimodal large model, so that the model can analyze the shape features and spatial distribution features of the defect scatter plot while understanding the semantic information of the prompt words, thereby outputting structured analysis results.

[0084] Specifically, the process of fusing visual and semantic features through the cross-modal feature alignment mechanism of a multimodal large model is as follows:

[0085] (4.21) Feature extraction is performed on the defect scatter plot to obtain visual feature vectors of reaction spatial distribution, aggregation morphology differences and density differences; and semantic modeling is performed on the prompt words to form semantic feature vectors.

[0086] (4.22) Visual feature vectors and semantic feature vectors are fused through cross-modal attention mechanism and feature alignment mechanism to achieve semantic understanding of defect clustering patterns.

[0087] (4.23) The semantic understanding output based on the defect pattern includes the clustering judgment result (whether there is a defect cluster, i.e., qualified or unqualified) and the clustering type prediction result (e.g., horizontal line, vertical line, ring, center, local, surrounding, diagonal line, etc.), and the judgment reason is attached.

[0088] (5) Perform performance evaluation and prompt word iterative optimization on the multimodal large model.

[0089] After the initial defect clustering analysis of the multimodal large model, in order to comprehensively evaluate the accuracy and robustness of its judgments, an independent validation set needs to be introduced to systematically validate the model's output results, and the system prompts should be iteratively optimized based on the evaluation feedback. This process aims to improve the model's generalization ability and semantic understanding consistency in real-world application scenarios, ensuring that it meets the accuracy requirements of industrial-grade quality control; the specific implementation method is as follows:

[0090] (5.1) Validation Set Construction: A set of scatter plots of glass panel defects that were not involved in the system prompt word design or example guidance was selected as the validation set to ensure data independence and evaluation objectivity. The validation set should cover a variety of typical defect clustering patterns, including but not limited to: horizontal clustering, vertical clustering, diagonal clustering, perimeter clustering, central clustering, local clustering, and normal distribution samples without significant clustering characteristics. Each type of sample should be representative, covering different densities, sizes, noise levels, and spatial distribution complexities to fully test the model's discrimination boundary.

[0091] (5.2) Performance index calculation: The output results of the multimodal large model on the validation set are quantitatively evaluated, and statistical analysis is carried out at two levels: ① Binary classification evaluation of whether there is clustering: The results of the model's judgment of clustering and non-clustering are compared with the manually labeled true values, and the accuracy, recall, precision and F1 score are calculated to measure the overall anomaly detection capability of the model; ② Multi-class evaluation of clustering type identification: For samples judged to have clustering, the accuracy of their category classification is further evaluated. The recall, precision and F1 score of each category are calculated by macro-average or weighted average to reflect the model's balance in identifying different clustering patterns; Finally, based on the quality control requirements of the actual production line, key evaluation indicators are selected as optimization targets.

[0092] (5.3) Misclassification Sample Analysis and Prompt Word Optimization: For samples that are misclassified or missed by the model, in-depth analysis is conducted to identify the causes such as visual feature ambiguity, pattern similarity, or semantic ambiguity. Combining domain expert knowledge, the model's judgment logic deviations are traced, and the semantic description rules in the system's prompt words are adjusted accordingly to enhance the ability to distinguish between categories and control the reasoning priority.

[0093] For example, if a sample shows locally dense defects located in the center of an image, and the model incorrectly classifies it as a local cluster while the true label is a central cluster, this indicates that the current prompt word has failed to effectively establish a priority relationship between spatial location and cluster type. To address this, explicit logical constraints can be introduced into the system prompt words. For instance, when the defect cluster area is located in the center of the image, it should be preferentially judged as a central cluster, even if it exhibits local high density; only when the cluster area deviates from the center (e.g., X and Y coordinates between -30000 and 30000 are considered central regions) should it be classified as a local cluster.

[0094] Through the above analysis and adjustments, a closed-loop iterative mechanism of evaluation, diagnosis, and optimization is formed. Multiple rounds of prompt word revisions can be completed collaboratively by process engineers and others, gradually improving the model's consistency in judgments under boundary cases and easily confused scenarios. This process continues until the model's core metrics on the validation set stably reach a preset threshold (e.g., F1 ≥ 0.95), completing the convergence and finalization of the prompt word strategy.

[0095] (6) Apply the iteratively optimized prompts and the multimodal large model of performance evaluation to panel defect clustering analysis to obtain the analysis results.

[0096] Using optimized prompt words, the prompt words are used as system prompt words, and the defect scatter plot is Base64 encoded as the user's question content. Together, they are input into the multimodal large model for judgment and analysis, and the cluster analysis results of the defect scatter plot of the glass substrate are obtained.

[0097] In the embodiments of the present invention, in order to more clearly demonstrate the purpose, technical solution and advantages of the present invention, the present invention will be described in detail below with specific data, in conjunction with the accompanying drawings and specific embodiments.

[0098] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0099] S10: Obtain the coordinate data of the defect points on the glass panel.

[0100] In this embodiment, a panel manufacturer needs to determine the aggregation type of the glass substrate, including: horizontal aggregation, vertical aggregation, diagonal aggregation, local aggregation, perimeter aggregation, center aggregation, and no obvious aggregation. Therefore, it is necessary to collect defect coordinate data for glass substrates with different aggregation types, as well as defect coordinate data for normal glass substrates (without obvious aggregation). Examples of the collected defect point coordinate data are shown in Table 2.

[0101] x y panel_id 79886.7 699024 panel_id_1 800736 669160 panel_id_1 848519 20643900 panel_id_1 … … … -742780 0393589 panel_id_120 -436793 337816 panel_id_120

[0102] Table 2 shows the coordinate data of multiple defect points on a glass substrate.

[0103] In this embodiment of the invention, a total of 525 glass substrate defect point coordinate data were collected, including 305 normal glass substrates, 28 glass substrates with defects clustered in the center, 27 glass substrates with defects clustered around the perimeter, 56 glass substrates with defects clustered in certain areas, 36 glass substrates with defects clustered diagonally, 44 glass substrates with defects clustered horizontally, and 29 glass substrates with defects clustered vertically.

[0104] S20: Convert the defect point coordinate data into a defect scatter plot.

[0105] In this embodiment, the defect point coordinate data of the glass substrate collected in step S10 is converted into a defect scatter plot. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 To illustrate different morphologies of the defect scatter plot distribution, visualization of scatter plots with various clustering types reveals that: scatter plots with horizontal clustering have clusters along horizontal lines; scatter plots with vertical clustering have clusters along vertical lines; scatter plots with diagonal clustering have clusters along diagonal lines; scatter plots with local clustering show a higher concentration of clusters in a specific area; scatter plots with clustering around the perimeter show a higher density of clusters around the perimeter compared to the center; scatter plots with clustering in the center show a clear clustering in the middle area; and normal scatter plots show no obvious clustering.

[0106] S30: Design prompt words according to different defect types.

[0107] In this embodiment, the designed prompt word template is shown in Table 3: the prompt words are designed according to the principles of prompt word design. In this embodiment of the invention, the designed prompt words are shown in Table 3:

[0108] Table 3 shows examples of system prompts for defect clustering analysis. It should be noted that the specific form and content of the system prompts in this embodiment of the invention are not limited and can be flexibly adjusted according to the complexity of the actual task, the difficulty of distinguishing between target categories, and the specific needs of the application scenario. For example, in cases of high recognition difficulty, a small number of typical example images can be introduced, these examples encoded as multimodal inputs, and embedded as contextual information in the system prompts, or submitted along with the user query, to achieve small-sample learning and thus enhance the model's understanding of specific patterns. Preferably, this invention adopts a zero-sample inference mode, that is, without any parameter fine-tuning or additional training of the multimodal large model, the model is driven to autonomously understand the defect scatter plot generated in step S20 and complete clustering judgment and type classification solely through the designed system prompts guided by natural language and vision. This method relies entirely on the visual perception and semantic reasoning capabilities already possessed by the pre-trained large model, achieving lightweight and generalized intelligent analysis without labeled data or model retraining, significantly improving the portability and industrial deployment efficiency of the method.

[0109] S40: Input the defect scatter plot and prompt words into the multimodal large model for analysis.

[0110] In this embodiment, the prompt word designed in step S30 is used as the system prompt word, and the defect scatter plot obtained in step S20 is Base64 encoded and used as the user input content. Both are input as the message body into the multimodal large model for joint analysis. Preferably, the present invention uses Qwen3-VL-8B-Instruct as the multimodal large model. After input into the multimodal large model, the multimodal large model performs joint analysis based on the system prompt word and the user input content, outputting a clustering determination result (whether defect clustering exists, OK or NG), a clustering type prediction result, and an accompanying reason for the determination.

[0111] S50: Performance evaluation and iterative optimization of prompt words based on the validation set.

[0112] In this embodiment, all scatter plots obtained in step S20 are used as the validation set. The multimodal large model from step S40 is used for analysis to calculate the validation set metrics, including precision, recall, and F1 score. The binary and multi-class validation metrics of the unoptimized multimodal large model on the validation set of 525 glass substrates are shown in Tables 4 and 5:

[0113] category Precision Recall F1-Score Sample size OK 0.93 0.88 0.90 305 NG 0.84 0.90 0.87 220

[0114] Table 4 shows the binary classification (OK / NG) indicators.

[0115] category Precision Recall F1 Sample size OK 0.93 0.88 0.90 305 Central 0.36 0.32 0.34 28 all around 0.47 0.30 0.36 27 Local 0.21 0.43 0.28 56 diagonal line 0.96 0.64 0.77 36 horizontal line 0.76 0.64 0.69 44 vertical line 0.70 0.48 0.57 29

[0116] Table 5 lists the various indicators for multiple categories.

[0117] As shown in Table 5, the multimodal large model exhibits certain misclassifications when determining cluster types, such as classifying local clusters as central clusters and surrounding clusters as local clusters. Based on these misclassifications, the prompt words are adjusted until the validation set metrics meet expectations. In this embodiment of the invention, the binary and multi-class validation metrics after adjusting and optimizing the prompt words are shown in Tables 6 and 7.

[0118] category Precision Recall F1-Score Sample size OK 0.96 0.95 0.96 305 NG 0.94 0.95 0.94 220

[0119] Table 6 shows the binary classification (OK / NG) metrics after optimizing the prompt words.

[0120] category Precision Recall F1-Score Sample size OK 0.96 0.95 0.96 305 Central gathering 0.68 0.82 0.74 28 Gathering around 0.62 0.67 0.64 27 Local clustering 0.57 0.64 0.61 56 Diagonal convergence 0.93 0.75 0.83 36 Horizontal lines converge 0.83 0.80 0.81 44 Vertical lines converge 0.84 0.72 0.78 29

[0121] Table 7 shows the multi-category indicators after optimizing the prompt words.

[0122] Please see Figure 3 , Figure 4 , Figure 3 This is a schematic diagram of the binary confusion matrix after optimizing the prompt words. Figure 4 The diagram shows the multi-class confusion matrix after optimizing the prompts. Combining the diagram and table, it can be seen that the adjusted model can better distinguish whether there are clusters in the defect scatter plot, and the type of clusters. Accuracy, recall, and F1 score are all improved for different cluster types. The prompts can be continuously adjusted according to needs until the validation set metrics reach the expected level.

[0123] S60: Apply the optimized prompts to the test samples to obtain the analysis results of the test samples.

[0124] In this embodiment, the prompt words optimized in step S50 are used as system prompt words, and the defect scatter plot obtained in step S20 is encoded using Base64 as the user's question content. Together, they are input into the multimodal large model for judgment and analysis to obtain the cluster analysis results of the defect scatter plot of the glass substrate.

[0125] Example 2

[0126] This invention provides a panel defect clustering analysis system based on a multimodal large model. The system corresponds one-to-one with the method described in Example 1. The system includes:

[0127] The defect scatter plot unit is used to acquire the coordinate data of defect points on the glass panel and convert the coordinate data of defect points into a defect scatter plot.

[0128] The prompt word design unit is used to distinguish different defect types by combining the shape characteristics and spatial distribution characteristics of defect clusters, and to design prompt words according to different defect types.

[0129] A multimodal large model unit is used to input defect scatter plots and prompt words into a multimodal large model for joint analysis, and to perform performance evaluation and prompt word iterative optimization on the multimodal large model.

[0130] The defect clustering analysis unit is used to apply the iteratively optimized prompts and the multimodal large model of performance evaluation to the panel defect clustering analysis to obtain the analysis results.

[0131] Example 3

[0132] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the panel defect clustering analysis method based on a multimodal large model as described in Embodiment 1.

[0133] The computer device provided in this embodiment can implement the method described in Embodiment 1. To avoid repetition, it will not be described again here.

[0134] Example 4

[0135] This invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the panel defect clustering analysis method based on a multimodal large model described in Embodiment 1.

[0136] The computer-readable storage medium provided in this embodiment can implement the method described in Embodiment 1. To avoid repetition, it will not be described again here.

[0137] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0138] The memory can be used to store the computer program and / or modules. The processor implements various functions of the two-stage parameter-level root cause localization system of the invention by running or executing the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart memory card, secure digital card, flash memory card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0139] If a panel defect clustering analysis system based on a multimodal large model is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program that can be stored in a computer-readable storage medium. When executed by a processor, this computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory, random access memory, dot carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0140] The basic concepts of this invention have been described. Obviously, for those skilled in the art, the above detailed disclosure is merely illustrative and does not constitute a limitation of this specification. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this specification. Such modifications, improvements, and corrections are suggested in this specification and therefore remain within the spirit and scope of the exemplary embodiments described herein.

Claims

1. A panel defect clustering analysis method based on a multimodal large model, characterized in that, The method The process includes the following steps: Obtain the coordinate data of defect points in the glass panel and convert the defect point coordinate data into a defect scatter plot; Different defect types are distinguished by combining the shape and spatial distribution characteristics of defect clusters, and prompt words are designed according to different defect types; The defect scatter plot and prompt words are input into the multimodal large model for joint analysis, and the performance of the multimodal large model is evaluated and the prompt words are iteratively optimized. The iteratively optimized prompts and the multimodal large model of performance evaluation were applied to panel defect clustering analysis to obtain the analysis results.

2. The panel defect clustering analysis method based on a multimodal large model according to claim 1, characterized in that, The process of converting defect point coordinate data into a defect scatter plot is as follows: Preprocessing and data augmentation of defect point coordinate data; For all sub-panels of the glass panel, the coordinate data of the defect points of the sub-panels are plotted as a two-dimensional defect scatter plot.

3. The panel defect clustering analysis method based on a multimodal large model according to claim 1, characterized in that, The process for designing prompts based on different defect types is as follows: The design of preset prompts follows certain principles; these principles include role positioning, task objectives, analysis process, and output requirements. For two-dimensional defect scatter plots, defect types are distinguished by combining the shape characteristics and spatial distribution characteristics of defect clusters, and prompt words are designed to guide the analysis of multimodal large models in accordance with the design principles of prompt words.

4. The panel defect clustering analysis method based on a multimodal large model according to claim 3, characterized in that, The defect types include horizontal clusters, vertical clusters, diagonal clusters, local clusters, perimeter clusters, and central clusters.

5. The panel defect clustering analysis method based on a multimodal large model according to claim 1, characterized in that, The process of inputting defect scatter plots and cue words into a multimodal large model for joint analysis is as follows: The defect scatter plot is encoded, and the prompt words and the encoded defect scatter plot are input into the multimodal large model; By using a cross-modal feature alignment mechanism in a multimodal large model, visual and semantic features are fused together. This allows the model to understand the semantic information of prompt words while analyzing the shape and spatial distribution features of the defect scatter plot, thereby outputting structured analysis results.

6. The panel defect clustering analysis method based on a multimodal large model according to claim 5, characterized in that, The process of fusing visual and semantic features through cross-modal feature alignment mechanism of multimodal large model is as follows: Feature extraction is performed on the defect scatter plot to obtain visual feature vectors representing spatial distribution, aggregation morphology differences, and density differences; and semantic modeling is performed on the prompt words to form semantic feature vectors. Visual feature vectors and semantic feature vectors are fused through cross-modal attention and feature alignment mechanisms to achieve semantic understanding of defect clustering patterns; The semantic understanding of defect patterns outputs analysis results including cluster determination and cluster type prediction.

7. The panel defect clustering analysis method based on a multimodal large model according to claim 1, characterized in that, The process for performance evaluation and iterative optimization of prompt words for large multimodal models is as follows: Construct an independent validation set, and calculate performance metrics based on the output of the multimodal large model using the validation set; Based on the performance index calculation results, the prompt words are iteratively optimized, and the design principles of the prompt words are adjusted to enhance the ability to distinguish defect categories and control reasoning priorities.

8. A panel defect clustering analysis system based on a multimodal large model, characterized in that, The system includes: A defect scatter plot unit is used to acquire the coordinate data of defect points on the glass panel and convert the coordinate data of defect points into a defect scatter plot. The prompt word design unit is used to distinguish different defect types by combining the shape characteristics and spatial distribution characteristics of defect clusters, and to design prompt words according to different defect types; A multimodal large model unit is used to input defect scatter plots and prompt words into a multimodal large model for joint analysis, and to perform performance evaluation and prompt word iterative optimization on the multimodal large model. The defect clustering analysis unit is used to apply the iteratively optimized prompts and the multimodal large model of performance evaluation to the panel defect clustering analysis to obtain the analysis results.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the panel defect clustering analysis method based on a multimodal large model as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the panel defect clustering analysis method based on a multimodal large model as described in any one of claims 1-7.

Images