Product yield control method and device based on large model

Abstract

The invention provides a product yield control method and device based on a large model, and relates to the field of quality control and artificial intelligence, and the method comprises the steps: obtaining various types of multi-modal data related to the product yield control, so as to construct a multi-modal yield data set; carrying out fine tuning training on the multi-modal fusion large model based on the multi-modal yield data set so as to construct a yield multi-modal large model; integrating various business analysis models and the yield multi-mode large model in an intelligent agent calling mode to form an integrated yield management and control system, and accessing various information systems to the integrated yield management and control system; and analyzing the user interaction data through an integrated yield management and control system to obtain a yield analysis management and control result. According to the method, the yield multi-mode large model is used as a basic base, flexible calling of the basic large model and the business analysis models is achieved through the intelligent agent, and the problem existing in existing product yield management and control is solved.

Description

Technical Field

[0001] The present invention relates to the fields of quality control and artificial intelligence, and in particular to a product yield control method and device based on a large model. Background Art

[0002] In the process of industrial product production and processing, the yield rate of products is often an important issue. Factories use statistical process control, root cause analysis and other means to control the process parameters of products during production and processing, and locate the root cause sites and parameters of defects. At the same time, combined with past historical experience and historical control records, process parameters are adjusted to improve product yield.

[0003] However, the existing product yield control methods have the following problems: (1) Yield data covers a wide range, and various types of data are currently scattered. Existing methods make it difficult to effectively integrate various types of yield-related data. (2) Abnormal monitoring of yield parameters and root cause parameter analysis require strong manual experience. Even if automated control is achieved, the operation of related systems still has a high knowledge threshold, and the operation process is relatively complex, and the cost of personnel training and learning is high. (3) Existing methods are mainly limited to a single data modality. For example, defect detection focuses on images, and root cause analysis focuses on parameter values. It is difficult to achieve comprehensive analysis of multi-modal data, and the accuracy of the analysis results needs to be improved. (4) The integration of different yield analysis models is difficult, and it is difficult to achieve multi-dimensional and comprehensive product yield analysis. Summary of the invention

[0004] The present invention provides a product yield control method and device based on a large model, which solves the problems existing in the existing product yield control.

[0005] In a first aspect, an embodiment of the present invention provides a product yield control method based on a large model, the method comprising the following process: Obtain various multimodal data related to product yield control to build a multimodal yield data set; Fine-tune and train the multimodal fusion model based on the multimodal yield data set to build a yield multimodal large model; Integrate various business analysis models and yield multimodal large models by using intelligent agent calls to form an integrated yield control system, and connect various information systems to the integrated yield control system; User interaction data is analyzed through an integrated yield control system to obtain yield analysis and control results.

[0006] In the above embodiment, the present invention takes the yield multimodal large model as the basic foundation, and realizes flexible calling of the basic large model and various business analysis models through intelligent agents, which not only ensures the independence between different models, but also facilitates the subsequent upgrading of the business analysis model.

[0007] As some optional embodiments of the present application, the various types of multimodal data related to the product yield control include data in document form, data in image form, data in numerical parameter form and data in network diagram form; wherein the data in document form includes process data, yield analysis reports and related research results of yield control; the data in image form includes defect images; the data in numerical parameter form includes production and processing data; the data in network diagram form includes production history data.

[0008] In the above embodiment, the yield multimodal big model used in the present invention makes full use of various types of multimodal data related to product yield control (such as yield process documents, yield history charts, defect graphs, etc.), which can better improve the big model's ability to understand specific professional background knowledge, and improve the big model's accuracy in question understanding, answer generation, and task understanding.

[0009] As some optional implementations of the present application, the method fine-tunes the open source multimodal fusion large model based on multimodal yield data, and privately deploys it to a local server to obtain a yield multimodal large model.

[0010] In the above embodiment, the present invention fine-tunes the open source multimodal fusion large model and privately deploys it to the local server, thereby solving the problem that the amount of corpus is difficult to meet the training needs and the training cost is very high when training is directly based on the factory's professional corpus.

[0011] As some optional implementation modes of the present application, the business analysis model includes a multimodal data retrieval model, a virtual measurement model, a yield prediction model and a root cause analysis model.

[0012] In the above embodiments, the present invention not only utilizes the highly innovative and latest large model technology, but can also be integrated with various traditional analysis methods and models, effectively utilizing the factory's original analysis methods and models, reducing the model development cost, and can be combined with various original analysis methods, experiences, and methods, which not only improves the accuracy of business analysis, but also improves the reliability and interpretability of yield analysis results.

[0013] As some optional implementation modes of the present application, the intelligent agent is used for task planning, task scheduling and user interaction.

[0014] In the above embodiments, the present invention is based on the method of intelligent agent calling, which fully utilizes the large model's powerful understanding and representation capabilities of natural language. Compared with various traditional analysis systems based on data screening, clicking, and switching, the yield control system based on intelligent agents allows users to directly input questions like using a search engine, and then get answers, and process and organize relevant results, thereby further enhancing the interactive experience between users and the yield control system, improving the humanization of the product, and reducing learning and training costs; at the same time, the intelligent agent can learn the analysis logic in the traditional analysis method, accumulate the experience and knowledge of experts, automatically select the corresponding business analysis model, and reduce the experience threshold of the yield control system.

[0015] As some optional implementation modes of the present application, the information system includes a machine vision and defect detection system, a statistical process control system, a manufacturing execution system, a quality management system, a database, a server and a storage system.

[0016] As some optional implementation methods of the present application, the process of analyzing user interaction data through an integrated yield control system to obtain yield analysis and control results is as follows: Receive multimodal user interaction data through the intelligent agent of the yield control system; The yield control system’s intelligent agent calls the yield multimodal big model, and uses the yield multimodal big model to parse multimodal user interaction data to obtain a unified embedded representation. Based on the unified embedded representation, the intelligent agent of the yield control system performs task planning and task scheduling to call various business analysis models, and performs yield analysis through the business analysis model to obtain the yield analysis results; The yield multimodal big model is called by the intelligent body of the yield control system, and the yield analysis results are uniformly processed by the yield multimodal big model to obtain the yield analysis control results; The yield analysis and control results are output through the intelligent agent of the yield control system.

[0017] In the above-mentioned embodiment, the present invention is based on a large yield multi-modal model, and data of various modalities such as images can be directly input, and the model processing process is fully integrated and automated; In a second aspect, the present invention provides a product yield control device based on a large model, the device comprising: A multimodal data acquisition unit, which is used to acquire various types of multimodal data related to product yield control to construct a multimodal yield data set; A large model fine-tuning training unit, wherein the large model fine-tuning training unit fine-tunes and trains a multimodal fusion large model based on a multimodal yield data set to construct a yield multimodal large model; A business model integration unit, which integrates various business analysis models and yield multimodal large models by intelligent agent calling to form an integrated yield control system, and connects various information systems to the integrated yield control system; The yield control and analysis unit analyzes the user interaction data through an integrated yield control system to obtain a yield analysis and control result.

[0018] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large model-based product yield control method when executing the computer program.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the large model-based product yield control method is implemented.

[0020] The beneficial effects of the present invention are as follows: 1. The present invention takes the yield multimodal large model as the basic foundation, and realizes flexible calling between the basic large model and various business analysis models through intelligent agents, which not only ensures the independence between different models, but also facilitates the subsequent upgrading of the business analysis model.

[0021] 2. The yield multimodal big model used in the present invention makes full use of various multimodal data related to product yield control (such as yield process documents, yield history charts, defect graphs, etc.), which can better improve the big model's ability to understand specific professional background knowledge, and improve the big model's accuracy in question understanding, answer generation, and task understanding.

[0022] 3. The present invention fine-tunes the open source multimodal fusion large model and privately deploys it to the local server, which can solve the problem that the amount of corpus is difficult to meet the training needs and the training cost is very high when training is directly based on the professional corpus of the factory.

[0023] 4. The present invention not only utilizes the highly innovative and latest large model technology, but can also be integrated with various traditional analysis methods and models, effectively utilizing the factory's original analysis methods and models, reducing the model development cost, and can be combined with various original analysis methods, experience, and methods, which not only improves the accuracy of business analysis, but also improves the reliability and interpretability of yield analysis results.

[0024] 5. The present invention is based on the method of intelligent agent calling, which makes full use of the large model's powerful understanding and representation capabilities of natural language. Compared with various traditional analysis systems based on data screening, clicking, and switching, the yield control system based on intelligent agents allows users to directly input questions like using a search engine, and then get answers, and process and organize relevant results, further enhancing the interactive experience between users and the yield control system, improving the humanization of products, and reducing learning and training costs; at the same time, the intelligent agent can learn the analysis logic of traditional analysis methods, accumulate the experience and knowledge of experts, automatically select the corresponding business analysis model, and reduce the experience threshold of the yield control system. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0026] Figure 1 It is an overall flow chart of the product yield control method according to an embodiment of the present invention; Figure 2 It is an overall architecture diagram of the yield control system according to an embodiment of the present invention; Figure 3 is an overall workflow diagram of the yield control system according to an embodiment of the present invention; Figure 4 It is a deployment diagram of the yield control system described in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0028] In order to solve the existing problems of product yield control. The embodiment of the present invention provides a product yield control method based on a large model, please refer to Figure 1 , Figure 1 The overall flow chart of the product yield control method is as follows: (1) Obtain various types of multimodal data related to product yield control to build a multimodal yield data set; Specifically, since yield control requires comprehensive consideration of defect detection, virtual measurement, yield prediction and early warning, root cause analysis, and process parameter adjustment, forming quality control throughout the entire process, it is possible to improve the timeliness and accuracy of yield control and form a closed loop of quality control.

[0029] In particular, in the yield control scenario of the entire process, a large amount of various multimodal data is often involved. Among them, various types of multimodal data related to product yield control include data in document form, data in image form, data in numerical parameter form, and data in network form. The data in document form includes process data (PPT, PDF, Word, etc.), yield analysis reports, and related research results of yield control (various Excel reports, reports, etc.). The data in image form includes defect images, etc. The data in numerical parameter form includes production and processing data, etc. The data in network diagram form includes production history data, etc. In order to improve the ability of the yield multimodal large model to understand specific business knowledge, it is necessary to use as much open source and closed source data as possible to train the model, thereby forming a multimodal yield data set.

[0030] In particular, the production and processing data in the form of numerical parameters include the process parameters of the factory production machines, the processing data recorded by various sensors, the measurement values ​​of the measuring machines, and the bad detection results (i.e. defect data) recorded in the quality inspection process, etc.; the production and processing data in the form of numerical parameters are the most important data in yield control. These data can not only support the training and construction of the model, but also users can query, analyze and calculate the yield data based on these data during the online use of the yield control system.

[0031] It should be emphasized that various types of multimodal data related to product yield control should not be limited to the factory's own data, but should also be widely collected from open source channels to further collect relevant industry information to make up for the lack of specific professional corpus. By making full use of relevant professional corpus from open source channels, the big model's ability to understand specific business knowledge can be improved. Relevant open source channels include but are not limited to: professional forums, professional books, paper databases, public patent databases, etc. At the same time, corpus includes but is not limited to Chinese, English, Korean, etc.

[0032] (2) Fine-tune and train the multimodal fusion large model based on the multimodal yield dataset to construct a yield multimodal large model.

[0033] Specifically, based on the collected multimodal yield data as corpus data, the open source multimodal fusion large model is fine-tuned and trained, and deployed to the local server to form a factory-privatized yield multimodal large model.

[0034] It should be noted that since the professional background knowledge of various business scenarios in industrial production scenarios is often complex, if the open source big model is used directly, the big model has limited understanding of professional knowledge and the effect of the big model is often not good. If the training is directly based on the professional corpus of the factory, not only the amount of corpus is difficult to meet the training needs, but also the cost of training is very high. At the same time, the factory environment has high requirements for data and network security, so the embodiment of the present invention adopts a solution of fine-tuning and private deployment based on the open source multimodal fusion big model.

[0035] Specifically, the multimodal fusion large model includes but is not limited to: Qianwen multimodal large model Qwen2-VL, BLIP-2, GPT-4V, etc. Preferably, the embodiment of the present invention selects Qianwen multimodal large model Qwen2-VL.

[0036] It should be pointed out that fine-tuning and training open source large models based on private corpus data in a specific field is a commonly used technical means in this field. Common fine-tuning training methods include instruction fine-tuning, alignment fine-tuning, etc. The embodiments of the present invention do not limit the fine-tuning training methods.

[0037] (3) Integrate various business analysis models and yield multimodal models by using intelligent agent calls to form an integrated yield control system, and connect various information systems to the integrated yield control system. Figure 2 , Figure 2 This is the overall architecture diagram of the yield control system.

[0038] Specifically, the yield multimodal big model constructed by fine-tuning training is used as the base, and the yield multimodal big model and various business analysis models are called by the agent to build a yield control system based on the yield multimodal big model.

[0039] It should be noted that the intelligent agent in the embodiment of the present invention is responsible for task planning, task scheduling (determining which business analysis model to select and call) and user interaction.

[0040] In addition, the function of the yield multimodal large model is mainly to understand the multimodal data input by the user and form a unified embedded representation; and further process the yield analysis results based on each business analysis model to form more interactive yield control data (for example, for the query results of the multimodal data retrieval model, enrich it into a descriptive language or generate statistical charts); and support the construction of a basic database to support the intelligent agent's understanding of multimodal information when performing task planning.

[0041] It should be emphasized that in addition to the base yield multimodal large model, various business analysis models in the yield control system can configure various artificial intelligence models as needed, including other large language models, and the embodiment of the present invention does not limit this.

[0042] It should be pointed out that the system applications corresponding to various business analysis models include, but are not limited to: defect detection systems, yield prediction and early warning systems, virtual measurement systems, root cause analysis systems, and process parameter adjustment systems. Various analysis methods can be flexibly integrated and updated in these systems, including traditional statistical analysis methods, machine learning methods, deep learning methods, optimization methods, complex network analysis methods, association rule mining and other data mining analysis methods, etc. The embodiments of the present invention do not limit this, and the specific means will not be repeated. At the same time, the methods in these systems can also be flexibly updated and iterated independently according to the results after the system is put into operation.

[0043] Specifically, after completing the functional test of the yield control system, the system will be deployed to the factory's production environment and various information systems will be connected.

[0044] In particular, the information system includes but is not limited to: machine vision and defect detection system (AOI defect detection system, mainly used for real-time monitoring of product defects), statistical process control system (SPC system, quality control based on industry standards and experience), manufacturing execution system (MES system, used for monitoring and scheduling of production processes), quality management system (QMS system, used for recording and collaborative processing of quality events), electronic design automation system (EDA system, used for design input, design simulation, synthesis, layout and routing, physical verification and analog circuit simulation of product circuits, etc.), multimodal databases, servers and storage systems, etc.

[0045] (4) Analyze user interaction data through an integrated yield control system to obtain yield analysis and control results.

[0046] Specifically, for the constructed yield control system, by receiving user questions (user interaction data), the corresponding yield analysis and control results can be given through intelligent agent calls.

[0047] In particular, user interaction data, namely, user analysis needs, include but are not limited to: querying the defects and defective rate of a batch of products, querying the historical defective rate and generating reports, inferring the root cause of a batch of product defects, locating the root cause site and parameters, giving process parameter adjustment suggestions for the root cause parameters that lead to product defects, predicting the yield of a batch of products based on the current measurement results, and issuing early warnings for the product processing process based on the current processing parameter detection results.

[0048] In particular, the yield analysis and control results include but are not limited to: product defect detection conclusions and defective rate results, historical yield results queried by users and related reports generated, root cause analysis conclusions for defects, production parameter adjustment suggestions, yield prediction results, yield early warning reminders, etc.

[0049] For details, please refer to Figure 3 , Figure 3 The overall workflow diagram of the yield control system is as follows: (4.1) Receive multimodal user interaction data through the intelligent agent of the yield control system.

[0050] (4.2) The yield control system’s intelligent agent calls the yield multimodal big model, and uses the yield multimodal big model to parse the multimodal user interaction data to obtain a unified embedded representation.

[0051] (4.3) Based on the unified embedded representation, the intelligent agent of the yield control system performs task planning and task scheduling to call various business analysis models, and performs yield analysis through the business analysis model to obtain the yield analysis results.

[0052] (4.4) The yield multimodal big model is called by the intelligent body of the yield control system, and the yield analysis results are uniformly processed by the yield multimodal big model to obtain the yield analysis control results.

[0053] (4.5) The yield analysis and control results are output through the intelligent agent of the yield control system.

[0054] Specifically, the core of the overall workflow of the yield control system lies in characterizing and analyzing multimodal data, understanding user input, and planning and scheduling tasks.

[0055] It should be pointed out that when asking questions, users can provide relevant interactive data, or specify in the questions that the business analysis model will directly mobilize the database for query analysis. The ways in which users provide interactive data include, but are not limited to: directly uploading files, providing data access links, etc. In their questions, users can specify the data range that needs to be analyzed, including, but not limited to: product batches, time ranges, processes where product defects occur, types of product defects, measurement points in the measurement process, etc., so that the yield control system can convert the user's input language into knowledge base query statements for query analysis. At the same time, when users ask questions, the richer the data information provided (such as the type of defect, the process where the defect occurs, defect defect images, the clustering of defect points, and Map diagrams, etc.), the more conducive it is for the yield control system to provide accurate yield control data.

[0056] It should be emphasized that the yield control system based on large models is more user-friendly and intelligent than the traditional system, and users can ask questions flexibly according to their needs. At the same time, in practice, even for the same analysis requirements, due to differences in experience levels, knowledge backgrounds, and expression habits of different users, users may ask questions in a variety of ways. Therefore, it is necessary to consider the diversity of user questions and ensure a good user experience, and to ensure that the user's questions are as accurate as possible so that the yield control system can give accurate answers. The embodiments of the present invention solve this problem from two aspects: technology and product.

[0057] Specifically, relevant means to improve the effectiveness of user questions include, but are not limited to: (1) Technical means: Based on natural language processing technology, use multi-round dialogue analysis technology to analyze the semantics of user input. When it is recognized that the user's question is unclear or incomplete, guide the user to complete the question through interactive dialogue; or preset some prompts to guide the yield multimodal large model to better understand the user's common expressions.

[0058] (2) Product-related measures: Providing users with “related question tips” and “frequently asked questions” to guide them to ask better questions; or using tool buttons or help buttons on the display interface to provide relevant question examples, guide tutorials, etc., or organizing training for users on how to use the system.

[0059] The above methods are relatively mature. Since they are not the focus of the embodiments of the present invention, the embodiments of the present invention do not limit them. They can be selected based on the performance in specific business background corpus and specific question-and-answer scenarios.

[0060] It should be noted that users can also specify in their questions the analysis methods to be used or the business analysis models to be called for analysis needs (such as "Please use the association rule mining method to analyze the root cause of a batch of defective products"), so that the yield control system can perform targeted analysis based on the user's needs. If the user does not specify it clearly, the yield control system will use the corresponding analysis methods or call the default business analysis model based on the agent's understanding of the analysis task.

[0061] The following provides several implementation forms of the embodiments of the present invention for business scenarios in the new display material processing and manufacturing industry. It should be noted that these implementation forms are only several implementation forms of the embodiments of the present invention, but not all implementation forms of the embodiments of the present invention. At the same time, the product yield control method proposed in the embodiments of the present invention is not limited to the new display material processing and manufacturing industry.

[0062] Specifically, in the semiconductor manufacturing industry, there are often problems such as data dispersion, complex processing procedures, large amounts of data, and difficulty in locating root causes, which seriously affect the improvement of product yield. Therefore, the embodiment of the present invention adopts a large model-based technology, and integrates the underlying multi-yield multi-modal large model, various business analysis models, and related data through intelligent agent calls to achieve intelligent control of the entire yield process.

[0063] In the detailed steps of the embodiment below, the method flow will be described in detail by taking three application scenarios, namely, "querying historical yield data and generating reports", "locating defect root causes and parameter control", and "daily prediction and early warning of yield data".

[0064] S10: Obtain various types of multimodal data related to product yield control to construct a multimodal yield data set, and the yield data set can be divided into a training set and a validation set.

[0065] In the embodiment of the present invention, for the processing and manufacturing industry of new display materials, examples of various types of multimodal data related to product yield control are obtained, see Table 1.

[0066] Table 1 shows the data needed in the training process of the yield multimodal large model

[0067] S20: Fine-tune and train the multimodal fusion large model based on the multimodal yield dataset to build a yield multimodal large model.

[0068] Specifically, based on the collected yield data set, the open source multimodal fusion model is locally fine-tuned and trained, and deployed to the local server to form a factory-privatized yield multimodal model.

[0069] Specifically, the process of model fine-tuning training is as follows: S201: Data preprocessing: perform preprocessing such as word segmentation and stop word removal on the collected text data, perform preprocessing such as size unification and normalization on the image data, standardize the numerical parameters, and construct training samples for text and image alignment.

[0070] S202: Model fine-tuning: Use fine-tuning technology to reduce memory consumption through quantization and efficient parameter fine-tuning. Specifically, use 8-bit quantization and double-precision adapters to significantly reduce video memory usage while maintaining performance, only fine-tune a small number of parameters, freeze the remaining parameters, and use a small learning rate (such as 1e-5~1e-6) for fine-tuning.

[0071] S203: Model training: A phased fine-tuning strategy is adopted, first fine-tuning the text and image encoders separately, and then fine-tuning them jointly. In implementation, it is recommended to use a warm-up strategy, learning to first increase and then decrease. At the same time, a progressive unfreezing strategy is adopted to gradually unfreeze the model parameters from shallow to deep layers, and gradient clipping is used to prevent gradient explosion.

[0072] S204: Loss function design for model training: For image understanding loss, contrastive learning loss is used. For cross-modal alignment loss during joint fine-tuning of text and image, the distance between text and image features is minimized. In addition, you can also choose according to business tasks, such as classification loss, regression loss, etc. In the loss function design, a multi-task learning framework is used to dynamically balance the weights of each loss item.

[0073] S205: Model training effect evaluation and monitoring: Regularly evaluate the model performance on the validation set, monitor the trend of each loss item until the model converges, check the gradient stability, and dynamically adjust the learning rate according to the validation set performance; It should be emphasized that model fine-tuning is an iterative process. According to the training effect of the model, the training strategy, loss function design, training loss function design, etc. need to be continuously adjusted (especially according to its performance in specific business scenarios) to achieve the best training effect.

[0074] S30: Integrate various business analysis models and yield multimodal models by using intelligent agent calls to form an integrated yield control system, and connect various information systems to the integrated yield control system. Please refer to Figure 4 , Figure 4 It is a deployment diagram of the yield control system.

[0075] Specifically, we build a yield control system based on a yield multimodal model based on new display industry data. First, we build an intelligent agent of the large model based on LangGraph, and then use various business analysis models as standard tools of the intelligent agent, so as to realize the call of various business analysis models by the intelligent agent.

[0076] Specifically, the process of integrating various business analysis models by using intelligent agent calls is as follows: S301: Construct the basic framework of the intelligent agent: define the intelligent agent base class and the intelligent agent workflow diagram, set the calling relationship between nodes and data flow rules, etc.

[0077] S302: Encapsulate various business analysis models into standard tool interfaces of intelligent entities: namely, yield data retrieval and analysis tools, yield prediction and early warning tools, defect root cause analysis tools, process parameter control tools, etc.

[0078] S303: Construct the workflow of the intelligent body: receive the user's yield analysis input, parse the user's natural language request, plan the required analysis tools and calling sequence according to the user's request, call the corresponding analysis tools according to the planned sequence, integrate the yield analysis results of each tool (each business analysis model), convert the integrated results into natural language response, and output the yield analysis report.

[0079] S304: Set up the agent's error handling and fallback mechanism: set up a timeout mechanism for tool calls, configure a retry strategy for failed calls, implement result verification and exception handling, and support manual intervention and manual fallback.

[0080] S305: System deployment and interface design: In order to improve the flexibility of deployment and later adjustments, the entire yield control system adopts a B / S architecture. Therefore, the intelligent agent is deployed as a Web service, and a standard API interface is designed to implement business user authentication and access control, and asynchronous calls and result push are supported to improve user experience and system maintainability.

[0081] Specifically, for the new display industry, the system applications corresponding to the business analysis model include but are not limited to: yield data retrieval and analysis system, yield data prediction system, yield prediction and early warning system, defect root cause analysis system, yield data process parameter control system, etc. The business analysis system here can be flexibly selected according to the business needs and data characteristics of each manufacturer, and can be continuously upgraded and expanded in the future.

[0082] For example: the yield data retrieval and analysis system includes but is not limited to the following functions: yield data retrieval, generation of yield analysis reports, summary of various yield analysis charts to generate summary reports, etc. The technical means to achieve the functions of the yield data retrieval and analysis system include but are not limited to: based on Text-to-SQL, generating database query statements for user input, statistical analysis based on statistical process control systems, generation of various statistical charts based on drawing plug-ins, and summary of charts based on chart understanding, etc.

[0083] For example: the yield data prediction and early warning system includes but is not limited to the following functions: based on historical yield data, yield prediction, yield early warning, yield fluctuation monitoring, etc. are performed for the current product processing parameters; the technical means to realize the functions of the yield data prediction and early warning system include but are not limited to: regression prediction based on deep neural networks, unsupervised anomaly detection, and time series data anomaly analysis.

[0084] For example: The defect root cause analysis system includes but is not limited to the following functions: based on various types of defects, the sites, machines (including equipment, sub-equipment, chambers, etc.), processing parameters, processing personnel, materials, processing time, etc. that are most related to product defects are located, that is, the root causes of defects are determined; the technical means for realizing the functions of the defect root cause analysis system include but are not limited to: root cause analysis based on association rule mining, root cause analysis based on decision trees, root cause analysis based on clustering analysis, root cause analysis based on neural networks, root cause analysis based on complex networks, root cause analysis based on maps, and root cause analysis based on statistical tests.

[0085] For example: the yield data process parameter control system includes but is not limited to the following functions: based on the defect root cause parameters determined by the defect root cause analysis system or the abnormally high parameters monitored by the yield prediction and early warning system, combined with the empirical values ​​of the value range required by the process and the historical control records, calculate the optimal process parameters. The technical means to realize the functions of the yield data process parameter control system include but are not limited to: parameter optimization based on genetic algorithms, parameter optimization based on Bayesian networks, parameter optimization based on neural networks, parameter optimization based on reinforcement learning, etc.

[0086] Specifically, after completing the functional test of the yield control system, the yield control system is deployed in the factory production environment, and the multimodal database, AOI defect detection system, MES system, SPC system, EDA system, etc. are integrated to form an integrated yield control system, such as Figure 3 shown.

[0087] S40: Analyze user interaction data through an integrated yield control system to obtain product yield control results.

[0088] Specifically, the yield control system can provide corresponding product yield control results based on the user's questions. The embodiment of the present invention takes three application scenarios as examples: "query of historical yield data and report generation", "location of defective root causes and parameter control", and "daily prediction and early warning of yield data" for detailed explanation.

[0089] Scenario 1: Generate relevant reports based on the query of yield data.

[0090] User input: "Please query the yield data of a certain model of OLED display from June 3, 2024 to June 15, 2024, and generate a yield analysis report; the analysis report includes a line graph of the daily yield and a proportional distribution graph of different defects in this production batch."

[0091] Specifically, after receiving the query demand input from the user, the intelligent agent of the yield control system first converts the query demand into a database query statement through the Text-to-SQL technology (converting natural language queries into structured query language SQL). It should be noted that if the logic of the user's query demand is relatively simple, or the user's database table structure is relatively simple, Text-to-SQL can be skipped, and the intelligent agent can directly generate SQL query statements based on the user input, and the embodiment of the present invention does not limit this.

[0092] Then, based on the database query statement, relevant yield data is retrieved from the database, as shown in Table 2 and Table 3. Table 2 records the daily panel yield data, and Table 3 records the proportion of different defects in the panel.

[0093] Table 2 shows the yield data (sample data)

[0094] Table 3 Defect data (sample data)

[0095] Finally, the yield control system generates relevant yield analysis reports based on the chart generation function according to the user's chart generation requirements (i.e., "a line chart of daily yield and a proportional distribution chart of different defects in this production batch"), such as a line chart of the corresponding daily yield fluctuations, a pie chart of different defect proportions, etc.

[0096] Scenario 2: Perform root cause analysis based on product defect data, process parameters of production machines, processing parameters, etc., and provide suggestions for adjusting process parameters.

[0097] User input: "Please analyze the root cause of the Mura defect in a certain model of OLED display from June 3, 2024 to June 15, 2024. The root cause site, root cause equipment, and root cause parameters must be located. Adjustment suggestions should be given for the process parameters that affect the defect. Defect images of Mura defects that occurred in the Array process should be obtained."

[0098] Among them, Mura is manifested as uneven brightness or color in the display area of ​​the display. The array process mainly involves forming a TFT array on a glass substrate through a series of process steps.

[0099] Specifically, after the intelligent agent of the yield control system receives the query input from the user, it first parses the question, and then performs TextToSQL conversion and calls the database in sequence according to a process similar to that in Scenario 1. At the same time, the intelligent agent determines that the input defective picture is a "linear Mura" based on the yield multimodal large model. The intelligent agent then inputs the queried data and the defect judgment result into the root cause analysis model. The root cause analysis model gives the root cause analysis result based on the input yield data. The intelligent agent then inputs the root cause analysis result (i.e., the located root cause site, equipment and parameters, as shown in Table 4) into the process parameter integrated control cockpit, thereby obtaining parameter adjustment suggestions, and outputs them to the user through the API interface.

[0100] Table 4 Root cause analysis results

[0101] Specifically, the yield control system outputs the determined root cause results to the process parameter integrated control cockpit, and the final effect is shown in Table 5.

[0102] Table 5 Output results of yield control system

[0103] Scenario 3: Yield prediction, detection and early warning based on daily production data.

[0104] User input: "Please predict the yield of a certain model of OLED display after the Array process before June 20, 2024, based on the yield data of the OLED display during the Array process from June 14, 2024 to June 16, 2024, and perform yield parameter fluctuation monitoring to point out abnormal parameters in the process, and specify the site where the parameters are located."

[0105] Note that the user's query input only mentions "yield prediction based on yield data" in general terms, without specifying the specific data type required. The yield management system will automatically select the data type related to the business based on the large amount of yield-related business knowledge learned and the optimization of the prompts of the large model, and perform yield prediction, height fluctuation monitoring, abnormal parameter analysis, etc.

[0106] For example, after receiving the user's query input, the intelligent agent of the yield control system first automatically determines the "processing parameters monitored by the Array process sensor, the monitoring data of various defects in the AOI link in the Array process, and the measurement values ​​of the measuring machine in the Array process" according to the user input, based on historical knowledge and prompt optimization, and for the user's unclear semantics. Then, it directly generates an SQL query statement, calls relevant data, and obtains data examples, as shown in Tables 6, 7, and 8.

[0107] Table 6 shows the data necessary for yield prediction 1-AOI defect monitoring data (example)

[0108] Table 7 shows the data required for yield prediction 2-processing parameters monitored by sensors (example)

[0109] Table 8 is the data required for yield prediction 3-Measurement values ​​of film thickness parameters by the measuring machine (example)

[0110] Then, based on these data, the intelligent agent directly calls the yield prediction model to predict the yield situation in this stage. The output results of the model (yield prediction results under a specific process site) are shown in Table 9.

[0111] Table 9 shows the yield prediction results

[0112] At the same time, based on the time series processing parameters in the above data, the intelligent agent directly calls the yield warning model (such as the unsupervised anomaly detection model based on isolation forest), monitors the abnormal parameters, and gives the site of the abnormal parameters. The output of the yield warning model is shown in Table 10.

[0113] Table 10 shows the output results of the yield warning model (example)

[0114] Finally, based on the above output results, the intelligent agent of the yield control system summarizes the relevant results and forms an analysis conclusion: "For the data currently monitored and queried by the user, the yield prediction found that the yield of site 18 and site 20 of the Array process may be significantly lower than that of other sites. At the same time, the monitoring and early warning of parameters found that there were abnormal processing parameters in sites 4, 7, 11, and 19 (see the table for details), which should be given special attention."

[0115] Through the above process, it can be seen that the yield control system can automatically realize yield prediction, early warning, and monitoring of abnormal parameters based on the understanding of the task.

[0116] In summary, the present invention takes the yield multimodal large model as the basic foundation, and realizes flexible calling between the basic large model and various business analysis models through intelligent agents, thereby solving the problems existing in the existing product yield control.

[0117] In addition, in one embodiment, based on the same inventive concept as the above embodiment, the embodiment of the present invention provides a product yield control device based on a large model, the device corresponds one-to-one with the method of embodiment 1, and the device includes: A multimodal data acquisition unit, which is used to acquire various types of multimodal data related to product yield control to construct a multimodal yield data set; A large model fine-tuning training unit, wherein the large model fine-tuning training unit fine-tunes and trains a multimodal fusion large model based on a multimodal yield data set to construct a yield multimodal large model; A business model integration unit, which integrates various business analysis models and yield multimodal large models by intelligent agent calling to form an integrated yield control system, and connects various information systems to the integrated yield control system; The yield control and analysis unit analyzes the user interaction data through an integrated yield control system to obtain a yield analysis and control result.

[0118] It should be noted that each unit in the product yield control device in this embodiment corresponds one-to-one to each step in the product yield control method in the aforementioned embodiment. Therefore, the specific implementation method and technical effects achieved by this embodiment can refer to the implementation method of the aforementioned product yield control method based on a large model, which will not be repeated here.

[0119] In addition, in one embodiment, the present application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the method in the aforementioned embodiment when executed by the processor.

[0120] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored, and the computer program implements the method in the aforementioned embodiment when executed by a processor.

[0121] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0122] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0123] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0124] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0125] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0126] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0128] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A product yield control method based on a large model, characterized in that: The method comprises the following steps: Obtain various multimodal data related to product yield control to build a multimodal yield data set; Fine-tune and train the multimodal fusion model based on the multimodal yield data set to build a yield multimodal large model; Integrate various business analysis models and yield multimodal large models by using intelligent agent calls to form an integrated yield control system, and connect various information systems to the integrated yield control system; User interaction data is analyzed through an integrated yield control system to obtain yield analysis and control results.

2. The product yield control method based on a large model according to claim 1, characterized in that: The various types of multimodal data related to the product yield control include data in document form, data in image form, data in numerical parameter form and data in network diagram form; wherein the data in document form includes process data, yield analysis reports and relevant research results of yield control; the data in image form includes defect images; the data in numerical parameter form includes production and processing data; the data in network diagram form includes production history data.

3. The product yield control method based on a large model according to claim 1, characterized in that: The method fine-tunes and trains an open-source multimodal fusion large model based on multimodal yield data, and privately deploys it to a local server to obtain a yield multimodal large model.

4. The product yield control method based on a large model according to claim 1, characterized in that: The business analysis model includes a multimodal data retrieval model, a virtual measurement model, a yield prediction model and a root cause analysis model.

5. The product yield control method based on a large model according to claim 1, characterized in that: The agent is used for task planning, task scheduling and user interaction.

6. The product yield control method based on a large model according to claim 1, characterized in that: The information system includes a machine vision and defect detection system, a statistical process control system, a manufacturing execution system, a quality management system, a database, a server and a storage system.

7. The product yield control method based on a large model according to claim 1, characterized in that: The process of analyzing user interaction data through an integrated yield control system to obtain yield analysis and control results is as follows: Receive multimodal user interaction data through the intelligent agent of the yield control system; The yield control system’s intelligent agent calls the yield multimodal big model, and uses the yield multimodal big model to parse multimodal user interaction data to obtain a unified embedded representation. Based on the unified embedded representation, the intelligent agent of the yield control system performs task planning and task scheduling to call various business analysis models, and performs yield analysis through the business analysis model to obtain the yield analysis results; The yield multimodal big model is called by the intelligent body of the yield control system, and the yield analysis results are uniformly processed by the yield multimodal big model to obtain the yield analysis control results; The yield analysis and control results are output through the intelligent agent of the yield control system.

8. A product yield control device based on a large model, characterized in that: The device comprises: A multimodal data acquisition unit, which is used to acquire various types of multimodal data related to product yield control to construct a multimodal yield data set; A large model fine-tuning training unit, wherein the large model fine-tuning training unit fine-tunes and trains a multimodal fusion large model based on a multimodal yield data set to construct a yield multimodal large model; A business model integration unit, which integrates various business analysis models and yield multimodal large models by intelligent agent calling to form an integrated yield control system, and connects various information systems to the integrated yield control system; The yield control and analysis unit analyzes the user interaction data through an integrated yield control system to obtain a yield analysis and control result.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the product yield control method based on a large model described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for controlling product yield based on a large model described in any one of claims 1 to 7 is implemented.

Images