3D printing process defect monitoring method and system based on multi-modal large model

Abstract

The invention discloses a 3D printing process defect monitoring method and system based on a multi-modal large model. The method comprises the steps that S1, before related operation is carried out, corresponding preposition work needs to be completed, wherein the preposition work comprises knowledge graph creation, retrieval enhancement generation, reasoning and action framework and fine adjustment of the large model; s2, collecting and preprocessing industrial camera cluster data; s3, according to the preprocessed image, segmenting the image, constructing a region adjacency graph, finding an optimal region combination by applying a random algorithm, and combining some regions to obtain image representation; s4, calling the multi-mode large model subjected to pre-training and specific field fine adjustment to realize real-time defect monitoring in the 3D printing process; s5, a detection report is given according to a defect detection result, user interaction intelligent consultation is opened, and the model is updated and adjusted according to printing data; according to the method, the problems of standard, efficiency, trust and supervision of defect monitoring in the 3D printing process can be effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of 3D printing defect detection, which belongs to the field of 3D printing technology, and more particularly to a 3D printing process defect monitoring method and system based on a multimodal large model. The system integrates multi-source data, fully utilizes contextual information of the printing process, and has good scenario adaptability. Background Art

[0002] Currently, the following problems still exist in 3D printing defect detection for binder jet forming:

[0003] (1) Failure to combine multiple sensor data from different sources (such as text, images, videos, audio, etc.) for training;

[0004] (2) It is impossible to perform zero-shot learning like a large multimodal model;

[0005] (2) Failure to utilize rich contextual information during the printing process, such as the physical properties of the printing material, G-code files, and dynamic changes during the printing process;

[0006] (3) In the field of 3D printing detection, detection methods based on multimodal large model technology have not yet been popularized, and existing deep learning models often need to be retrained or fine-tuned on new data sets to adapt to new application scenarios, which limits the widespread application of scientific and effective detection technologies.

[0007] (4) Hallucination fabrication and outdated knowledge are inherent defects of large models, which have a serious impact on the output of results in 3D printing with high precision requirements.

[0008] Therefore, there is an urgent need for a method to monitor defects in real time throughout the entire 3D printing process based on multimodal large models. Summary of the Invention

[0009] The technical problem to be solved by this invention is to realize real-time automatic identification and monitoring of 3D printing defects in the binder jet forming method. In addition to integrating multi-source data, it can also make full use of the contextual information of the printing process and has good scene adaptability.

[0010] Currently, defect detection methods for binder jet forming 3D printing have the following problems:

[0011] (1) Defect detection methods for binder jet forming 3D printing cannot achieve the integration of multiple sensor data for training. Data from different sources (such as text, images, video, and audio) have different feature representations and data structures, resulting in limited effectiveness of the model in integrating multi-sensor data, limiting its application potential in binder jet forming 3D printing.

[0012] The complexity of this information makes it extremely challenging to leverage the rich contextual information from the printing process, such as the physical properties of the printing material, GCode files, and dynamic changes, for defect detection. The lack of effective methods for effectively incorporating this complex contextual information into model training and inference has hindered the further development of defect detection methods for binder jet forming 3D printing.

[0013] (2) Retraining or fine-tuning deep learning models to adapt to new application scenarios is an expensive and tedious process. In particular, deep learning models cannot adapt to and generalize new application scenarios when faced with different printing and process parameters. Retraining requires a large number of new annotated datasets and consumes a lot of time and computing resources. When processing large datasets, it is impossible to identify models and respond quickly. Deep learning models often lack transparency in their decision-making process.

[0014] In order to solve the above technical problems, the present invention provides a 3D printing process defect monitoring method and system based on a multimodal large language model.

[0015] According to one aspect of the present invention, a method for monitoring defects in a 3D printing process based on a multimodal large language model is provided, which specifically includes the following steps:

[0016] S1: Before performing related operations, the corresponding preparatory work needs to be completed: creating a knowledge graph, generating search enhancements, reasoning and action frameworks, and fine-tuning the large model;

[0017] S2: Industrial camera cluster data acquisition and preprocessing;

[0018] S3: Based on the preprocessed image, the image is segmented, a region adjacency graph is constructed, a random algorithm is applied to find the optimal region combination, and some regions are merged to obtain the image representation;

[0019] S4: Invoke a multimodal large model that has been pre-trained and fine-tuned for a specific domain to achieve real-time defect detection in the 3D printing process;

[0020] S5: Issue a test report based on the defect detection results, provide user interactive intelligent consultation, and update and adjust the model based on the printing data;

[0021] Furthermore, according to the above aspects and any possible implementation, an implementation is further provided, wherein step S1 specifically includes:

[0022] S11 Create a knowledge graph

[0023] Knowledge modeling: This study uses ontology and semantic web technology to perform structured modeling on knowledge collected from multiple channels and construct a knowledge graph with rich semantic information.

[0024] Integration and verification: Combine professional knowledge such as 3D printing defect types and their causes with existing knowledge graphs and conduct strict verification to ensure the accuracy and completeness of knowledge.

[0025] S12 builds a retrieval-enhanced generation model

[0026] Combined with knowledge graph: Tightly combine knowledge graph with retrieval enhancement generation technology and multimodal large models, and use its structured data as an external knowledge source to improve the retrieval ability of the model.

[0027] Design retrieval module: This study designed an efficient retrieval module that can quickly retrieve contextual information related to defects from the knowledge graph, such as defect type, cause, solution, etc.

[0028] Generation module optimization: The retrieved information is fed into the generation module to produce more accurate and contextually appropriate defect detection results and improvement suggestions.

[0029] S13 adds reasoning and action

[0030] Integrated reasoning and action framework: Integrating the reasoning and action framework into the system gives large models the ability to handle complex tasks through reasoning and action interaction.

[0031] Multi-step reasoning: During the defect detection process, the reasoning and action framework guides the model to perform multi-step reasoning, integrating information retrieved from the knowledge graph and retrieval enhancement generation technology to gradually analyze the cause of the defect and the solution strategy.

[0032] Dynamic Generation: Through the reasoning and action framework, the model can dynamically generate reasoning steps and action plans, such as adjusting 3D printing parameters or triggering equipment shutdown operations based on defect characteristics.

[0033] S14 fine-tuning large model

[0034] Fine-tuning for specific scenarios: Based on the output of knowledge graphs, retrieval enhancement generation, and reasoning and action frameworks, large models are fine-tuned for scenario adaptability to improve their performance in practical applications.

[0035] Training with professional datasets: The model is deeply trained using professional datasets from the 3D printing industry (including images, sensor data, process parameters, etc.) to ensure that the model can accurately identify and classify various types of printing defects.

[0036] Real-time feedback optimization: During the fine-tuning process, a real-time feedback mechanism is introduced to continuously adjust and optimize the model based on actual test results and user feedback to ensure continuous performance improvement.

[0037] Furthermore, according to the above aspects and any possible implementation, an implementation is further provided, wherein step S2 specifically includes:

[0038] S21 Industrial Camera Cluster Deployment and Data Collection: Multiple industrial cameras are deployed at key locations on the 3D printer to form a comprehensive monitoring network. The selection of these cameras should take into account parameters such as resolution, frame rate, and field of view to ensure that every detail of the printing process can be captured. Hardware or software synchronization ensures that all cameras capture images at the same time to avoid image misalignment caused by time differences. Image data is continuously collected during the printing process, forming a dataset that includes multiple print batches and various printing materials.

[0039] Initial processing of the S22 dataset: Remove blurry, missing, or invalid images due to camera failure, insufficient lighting, or obstruction by printed materials. Calibrate images from each camera to eliminate lens distortion and image warping, ensuring spatial consistency across all images. Timestamp each image and sort all images by timestamp in the order they were printed.

[0040] S23 Data Augmentation and Annotation: Data augmentation methods such as rotation, scaling, flipping, and noise addition are used to expand the image dataset and improve the model's generalization capabilities. Two cameras are used to capture the same print area from different angles. By comparing and fusing the two images, defect areas within the images are more accurately annotated. Annotations should include information such as defect location, size, type, and severity. Annotation results are manually reviewed to ensure accuracy and consistency.

[0041] The S24 uses UDP for data transmission: Considering real-time and low-latency requirements, UDP was chosen as the data transmission protocol. UDP enables faster data transmission and is suitable for scenarios requiring high real-time performance. UDP sending and receiving functions are implemented on the camera and server, respectively. The camera encapsulates captured image data into UDP packets and sends them. The server receives and parses these packets, saving the image data locally or processing it further. Data transmission status and speed are monitored in real time to ensure stable and fast data transmission to the server. Any anomalies during transmission are also recorded for troubleshooting and repair.

[0042] Furthermore, according to the above aspects and any possible implementation, an implementation is further provided, wherein step S3 specifically includes:

[0043] S31 image preprocessing: performing operations such as improving image quality, reducing noise, and standardizing image data;

[0044] S32 Initial image segmentation: Use an algorithm to segment the image into several superpixels, which are small areas with similar properties (such as color and texture);

[0045] S33 builds a region adjacency graph: creates nodes for the segmented regions and establishes relationships between each node;

[0046] S34 optimizes segmentation results: finds the optimal region combination by minimizing the energy function;

[0047] S35 post-processing: optimize the segmented boundaries and simplify the final result;

[0048] Furthermore, according to the above aspects and any possible implementation, an implementation is further provided, wherein step S4 specifically includes:

[0049] S41: Large Model Call Preparation and Integration Initialization: Load the pre-trained and fine-tuned multimodal large model and large language model from storage and deploy them into the system for integration, forming a complete defect detection system. Construct a suitable environment to ensure the model can run stably and receive multimodal data input from various sensors, including images, videos, and G-code files. Furthermore, configure an efficient and appropriate environment for the model to ensure the accuracy and precision of the real-time monitoring process.

[0050] S42 Data Preparation: Accurately convert collected multimodal data, such as images, videos, G-code files, and sliced models. For images or videos, advanced preprocessing steps are used to extract key feature vectors to highlight important information in the data.

[0051] S43: During the large-scale model reasoning process, we employ a method of decomposing intermediate stages to handle tasks and introduce intermediate reasoning steps to achieve powerful complex reasoning capabilities. This process relies on specific prompting techniques to accurately identify the necessary steps in the reasoning process, thereby constructing an intermediate, coherent chain of reasoning. Based on the actual status of the current 3D printing task, we construct a query containing contextual information and the data fragments to be analyzed, providing comprehensive and accurate input for the model. Furthermore, during the model reasoning process, we utilize prompting techniques to identify the intermediate reasoning steps necessary to complete defect determination. After executing the inference, the model returns the predicted results. The receiving process is broken down into result reception, format verification, and content parsing. The result data returned by the model is first received, checked for integrity and formatting, and the result content is parsed. The numerical and symbolic information output by the model is converted into information regarding defect determination, defect type, location, and severity. This information is then categorized and organized to provide a clear basis for subsequent decision-making.

[0052] S44 Integrated Defect Detection: Integrate the defect detection results obtained by calling the large model with the reasoning steps and action plans generated by the context information retrieved by the knowledge graph and retrieval enhancement generation technology to form a complete 3D printing process defect monitoring report.

[0053] S45 displays results and provides suggestions: The monitoring results are displayed in real time through the system interface, providing users with intuitive defect information and improvement suggestions.

[0054] S46 establishes a real-time feedback loop: During the printing process, new multimodal data is continuously collected, including real-time images and sensor data from the printing process. This collected data is transmitted to the system to ensure its timeliness and accuracy. This transmitted data is analyzed in real time, using reasoning steps and methods based on thought chaining technology to promptly identify potential problems. As data is continuously processed, the knowledge base update process is broken down into knowledge extraction, knowledge verification, and knowledge fusion using retrieval-enhanced generation technology. The extracted knowledge is verified through multiple experiments and comparative analysis to ensure its reliability. Finally, the verified new knowledge is integrated into the existing knowledge graph and other auxiliary tools to update the system's knowledge base, enhancing the system's adaptability to different printing scenarios and the accuracy of defect detection. System parameters are adjusted and optimized based on the specific needs of the module to ensure compatibility with the existing system while improving the overall system performance and responsiveness.

[0055] According to another aspect of the present invention, a 3D printing process defect monitoring system based on a multimodal large language model is provided, specifically comprising:

[0056] 3D printing feature data acquisition module: used to collect 3D printing process data through multi-angle according to industrial-grade cameras;

[0057] Machine vision module based on a multimodal large language model: used to automatically identify defects generated during the printing process and locate them;

[0058] Analysis report module: Generates test reports based on defect detection results;

[0059] Compared with the prior art, one or more embodiments of the above solutions may have the following advantages or beneficial effects:

[0060] (1) The present invention can effectively solve the problems of standards, efficiency, trust, and supervision in defect monitoring during 3D printing.

[0061] (2) The present invention can support real-time feedback of system monitoring during the 3D printing process.

[0062] (3) The present invention can realize efficient and accurate collection, monitoring and recording of the entire process of 3D printing.

[0063] (4) The present invention can realize accurate and efficient monitoring of defects in the 3D printing process by combining photos and three-point cloud data sets.

[0064] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purposes and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the summary and embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0066] Figure 1 Shown is a flow chart of the present invention; DETAILED DESCRIPTION

[0067] The following describes the embodiments of the present invention in detail with reference to the accompanying drawings and examples, so that the present invention can fully understand how to apply technical means to solve technical problems and achieve technical effects, and thus implement the invention accordingly. It should be noted that, as long as no conflict exists, the various embodiments of the present invention and the various features of the embodiments can be combined with each other, and the resulting technical solutions are all within the scope of protection of the present invention.

[0068] The technical solution adopted by the present invention is a 3D printing process defect monitoring method and system based on a multimodal large model. The method includes:

[0069] S1: Before performing related operations, the corresponding preparatory work needs to be completed: creating a knowledge graph, search enhancement generation, reasoning and action framework, and fine-tuning the large model

[0070] S11 Knowledge Graph Creation

[0071] Data Collection: This study comprehensively collected knowledge in the field of 3D printing from multiple perspectives, including academic literature, industry expert knowledge bases, and historical data. This diverse data source helped to build a comprehensive and in-depth knowledge graph.

[0072] Ontology Design: By clearly defining entities within the domain, organizing relationships, and categorizing attributes, this study constructed a framework for the 3D printing knowledge graph. This ontology design provides a solid structural foundation for subsequent applications of the knowledge graph.

[0073] Knowledge fusion: In order to ensure the consistency and accuracy of the knowledge graph, this study adopted a comprehensive knowledge fusion strategy to integrate knowledge from different sources and achieve harmonious coexistence between data through algorithm optimization.

[0074] Verification tools: With the help of automated verification tools, this study conducted a comprehensive quality check on the constructed knowledge graph to ensure the completeness and accuracy of the graph.

[0075] S12 search enhancement generation process construction

[0076] Model selection: In the selection of dialogue system model architecture, this study adopted a multimodal large model, with its powerful knowledge processing capabilities as the research basis.

[0077] Knowledge injection: This study injects data from the knowledge graph into the 3D domain knowledge base as an external knowledge source for the model to enhance its dialogue generation capability and accuracy.

[0078] Retrieval algorithm optimization: In response to the requirements of retrieval speed and accuracy, this study optimized the retrieval algorithm to ensure the efficiency of the retrieval enhancement generation technology model in the knowledge retrieval process.

[0079] Improvement of the generation module: By improving the generation module, this study improves the accuracy and practicality of the content generated by the retrieval-enhanced generation technology model, thereby better meeting the needs of users.

[0080] S13 Reasoning and Action Framework Integration

[0081] Framework configuration: In order to achieve seamless integration with existing systems, this study carefully configured the reasoning and action framework to ensure that its application in the 3D printing field can be smooth and efficient.

[0082] Reasoning logic design: This study designed multi-step reasoning logic to support the model's step-by-step reasoning in the process of complex problem solving, thereby improving the accuracy of problem solving.

[0083] Action Planning: This study provides the model with the ability to dynamically generate reasoning steps and action plans, enabling it to flexibly respond to decision-making needs in different scenarios.

[0084] S14 fine-tuning of multimodal large models

[0085] Data preparation: We carefully selected and integrated high-resolution images to construct a comprehensive and targeted dataset, providing a solid foundation for subsequent model fine-tuning.

[0086] Model training: This study adopts a phased training strategy, first fine-tuning the multimodal large model, and then deeply fine-tuning it using a proprietary dataset for the 3D printing field. This ensures that the model can deeply understand the complex characteristics unique to the industry, thereby improving its recognition accuracy and classification capabilities in the 3D printing field.

[0087] Feedback loop: A dynamic feedback loop mechanism is established, which not only collects user feedback in real time, but also automatically records system operation data. Through this feedback data, the model can continuously learn and adjust, gradually improve its performance and user experience, and achieve self-optimization and upgrade of the model.

[0088] S2: Industrial camera cluster data acquisition and preprocessing;

[0089] S21 Industrial Camera Cluster Deployment and Data Collection: Multiple industrial cameras are deployed at key locations on the 3D printer to form a comprehensive monitoring network. The selection of these cameras should take into account parameters such as resolution, frame rate, and field of view to ensure that every detail of the printing process can be captured. Hardware or software synchronization ensures that all cameras capture images at the same time to avoid image misalignment caused by time differences. Image data is continuously collected during the printing process, forming a dataset that includes multiple print batches and various printing materials.

[0090] Initial processing of the S22 dataset: Remove blurry, missing, or invalid images due to camera failure, insufficient lighting, or obstruction by printed materials. Calibrate images from each camera to eliminate lens distortion and image warping, ensuring spatial consistency across all images. Timestamp each image and sort all images by timestamp in the order they were printed.

[0091] S23 Data Augmentation and Annotation: Data augmentation methods such as rotation, scaling, flipping, and noise addition are used to expand the image dataset and improve the model's generalization capabilities. Two cameras are used to capture the same print area from different angles. By comparing and fusing the two images, defect areas within the images are more accurately annotated. Annotations should include information such as defect location, size, type, and severity. Annotation results are manually reviewed to ensure accuracy and consistency.

[0092] The S24 uses UDP for data transmission: Considering real-time and low-latency requirements, UDP was chosen as the data transmission protocol. UDP enables faster data transmission and is suitable for scenarios requiring high real-time performance. UDP sending and receiving functions are implemented on the camera and server, respectively. The camera encapsulates captured image data into UDP packets and sends them. The server receives and parses these packets, saving the image data locally or processing it further. Data transmission status and speed are monitored in real time to ensure stable and fast data transmission to the server. Any anomalies during transmission are also recorded for troubleshooting and repair.

[0093] S3: Based on the preprocessed image, the image is segmented, a region adjacency graph is constructed, a random algorithm is applied to find the optimal region combination, and some regions are merged to obtain the image representation.

[0094] S31 image preprocessing: performing operations such as improving image quality, removing noise from images, and standardizing image data;

[0095] S32 Initial image segmentation: Use an algorithm to segment the image into several superpixels, which are small areas with similar properties (such as color and texture);

[0096] S33 builds a region adjacency graph: creates nodes for each segmented region and establishes relationships between each node;

[0097] S34 optimizes segmentation results: finds the optimal region combination by minimizing the energy function;

[0098] S35 post-processing: optimize the segmented boundaries and simplify the final result;

[0099] S4: Invoke a multimodal large model that has been pre-trained and fine-tuned for a specific domain to achieve real-time defect detection in the 3D printing process.

[0100] S41: Large Model Call Preparation and Integration Initialization: Load the pre-trained and fine-tuned large multimodal model from storage and deploy it into the system for integration, forming a complete defect detection system. Construct a suitable environment to ensure the model can run stably and receive multimodal data input from various sensors, including images, videos, G-code files, and sliced models. Furthermore, configure an efficient and appropriate environment for the model to ensure the accuracy and precision of the real-time monitoring process.

[0101] S42 Data Preparation: Accurately convert collected multimodal data, such as images, videos, and G-Code files. For images or videos, advanced preprocessing steps are used to extract key feature vectors to highlight important information in the data.

[0102] S5: Issue a test report based on the defect detection results, provide user interactive intelligent consultation, and update and adjust the model based on the printing data;

[0103] S51 issues a test report: Based on the test results of multimodal large model reasoning, a defect detection report is issued, including problem location, problem cause analysis, and problem improvement measures;

[0104] S52 Open Interactive Consultation: It provides a natural language interactive interface to answer user questions, conducts inference based on test data and knowledge graphs, and provides feedback on defect types and causes to users, helping them understand problems encountered during printing.

[0105] S53 optimizes and adjusts the model: adjusts the model in the platform according to the test data results; and at the same time organizes and learns the new data obtained in S3, feeds back the defect information detected during the printing process into the knowledge graph, enriches the content of the knowledge graph, and updates the model in the platform.

[0106] The step S31 specifically includes:

[0107] For images, contrast adjustment is performed through adaptive histogram equalization, image sharpening is performed through high-pass filtering, and noise in images is removed using linear filtering and nonlinear filtering;

[0108] The step S32 specifically includes:

[0109] Generate superpixels using a suitable superpixel generation algorithm, perform initial segmentation on the preprocessed image, and display the segmentation results;

[0110] The step S33 specifically includes:

[0111] Build a graph structure representing superpixels and their adjacent relationships, create a node for each superpixel, create an edge for adjacent superpixels in the image, and define the edge weight based on the difference between adjacent regions;

[0112] The step S34 specifically includes:

[0113] Combine knowledge graph and retrieval enhancement generation technology to optimize segmentation results, and use functions to gradually merge nodes to minimize the energy function;

[0114] The step S35 specifically includes:

[0115] The segmented boundaries are smoothed, and the similarity of adjacent regions is evaluated through knowledge graph and retrieval enhancement generation technology. Regions with high similarity are merged according to a certain threshold, and boundary information is obtained by calculating the gradient map for boundary refinement.

[0116] The step S42 specifically includes:

[0117] S43: During the large-scale model reasoning process, we use a method to decompose the intermediate stages of the task and introduce intermediate reasoning steps to achieve powerful complex reasoning capabilities. This process relies on specific prompting techniques to accurately identify the necessary steps in the reasoning process, thereby constructing an intermediate, coherent reasoning chain. Step S43 specifically includes:

[0118] Based on the actual status of the current 3D printing task, a query containing contextual information and the data fragments to be analyzed is constructed to provide comprehensive and accurate input for the model. At the same time, during the model reasoning process, hint technology is used to identify the various intermediate reasoning steps necessary to complete defect judgment.

[0119] After the model performs inference, it returns a prediction result. The receiving process is broken down into result reception, format verification, and content parsing. First, the result data returned by the model is received, checked for integrity and format, and the result content is parsed. The numerical and symbolic information output by the model is converted into information about defect determination, defect type, location, and severity. This information is then categorized and organized to provide a clear basis for subsequent decision-making.

[0120] S44 Integrated Defect Detection: Integrates the defect detection results obtained by calling the large model with the knowledge graph, contextual information retrieved by the retrieval enhancement generation technology, and the reasoning steps and action plans generated after calling the large model tool to form a complete 3D printing process defect monitoring report.

[0121] S45 displays results and provides suggestions: The monitoring results are displayed in real time through the system interface, providing users with intuitive defect information and improvement suggestions.

[0122] The step S45 specifically includes:

[0123] S45 establishes a real-time feedback loop: During the printing process, new multimodal data is continuously collected, including real-time images of the printing process, sensor data, etc. The collected data is transmitted to the system to ensure the timeliness and accuracy of the data. The transmitted data is analyzed in real time, and potential problems are promptly identified using the same reasoning steps and methods as before;

[0124] As data is continuously processed, the knowledge base update process is broken down into knowledge extraction, knowledge verification, and knowledge fusion using retrieval-enhanced generation technology. The extracted knowledge is verified through multiple experiments and comparative analysis to ensure its reliability. Finally, the verified new knowledge is integrated into the existing knowledge graph and other auxiliary tools to update the system's knowledge base, enhancing its adaptability to different printing scenarios and the accuracy of defect detection.

[0125] The step S51 specifically includes:

[0126] Design a defect detection report template and use the text generation capability of the big model to fill the detected defect data such as defect type, location, size, etc. into the pre-designed report template;

[0127] The step S52 specifically includes:

[0128] Provides interactive methods such as asking questions based on image recognition results, clicking on preset questions, asking questions using natural language voice, and asking questions using natural language input. Answers are provided through the reasoning and text generation functions of large models to optimize the user experience.

[0129] The step S53 specifically includes:

[0130] The new printing data obtained in step S3 is cleaned and converted. The processed data is integrated with the test results and normalized. The detected defect information, including defect type, characteristics, location, etc., is updated to the knowledge graph to enrich the content of the knowledge graph so that it can more comprehensively reflect various problems and knowledge in the printing process.

[0131] Real-time defect monitoring in 3D printing processes based on multimodal large models overcomes the following limitations of traditional methods that rely solely on deep learning models:

[0132] 1) Failure to combine data from different sources (such as text, images, video, audio, etc.) for training. In 3D printing defect detection, this means fusing data from multiple sensors (such as cameras, sound sensors, temperature sensors, etc.). Compared to deep learning models, multimodal large models can leverage the complementary information in these different data sources to provide more comprehensive and accurate defect detection.

[0133] 2) Because the multimodal large model can process multiple types of data, it can capture richer contextual information during the printing process than deep learning models, such as the physical properties of the printing material, the conditions of the printing environment (such as temperature and humidity), G Code files, dynamic changes during the printing process, and the relationship between different areas of the image.

[0134] 3) Faced with different application scenarios, traditional deep learning models need to be retrained or transferred on a large number of new data sets. In addition, there is a natural connection between natural language and vision. Most problems require joint modeling of the two streams to produce satisfactory results. In contrast, the design of the multimodal large model enables it to easily adapt to new data sources and new application scenarios with relatively small data sets, and it is easy to interact with the model or obtain feedback in natural language. At the same time, by constructing a knowledge graph, the multimodal large model can generate output based on the reasoning results, reduce the impact of hallucinations and outdated knowledge, and ensure the authenticity and effectiveness of the feedback obtained by interacting with the large model. Therefore, in general, the multimodal large model is of great significance to improving the stability of the 3D printing process, improving product quality, ensuring the yield of parts, and optimizing the user experience.

[0135] Lack of immediate feedback and dynamic adjustment capabilities: After detecting a defect, traditional deep learning models often require manual intervention or offline analysis after a period of time to determine the cause and take action. However, approaches that integrate intelligent analysis with timely response mechanisms can respond quickly upon defect detection through immediate cause analysis and action formulation, reducing the delay of manual intervention and improving the system's dynamic adjustment capabilities.

[0136] The present invention is specifically described by enumerating the following embodiments:

[0137] Example

[0138] In order to solve the problem of real-time automatic identification and monitoring of 3D printing defects, the present invention provides a defect detection method based on a multimodal large model.

[0139] The specific description of the embodiments of the present invention is as follows:

[0140] S1: Before performing related operations, you need to complete the corresponding preparatory work: creating a knowledge graph, search enhancement generation technology, reasoning and action framework, and fine-tuning the large model

[0141] S2: Industrial camera cluster data acquisition and preprocessing;

[0142] S3: Based on the preprocessed image, the image is segmented, a region adjacency graph is constructed, a random algorithm is applied to find the optimal region combination, and some regions are merged to obtain the image representation;

[0143] S4: Invoke a multimodal large model that has been pre-trained and fine-tuned for a specific domain to achieve real-time defect detection in the 3D printing process.

[0144] S5: Issue a test report based on the defect detection results, provide user interactive intelligent consultation, and update and adjust the model based on the printing data;

[0145] The step S1 specifically includes:

[0146] S11: Create a knowledge graph: Build a structured knowledge graph that includes core concepts, term definitions, defect types, and their causes in the 3D printing field. Model the collected knowledge using ontology and semantic web technologies to ensure the accuracy and completeness of the knowledge graph.

[0147] S12 Retrieval-enhanced generation: Retrieve relevant information from an external knowledge base and input it as prompts to a large language model to enhance the model's ability to handle knowledge-intensive tasks.

[0148] S13 Reasoning and Action Framework: Design a reasoning and action framework that can analyze the causes of defects and propose corresponding action suggestions based on the detected defect information and the knowledge in the knowledge graph.

[0149] S14 Fine-tuning large models: Using domain-specific fine-tuning techniques, the pre-trained multimodal large model is adjusted to make it more suitable for 3D printing defect monitoring tasks and improve the accuracy and efficiency of the model.

[0150] The step S2 specifically includes:

[0151] S21 Industrial Camera Cluster Deployment: Multiple industrial cameras are deployed around the 3D printing equipment to form a camera cluster to capture image data from different angles during the printing process.

[0152] S22 Data Acquisition: It collects image data captured by industrial cameras in real time and simultaneously collects environmental parameters such as temperature and humidity during the printing process to form a multimodal dataset.

[0153] S23 Data Preprocessing: Perform preprocessing operations such as denoising and contrast enhancement on the collected image data to improve data quality. At the same time, the environmental parameter data is normalized so that it can be combined with the image data.

[0154] The step S3 specifically includes:

[0155] S31 Image Segmentation: Use deep learning technology to segment the preprocessed image, separate different areas in the image, and provide a basis for subsequent area merging.

[0156] S32 constructs a region adjacency graph: constructs a region adjacency graph based on the image segmentation results, which reflects the spatial relationship between different regions.

[0157] S33 Region Merging: Apply a random algorithm to analyze the region adjacency graph, find the optimal region combination, and merge certain regions to obtain a more realistic image representation.

[0158] The step S4 specifically includes:

[0159] S41 Model loading and initialization: Load the pre-trained and domain-specific fine-tuned multimodal large model into the system and initialize it to ensure that the model is in the best working condition.

[0160] S42 Data input: Input the preprocessed image data and multimodal environmental parameter data into the multimodal large model to provide the model with necessary information for defect monitoring.

[0161] S43 Defect Monitoring: The multimodal large model performs real-time analysis based on input data, identifies defects in images, and monitors the development trend of defects.

[0162] S44 Result Output: The model outputs defect monitoring results, including defect type, location, severity and other information, providing a basis for subsequent defect handling.

[0163] S45 Real-time feedback: Monitoring results are fed back to users in real time, and the system records monitoring data to provide data support for continuous optimization of the model.

[0164] S5: Issue a test report based on the defect detection results, provide user interactive intelligent consultation, and update and adjust the model based on the printing data.

[0165] S51 issues a test report: Based on the test results of multimodal large model reasoning, a defect detection report is issued, including problem location, problem cause analysis, and problem improvement measures;

[0166] S52 Open Interactive Consultation: It provides a natural language interactive interface to answer user questions, conducts inference based on test data and knowledge graphs, and provides feedback on defect types and causes to users, helping them understand problems encountered during printing.

[0167] S53 optimizes and adjusts the model: adjusts the model in the platform according to the test data results; and at the same time organizes and learns the new data obtained in S3, feeds back the defect information detected during the printing process into the knowledge graph, enriches the content of the knowledge graph, and updates the model in the platform.

[0168] The step S51 specifically includes:

[0169] Design a defect inspection report template, including but not limited to:

[0170] Hardware environment: device model and configuration, sensor type and accuracy, etc.

[0171] Software environment: operating system version, detection software version, related dependent software version, etc.

[0172] Defect List: Presents all defects found in a table format, including but not limited to the following columns:

[0173] Defect number, defect title, module, defect description, severity, and discovery time.

[0174] Defect Cause: There are many reasons for this type of defect.

[0175] Improvement measures: including but not limited to adjusting printing parameters, optimizing shape design, adjusting temperature, etc.

[0176] Leverage the text generation capabilities of the large model to fill the detected defect data into pre-designed report templates, and use visualization tools such as bar charts and line charts to display defect detection distribution and statistical results;

[0177] The step S52 specifically includes:

[0178] Design a user-friendly interactive interface, integrate multiple interfaces, and provide interactive methods such as asking questions based on image recognition results, clicking preset questions, asking questions using natural language voice, and asking questions using natural language input. A series of frequently asked questions is designed and stored, and the preset questions are displayed in a categorized manner on the user interface. Users enter questions through the interactive interface, and the system answers them using the large model's reasoning and text generation capabilities, and the results are fed back to the interactive interface.

[0179] The step S53 specifically includes:

[0180] Data cleaning is performed using methods such as removing duplicate data, handling missing values, and correcting erroneous data. Numerical data is normalized and scaled to a fixed range. During model analysis, the impact of various parameters is compared and evaluated, and appropriate parameters are adjusted to update the model. Frequently occurring and accurately identified defect information is selected and integrated and updated based on the knowledge graph.

Claims

1. A 3D printing process defect monitoring method based on a multimodal large language model, characterized in that: The following steps are involved: S1: Preliminary work: creating knowledge graphs, retrieval enhancement generation, reasoning and action frameworks, and fine-tuning large models; S2: Industrial camera cluster data acquisition and preprocessing; S3: Based on the preprocessed image, the image is segmented, a region adjacency graph is constructed, a random algorithm is applied to find the optimal region combination, and some regions are merged to obtain the image representation; S4: Invoke a multimodal large model that has been pre-trained and fine-tuned for a specific domain to achieve real-time defect detection in the 3D printing process; S5: Issue a test report based on the defect detection results, provide user interactive intelligent consultation, and update and adjust the model based on the printing data.

2. A 3D printing process defect monitoring method based on a multimodal large language model according to claim 1, characterized in that: The step S1 specifically includes: S11 creates a knowledge graph; Knowledge modeling: Using ontology and semantic web technologies, we structure the knowledge collected from multiple channels and create a knowledge graph. Integration and verification: Integrate professional knowledge such as 3D printing defect types and their causes with existing knowledge graphs, and verify them to ensure the accuracy and completeness of the knowledge; S12 builds a retrieval enhancement generation model; Combining knowledge graphs: Tightly integrate knowledge graphs with retrieval-enhanced generation technology and multimodal large models, using their structured data as an external knowledge source to improve the model's retrieval capabilities; Design a retrieval module: Design a retrieval module that can quickly retrieve contextual information related to defects from the knowledge graph, including defect type, cause, and solution; Generation module optimization: The retrieved information is fed into the generation module to generate more accurate and context-appropriate defect detection results and improvement suggestions; S13 adds reasoning and action; Integrated reasoning and action framework: Integrating the reasoning and action framework into the system enables large models to handle complex tasks through the interaction of reasoning and action; Multi-step reasoning: During the defect detection process, the reasoning and action framework guides the model to perform multi-step reasoning, integrating information retrieved by the knowledge graph and retrieval enhancement generation technology to gradually analyze the cause of the defect and the solution strategy; Dynamic Generation: Through the reasoning and action framework, the model can dynamically generate reasoning steps and action plans, such as adjusting 3D printing parameters or triggering equipment shutdown operations based on defect characteristics; S14 fine-tuned the large model; Fine-tuning for specific scenarios: Fine-tuning large models for scenario adaptability based on the output of knowledge graphs, retrieval enhancement generation, and reasoning and action frameworks; Training with professional datasets: Utilizing professional datasets from the 3D printing industry, including images, sensor data, and process parameters, the model is deeply trained to ensure accurate identification and classification of various printing defects. Real-time feedback optimization: During the fine-tuning process, a real-time feedback mechanism is introduced to continuously adjust and optimize the model based on actual test results and user feedback to ensure continuous performance improvement.

3. The 3D printing process defect monitoring method based on a multimodal large language model according to claim 1, characterized in that: The step S2 specifically includes: S21 industrial camera cluster deployment and data acquisition: Multiple industrial cameras are deployed at key locations on the 3D printer to form a monitoring network. Camera selection takes into account resolution, frame rate, and field of view to ensure that every detail of the printing process can be captured. Hardware or software synchronization ensures that all cameras capture images at the same time to avoid image misalignment caused by time differences. Image data is continuously collected during the printing process to form a data set that includes multiple print batches and various printing materials. Initial processing of the S22 dataset: removing blurry, missing, or invalid images caused by camera failure, insufficient lighting, or obstruction by printed materials; calibrating the images of each camera to eliminate lens distortion and image warping, ensuring spatial consistency of all images; adding a timestamp to each image and sorting all images in the order in which they were printed based on the timestamp; S23 Data Augmentation and Annotation: This approach uses data augmentation methods such as rotation, scaling, flipping, and noise addition to expand the image dataset and improve the model's generalization capabilities. Two cameras are used to capture the same print area from different angles. By comparing and fusing the information from the two images, defect areas within the images are more accurately annotated. The annotations should include information about the defect's location, size, type, and severity. The annotation results are manually reviewed to ensure accuracy and consistency. S24 data transmission based on UDP protocol: Taking into account the requirements of real-time and low latency, UDP protocol is selected as the data transmission protocol; UDP protocol can transmit data faster and is suitable for scenarios with high real-time requirements; the sending and receiving functions of UDP protocol are implemented on the camera and server sides respectively; the camera side encapsulates the collected image data into UDP data packets and sends them, and the server side receives and parses these data packets, saving the image data locally or processing it; the status and speed of data transmission are monitored in real time to ensure that the data can be transmitted to the server side stably and quickly; at the same time, any abnormal conditions during the transmission process are recorded for troubleshooting and repair.

4. The 3D printing process defect monitoring method based on a multimodal large language model according to claim 1, characterized in that: The step S3 specifically includes: S31 image preprocessing: improve image quality, reduce noise, and standardize image data operations; S32 Initial Image Segmentation: Use an algorithm to segment the image into several superpixels, which are small areas with similar attributes. S33 builds a region adjacency graph: creates nodes for the segmented regions and establishes relationships between each node; S34 optimizes segmentation results: finds the optimal region combination by minimizing the energy function; S35 post-processing: optimize the segmented boundaries and simplify the final result.

5. The 3D printing process defect monitoring method based on a multimodal large language model according to claim 1, characterized in that: The step S4 specifically includes: S41: Large model call preparation and integration initialization: Load the multimodal large model and large language model that have been pre-trained and fine-tuned in the previous steps from the storage location and deploy them into the system for integration to form a complete defect detection system; construct a suitable environment to ensure that the model can run stably and receive multimodal data input from different sensors, including images, videos, and G Code files; at the same time, configure an efficient and appropriate environment for the model to meet the accuracy and precision of the real-time monitoring process; S42 Data Preparation: Accurately convert the collected multimodal data, including images, videos, G-code files, and slice models. For images or videos, a preprocessing step is used to extract key feature vectors to highlight important information in the data. S43: In the large-scale model reasoning process, we use the method of decomposing intermediate stages to handle tasks and introduce intermediate reasoning steps to achieve powerful complex reasoning capabilities. We rely on hinting technology to accurately identify the necessary steps in the reasoning process, thereby constructing an intermediate and coherent reasoning chain. S44 Integrated Defect Detection: Integrates the defect detection results obtained by calling the large model with the reasoning steps and action plans generated by the context information retrieved by the knowledge graph and retrieval enhancement generation technology to form a complete 3D printing process defect monitoring report; S45 displays results and provides suggestions: The system interface displays monitoring results in real time, providing users with intuitive defect information and improvement suggestions; S46 establishes a real-time feedback loop: during the printing process, new multimodal data is continuously collected, including real-time images and sensor data during the printing process.

6. The 3D printing process defect monitoring method based on a multimodal large language model according to claim 5, characterized in that: In the S44, based on the actual status of the current 3D printing task, a retrieval input including context information and data fragments to be analyzed is constructed; during the model reasoning process, with the help of prompt technology, the various intermediate reasoning steps necessary to complete defect judgment are identified; after the model executes the reasoning, the prediction result is returned, and the receiving process is decomposed into result reception, format verification and content parsing; first, the result data returned by the model is received, and the integrity and format of the data are checked to see if they meet expectations, and the result content is parsed, and the numerical and symbolic information output by the model is converted into information about defect judgment, defect type, location and severity, and classified and sorted to provide a basis for subsequent decision-making.

7. The 3D printing process defect monitoring method based on a multimodal large language model according to claim 5, characterized in that: In the above-mentioned S46, the collected data is transmitted to the 3D printing process defect monitoring system to ensure the timeliness and accuracy of the data; the transmitted data is analyzed in real time, and the reasoning combined with the thinking chain technology is used to timely discover potential problems; as the data is continuously processed, the process of updating the knowledge base is decomposed into knowledge extraction, knowledge verification and knowledge fusion with the help of retrieval enhancement generation technology; the extracted knowledge is verified, and the reliability of the knowledge is ensured through multiple experiments and comparative analysis; finally, the verified new knowledge is integrated into the existing knowledge graph and other auxiliary tools to update the knowledge base of the 3D printing process defect monitoring system, thereby enhancing the system's adaptability to different printing scenarios and the accuracy of defect detection.

8. A 3D printing process defect monitoring system based on a multimodal large language model using the method of any one of claims 1 to 6, comprising: 3D printing feature data acquisition module: used to collect 3D printing process data through multi-angle according to industrial-grade cameras; Machine vision module based on a multimodal large language model: used to automatically identify defects generated during the printing process and locate them; Analysis report module: Generates inspection reports based on defect inspection results.

Images