Manufacturing specific AI agent based interaction system utilizing multimodal and extended reality

Abstract

A manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality according to one embodiment of the present invention is a manufacturing-specialized AI agent-based interaction system equipped with a manufacturing process-specialized small language model for providing answers to questions related to natural language-based manufacturing processes. The manufacturing-specialized AI agent-based interaction system uses the manufacturing process-specialized small language model as an inference engine to generate answers to questions related to natural language-based manufacturing processes by searching for data corresponding to the purpose of the questions related to the manufacturing process from a manufacturing process-specialized language database and performing semantic search on the search results based on a Retrieval-Augmented Generation (RAG) method. It can provide a manufacturing-specialized service to a user that generates natural language-based answers and augmented reality-based answers, respectively, to questions related to natural language-based manufacturing processes based on interaction with an augmented reality providing means and a digital twin that interacts with the equipment of the actual factory in real time by linking with a multimodal processing system.

Description

Technology Field

[0001] The present invention relates to a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality, and more specifically, to a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality that uses a small-scale language model as an inference engine, integrates manufacturing AI and digital twin solutions, and can interact with a user through multimodal and augmented reality. Background Technology

[0002] Typically, manufacturing processes involve various procedures such as process control, quality monitoring, and safety management, which require significant manpower and time. Furthermore, human error, disparities in capabilities among field engineers, and differences in work methods make it difficult to maintain quality, leading to frequent disruptions in process operations.

[0003] Companies operating these manufacturing processes seek to maintain the smooth operation of the processes, but they face difficulties in maintaining consistent production conditions and quality, as well as securing field engineers, and feel an economic burden regarding personnel recruitment.

[0004] Furthermore, while there is a high possibility of casualties in the event of safety accidents at manufacturing sites, there is a problem in that the risk is further amplified if response procedures are inadequate or safety education and training are insufficient. Prior art literature

[0005] Korean Patent Publication No. 10-2379259 (Registered on March 23, 2022) The problem to be solved

[0006] Accordingly, the present invention has been devised to solve the aforementioned problems, and the objective of the present invention is to provide a manufacturing-specific AI agent-based interaction system utilizing multimodal and augmented reality that can maximize the operational efficiency of the manufacturing process by minimizing human error while simultaneously saving time and manpower by minimizing the intervention of field engineers in the manufacturing process.

[0007] More specifically, the objective of the present invention is to provide a manufacturing-specific AI agent-based interaction system utilizing multimodal and augmented reality that uses a small-scale language model as an inference engine to maximize the operational efficiency of a manufacturing process, integrates manufacturing AI and digital twin solutions, and can interact with users through multimodal and augmented reality.

[0008] However, the technical problems to be solved by the present invention are not limited to those mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art to which the present invention belongs from the description below. means of solving the problem

[0009] As a technical means for achieving the above-mentioned purpose, a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality according to an embodiment of the present invention comprises a manufacturing-specialized AI agent equipped with a manufacturing process-specialized small language model for providing answers to questions related to natural language-based manufacturing processes. The manufacturing-specialized AI agent-based interaction system utilizes the manufacturing process-specialized small language model as an inference engine to generate answers to questions related to natural language-based manufacturing processes by searching for data corresponding to the purpose of the questions related to manufacturing processes from a manufacturing process-specialized language database and performing semantic search on the search results based on a Retrieval-Augmented Generation (RAG) method. It can provide a manufacturing-specialized service to a user that generates natural language-based answers and augmented reality-based answers, respectively, to questions related to natural language-based manufacturing processes based on interaction with an augmented reality providing means and a digital twin interacting with the equipment of the actual factory in real time by linking with a multimodal processing system.

[0010] In addition, the manufacturing-specialized AI agent-based interaction system may be equipped with multiple AI models to provide natural language-based answers to the user regarding natural language-based questions related to the manufacturing process using the manufacturing process-specialized small-scale language model as an inference engine, or to link with the multimodal processing system.

[0011] And the above plurality of AI models may include: a vision inspection AI that provides answers to natural language-based questions for requesting a table of results of monitoring and analyzing the appearance quality of products produced from the equipment; a quality prediction AI that provides answers to natural language-based questions for requesting a table of results of analyzing the status of products produced from the equipment by product and by defect type of defective products; a process optimization AI that provides answers to natural language-based questions for requesting optimal manufacturing process conditions in conjunction with a manufacturing process condition simulator; and an industrial environment safety AI that detects and responds to accident events by performing multi-class classification based on natural language descriptions of the video and voice data of the actual factory generated by a video and voice collection unit, which collects video and voice data of the actual factory in real time as a multimodal processing system, performing image captioning and automatic voice documentation.

[0012] Furthermore, the above-described manufacturing-specialized AI agent-based interaction system can provide the user with a table of results for monitoring and analyzing the appearance quality of products produced from actual factory equipment that matches the equipment of the digital replica generated by the vision inspection AI performing machine vision and AI-based product appearance inspection services and monitoring the equipment of the digital replica displayed in the digital twin based on interaction with the digital twin.

[0013] In addition, the machine vision and AI-based product appearance inspection service may display a table of appearance quality monitoring and analysis results of a product produced from actual factory equipment that matches the equipment of the digital replica generated by the vision inspection AI, in a form where the digital twin is overlapped over the equipment of the digital replica, and the augmented reality providing means may visualize the table of appearance quality monitoring and analysis results of the product in augmented reality based on interaction with the digital twin and provide it to the user.

[0014] And the table of the results of the external quality monitoring and analysis of the above-mentioned product may be data in the form of a table and a graph that displays the results of the external inspection of the product, calculated by checking the external condition of the product produced from the equipment of the actual factory.

[0015] In addition, the manufacturing-specialized AI agent-based interaction system can provide the user with a table of status analysis results by product and defect type of actual factory equipment that matches the equipment of the digital replica generated by the quality prediction AI performing AI-based quality cause analysis and quality prediction services and monitoring the equipment of the digital replica displayed in the digital twin based on interaction with the digital twin.

[0016] Furthermore, the above AI-based quality cause analysis and quality prediction service displays a table of status analysis results by product and defect type of actual factory equipment that matches the equipment of the digital replica generated by the quality prediction AI, in a form where the digital twin is overlaid on the equipment of the digital replica, and the augmented reality providing means can visualize the table of status analysis results by product and defect type of defect in augmented reality based on interaction with the digital twin and provide it to the user.

[0017] In addition, the results of the analysis of the status by product and defect type of the above-mentioned defective products may be data including information on the number of good products, the number of defects, the production rate of products, the production time per unit product, and the number of productions according to the production conditions of the equipment of the actual factory.

[0018] Furthermore, the process optimization AI performs an AI-based process optimization solution provision service, receives an optimal manufacturing process condition information query to search for or extract optimal manufacturing process condition setting values ​​from the digital twin based on interaction with the digital twin, and then performs a simulation in conjunction with the manufacturing process condition simulator to calculate the optimal manufacturing process condition setting values ​​and the predicted defect rate or production volume according to the optimal manufacturing process based on the optimal manufacturing process condition information query, and the manufacturing-specific AI agent-based interaction system can provide the optimal manufacturing process condition setting values ​​and the predicted defect rate or production volume according to the optimal manufacturing process, which are the results of the simulation performed by the process optimization AI, to the user.

[0019] In addition, the AI-based process optimization solution providing service may display the results of the simulation performed by the process optimization AI in a form where the digital twin is overlapped over the equipment of the digital replica, and the augmented reality providing means may visualize the results of the simulation in augmented reality based on interaction with the digital twin and provide them to the user.

[0020] And the above optimal manufacturing process condition information query may be an information query regarding setting values ​​such as the number of good products, the number of defects, the product production rate, the production time per unit product, and the number of production cycles for each production condition applicable to the manufacturing process.

[0021] In addition, the manufacturing process-specific small-scale language model is constructed through manufacturing process specialization and Korean fine-tuning of the small-scale language model, and the small-scale language model can be fine-tuned for the manufacturing process through at least one of Instruction Fine-tuning and PEFT (Parameter-Efficient Fine-Tuning). Effects of the invention

[0022] The present invention can maximize the operational efficiency of the manufacturing process by minimizing the intervention of field engineers, thereby saving time and manpower, while simultaneously minimizing human error.

[0023] In addition, the present invention can implement a machine vision and AI-based product appearance inspection service that detects and responds to defective products early, thereby reducing the production loss rate of products while maintaining product quality.

[0024] Furthermore, the present invention can implement an AI-based quality cause analysis and quality prediction service that provides a report on countermeasures to the user to quickly resolve and prevent recurring defect problems in the manufacturing process.

[0025] In addition, the present invention can implement an AI-based process optimization solution service that can induce an improvement in product production rates by recommending optimal manufacturing process conditions to the user.

[0026] Furthermore, the present invention can implement an industrial environment safety solution service that enables real-time detection and immediate response to the occurrence of accident events in actual factories, and supports real-time interpretation to protect workers using other languages ​​from accident events.

[0027] However, the effects obtainable from the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0028] FIG. 1 is a diagram illustrating a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a machine vision and AI-based product appearance inspection service according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of an AI-based quality cause analysis and quality prediction service according to an embodiment of the present invention. FIG. 4 is a diagram illustrating an example of an AI-based process optimization solution provision service according to an embodiment of the present invention. FIG. 5 is a diagram illustrating an example of an industrial environment safety solution provision service according to an embodiment of the present invention. Specific details for implementing the invention

[0029] Hereinafter, embodiments of the present invention are described in detail with reference to the attached drawings so that those skilled in the art can easily implement the present invention. However, since the description of the present invention is merely an example for structural or functional explanation, the scope of the present invention should not be interpreted as being limited by the embodiments described in the text. That is, since the embodiments are subject to various modifications and may take various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical concept. Furthermore, the objectives or effects presented in the present invention do not imply that a specific embodiment must include all of them or only such effects; therefore, the scope of the present invention should not be understood as being limited by them.

[0030] The meaning of the terms described in this invention should be understood as follows.

[0031] Terms such as "first" and "second" are intended to distinguish one component from another, and the scope of rights shall not be limited by these terms. For example, the first component may be named the second component, and similarly, the second component may be named the first component. When a component is referred to as being "connected" to another component, it should be understood that it may be directly connected to that other component, or that there may be other components in between. Conversely, when a component is referred to as being "directly connected" to another component, it should be understood that there are no other components in between. Meanwhile, other expressions describing the relationship between components, such as "between" and "exactly between," or "adjacent to" and "directly adjacent to," shall be interpreted in the same manner.

[0032] A singular expression should be understood to include a plural expression unless the context clearly indicates otherwise, and terms such as "include" or "have" are intended to specify the existence of the set-up features, numbers, steps, actions, components, parts, or combinations thereof, and should be understood not to preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0033] Unless otherwise defined, all terms used herein have the same meaning as generally understood by those skilled in the art to which this invention pertains. Terms defined in commonly used dictionaries should be interpreted as having meanings consistent with the context of the relevant technology and should not be interpreted as having an ideal or overly formal meaning unless explicitly defined in this invention.

[0035] AI Agent-Based Interaction System

[0036] Hereinafter, the configuration of a preferred embodiment will be described in detail with reference to the attached drawings.

[0037] FIG. 1 is a diagram illustrating a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality according to an embodiment of the present invention.

[0038] Referring to FIG. 1, an AI agent-based interaction system according to one embodiment of the present invention can be constructed through a manufacturing-specific AI agent-based interaction system (10), a digital twin (20), an extended reality providing means (30), and equipment (40) to maximize the operational efficiency of a manufacturing process.

[0039] In one embodiment, the manufacturing-specific AI agent-based interaction system (10) may be a manufacturing-specific AI agent-based interaction system utilizing multimodal and augmented reality that uses a small language model (100) as an inference engine, integrates manufacturing AI and digital twin solutions, and can interact with a user (1) through multimodal and augmented reality.

[0040] A small language model (100) is a concept contrasted with a large language model (LLM), and refers to a language model trained with a relatively small number of parameters and data compared to the large language model. It requires less computing resources than a large model and has the advantage of being able to be used more efficiently in specific tasks or limited environments. More specifically, the small language model (100) has advantages such as computing efficiency, fast learning speed, and ease of deployment, and can be optimized for specific tasks or domains.

[0041] In one embodiment, the small language model (100) can be constructed as a manufacturing process-specific small language model (100) with improved accuracy of answers to questions related to the manufacturing process by performing manufacturing process specialization and Korean fine tuning, so that it is used as an inference engine of a manufacturing-specific AI agent-based interaction system (10) for providing answers to questions related to the manufacturing process.

[0042] More specifically, the small language model (100) can be constructed as a manufacturing process-specific small language model (100) capable of providing answers to questions related to the manufacturing process by being fine-tuned for the manufacturing process through at least one of Instruction Fine-tuning and PEFT (Parameter-Efficient Fine-Tuning).

[0043] Here, Instruction Fine-tuning is a technique for adjusting a pre-trained language model to a specific task, referring to a method in which the model is trained to generate responses by providing clear instructions or prompts. This approach is primarily used in Natural Language Processing (NLP) applications and can be effectively utilized in various fields, particularly chatbots, Q&A systems, text summarization, and translation. The Instruction Fine-tuning method involves selecting a pre-trained language model such as BERT, GPT, or T5, preparing a dataset containing prompts and expected responses, and performing preprocessing by tokenizing the text for the model and generating padding and attention masks as necessary. Furthermore, the Instruction Fine-tuning method fine-tunes the model using the prepared dataset, and in this process, the model can be trained to generate appropriate responses that match the prompts.

[0044] Additionally, the Instruction Fine-tuning method may be a process of fine-tuning a small language model (100) to match specific instructions or commands so that it can provide answers to questions related to the manufacturing process.

[0045] And the PEFT method may be a process of fine-tuning a small language model (100) so as to provide answers to questions related to the manufacturing process by adjusting only some parameters, rather than adjusting all parameters of the small language model (100).

[0046] That is, the manufacturing-specialized AI agent-based interaction system (10) is equipped with a manufacturing process-specialized small language model (100) that has undergone manufacturing process-specialized fine-tuning through at least one of Instruction Fine-tuning and PEFT (Parameter-Efficient Fine-Tuning), and the manufacturing process-specialized small language model (100) can be used as an inference engine.

[0047] In one embodiment, the manufacturing process specialized small language model (100) can be linked with a manufacturing process specialized language database (not shown) so as to provide answers to questions related to the manufacturing process to the user (1).

[0048] At this time, the manufacturing process specialized language database refers to a database that stores as storage data documents containing data such as text, table data, reference images, and graphs corresponding to the purpose of the questions related to the manufacturing process conveyed by the user (1).

[0049] In addition, the manufacturing process specialized small language model (100) can search for data corresponding to the purpose of a question related to the manufacturing process from the manufacturing process specialized language database, and then perform a semantic search on the search results using a Retrieval-Augmented Generation (RAG) method.

[0050] Here, Search Augmented Generative (RAG) is a technology in the field of Natural Language Processing (NLP) that combines search and generative models to achieve better performance. This approach combines the strengths of search-based information retrieval and generative-based text generation, enabling high performance, particularly in information retrieval and conversational AI systems. Such Search Augmented Generative (RAG) can be broadly composed of a Retriever and a Generator.

[0051] Searchers can search for relevant documents in large text corpora. This is similar to traditional information retrieval systems and can be used to find documents suitable for a query, primarily by using efficient search algorithms. For example, dense passage-based searchers such as Dense Passage Retrieval (DPR) using pre-trained language models like BERT, or searchers using keyword matching-based ranking algorithms such as BM25, may be used.

[0052] The generator can produce a final response based on the retrieved documents and can utilize pre-trained language models, such as GPT-3 or T5. Additionally, it can take the retrieved documents as input and generate meaningful and consistent text based on them.

[0053] Furthermore, when a query is entered, Search Augmented Generation (RAG) searches the manufacturing process-specific language database based on the input query, measures the similarity to the query, and selects highly relevant information. At this time, the data stored in the manufacturing process-specific language database is encoded as dense vectors, and Search Augmented Generation (RAG) uses these vectors to calculate the similarity with the query to select the most relevant information, and selects the most relevant information based on the similarity between the selected query and the information.

[0054] This Search Augmented Generation (RAG) generates responses by directly extracting information from retrieved data, thereby enhancing response accuracy and relevance. It also improves information access speed by enabling the rapid retrieval and utilization of necessary information from manufacturing process-specific language databases, and offers the advantage of diverse applications in various NLP fields such as QA systems, conversational AI, and text summarization. Furthermore, because the generative model in Search Augmented Generation (RAG) accurately understands the context of the retrieved information and generates answers based on this understanding, it can provide consistent responses that are appropriate to the context.

[0055] In one embodiment, when a manufacturing process-specific small language model (100) receives a question related to a manufacturing process, it performs a semantic search and transmits the context within a document most similar to the question related to the manufacturing process among the stored data stored in the manufacturing process-specific language database to a searcher, and a generator can generate a natural language-based answer to the question related to the manufacturing process based on the document searched from the searcher and transmit it to the manufacturing process-specific small language model (100).

[0056] That is, the manufacturing process specialized small-scale language model (100) can provide the user (1) with answers to natural language-based questions related to the manufacturing process received from the generator.

[0057] As such, the manufacturing process specialized small-scale language model (100) can provide answers to questions related to the manufacturing process based on the most accurate and up-to-date information using search augmented generation (RAG), and thus has the advantage of improving the reliability of the answers by resolving problems such as hallucinations, answers from unreliable sources, knowledge gaps, and inability to provide up-to-date information.

[0058] A manufacturing-specific AI agent-based interaction system (10) that uses such a manufacturing process-specific small-scale language model (100) as an inference engine can be provided to a user (1) in an on-premises or cloud manner depending on the characteristics of the industrial site.

[0059] Here, the on-premise method is a method in which the company directly manages the stored data to prevent leakage of the company's stored data that deploys the manufacturing-specialized AI agent-based interaction system (10), and the cloud method is a method in which the company receives the stored data from an external cloud and operates it.

[0060] The manufacturing-specialized AI agent-based interaction system (10) of the present invention can minimize the intervention of field engineers to save time and manpower, minimize human error to maximize the operational efficiency of the manufacturing process, and improve the stability of the industrial site to support the operation of the manufacturing process.

[0061] And, a manufacturing-specialized AI agent-based interaction system (10) of one embodiment can provide a manufacturing-specialized service that can interact with a user (1) through multimodal and extended reality (XR) by integrating manufacturing AI and digital twin solutions based on interaction with a digital twin (20) that interacts with an extended reality providing means (30) and equipment (40) as shown in FIG. 1.

[0062] Here, the manufacturing-specialized services to be provided to the user (1) by the manufacturing-specialized AI agent-based interaction system (10) are not limited, but in one embodiment, the machine vision and AI-based product appearance inspection service (S11) shown in FIG. 2, the AI-based quality cause analysis and quality prediction service (S12) shown in FIG. 3, the AI-based process optimization solution provision service (S13) shown in FIG. 4, and the industrial environment safety solution provision service (S14) shown in FIG. 5 are included.

[0063] In one embodiment, a manufacturing-specific AI agent-based interaction system (10) may be equipped with a plurality of AI models that use a manufacturing-specific small language model (100) as an inference engine to provide various services related to the manufacturing process to a user (1).

[0064] Here, the plurality of AI models mounted on the manufacturing-specialized AI agent-based interaction system (10) are not limited, but in one embodiment, the system includes a vision inspection AI (11) for providing a machine vision and AI-based product appearance inspection service (S11), a quality prediction AI (12) for providing an AI-based quality cause analysis and quality prediction service (S12), a process optimization AI (13) for providing an AI-based process optimization solution service (S13), and an industrial environment safety AI (14) for providing an industrial environment safety solution service (S14).

[0065] In one embodiment, the digital twin (20) is a digital replica of an actual factory including an industrial site where a manufacturing process is carried out based on equipment (40), and is capable of interacting with a plurality of AI models (11-14) mounted on a manufacturing-specialized AI agent-based interaction system (10).

[0066] At this time, the digital twin (20), which is a digital replica of the actual factory, is continuously updated through real-time data collected by monitoring the equipment (40) of the actual factory in real time, and can reflect the status of the product produced from the manufacturing process of the actual factory and the operation of the equipment (40).

[0067] In addition, the interaction between the digital twin (20) and the AI ​​models (11-14) means that the digital twin (20) grants data access rights to the AI ​​models (11-14), and additionally, the AI ​​models (11-14) of the manufacturing-specialized AI agent-based interaction system (10) automatically monitor the operating status and setting values ​​of the equipment (40) and the production data of the product produced in the manufacturing process in the digital twin (20), perform simulations based on the operating settings of the equipment (40), and perform optimization and autonomous operation of the equipment (40) based on the results of the automatic monitoring and simulations.

[0068] In one embodiment, the extended reality providing means (30) may be a means for visualizing the results of automatic monitoring, simulation, optimization, and autonomous operation of the equipment (40) performed by AI models (11-14) in the digital twin (20) based on extended reality (XR) technology and providing them to the user (1).

[0069] In this case, Extended Reality (XR) may be a concept that includes Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR).

[0070] That is, the extended reality providing means (30) can visualize the results of automatic monitoring, simulation, optimization, and autonomous operation of the equipment (40) performed by the AI ​​models (11-14) in the digital twin (20) through at least one of virtual reality (VR), augmented reality (AR), and mixed reality (MR).

[0071] Additionally, the extended reality providing means (30) is not limited to a means of providing the user (1) with the results of automatic monitoring, simulation, optimization, and autonomous operation of the equipment (40) performed by the AI ​​models (11-14) in the digital twin (20) in the extended reality (XR), but in one embodiment, it may be implemented as a goggle display worn on the face of the user (1).

[0072] And the extended reality providing means (30) can interact with the digital twin (20) to provide equipment information (31) and production status (32) to the user (1) in the extended reality (XR).

[0073] At this time, the interaction between the extended reality providing means (30) and the digital twin (20) means that the digital twin (20) transmits to the extended reality providing means (30) the operating status and setting values ​​of the equipment (40), which are the results of automatic monitoring performed by the AI ​​models (11-14), and the production data of the product produced in the manufacturing process. Additionally, the extended reality providing means (30) means providing to the user (1) in the extended reality (XR) the equipment information (31), which is the operating status and setting values ​​of the equipment (40) received from the digital twin (20), and the production status (32), which is the production data.

[0074] In one embodiment, facility information (31) and production status (32) can be overlaid on the facility (40) displayed in the digital twin (20) provided to the user (1) through extended reality (XR).

[0075] In addition, among the equipment information (31), the setting value of the equipment (40) may be the real-time temperature, pressure, vibration, noise, etc. of the equipment (40).

[0076] And the production data, which is the production status (32), may be data including the results of the analysis of the status of products including good and defective products and the defect types of the defective products.

[0077] At this time, the product status analysis result may be data including the number of good products, number of defective products, product production rate, production time per unit product, and production frequency information for each production condition of the equipment (40), and the unit product refers to the minimum sales unit product finally produced in the manufacturing process. As a specific example, in the automobile manufacturing process, the unit product may be one automobile.

[0078] And the analysis result of the defect type status of the defective product may be data including the defect type of the defective product produced from the actual factory equipment (40) during a preset period (e.g., at least 1 day, at most 1 year) and the cause of the defect type of the defect.

[0079] Additionally, the augmented reality providing means (30) of one embodiment can transmit a question related to a manufacturing process generated by a user (1) to a manufacturing-specialized AI agent-based interaction system (10).

[0080] Additionally, when the extended reality providing means (30) receives an answer to a question related to a manufacturing process inferred from a manufacturing process specialized small language model (100) mounted on the manufacturing specialized AI agent-based interaction system (10), it can provide the answer to the received question related to the manufacturing process to the user (1) in the extended reality (XR).

[0081] In addition, the extended reality providing means (30) of one embodiment can provide the user (1) with the function of controlling the operation of the equipment (40) displayed in the digital twin (20) through the extended reality (XR), and the digital twin (20) enables the equipment (40) of the actual factory that matches the equipment (40) of the digital replica controlled by the extended reality providing means (30) to be controlled identically through interaction between the extended reality providing means (30) and the equipment (40) of the actual factory.

[0082] In one embodiment, the equipment (40) can interact with the digital twin (20) as a device installed in the actual factory to produce a unit product of the minimum sales unit by performing a manufacturing process carried out in the actual factory.

[0083] At this time, the interaction between the equipment (40) and the digital twin (20) means that the equipment (40) provides the digital twin (20) with equipment information (31) and production status (32) to be provided to the user (1) through extended reality (XR), and additionally, the digital twin (20) controls the equipment (40) of the actual factory that matches the equipment (40) when the user (1) controls the equipment (40) displayed in the digital twin (20) using the extended reality providing means (30).

[0085] Machine Vision and AI-based Product Appearance Inspection Service

[0086] Below, we will describe in detail the machine vision and AI-based product appearance inspection service (S11), which is one of the manufacturing-specialized services that the manufacturing-specialized AI agent-based interaction system (10) will provide to the user (1).

[0087] FIG. 2 is a drawing illustrating an example of a machine vision and AI-based product appearance inspection service according to an embodiment of the present invention.

[0088] Referring to FIG. 2, the user (1) can request (or input) a natural language-based question to the manufacturing-specific AI agent-based interaction system (10) to request a table (table and graph) of the results of monitoring and analyzing the appearance quality of a product produced from the equipment (40) after wearing an augmented reality providing means (30) to perform a machine vision and AI-based product appearance inspection service (S11).

[0089] After that, the vision inspection AI (11) mounted on the manufacturing-specialized AI agent-based interaction system (10) monitors the equipment (40) of the digital replica displayed in the digital twin (20) based on interaction with the digital twin (20), and can generate a table of results for monitoring and analyzing the appearance quality of products produced from the actual factory equipment (40) that matches the equipment (40) of the digital replica.

[0090] After that, the manufacturing-specialized AI agent-based interaction system (10) can provide the user (1) with a table of results of monitoring and analyzing the appearance quality of a product produced from an actual factory facility (40) that matches the facility (40) of a digital replica generated by the vision inspection AI (11), as a natural language-based answer (inference result) to a question generated by the user (1).

[0091] In this way, the manufacturing-specialized AI agent-based interaction system (10) provides a natural language-based answer to the user (1), while the digital twin (20) and the extended reality providing means (30) can provide an answer to a natural language-based question generated by the user (1) in extended reality (XR).

[0092] As a specific example, the digital twin (20) is overlaid on the equipment (40) of the digital replica to display a table of results of monitoring and analyzing the appearance quality of a product produced from the actual factory equipment (40) that matches the equipment (40) of the digital replica generated by the vision inspection AI (11), and the extended reality providing means (30) can provide the table of results of monitoring and analyzing the appearance quality of a product produced from the actual factory equipment (40) that matches the equipment (40) of the digital replica to the user (1) by visualizing it in extended reality (XR) based on interaction with the digital twin (20).

[0093] That is, the machine vision and AI-based product appearance inspection service (S11) can generate a natural language-based answer and an extended reality (XR)-based answer, respectively, for a natural language-based question requesting a table (table and graph) of the results of the appearance quality monitoring and analysis of a product produced from the equipment (40) created by the user (1), and provide them to the user (1).

[0094] At this time, the table of product appearance quality monitoring and analysis results refers to data that displays the product appearance inspection result value (good product or defective product) calculated by checking the appearance condition of the product produced from the equipment (40) in the form of a table and a graph, and the items of the table may include product information such as the product name and ID, and this can be changed through settings by the user (1).

[0095] In addition, the manufacturing-specialized AI agent-based interaction system (10) can provide a table of product appearance quality monitoring and analysis results at a preset interval to a user (1) wearing an augmented reality providing means (30) in a machine vision and AI-based product appearance inspection service (S11).

[0096] At this time, the preset cycle for providing a table of product appearance quality monitoring and analysis results is preferably 1 to 10 minutes, but this can be changed through a setting by the user (1).

[0097] In addition, the manufacturing-specialized AI agent-based interaction system (10) provides a table of product appearance quality monitoring and analysis results to the user (1) in order to respond to cases where the product appearance inspection result value continuously deviates from a threshold value and is determined to be a defective product, thereby maintaining product quality and reducing the production loss rate of the product.

[0098] As a specific example, the manufacturing-specialized AI agent-based interaction system (10) monitors the equipment (40) of a digital replica, and when the product's appearance inspection result value in the table of product appearance quality monitoring and analysis results is determined to be a defective product because it exceeds the threshold value for 5 consecutive times, an alert is generated in the extended reality (XR) to track the location of the equipment (40) that produced the product whose appearance inspection result value exceeded the threshold value for 5 consecutive times. The system then performs optimization and autonomous operation of the equipment (40) so that the product's appearance inspection result value does not exceed the threshold value, and at the same time, transmits the time information of the alert generation and the autonomous operation of the equipment (40) to the extended reality providing means (30) so that the time information of the alert generation and the autonomous operation of the equipment (40) is displayed in the extended reality (XR).

[0099] As another specific example, a manufacturing-specialized AI agent-based interaction system (10) can control the operation of the equipment (40) that produced the product whose appearance inspection results exceeded the threshold for 10 consecutive times in a table of product appearance quality monitoring and analysis results generated by monitoring the equipment (40) of the digital replica to stop the operation of the equipment (40) that produced the product whose appearance inspection results exceeded the threshold for 10 consecutive times.

[0100] At this time, the number of product appearance inspection results that deviate from the threshold value for determining the product as defective, the time of occurrence of the notification that can track the location of the equipment (40) that produced the product that deviates from the threshold value, the threshold value, the time of display of time information, and the display method can be changed through settings by the user (1).

[0101] In one embodiment, the machine vision and AI-based product appearance inspection service (S11) can detect and respond to defective products early, thereby reducing the production loss rate of products while maintaining product quality.

[0103] AI-based quality cause analysis and quality prediction service

[0104] Below, we will explain in detail the AI-based quality cause analysis and quality prediction service (S12), which is one of the manufacturing-specialized services that the manufacturing-specialized AI agent-based interaction system (10) will provide to the user (1).

[0105] FIG. 3 is a diagram illustrating an example of an AI-based quality cause analysis and quality prediction service according to an embodiment of the present invention.

[0106] Referring to FIG. 3, the user (1) can request (or input) a natural language-based question to the manufacturing-specialized AI agent-based interaction system (10) to request a table (table and graph) of the results of the analysis of the status of products and defect types of defective products produced from the equipment (40) after wearing an augmented reality providing means (30) to perform an AI-based quality cause analysis and quality prediction service (S12).

[0107] After that, the quality prediction AI (12) installed in the manufacturing-specialized AI agent-based interaction system (10) can generate quality information of products produced from the actual factory equipment (40) that matches the digital replica equipment (40) by monitoring products produced for a preset period (e.g., one week) from the digital replica equipment (40) displayed in the digital twin (20) based on interaction with the digital twin (20).

[0108] After that, the quality prediction AI (12) can analyze the results of the analysis of the status of products and defect types based on the quality information of the products and defect types to generate a table of the results of the analysis of the status of products and defect types produced from the equipment (40) of the actual factory that matches the equipment (40) of the digital replica.

[0109] After that, the manufacturing-specialized AI agent-based interaction system (10) can provide the user (1) with a table of analysis results of the status of product by product and defect type produced from the actual factory equipment (40) that matches the digital replica equipment (40) generated by the quality prediction AI (12), as a natural language-based answer (inference result) to a question generated by the user (1).

[0110] In this way, the manufacturing-specialized AI agent-based interaction system (10) provides a natural language-based answer to the user (1), while the digital twin (20) and the extended reality providing means (30) can provide an answer to a natural language-based question generated by the user (1) in extended reality (XR).

[0111] As a specific example, the digital twin (20) is overlaid on the equipment (40) of the digital replica to display a table of the results of the analysis of the status of products and defect types produced from the actual factory equipment (40) that matches the equipment (40) of the digital replica generated by the quality prediction AI (12), and the extended reality providing means (30) can provide the table of the results of the analysis of the status of products and defect types produced from the actual factory equipment (40) to the user (1) by visualizing it in extended reality (XR) based on interaction with the digital twin (20).

[0112] That is, the AI-based quality cause analysis and quality prediction service (S12) can generate a natural language-based answer and an extended reality (XR)-based answer, respectively, for a natural language-based question requesting a table (table and graph) of the results of the analysis of the status of products and defect types produced from the equipment (40) created by the user (1), and provide them to the user (1).

[0113] In one embodiment, the analysis result of the status of each product and the defect type of the defective product refers to production data, which is a production status (32) provided to the user (1) through extended reality (XR). The analysis result of the status of each product is data that includes information on the number of good products, the number of defective products, the production rate of the product, the production time per unit product, and the number of productions for each production condition of the equipment (40). The analysis result of the defect type of the defective product may be data that includes the defect type of the defective product produced from the equipment (40) of the actual factory during a preset period and the cause of the defect of the defect type.

[0114] Additionally, when the quality prediction AI (12) generates a table of results of the analysis of the status of products and defect types of defective products, it can receive an information query from the digital twin (20) to search for or extract the top n products and defect types with the highest defect rates from the results of the analysis of the status of products and defect types of defective products based on interaction with the digital twin (20).

[0115] Here, the number of top defect rates in the information query can be changed through settings by the user (1), but in one embodiment, it will be explained assuming that there are three top defect rates.

[0116] After that, the quality prediction AI (12) can retrieve a report from a database (not shown) that analyzes the top 3 products and defect types with the highest defect rates and the most similar products and defect types based on an information query on the top 3 products and defect types with the highest defect rates.

[0117] After that, the manufacturing-specialized AI agent-based interaction system (10) can automatically generate a response plan report containing the top 3 products with the highest defect rates and the causes of defect types based on reports retrieved from the database, and a response plan report containing a response plan to resolve the causes of defects, and then send the response plan report to the user (1) in the form of an email by linking with an email sending server (not shown).

[0118] At this time, the time of sending the response plan report can be changed through settings by the user (1).

[0119] In one embodiment, the AI-based quality cause analysis and quality prediction service (S12) can quickly resolve and prevent recurring defect problems in the manufacturing process by providing a response plan report to the user (1).

[0121] AI-based process optimization solution provision service

[0122] Below, we will explain in detail the AI-based process optimization solution provision service (S13), which is one of the manufacturing-specialized services that the manufacturing-specialized AI agent-based interaction system (10) will provide to the user (1).

[0123] FIG. 4 is a diagram illustrating an example of an AI-based process optimization solution providing service according to an embodiment of the present invention.

[0124] Referring to FIG. 4, the user (1) can request (or input) a natural language-based question to the manufacturing-specialized AI agent-based interaction system (10) to request optimal manufacturing process conditions after wearing an extended reality providing means (30) to proceed with the AI-based process optimization solution providing service (S13).

[0125] After that, the process optimization AI (13) installed in the manufacturing-specialized AI agent-based interaction system (10) can receive an optimal manufacturing process condition information query to search for or extract optimal manufacturing process condition setting values ​​from the digital twin (20) based on interaction with the digital twin (20).

[0126] After that, the process optimization AI (13) can perform a simulation to calculate the optimal manufacturing process condition setting value and the predicted defect rate or production volume according to the optimal manufacturing process based on the optimal manufacturing process condition information query received from the digital twin (20) by linking with a manufacturing process condition simulator (not shown) based on the optimal manufacturing process condition information query.

[0127] After that, the manufacturing-specialized AI agent-based interaction system (10) can provide the user (1) with an optimal manufacturing process condition setting value, which is the result of a simulation performed by the process optimization AI (13), and a predicted defect rate or production volume according to the optimal manufacturing process as a natural language-based answer (inference result) to a question generated by the user (1).

[0128] In this way, the manufacturing-specialized AI agent-based interaction system (10) provides a natural language-based answer to the user (1), while the digital twin (20) and the extended reality providing means (30) can provide an answer to a natural language-based question generated by the user (1) in extended reality (XR).

[0129] As a specific example, the digital twin (20) can display the results of a simulation performed by the process optimization AI (13) in a form that overlaps the equipment (40) of the digital replica, and the extended reality providing means (30) can provide the results of the simulation to the user (1) by visualizing them in extended reality (XR) based on interaction with the digital twin (20).

[0130] That is, the AI-based process optimization solution providing service (S13) can generate a natural language-based answer and an extended reality (XR)-based answer, respectively, for a natural language-based question requesting optimal manufacturing process conditions generated by the user (1), and provide them to the user (1).

[0131] In one embodiment, the optimal manufacturing process condition information query refers to an information query regarding setting values ​​such as the number of good products, the number of defects, the product production rate, the production time per unit product, and the number of production cycles for each production condition applicable to the manufacturing process.

[0132] In addition, the manufacturing-specialized AI agent-based interaction system (10) not only provides optimal manufacturing process condition setting values ​​and predicted defect rates or production volumes according to the optimal manufacturing process in the AI-based process optimization solution provision service (S13), but also allows the selection of the manufacturing process to be operated under the conditions of the optimal manufacturing process.

[0133] And the manufacturing-specialized AI agent-based interaction system (10) can present the optimal manufacturing process condition setting value to the user (1) to optimize the manufacturing process through a report in which the optimal manufacturing process condition setting value is displayed in the form of text, table, graph, etc. during the process of selecting the manufacturing process.

[0134] These optimal manufacturing process condition settings include process variables and environmental variables. The process variables may include the temperature and pressure of the injection molding process, the speed of the product production line, the reaction for producing the product, heating or cooling, and processing time, which directly affect the performance of the manufacturing process and the quality of the product. The environmental variables may include environmental conditions outside the manufacturing process, such as the temperature, humidity, air quality, vibration, and noise of the industrial site where the manufacturing process takes place.

[0135] When a user (1) selects to operate the manufacturing process under optimal manufacturing process conditions, the manufacturing process can be operated as an optimal manufacturing process by changing the setting value of the equipment (40) of the digital replica based on interaction with the digital twin (20), thereby changing the setting value of the equipment (40) of the actual factory that matches the equipment (40) of the digital replica.

[0136] At this time, if the setting value of the equipment (40) is set to the optimal formulation process conditions before the user (1)'s request, the setting value of the equipment (40) can be maintained.

[0137] In one embodiment, the AI-based process optimization solution providing service (S13) can induce an improvement in the production rate of a product by recommending optimal manufacturing process conditions to the user (1).

[0139] Industrial Environment Safety Solution Provision Service

[0140] Below, we will explain in detail the industrial environment safety solution provision service (S14), which is one of the manufacturing-specialized services that the manufacturing-specialized AI agent-based interaction system (10) will provide to the user (1).

[0141] FIG. 5 is a drawing illustrating an example of an industrial environment safety solution provision service according to an embodiment of the present invention.

[0142] Referring to FIG. 5, the video and voice collection unit (200) linked to the manufacturing-specialized AI agent-based interaction system (10) of the present invention is a multimodal processing system that is installed in an actual factory and can collect video data and voice data generated in the actual factory in real time, and then process the noise of the video data and voice data.

[0143] After that, the video and audio collection unit (200) can perform image captioning (S14a) and automatic voice documentation (S14b) to generate text-based natural language descriptions from the video data and audio data of the actual factory.

[0144] At this time, image captioning (S14a) may be a step of extracting key features from images of video data collected from an actual factory based on Convolutional Neural Networks (CNN), and generating text related to the visual information of the image using a model such as Recurrent Neural Networks (RNN) or Transformer based on the extracted image features.

[0145] Additionally, automatic speech documentation (S14b) may be a step of generating natural language descriptions from voice data collected from an actual factory based on voice-to-text conversion and automatic speech recognition (ASR).

[0146] After that, the video and voice collection unit (200) can transmit a natural language description of the generated video data and voice data of the actual factory to the manufacturing-specialized AI agent-based interaction system (10).

[0147] After that, the industrial environment safety AI (14) installed in the manufacturing-specialized AI agent-based interaction system (10) can perform multi-class classification (S14c) based on natural language descriptions of video data and voice data of the actual factory, and analyze the types of accident events occurring in the actual factory.

[0148] At this time, multi-class classification (S14c) may be a step of extracting risk factor information from natural language descriptions of image data and voice data based on natural language processing (NLP) and machine learning techniques, and analyzing the types of accident events occurring in the actual factory based on the risk factor information.

[0149] After that, the manufacturing-specialized AI agent-based interaction system (10) can perform different responses depending on the type of accident event that occurred in the actual factory when the industrial environment safety AI (14) determines that an accident event has occurred in the actual factory through multi-class classification (S14c) based on natural language description of video data and voice data (S14d).

[0150] As a specific example, the manufacturing-specialized AI agent-based interaction system (10) can perform the role of a chatbot that supports on-site emergency measures for workers located at the actual factory and, if necessary, supports real-time simultaneous interpretation with workers using other languages. This is achieved through multi-class classification (S14c) based on natural language descriptions of video data and voice data, so that when a fire occurs in an actual factory and an accident response step (S14) must be carried out, it can automatically make an emergency call to the fire station closest to the actual factory and send an emergency text message to a terminal (e.g., smartphone, PC, tablet, etc.) equipped by the safety officer at the actual factory.

[0151] In this case, accident events include the collapse of a worker, fire and building collapse, entry of a worker or objects other than the worker into a danger zone, failure of a worker to wear a safety helmet, and failure to meet the minimum number of personnel.

[0152] In one embodiment, the industrial environment safety solution providing service (S14) is linked with a manufacturing-specialized AI agent-based interaction system (10) and a video and voice collection unit (200), which is a multimodal processing system, thereby enabling real-time detection of accident events in an actual factory and immediate response, and supporting real-time simultaneous interpretation to protect workers who speak other languages ​​from accident events.

[0154] As described above, the detailed description of the preferred embodiments of the present invention disclosed is provided to enable those skilled in the art to implement and practice the present invention. Although the present invention has been described with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the present invention without departing from the scope of the invention. For example, those skilled in the art may utilize each configuration described in the embodiments described above in combination with one another. Accordingly, the present invention is not intended to be limited to the embodiments shown herein, but to be given the broadest scope consistent with the principles and novel features disclosed herein.

[0155] The present invention may be embodied in other specific forms without departing from the technical spirit and essential features of the invention. Accordingly, the above detailed description should not be interpreted restrictively in all respects but should be considered exemplary. The scope of the invention shall be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the invention are included within the scope of the invention. The invention is not intended to be limited to the embodiments shown herein, but to be given the broadest possible scope consistent with the principles and novel features disclosed herein. Furthermore, embodiments may be constructed by combining claims that are not explicitly related in the claims, or included as new claims through amendments made after filing. Explanation of the symbols

[0156] 1: User, 10: Manufacturing-specialized AI agent-based interaction system, 11: Vision inspection AI, 12: Quality prediction AI, 13: Process optimization AI, 14: Industrial environment safety AI, 20: Digital twin, 30: Means of providing augmented reality, 31: Equipment information, 32: Production status, 40: Equipment, 100: Manufacturing process-specialized small-scale language model, 200: Video and audio collection unit.

Claims

Claim 1 A manufacturing-specialized AI agent-based interaction system equipped with a manufacturing process-specialized small language model for providing answers to natural language-based questions related to manufacturing processes, wherein the manufacturing-specialized AI agent-based interaction system uses the manufacturing process-specialized small language model as an inference engine to generate answers to natural language-based questions related to manufacturing processes by searching for data corresponding to the purpose of the questions related to manufacturing processes from a manufacturing process-specialized language database and performing semantic search on the search results based on a Retrieval-Augmented Generation (RAG) method, and provides a manufacturing-specialized service to a user that generates natural language-based answers and augmented reality-based answers, respectively, to natural language-based questions related to manufacturing processes based on interaction with an augmented reality providing means and a digital twin interacting with the equipment of the actual factory, in conjunction with a multimodal processing system to detect and respond to the occurrence of accident events in real time in an actual factory, and based on interaction with an augmented reality providing means and a digital twin interacting with the equipment of the actual factory. Claim 2 In claim 1, the manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality is characterized by being equipped with a plurality of AI models for providing natural language-based answers to the user regarding natural language-based questions related to the natural language-based manufacturing process using the manufacturing process-specialized small-scale language model as an inference engine, or for linking with the multimodal processing system. Claim 3 In claim 2, the plurality of AI models comprises: a vision inspection AI that provides answers to natural language-based questions for requesting a table of results of monitoring and analyzing the appearance quality of products produced from the equipment; a quality prediction AI that provides answers to natural language-based questions for requesting a table of results of analyzing the status of products produced from the equipment by product type and defect type of defective products; a process optimization AI that provides answers to natural language-based questions for requesting optimal manufacturing process conditions in conjunction with a manufacturing process condition simulator; and an industrial environment safety AI that detects and responds to accident events by performing multi-class classification based on natural language descriptions of the video and voice data of the actual factory generated by a video and voice collection unit, which collects video and voice data of the actual factory in real time as a multimodal processing system, performing image captioning and automatic voice documentation. Claim 4 In claim 3, the manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality is characterized by the vision inspection AI performing machine vision and AI-based product appearance inspection services, and providing the user with a table of appearance quality monitoring and analysis results of a product produced from actual factory equipment that matches the equipment of the digital replica generated by monitoring the equipment of the digital replica displayed in the digital twin based on interaction with the digital twin. Claim 5 In claim 4, the machine vision and AI-based product appearance inspection service is characterized by displaying a table of appearance quality monitoring and analysis results of a product produced from actual factory equipment that matches the equipment of the digital replica generated by the vision inspection AI in a form where the digital twin is overlapped over the equipment of the digital replica, and by the augmented reality providing means visualizing the table of appearance quality monitoring and analysis results of the product in augmented reality based on interaction with the digital twin and providing it to the user, thereby creating a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality. Claim 6 A manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality, characterized in that, in claim 4, the table of the product appearance quality monitoring and analysis results is data that displays the product appearance inspection result values ​​calculated by checking the appearance condition of the product produced from the actual factory equipment in the form of a table and a graph. Claim 7 In claim 3, the manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality is characterized by the quality prediction AI performing AI-based quality cause analysis and quality prediction services, and providing the user with a table of status analysis results by product and defect type of actual factory equipment matched with the equipment of the digital replica generated by monitoring the equipment of the digital replica displayed in the digital twin based on interaction with the digital twin. Claim 8 In claim 7, the AI-based quality cause analysis and quality prediction service is characterized by displaying a table of status analysis results by product and defect type of actual factory equipment that matches the equipment of the digital replica generated by the quality prediction AI in a form where the digital twin is overlapped over the equipment of the digital replica, and by the augmented reality providing means visualizing the table of status analysis results by product and defect type of defect of actual factory equipment in augmented reality based on interaction with the digital twin and providing it to the user, thereby creating a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality. Claim 9 A manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality, characterized in that, in claim 7, the results of the status analysis by product and defect type of the above-mentioned product are data including information on the number of good products, the number of defects, the production rate of products, the production time per unit product, and the number of productions according to the production conditions of the equipment of the above-mentioned actual factory. Claim 10 In claim 3, the process optimization AI performs an AI-based process optimization solution provision service, receives an optimal manufacturing process condition information query to search for or extract optimal manufacturing process condition setting values ​​from the digital twin based on interaction with the digital twin, and then performs a simulation in conjunction with the manufacturing process condition simulator to calculate optimal manufacturing process condition setting values ​​and a predicted defect rate or production volume according to the optimal manufacturing process based on the optimal manufacturing process condition information query, and the manufacturing-specialized AI agent-based interaction system provides the optimal manufacturing process condition setting values ​​and the predicted defect rate or production volume according to the optimal manufacturing process, which are the results of the simulation performed by the process optimization AI, to the user. Claim 11 In claim 10, the AI-based process optimization solution providing service is characterized in that the digital twin displays the results of a simulation performed by the process optimization AI in a form that overlaps on the equipment of a digital replica displayed in the digital twin based on interaction with the digital twin, and the augmented reality providing means visualizes the results of the simulation in augmented reality based on interaction with the digital twin and provides them to the user, thereby creating a manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality. Claim 12 A manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality, characterized in that, in claim 10, the optimal manufacturing process condition information query is an information query for setting values ​​such as the number of good products, number of defects, product production rate, production time per unit product, and production count information for each production condition applicable to the manufacturing process. Claim 13 A manufacturing-specialized AI agent-based interaction system utilizing multimodal and augmented reality, characterized in that, in claim 1, the manufacturing process-specialized small language model is constructed through manufacturing process specialization and Korean fine-tuning of the small language model, and the small language model is fine-tuned for manufacturing process specialization through at least one of Instruction Fine-tuning and PEFT (Parameter-Efficient Fine-Tuning).

Images