Method and system for monitoring a production process for a product

Abstract

The invention relates to a computer-implemented method for monitoring a production process for a product, comprising the following steps: a) recording (II) image data of the produced product comprising components; b) providing (RET) a material list containing the components of the produced product; c) encoding (I_ENC) the image data by applying a model on the basis of artificial intelligence and providing the encoded image data as image embedding; d) encoding (P_ENC) the material list by applying a provided model on the basis of artificial intelligence and providing the encoded material list as an input request into a basic model on the basis of artificial intelligence; e) decoding (M_DEC) the image encoding by means of the input request into the provided basic model; f) instance segmentation (SML) of the decoded image encoding and generating segmentation masks and segmentation identifiers for the components of the decoded image embedding; g) converting (CML) the masks and the identifiers for the components in accordance with the material list; and, h) checking (CHK) whether the converted masks and identifiers of the components correspond to the components in the material list, and outputting the result of the check.

Description

[0001] Method and system for monitoring a manufacturing process for a product

[0002] The invention relates to a computer-implemented method for monitoring a manufacturing process for a product and a system for this purpose.

[0003] Monitoring a manufacturing process typically involves the use of various technologies to monitor and control the production of products during the assembly or production phase.

[0004] It is important to ensure that the assembly process meets quality standards, adheres to specifications, and operates efficiently.

[0005] In the context of image processing for assembly monitoring, the state of the art focuses on the use of computer vision technologies to improve and automate monitoring tasks.

[0006] However, a common deficiency in the state of the art is that available computer vision systems for assembly monitoring have only a very limited adaptability to highly complex assembly processes, especially with different product designs or components.

[0007] Adaptability in complex assembly processes is often limited to the following approaches in the current state of the art:

[0008] • Hand-crafted functions and rules: Classical computer vision systems often rely on manually designed functions and rules to interpret visual data. Engineers identify specific patterns or features relevant to the assembly process and code them into the system. While this approach is effective for clearly defined scenarios, it lacks adaptability to changes or variations.

[0009] • Heuristic-based approaches: Heuristic methods involve the use of predefined rules or strategies based on domain knowledge. These rules guide the decision-making process of the image processing system in response to specific visual cues. While heuristics can be effective in controlled environments, they struggle to adapt to unforeseen fluctuations.

[0010] • Template matching: Template matching is a technique that compares predefined templates of objects or features with the observed visual data. While suitable for certain applications with consistent patterns, it may lack robustness when faced with variations or deviations from the expected templates.

[0011] • Rule-based systems: Classical machine vision often uses rule-based systems, where decision-making is guided by a set of predefined rules. These rules are typically created based on the expected behavior of the assembly process. However, adapting these rules to changes can be time-consuming and challenging.

[0012] • Manual calibration: Traditional image processing systems often require manual calibration to account for changes in lighting conditions, camera angles, or other environmental factors. This manual adjustment is a reactive approach and may not offer real-time adaptation to dynamic assembly processes.

[0013] It is important to note that classic computer vision systems may encounter limitations when dealing with the complexity and variability of modern manufacturing environments.

[0014] In contrast to advanced machine learning approaches, classical methods may require extensive manual intervention and may not be easily adapted to unforeseen changes in assembly processes.

[0015] The transition from classical computer vision to more adaptive and learning-based approaches represents a significant step forward in addressing the challenge of adaptability.

[0016] Publication US 2023 / 237642 A1 relates generally to the quality of electronic system resources and, in particular, to systems and methods for identifying and tracing components of electronic assemblies, the method comprising obtaining an image of a component of the electronic assembly, generating identification information of the component based on visual features of the component, authenticating the component based on the identification information, generating correlating information by assigning the component to an electronic assembly, and classifying the component into a group based on the visual features of the component.

[0017] In the publication PASUNURI ABINAI ET AL: "A Comparison of Neural Networks for PCB Component Segmentation", 2021 IEEE INTERNATIONAL SYMPOSIUM ON HARDWARE ORIENTED SECURITY AND TRUST (HOST), IEEE, December 12, 2021 (2021-12-12), pages 113-123, XP034084744, DOI: 10.1109 / HOST49136.2021.9702286, a method for semantic image segmentation, also known as pixel-level image labeling, is described, particularly for accurate image segmentation for electronic circuit boards.

[0018] The object of the invention is therefore to provide a method that is particularly easy to adapt to changes in product design and can therefore be adapted more easily and efficiently to a product to be manufactured, and which is also automatable.

[0019] The problem according to the invention is solved by a computer-implemented method for monitoring a manufacturing process for a product, comprising the steps: a) capturing image data of the manufactured product with components that define an image, b) providing a bill of materials with the components of the manufactured product, c) encoding the image data by applying a provided first model based on artificial intelligence and providing the encoded image data as an image embedding, d) encoding the bill of materials by applying a provided second model based on artificial intelligence and providing the encoded bill of materials as an input prompt to a basic model based on artificial intelligence, e) decoding the image encoding using the input prompt to the provided basic model, f) instance segmentation of the decoded image encodings.in which individual objects within the image are identified and delimited, and segmentation masks and segmentation markers are generated for the components of the decoded image embeddings, whereby the identification and delimitation of individual objects within the image is performed, g) conversion of the masks and markers for the components according to the bill of materials, h) checking whether the converted masks and markers of the components match the components of the bill of materials, and outputting the result of the check.

[0020] The adaptability of production monitoring to product variations is improved by integrating the retrieval and coding of the bill of materials as an input prompt.

[0021] This adaptability is a crucial advantage over traditional image processing systems, which often rely on fixed rules and struggle to handle changes in product designs or component configurations. Furthermore, the integration of advanced learning techniques, including instance segmentation and bill of materials comparison, enhances the system's ability to learn from data and dynamically adapt to complex assembly scenarios.

[0022] This integration surpasses classic image processing systems, which can rely on manual feature engineering and rule-based methods.

[0023] The segmentation or instance segmentation in step f) is carried out using a "promptable segmentation model".

[0024] The core concept of prompt segmentation utilizes the power of "Natural Language Processing" (NLP for short) and "Deep Learning" to segment images based on user-provided prompts.

[0025] Instead of training a model with a fixed set of object classes, they can define the target object using text descriptions or even an image as a reference.

[0026] This approach offers greater flexibility and allows for the segmentation of a wider range of objects without requiring extensive retraining for new datasets.

[0027] Additionally, generalizability improves because the models have the potential to learn and segment invisible object categories based on the provided prompts.

[0028] In an image-based input prompt, the user, manually or automatically, provides an image as a reference (e.g., an image of a correctly assembled component), and the model learns to identify objects similar to the reference image within the target image and generates a segmentation mask accordingly.

[0029] Similarly, segmentation indicators or segmentation markers can be generated.

[0030] The generation and training of the ML model provided in step d) can be done using appropriately labeled image data, which are marked, for example, as "good" and "bad" components.

[0031] It may be intended that different individual components are used for different products.

[0032] The machine learning model itself is an artificial intelligence-based model and can be described by a neural network in the form of model nodes and model weights. In mathematics, an embedding is an instance of a mathematical structure that is contained within another instance, for example, a group that is a subgroup.

[0033] Image embedding is a numerical representation of an image that encodes the semantics of the content in the image.

[0034] Embeddings are calculated by computer vision models, which are typically trained on large datasets of text and image pairs.

[0035] In other words, image embeddings are numerical representations of images that capture their semantic meaning and visual features and can be used, for example, for applications such as image search, image classification, image recovery, and image similarity.

[0036] The invention further provides that the segmentation process involves identifying and delimiting individual objects within an image.

[0037] This separates the product components for analysis and comparison, improving the flexibility and accuracy of manufacturing monitoring.

[0038] In step c) of the invention, the coding is to be carried out by applying a first model based on artificial intelligence.

[0039] The invention further provides in step d) to encode the bill of materials (BOM) using a second model based on artificial intelligence.

[0040] The invention also provides in step e) to decode the image encoding using the input prompt to the provided basic model IFM.

[0041] By applying three models based on artificial intelligence, the invention simplifies the generation of relevant data for monitoring the manufacturing process for the product and also allows for the achievement of higher quality results, since the models can react more dynamically and accurately to the data used than, for example, the use of static information regarding classification.

[0042] By using the individual models, it is possible to achieve a simpler way of encoding and decoding data or converting masks or identifiers than is known in the prior art.

[0043] It is clear that the aforementioned models must first be generated and trained with appropriate data, whereby the trained models can subsequently be used for several different products and manufacturing processes. Accordingly, in a further development of the invention, it is provided that the first model and the second model are pre-trained with valid and / or invalid image data or material data, respectively.

[0044] The basic model is a large learning model from the field of "computer vision", which is created and trained for various tasks and solves tasks posed by appropriate input prompts.

[0045] A prompt is a natural language instruction that instructs a large language model (LLM) to perform a task and is also referred to as instruction optimization.

[0046] The model follows, for example, the prompt to determine the structure and content of the text or image to be generated.

[0047] Prompt engineering is the process of creating and refining the prompt used by the model.

[0048] In a further development of the invention, it is provided that the objects are components of the manufactured product.

[0049] In a further development of the invention, it is provided that the model is a basic model.

[0050] It is assumed that special models, such as industrial foundation models (IFM), can be used particularly advantageously for this purpose.

[0051] Examples of a basic model include a large language model (LLM) or a so-called "vision foundation model" (VFM for short).

[0052] Therefore, it is particularly advantageous if the model provided in step d) is an IFM.

[0053] A basic or foundational model is a machine learning or deep learning model that is trained on broad data so that it can be applied in a wide variety of use cases.

[0054] Basic models are general-purpose technologies that can support a wide variety of use cases.

[0055] Creating fundamental models is often very resource-intensive. Therefore, it can be significantly more efficient to adapt an existing base model for a specific use case or even to use it directly.

[0056] The object of the invention is also solved by a system for monitoring a manufacturing process for a product which is described by a list of materials, comprising a production system which is set up to manufacture the product, and further comprising a monitoring device which has a sensor and a computing device with a processor and a memory, and the sensor which is set up to capture the product with image data, and the computing device which is set up to execute the method of the invention.

[0057] The invention will now be explained in more detail using an exemplary embodiment. The figures show in

[0058] Fig. 1 shows a block diagram for a system according to the invention,

[0059] Fig. 2 is a flowchart for an embodiment of the method according to the invention,

[0060] Fig. 3 is a flowchart with details for the embodiment of Fig. 2.

[0061] Fig. 1 shows a block diagram for a system S according to the invention for monitoring a manufacturing process for a product PR.

[0062] In this embodiment, product PR is a printed circuit board which is populated with components K.

[0063] The product PR is described by a material list (BOM).

[0064] The system comprises a production system MS, for example a device for assembling provided electronic circuit boards with designated electronic components K, and a monitoring device MON for monitoring and visual control of the assembly and connections made between a circuit board and the assembled components.

[0065] The MS production system is therefore set up to manufacture the product PR.

[0066] The MON monitoring device includes an imaging sensor CAM, such as a camera, and a computing device with a processor and memory.

[0067] However, the sensor can also be another imaging sensor, such as a radar sensor or an ultrasonic sensor, whereby the image resolution of the sensor can be high or low depending on the application.

[0068] The CAM sensor is set up to capture the product PR with image data IMG.

[0069] The computing device is designed to carry out the method according to the invention as explained below.

[0070] Fig. 2 shows a flowchart for an embodiment of the inventive method for monitoring a manufacturing process for a product PR, comprising the steps: a) Acquiring image data IMG of the manufactured product PR with components K, which define an image; b) Providing a bill of materials (BOM) with the components K of the manufactured product PR; c) Encoding the image data IMG by applying a provided first model based on artificial intelligence and providing the encoded image data as an image embedding IMG_EMB; d) Encoding the bill of materials (BOM) by applying a provided second model based on artificial intelligence and providing the encoded bill of materials as an input prompt to a basic model IFM based on artificial intelligence; e) Decoding the image encoding using the input prompt to the provided basic model IFM.f) Instance segmentation (SML) of the decoded image encodings, in which individual objects within the image are identified and delimited, and generation of segmentation masks and segmentation indicators for the components K of the decoded image embeddings (IMG_EMB), whereby the identification and delimitation of individual objects within the image is performed; g) Conversion of the masks and indicators for the components K to CML according to the bill of materials (BOM); h) CHK check whether the converted masks and indicators of the components K match the components K of the BOM, and output of the result of the check.

[0071] One or more steps of the process are computer-implemented.

[0072] Instance segmentation involves identifying and separating individual objects within an image, where the objects are components K of the manufactured product PR and the image is defined by the image data IMG.

[0073] The model used is a provided basic model IFM (industrial foundation model).

[0074] Fig. 3 shows a flowchart with details for the embodiment of the preceding figure with an assembly monitoring system AM.

[0075] The statements regarding the preceding figures also apply.

[0076] In step a), an image (IMG) of the assembled printed circuit board (PR) is captured, and this image (IMG) serves as input data in the subsequent steps. In step b), the bill of materials (BOM) for product PR with components K is provided and converted into an input prompt by a BOM2P conversion.

[0077] The bill of materials (BOM) is a list of the components K and their quantities required for the assembly or product PR.

[0078] By retrieving the specific bill of materials relevant to the current assembly step, it is ensured that the AI ​​system has the correct reference for the subsequent comparison with the captured image data.

[0079] In step c), the image data IMG is encoded by applying a KL model.

[0080] In step d), the retrieved bill of materials (BOM) is encoded using the Kl model into a format that serves as an input prompt for the Kl system.

[0081] This coding helps the system understand the expected components and all other relevant details that are essential for subsequent analysis.

[0082] In step e), the image encoding is decoded M_DEC using the Kl model.

[0083] In step f), the instance segmentation is performed.

[0084] Instance segmentation is a computer vision task in which individual objects within an image are identified and separated.

[0085] As part of monitoring the printed circuit board assembly, this step aims to precisely locate and outline each component on the circuit board and to distinguish them from one another.

[0086] In step g), the masks and the labels are converted to CML.

[0087] The instance segmentation process generates masks and labels that correspond to each identified component on the circuit board.

[0088] These masks and labels are then converted into a format aligned with the bill of materials, creating a representation of the assembly as it is checked by the image processing system.

[0089] In step h), the system S compares the tested assembly PR, which was derived from the segmentation results, with the bill of materials BOM from the PLM system.

[0090] The system checks for discrepancies such as missing components, incorrect placements, or deviations from the expected assembly configuration. Essentially, this step verifies whether the PR assembly meets the specified requirements.

[0091] This is done by comparing the material list COMP_BOM with the image data IMG captured by the camera inspection COMPJNSP.

[0092] Optionally, in step i) the result can be written to a memory of a quality management system (QMS) and / or to a memory of the production system (MS), for example the result of the assembly inspection, whether it confirms correctness or detects discrepancies.

[0093] Storage in a memory serves to record and document the manufacturing process for the product PR, in order to ensure traceability of errors or defects and to provide valuable data for process improvement and analysis.

[0094] In summary, these steps outline a comprehensive process for monitoring printed circuit board assembly using computer vision.

[0095] The integration of bill of materials information, instance segmentation and comparison with expected assembly configurations enables automated quality control and increases the efficiency of the manufacturing process.

[0096] The final step, writing the results into the quality management system (QMS), ensures that the insights from the image processing system contribute to general quality assurance and continuous improvement.

[0097] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included.

[0098] Reference symbol list

[0099] AM assembly monitoring (English: “quality assembly monitoring”)

[0100] BOM (Bill of Materials),

[0101] BOM2P conversion of the bill of materials (English: “conversion bill of material”),

[0102] Conversion to command prompt

[0103] CAM camera

[0104] CHK Check of the assembly (English: "check assembly")

[0105] CML conversion of masks and labels

[0106] COMP_BOM Comparison of the bill of materials (English: "compare bill of material")

[0107] COMPJNSP Comparison of inspection (English: "compare inspection")

[0108] II Image capture (“ingest image”) l_ENC Image encoder (“image encoder”)

[0109] IFM industrial foundation model

[0110] IMG image (English "image")

[0111] IMG_EMB Image embedding

[0112] K components of the product

[0113] M_DEC mask decoder

[0114] M2BOM conversion of masks to bill of materials

[0115] MON monitoring device

[0116] MS production system (English: “manufacturing system”), production device

[0117] P_ENC Prompt Encoder

[0118] PR product

[0119] QMS Quality Management System (English: "quality management system")

[0120] RET Provision

[0121] S System

[0122] SML segmentation of masks and labels (English: "segmentation masks and labels"), English: "perform instance segmentation"

[0123] WR data backup to storage

Claims

Patent claims 1. A computer-implemented method for monitoring a manufacturing process for a product (PR), comprising the steps of: a) capturing (II) image data (IMG) of the manufactured product (PR) with components (K) defining an image, b) providing (RET) a bill of materials (BOM) with the components (K) of the manufactured product (PR), c) encoding (l_ENC) the image data (IMG) by applying a provided first model based on artificial intelligence and providing the encoded image data as an image embedding (IMG_EMB), d) encoding (P_ENC) the bill of materials (BOM) by applying a provided second model based on artificial intelligence and providing the encoded bill of materials as an input prompt to an AI-based basic model (IFM), e) decoding (M_DEC) the image encoding using the input prompt to the provided basic model (IFM), f) instance segmentation (SML) of the decoded Image encodings,in which individual objects within the image are identified and delimited, and segmentation masks and segmentation indicators are generated for the components (K) of the decoded image embeddings (IMG_EMB), whereby the identification and delimitation of individual objects within the image is performed, g) conversion (CML) of the masks and indicators for the components (K) according to the bill of materials (BOM), h) check (CHK) whether the converted masks and indicators of the components (K) match the components (K) of the bill of materials (BOM), and output of the result of the check.

2. Method according to the preceding claim, wherein the objects are components of the manufactured product.

3. Method according to one of the preceding claims, wherein the first model and the second model are pre-trained by valid and / or invalid image data or material data.

4. A system for monitoring a manufacturing process for a product (PR) described by a bill of materials (BOM), comprising a production system (MS) set up to manufacture the product (PR), and further comprising a Monitoring device (MON) comprising a sensor (CAM) and a computing device with a processor and a memory, and the sensor (CAM) being configured to capture the product (PR) with image data (IMG), and the computing device being configured to perform the method according to one of the preceding claims.

Images