Computer vision guided manufacturing

Abstract

A system for generating and managing a manufacturing file, comprising a rule-based user interface for engineering procedures, a manual validation module for BOM and assembly validation, a checklist module for technician guidance, and a hybrid model with Al and machine vision for automating instruction validation across all production phases.

Description

COMPUTER VISION GUIDED MANUFACTURINGFIELD OF THE INVENTION

[0001] The present invention relates in general to the field of guided manufacturing and maintenance process, and in particular to generation of manufacturing and maintenance processes using machine vision and machine learning.BACKGROUND

[0002] The field of manufacturing and maintenance processes involves complex procedures necessitating precise management of numerous activities, including assembly, fault isolation, rework, calibration, testing, and checklist compilation. These processes require continuous updating, authorization, testing, and modification by various organizational personnel. Given the geographic dispersion and utilization of divergent systems, the task is further complicated by the need for consistency and systematic validation across diverse operational phases, often spanning months or years.

[0003] Existing systems employ manual creation, validation, and guidance of engineering instructions and are prone to errors, while the processes still rely on significant human intervention for validation and verification, underscoring the limitations in current methodologies.

[0004] The intricacies of generating and maintaining up-to-date manufacturing files have proven to be substantial challenges under existing systems, particularly in environments characterized by rapid technological advancements and evolving product designs. Current solutions fail to adequately address the need for automated and real-time validation and verification, resulting in suboptimal first-time yield rates and frequent need for rework. Consequently, there is a pressing requirement for a more robust system that minimizes human error and enhances the consistency and accuracy of manufacturing documentation throughout the product lifecycle.

[0005] What is needed is an advanced system that leverages artificial intelligence and computer vision to automate the validation and verification processes across all phases of manufacturing and maintenance. Such a system would reduce human error, enhance thestandardization of engineering instructions, streamline the updating and approval processes, and ensure real-time verification of assembly and maintenance activities. This enhancement would significantly improve first-time yields, minimize rework, and elevate the overall quality and efficiency of manufacturing and maintenance operations.SUMMARY

[0006] In one aspect, a system is provided for generating and managing a manufacturing file comprising a plurality of standard engineering instructions. The system includes a rule-based user interface configured to guide engineers and documentors in the creation of a production tree and step-by-step varied engineering procedures. Additionally, the system includes a manual validation module configured to guide assembly workers on a production line through the validation of a bill of materials (BOM) components and step-by-step assembly results. Further, the system incorporates a checklist module to guide technicians through single-step checklist manual validation processes, and a hybrid model comprising artificial intelligence (Al) and machine vision algorithms, which automates the validation and verification of engineering instructions at all phases from engineering to production to maintenance.

[0007] In another aspect, the system's rule-based user interface features a graphical user interface (GUI) that allows for the display of engineering procedures and includes options for manual input and adjustments. This user interface is designed to improve interaction with the engineering and documentation processes, ensuring that all instructions are accurately captured and easily modifiable.

[0008] One object of the system and method is to enhance the consistency and reliability of manufacturing files by employing advanced Al and machine vision technologies. These technologies help in identifying and flagging potential errors within the engineering instructions, ensuring that discrepancies are caught and corrected early.

[0009] In an embodiment, the manual validation module of the system provides real-time feedback to assembly workers on BOM component validation and assembly results. This feedback mechanism helps in reducing the errors and rework typically associated with manual assembly processes. Similarly, the checklist module presents a series of verification steps that must be acknowledged and validated by the technician, ensuring thorough and systematic validation of each step in the production process.

[0010] One object of the system is to streamline the manufacturing documentation process by integrating Al-driven automation with manual oversight, thereby reducing the time and effort required for the generation and maintenance of manufacturing files. The hybrid model, which combines Al and machine vision algorithms, not only identifies discrepancies but also provides real-time verification of assembly results, thus enhancing the overall accuracy and efficiency of the manufacturing process.

[0011] In an embodiment, the system further comprises a machine vision module that captures images of manufacturing components during various production processes. These images are analyzed by a machine learning module that identifies patterns or defects, thereby facilitating real-time fault detection and diagnosis. The analyzed data and images are stored in a database, which is accessible for continuous learning and pattern recognition improvements.

[0012] In yet another aspect, the system includes a user interface that provides graphical representations of captured images, analyzed data, and identified patterns or defects. This interface enables users to manually oversee and adjust the engineering instructions based on real-time data, thus ensuring high standards of quality and consistency in the production process. The system may also include a communication interface to facilitate the seamless transmission and receipt of data related to manufacturing processes among various modules.

[0013] These aspects collectively offer a comprehensive solution for managing and optimizing manufacturing and maintenance processes, leveraging advanced Al and machine vision technologies to improve accuracy, efficiency, and overall quality.BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Fig. 1 is a flowchart depicting part identification computer vision deep neural network training and usage.

[0015] Fig. 2 is a block diagram illustrating the workflow of a system for generating and managing manufacturing processes, showing the communication and functional roles between various servers, vision modules, and users including project managers, documentors, line planners, line managers, runners, and technicians.

[0016] Fig. 3 is a flowchart illustrating the artificial intelligence algorithm and database interface with main users including a project manager, documentor, line planner, line manager, runner, and technician, detailing their interactions with various system modules and functions.

[0017] Fig. 4 is a block diagram illustrating the architecture of a classification and segmentation network, detailing the flow of input points through various transformations, multi-layer perceptrons (MLPs), pooling operations, and feature extraction modules to produce output scores.

[0018] Fig. 5 shows a series of schematic diagrams labeled (a), (b), and (c), each depicting different configurations of an X-Conv layer-based neural network architecture optimized for various parameters including the number of neurons (N), channels (C), filters (K), and dimensions (D), with associated loss functions at the output.

[0019] Fig. 6 shows a detailed illustration of an X-Conv operator algorithm along with a graphical representation of its application in both regular and irregular data structures.

[0020] Fig. 7 is a graph showing a normalized confusion matrix for various engineering components detected by a system, including bearings, bushings, clamps, and gears.

[0021] Fig. 8 shows a side-by-side comparison of ground truth and prediction images for various parts in a manufacturing system, highlighting the accuracy of component identification.

[0022] Fig. 9 is a block diagram depicting the process flow involving 3D CAD models, camera images, and a hybrid model for class label generation.

[0023] Fig. 10 shows that most false predictions, as seen in Table 2c, are identified as background.DETAILED DESCRIPTIONItem Classification

[0024] Manufacturing relies on knowledge specializing in many different fields. The present invention employs computer vision for the purpose of image classification and identifying mechanical parts that are required for assembly in production lines. The system facilitates the automated classification of single parts, composite parts, materials, tools, and jigs using an integrated approach that leverages convolutional neural networks (CNNs). Central to this classification are hybrid models that incorporate several CNN architectures, including PointNet and PointCNN for 3D CAD models, and CADNet and YOLO for 2D images and real-world captures. This multi-faceted approach allows for a robust classification process by integrating and weighing the outputs from various CNNs to ensure high accuracy and robust identification of mechanical parts utilized in manufacturing processes.

[0025] PointNet and PointCNN operate primarily to manage the 3D CAD models, providing classification capabilities by analyzing the three-dimensional structure and geometric features. These networks are adept at handling point cloud data and identifying subtle differences among similar parts, which is crucial for generating accurate engineering instructions. On the other hand, CADNet and YOLO are optimized for 2D images, providing a complementary perspective by processing real-world photographs and blueprints. YOLO's strength in object detection in real-time scenarios enhances the system's ability to function effectively on the production floor.

[0026] The synthesis of these models results in comprehensive and reliable part identification. The combined output from these neural networks is processed through a weighting algorithm that evaluates the confidence levels of each classification. The "winner takes all" mechanism is employed here, where the highest probability classification is chosen, ensuring that only the most reliable identifications are used in generating manufacturing instructions. This method mitigates the risk of errors in parts identification, which is critical to maintaining process integrity.

[0027] Additionally, the system's classification capabilities extend beyond single parts to include composite parts, materials, tools, and jigs, providing a unified solution for the entirety of the manufacturing ecosystem. The classifiers are trained on diverse datasets encompassing various shapes, sizes, and properties, enabling the system to adapt to a wide range of manufacturing contexts. This adaptability is crucial to ensuring that the system can handle different types of components irrespective of their complexity or specific application requirements.

[0028] The automated classification system represents a significant improvement over manual methods, enhancing efficiency, reducing errors, and standardizing the process of generating engineering instructions. By combining advanced computer vision techniques with deep learning algorithms, the system provides a scalable solution that supports the evolving needs of manufacturing industries. The integration of multiple CNN models ensures that all aspects of part identification are addressed, promoting higher accuracy and reliability in manufacturing processes, thereby contributing to improved first-time yield rates and reducing the need for rework.Model Database Classe Net Train Train Vai Vai Vai Test Comment size loss acc loss acc acc acc per cisPointCNN CADNet 43 32*32 0.15 0.94 0.63 0.9 0.84 0.83PointCNN MCB B 25 32*32 1.1 0.65 1.12 0.68 0.61 0.64CADNet CADNet 30 32*32 0.1 0.95 0.54 0.91(MCB B cat)CADNet CADNet 30 32*64 0.21 0.9 0.6 0.85(MCB B cat)CADNet MCB B 25 32*32 0.25 0.87 Nebius cloudYOLO MCB B 25 200 0.5 0.9 0.4 0.95

[0029] Table 1. shows part identification networks meta-parameters and performance. The table details some of the main deep neural networks used by our system to classify mechanical parts, the differences between the networks and the main performance criteria (validation acc - % of parts classified accurately), assigning each of the best performing networks to the corresponding task. Network size is number of layers times (if relevant) width of the first layer (number of convolution filters). Loss is calculated by distance (in various metrics) to classifying all database entries at 100% probability, i.e. not just getting highest probability in the correct class for each item, but getting 0% probability in every other class. Accuracy is calculated as the number of correct classifications divided by all classifications to one class (correct + false positives).

[0030] Fig. 1 discloses an overview of part identification computer vision deep neural network training and usage. Each different network model (blue) is trained on a different (sometimes overlapping) 3D CAD model or image database (green), fine-tuned, tested on different datasets, then put to use in various part identification tasks (orange), set by corresponding formats and optimal performance.

[0031] Fig. 1 shows a high-level workflow of a system for generating and managing manufacturing processes, particularly focused on the generation, fine-tuning, testing, and usage phases of deep learning models for image classification and assembly verification. The system initiates with data input from several sources under the "Data" section. These sources include 3D models, specifically from the MCB with 43 classes and from the CADNet DB with 25 classes, as well as inputs from an image grabber.

[0032] The "3D Models" are subdivided into different file formats, such as *.obj, *.stl, and *.stl, which are fed into CNN models. This process starts with "PointCNN" and "PointNet" for 3D models, and "CADNet" and "YOLOv" for CAD models and real images. The "CNN Model" section details these respective models— PointCNN, PointNet, CADNet, and YOLOv— that process the input data; models PointCNN and PointNet handle "30 conjoined classes and 26 views" from 3D model inputs.

[0033] Continuing, the "Fine tuning" phase encompasses processes such as "Pruning Uninformative Views" and "Pruning Non-engineering object classes" which help refine the trained models by eliminating irrelevant or non-informative data points. The subsequent "Testing" phase incorporates additional data like models from other databases (indicated as "Other DB"), "Extracted images from instructions," and "Self-documented photos." During this phase, the system verifies the model performance against real-world examples captured by an "Image Grabber."

[0034] Lastly, the "Usage" section demonstrates various practical applications of the refined models. This includes generating "3D Models from instructions," creating "Reconstructed 3D Models," utilizing "Extracted images," and employing "Assembly-line taken photos" for real-time verification and validation in manufacturing settings. These applications ensure that the automated system effectively supports manufacturing processes by providing accurate visual inspections and instructions, significantly improving the overall assembly and verification tasks.

[0035] In one embodiment, PointNet and PointCNN are configured to process 3D CAD models of mechanical parts by extracting and analyzing geometric features from point cloud data. These features include surface curvature, edge sharpness, and volume, which are used to differentiate between parts with similar appearances. This analysis is critical for generating precise engineering instructions that ensure proper part placement and orientation during assembly.

[0036] In another embodiment, CADNet and YOLO are integrated to handle 2D images in different contexts, such as real-world photographs and technical blueprints. CADNet is capableof extracting detailed features from blueprint drawings, such as line thickness, angles, and dimensions, while YOLO excels in real-time object detection from camera feeds on the production floor. The combination of these two models allows for comprehensive image data processing, facilitating accurate part identification even in diverse lighting and environmental conditions.

[0037] In yet another embodiment, the system employs a weighting algorithm to synthesize the outputs from PointNet, PointCNN, CADNet, and YOLO. Each model's output is assigned a confidence level based on its performance and relevance to the specific type of data being analyzed. The weighting algorithm evaluates these confidence levels and employs a "winner takes all" approach, selecting the classification with the highest probability. This ensures that the system leverages the strengths of each CNN model to achieve the most reliable part identification results.

[0038] In a further embodiment, the training datasets for the classifiers include a wide range of parts, composite parts, materials, tools, and jigs. These datasets are curated to encompass various shapes, sizes, textures, and functionalities, allowing the classifiers to learn and adapt to different manufacturing scenarios. This extensive training enables the system to maintain high accuracy in identifying components across multiple industries and applications, ensuring that the classification process remains robust and reliable.

[0039] In an additional embodiment, the automated classification system is integrated with a manufacturing execution system (MES), allowing for real-time updates and feedback on parts identification. This integration enables dynamic adjustments in the production line, such as reordering parts supply or modifying assembly instructions based on updated part classifications. Consequently, the system enhances overall manufacturing efficiency by reducing delays and preventing errors associated with incorrect part identification.

[0040] Two models run classification specifically on 3D CAD models PointNet and PointCNN as shown in Figs. 4-6. They both access the 3D model data in form of a point cloud, whereas other networks only have access to flattened 2D views (images) of the model from various angles. Point cloud representation keeps 3D model data by location of all edge points in 3D, instead of all adjacent points on the same straight lines, which may simplify data storage and loading (and thus learning) of mechanical parts because of their many straight lines.

[0041] The PointCNN classification neural network incorporates X-convolution to enhance accuracy and generalization. This technique reduces the dimensionality of 3D CAD model datathat will be incorporated into the WizGen, akin to the conventional 2D convolution used in image processing. It achieves this by substituting cloud points with mean locations of the original adjacent points, a process similar to 2D convolution's averaging of adjacent pixel values in images.

[0042] However, it's important to note that the specialized data format required for this network may limit its application for live image identification, unless a comprehensive 3D reconstruction is generated from a revolving camera / platform (that we intend to use in task 3 for the step by step result validation). This highlights a constraint in the network's utility, emphasizing the need for a pre-existing 3D representation for effective operation.

[0043] Fig. 4 shows a schematic diagram of a combined classification and segmentation network designed for processing input point clouds. Fig. 4 illustrates the flow of data through two separate networks: Classification Network and Segmentation Network.

[0044] The Classification Network begins with an input transform applied to input points in the form of a 3x3 matrix multiplication utilizing the T-Net module. The transformed points are passed through a series of multi-layer perceptrons (MLP), specifically denoted as mlp (64,64). Subsequently, the transformed data follows a feature transform pathway, which involves another T-Net module with 64x64 transforms. These features, now in nx64 format, are further processed through another set of MLP layers identified as mlp (64,128,1024). Afterward, a max pooling operation aggregates the data into a global feature of 1024 dimensions. This global feature is then passed to the final MLP layers, expressed as mlp (512, 256, k), producing output scores.

[0045] The Segmentation Network utilizes point features propagated in a similar fashion, initially with nxl088 features which pass through consecutive MLP layers referred to as mlp (512,256). These features are shared and input into additional MLP layers called mlp (128, m). The segmentation network finally outputs nxm output scores.

[0046] Turning to Fig. 5, it illustrates three variations (a, b, and c) of a convolutional neural network architecture that leverages X-Conv layers for feature extraction and processing.

[0047] Fig. 5a displays a simple two-layer X-Conv architecture. The first X-Conv layer is specified as X-Conv(N=4, C=C1, K=4) and the second layer is X-Conv(N=l, C=C2, K=4). Both layers have outgoing connections leading to fully connected layers (FCs) which culminate in a loss function for training purposes.

[0048] Fig. 5b shows a more complex architecture with three X-Conv layers specified as X- Conv(N=7, C=C1, K=4), X-Conv(N=4, C=C2, K=4, D=2), and X-Conv(N=4, C=C2, K=4). Each of these layers is connected, propagating the processed features through multiple paths before leading them to a series of FCs and a loss function.

[0049] Fig. 5c depicts the most intricate configuration with four sequential X-Conv layers characterized as X-Conv(N=7, C=C1, K=4), X-Conv(N=4, C=C2, K=4, D=2), X-Conv(N=7, C=C3, K=3), and X-Conv(N=10, C=C4, K=3). The architecture integrates multiple inter-layer connections signifying a deep, multi-path network flow. The outputs from each layer converge towards FCs and ultimately the loss function.

[0050] Turning to Fig. 6, it elaborates on the X-Conv operator algorithm for convolutional neural networks. The algorithm is outlined in both procedural steps and visual representation.

[0051] The upper section details the input parameters, which include K, p, P, and F, and provides an output Fp. The procedural steps involve computing P' by translating P relative to p, then calculating F6 via MLP6(P'). These features are concatenated (denoted as F*) and processed through MLP to derive X. The features Fx are then computed as the product of X and F*. Finally, typical convolution (Conv) between K and Fx yields Fp.

[0052] The lower visual schematic delineates an example illustrating this convolution process. From Fl features, X-Conv maps Fl to different representations, indexed as F2 through Conv operations, showcasing feature transformation and aggregation. The result, displayed as {F2, Conv], emphasizes the hierarchical mode of operation in computing localized features, culminating in the spatial convolution X-Conv mechanism.Multi-model based database for general model training

[0053] In some embodiments, a unified database of mechanical parts in image and 3D-model formats is created, using available online database sources, together with images (photographs, graphics, videos) from across the internet, images extracted from assembly instruction documents and captured first-hand by camera. Merging these data sources requires captioning, comparing categories labels and physical properties, and converting all items to a uniform format compatible with the image classification algorithm at hand and allowing high performance. Four different deep neural network designs were identified for mechanical part classification each requiring a different format: natural (or background / floor detached) images,various 3D-model formats, with or without a frame wire, 2D-view images produced by a custom algorithm from the 3D-models, and reduced-dimension point cloud compression of the models.

[0054] It is important to is to identify Bill of Materials (BOM) mechanical parts and tools specified for a particular assembly. The system incorporates a multi-model based database designed to facilitate general model training, leveraging the integration of diverse convolutional neural network (CNN) architectures to enhance the classification and identification accuracy for manufacturing components. This database stores the output from multiple CNN models, such as PointNet, PointCNN, CADNet, and YOLO, each specializing in different dimensional data perspectives, including 3D CAD models and 2D images. The multi-faceted dataset enables the training models to account for a spectrum of geometric and visual information, thereby enabling comprehensive analyses and robust identification of various parts, materials, tools, and jigs.

[0055] The PointNet and PointCNN models are specifically designed to manage and classify 3D CAD model data by analyzing the structural and geometric features inherent to point cloud data. These models excel in recognizing intricate details necessary for distinguishing subtly different components, which is essential for generating accurate engineering instructions. Conversely, CADNet and YOLO focus on processing 2D images and real-world captures, with YOLO optimized for real-time object detection. This allows the system to seamlessly identify and classify components on the production floor, incorporating real-time operational datasets.

[0056] The database aggregates outputs from these CNN models, applying a weighting algorithm to evaluate the confidence levels of each classification. Utilizing a "winner takes all" mechanism, the system ensures the highest probability classifications are selected, thus minimizing errors in parts identification. This method streamlines the generation of engineering instructions and maintains process integrity by leveraging the strengths of each CNN model, effectively integrating various perspectives to produce reliable identification outputs. This not only augments the system's overall accuracy but also ensures its scalability across diverse manufacturing scenarios.

[0057] Furthermore, the database supports continuous learning through the storage of analyzed data and images, facilitating ongoing improvements in pattern recognition and classification accuracy. This continuous learning capability is enhanced by the system's ability to import and integrate data from existing manufacturing documents and processes, providing a comprehensive dataset for model training. By enabling the assimilation of historical data and real-time captures, the database contributes to optimizing yields and reducing rework, therebyenhancing the overall quality and efficiency of manufacturing operations. The multi-model approach ensures that the system can adapt and remain effective in the face of evolving manufacturing complexities and technological advancements.Selecting most informative views for step image and step result generation

[0058] The WIZ-GEN ZERO view extraction algorithm of the invention ensures automatic generation of optimal images from 3D CAD models. It not only enhances part identification but also provides users with accessible images through the WIZ-GEN image tool of the invention. The process of selecting the most informative views for step image and step result generation involves a systematic algorithm designed to optimize the visual data utilized in manufacturing and maintenance procedures. This selection process begins with the thorough analysis of 3D CAD models to identify key geometric features and spatial relationships that are critical for part identification and assembly. The algorithm captures multiple angles and perspectives of the 3D models to ensure comprehensive visual representation. Each view is evaluated based on its potential to convey essential information about the part's structure, orientation, and assembly requirements. By prioritizing views that offer the highest diagnostic utility, the system enhances the clarity and accuracy of the manufacturing instructions generated from these images.

[0059] The WIZ-GEN ZERO view extraction algorithm employs a set of criteria to discard irrelevant or redundant perspectives, focusing on those that provide optimal clarity. These criteria include the visibility of assembly points, facial features, and mechanical joints that are pivotal during the assembly process. The algorithm integrates principles from computer vision to automatically detect and rank views according to their informativeness. This procedure ensures that the views selected for step images and step results are those that most effectively support the assembly workers and technicians in understanding and executing the instructions. By standardizing the view selection process, the system minimizes ambiguity and enhances the consistency of the visual instructions provided.

[0060] In addition to geometric analysis, the system incorporates feedback mechanisms to continuously refine the view selection process. Data from previous assembly tasks, including error rates and user feedback, are analyzed to identify which views were most effective in aiding assembly and which were less helpful. This input is used to adjust the algorithm's parameters,improving its ability to select the most informative views over time. The ongoing refinement process ensures that the system adapts to different types of components and assembly scenarios, maintaining high levels of accuracy and utility across various manufacturing contexts. This adaptability is crucial in environments where product designs and assembly processes are continuously evolving.

[0061] Moreover, the system's capability to generate and select part views is augmented by the use of deep learning models that predict the visual needs of assembly sequences. These models are trained on extensive datasets comprising various parts, assemblies, and their corresponding effective views. Using these datasets, the models learn to identify patterns and features that indicate the most informative angles. This predictive capability enables the system to anticipate the visual requirements of subsequent assembly steps, ensuring that the step images and step results not only support the current task but also provide continuity and foresight for the entire assembly process. This holistic approach to view selection significantly enhances the overall efficiency and accuracy of manufacturing instructions.

[0062] Finally, the system ensures that the selected views are seamlessly integrated into the step image and step result generation process. High-resolution images of the selected views are captured and annotated with relevant information such as part numbers, tool requirements, and procedural steps. These annotated images form the basis of the engineering instructions presented to the assembly workers and technicians. By automating the integration and annotation of these views, the system streamlines the documentation process, reducing the manual effort required and minimizing the risk of human error. The resultant visual instructions are clear, precise, and comprehensive, greatly facilitating the effective implementation of manufacturing and maintenance tasks.Automatic labelling of pictures or views, of tools, materials and jigs

[0063] One of the system's goals involves importing materials, for example, into the WizSuite of the invention, for use in the WizGen of the invention and for training the systems' algorithms on actual client real data. The system then proceeds to automatically classify and label uploaded pictures and models (or generated views) of products, parts, tools, materials and even jigs. Next, the system proceeds to connect images with descriptions available in manually created instructions and particularly in the BOM.

[0064] Automatic labeling of pictures or views, of tools, materials, and jigs involves a systematized process that integrates various technologies to enhance the identification and annotation of visual data within a manufacturing context. The system begins by importing material data into the WizSuite of the invention, which encompasses diverse formats such as images, 3D models, and real-time video captures. Using deep learning models, particularly convolutional neural networks (CNNs) such as PointCNN, PointNet, YOLO, and CADNet, the system automates the classification and labeling tasks. This integration of models allows for multi-dimensional data analysis, leveraging the strengths of each neural network to provide comprehensive and accurate labeling of components such as mechanical parts, assembly tools, required materials, and assembly jigs.

[0065] Upon importing the data, the system employs advanced image processing techniques to extract relevant features and descriptors from the visual inputs. The extracted features are then processed using pre-trained deep learning models optimized for various classification tasks. For instance, PointCNN and PointNet are used for three-dimensional CAD model analysis, focusing on geometric and spatial features essential for identifying unique parts. In contrast, CADNet and YOLO are tailored for two-dimensional image processing, facilitating the detection and labeling of objects in real-world scenarios. This hybrid approach ensures that the system can effectively manage both 2D images and 3D models, enhancing its labeling accuracy and applicability across different manufacturing environments.

[0066] The automatic labeling system further refines its efficiency through iterative learning processes. The system analyzes the initial classification outputs, compares them to existing labels and descriptions in the manually created instructions, such as the Bill of Materials (BOM), and aligns them for consistency. This comparison is achieved using a set of heuristic algorithms that match visual features to text descriptors, ensuring that the labeled images correspond accurately to the expected components. The system then updates its training datasets and recalibrates its neural network weights based on feedback from these comparisons, continuously improving its accuracy and reliability in subsequent labeling tasks.

[0067] Deep convolutional neural network designs tested, and conjointly classifying images are PointCNN, PointNet, YOLO and CADNet. Each design was tested and used to classify data of slightly different types and formats on which it is expected to perform best. Several architecture and meta parameter options were examined for maximal performance, making use of the adequate hardware and software packages for realistic timeline training and inference. Allnetworks converged with accuracies above a set threshold of 90% in a matter of hours to a few days, running learning algorithms on standalone GPUs. CADNet was latter trained in cloud computing solutions on additional datasets used for the other networks and more realistic images, requiring conversion of 3D models, point clouds and RGB images into the suitable resolution and angular viewpoints grayscale images. These procedures assured adequate classification performance and accuracy of a single trained network realization on several data types, sources and formats. The ultimate persistent and robust mechanical part classification solution is, however, the joint inference of the various trained deep neural networks capable of identifying each mechanical part type and format. Le., a "winner takes al l"-type algorithm compares both classification confidence (distance between scores of the best class and all others) and prior tested accuracy of each network in the specific class. This multisource integrated classification method provides maximum accuracy and robustness with resistance to biases.

[0068] The algorithm to select the most probable class identification for a mechanical part calculates scores, as following. Prior to choosing the predicted class, the inference network run computes scores for each output unit in the final layer, i.e. each class. The chosen output is taken by simple maximum, but for computing probabilities the scores are normalized. The system takes each class probability and applies a weight to it, for example, by multiplying it with the average accuracy measured for that class during network testing. This weighing allows preference between network models, since their accuracy rates differ. Then the class for which this multiplication is highest, across network models, is the hybrid winner-takes-all. The equivalent mathematical representation follows: [MATH FORMULA]

[0069] In order to integrate the labeled data effectively into the WizSuite, the system automates the generation of comprehensive annotations for each identified component. These annotations include detailed descriptions, part numbers, tool specifications, and procedural steps, which are essential for creating cohesive manufacturing instructions. The system utilizes natural language processing (NLP) techniques to generate contextually relevant text that complements the visual data, ensuring that all necessary information is included in the final documentation. This automated annotation process significantly reduces the manual effort required from documentation teams and minimizes the risk of human error, leading to more reliable and consistent engineering instructions.

[0070] Finally, the automatic labeling system is designed to operate in real-time, allowing for the continuous updating and validation of visual data as new components are introduced into the manufacturing process. The system employs robust communication interfaces to transmit and receive data among various modules, ensuring seamless integration with existing workflows. By providing real-time feedback and allowing for dynamic reclassification based on the latest data, the system enhances the agility and responsiveness of the manufacturing operations. This capability ensures that the visual documentation remains current and reflects accurately the on-ground realities of the production environment, thus facilitating more efficient and effective manufacturing processes.

[0071] Fig 3. shows an embodiment of an interface of artificial intelligence algorithms with the WizSuite and its various users. This diagram relates the WizSuite+ML architecture to the artificial intelligence algorithms performing the task (green) and illustrates the required database, model or tree structure each algorithm uses for training to optimize its performance on the corresponding task.

[0072] In an embodiment, the interface depicted in Fig. 3 integrates a series of artificial intelligence algorithms with the WizSuite platform. The figure outlines how these algorithms interface with various users through the WizSuite+ML architecture. Each artificial intelligence algorithm is represented in green, indicating its role in executing a specific task within the system.

[0073] In another embodiment, Fig. 3 illustrates that each algorithm accesses and utilizes distinct data sources, such as databases, models, or tree structures, for training purposes. This training process is necessary to optimize the performance of the algorithms on their respective tasks. For instance, the algorithms may make use of historical data stored in a database to predict future trends or employ decision tree models to categorize user inputs efficiently.

[0074] In yet another embodiment, the depicted architecture includes multiple modules that interact with the artificial intelligence algorithms. These modules may include, but are not limited to, data preprocessing units, feature extraction components, and performance evaluation tools, each designed to enhance the overall functionality and accuracy of the algorithms. The data preprocessing units may be responsible for cleaning and organizing raw input data, while feature extraction components identify and isolate relevant characteristics from the data. Performance evaluation tools, in turn, may assess the algorithms' efficacy, providing feedback for further optimization.

[0075] In a further embodiment, the interface may support interaction with a plurality of different user types, such as administrators, data scientists, and end-users. Each user type may access specific functionalities tailored to their needs. For example, administrators may manage system operations and monitor performance metrics, data scientists may develop and refine machine learning models, and end-users may utilize the insights generated by the algorithms to make informed decisions.

[0076] In an additional embodiment, the architecture of Fig. 3 may allow for real-time data processing and decision-making. This capability could be facilitated by the integration of streaming data inputs, enabling the system to continuously update its predictions and recommendations as new information becomes available. This real-time processing feature may significantly enhance the responsiveness and adaptability of the artificial intelligence algorithms within the WizSuite platform.Automatic identification of components, parts and their relationships

[0077] Creating engineering instructions is very tedious, complex, time consuming and prone to errors task. The documentor, sitting at an office, is expected to envision each action that should be done during the assembly process and document it in such a way that every assembly worker at the production line will perform the same actions with the same BOM achieving the same successful result. However, (confidential) clients testimonies report that FTY (first time yields) run from 20+ to 60+ percentage rate at the highest. Clients then find themselves facing many runs of rework to reach the minimal expected yield (from 80 to 90+ percentage) set by the organization. The system however, makes this documentation process both faster and more accurate, saving both expensive documentor time and possible assembly line errors. For example, similar parts may be involved in the assembly instruction, differentiated by small details or angles, creating a need to monitor and assure correct assignment and labelling.

[0078] Fig 2. illustrates integration of machine learning with WizGen, WizPlan, WizShield, WizField and WizOpt, with main users. An extension for the WizShield and WizField architecture (orange servers), this diagram shows each machine learning capability (blue rectangles) connected to its main users (smileys). Additionally, it illustrates with yellow arrows which tasks require inputs that include some output of a previous task.Y1

[0079] In an embodiment, Fig. 2 illustrates the integration of machine learning with the modules WizGen, WizPlan, WizShield, WizField, and WizOpt, along with the primary users associated with each module. The architecture includes extensions for WizShield and WizField, represented by orange servers, which host machine learning capabilities, depicted as blue rectangles. Each machine learning capability is connected to its main users, represented by smiley icons, to delineate Fig. 2 the interactive flow of tasks.

[0080] In yet another embodiment, further illustrates how yellow arrows indicate the dependency of certain tasks on the outputs derived from preceding tasks. These arrows signify the necessity for sequential data processing wherein the output of one task serves as an input for another. This interconnected workflow ensures a coherent and integrated process across different modules of the system. For instance, the output from a machine learning capability in WizShield may be required as an input for a task in WizField, indicating a collaborative exchange of information between modules.

[0081] In a further embodiment, each module, WizGen, WizPlan, WizShield, WizField, and WizOpt, may include specific machine learning functionalities designed to enhance their respective operations. For example, WizGen might incorporate predictive analytics, WizPlan could utilize optimization algorithms, WizShield might employ security detection algorithms, WizField could leverage field data analysis, and WizOpt may adopt operational optimization strategies. These functionalities may be dynamically adjusted based on real-time user interactions and system requirements.

[0082] In an additional embodiment, the diagram demonstrates the scalability of the integration, where additional machine learning capabilities can be incorporated without reconfiguring the existing system architecture. This modular approach allows for continuous improvement and adaptation of the system to emerging technologies and evolving user needs.

[0083] In another embodiment, the main users associated with each module may have varying levels of interaction with the machine learning capabilities, tailored to their specific roles and responsibilities. For instance, a user interfacing with WizPlan might have access to strategic planning insights, whereas a user operating within WizField may interact with real-time data analytics for field operations. This role-based access ensures that users receive relevant and actionable information pertinent to their tasks.

[0084] In yet another embodiment, the depiction of yellow arrows indicating task dependencies may also encompass feedback loops where the output of a downstream taskprovides insights or corrective measures for an upstream task, thereby fostering a selfimproving system. This iterative process ensures continuous learning and enhancement of system performance through constant monitoring and refinement based on empirical data and outcomes.Product Tree

[0085] In some embodiments, an automatic generated IPCI graph displayed as a product tree checks the integrity and correctness of the parts and the procedures, shows statuses, and proactively prompts the user for completeness. Its creation depends on both the internal structure of the assembly procedure and hierarchical connectivity trees of the IPCIs, i.e., which nuts go with which screws and into or between which larger complex objects and surfaces. Learning this structure considering the individual needs and factors of safety (FOS) of an assembly line, allows detection of defects in the procedure, matching substitutes, and replacement of steps in the assembly order or from without it.

[0086] Workflows and procedures created using the WizGen module of the invention are senior completely automated by the machine-learning (ML) suggestion system, relying on the most probable and all correctly completed allowed processes including thorough version history, previously learned by the system from existing and in-house or external previously created procedures. The suggestion process enables generation of procedures only though specification of available parts and desired result, so as not to recreate sections of product assemblies from other clients. The system learns previously completed instructions, but also user behaviour and preference from logger. This allows faster and safer generation of procedures, and reveals other possibilities not always intuitive for any engineer approaching to create novel instructions.

[0087] In addition to suggesting substitute steps and methods, the system is used to create whole assembly instructions simply by designating a desired final product and an accompanying BOM of all required parts and materials. In some embodiments, the system is able to suggest an accompanying BOM on its own. The system then creates a "product tree" including all parts and components, hierarchically ordered by their relationships, i.e. assemblies and subassemblies. Each atomic part can join iteratively in any number of duplicates, and to various assemblies and subassemblies. Then the use of individual components can be subjected to novel processes, by direct and complete reuse, or shared in several in instructions making any change propagateinstantaneously and saving all previous and altered versions for future and colleague use and reference.

[0088] The product tree allows use of parts and materials in all configurations allowed for a manufacturing step creation. The system crops out any non-useful options by retaining constraints on subassemblies (order of assembly, minimal number of steps, etc.) and on the final result. Le., only efficient, logical and possible assemblies will be suggested automatically. A threshold for suggestions can be lowered in some embodiments to allow the generation and recommendation of steps more distant from the most probable and efficient selected model. The system takes into account both the general preference documented, and the specific actions taken by the current user in the past, to generate and display steps and assemblies with high probability of being put into use successfully. Additionally, the system mimics changes made after previous deliveries of suggestions, to generate a procedure closer to the desired final state.

[0089] The product tree connects all possible interactions between isolated and / or composite parts, that may be used in designing an assembly (manufacturing) instruction and in putting an assembly in action. The final products are generally on top, and connect all different parts of the tree. Below are large or progressive states of assembly, and at the bottom are all individual (and mostly generic) mechanical parts participating in the assembly. At first generation, an algorithm walking through the product tree will only allow steps that are part of the original instruction. Then, by recurrence of parts in various steps and more configurations, alternative subassemblies and order of steps within and between them will be generated as tentative possibilities. Only when chosen, their ranking out of all step formation probabilities will rise, and then after appearing at a confirmed instruction, they will henceforth be commonly recommended in following generations of the same product and other projects. After extensively pruning all disallowed actions, and selecting the best choices and relevant alternatives, a slightly different final assembly product may appear. If chosen for instruction generation, it will prompt more high-level recommendations and extensions for an existing assembly line and for new uncharted projects.

[0090] By limiting recommendations by proximity to BOM and final product model, the system constrains the model not to allow recommendations resembling anything from other clients' instructions. Other methods of these constrains include integration of a smaller percentage of data from instruction, and in distant steps and parts. This limit is picked in conjunction with aspecific client, and in relation to the thoroughness of recommendations the client is willing to accept himself. Whereas the recommended default percentage is between 10 percent and 20 percent, some generic products that the clients are willing to keep open for innovation may choose a larger portion, for example, of 50 percent or more, and in the other extreme, extrasensitive novel and highly specific instructions may want to keep a single digit for the percentage of data available to the system's algorithms.

[0091] The optimized generation of assembly procedures (AKA "steps") creates a consistent instruction flow that best serves documentors and related engineering roles, manufacturing line operators and most of all the assembly process accessibility and performance measures (yield, time, cost, rework instances etc.).

[0092] The system of the invention automates the generation of assembly process flows, while standardizing the way images are generated and integrated into assembly steps one or more corresponding images along with a step result for future validation by WIZ SHIELD CV algorithms of the invention. In addition to CV and deep convolutional neural networks, the system involves creation exploration and pruning of an undirected graph or decision tree, which hierarchically and temporally connect all possible configurations and user actions available for the participating parts in an assembly, then confound by various types of constraints to produce only the correct and most efficient FOSs.Automatic disassembly and rework generation with integrated computer vision

[0093] Things may not always work as planned during assembly in a production floor. Assembly workers may receive a faulty part, be distracted, or make a human mistake or a part can break or be damaged. In those cases, a rework instruction is necessary. However, rework may require first disassembling the current, unfinished step and only after reaching the previous step t, the must redo the steps to continue the assembly. Computer vision is integrated in every step result, initiates the rework and assesses the correctness the parts, process parts and the final step result correction. In the past, using WIZ GEN the documentor wrote the tests to be done and couldn't add exact image to support the whole process.

[0094] In an embodiment, a computer vision system may be utilized to monitor each step of the assembly process on the production floor. This system may capture images of the assembledparts at each step and compare them to a database of reference images to detect any discrepancies or errors.

[0095] In another embodiment, when an error is detected, the computer vision system may generate a rework instruction that includes visual aids, such as annotated images, to guide the assembly worker through the disassembly and reassembly process. This system may also track the rework progress and ensure that each corrective step is executed accurately.

[0096] In yet another embodiment, the rework instructions generated by the computer vision system may be dynamically adjusted according to the specific error detected. This ensures that the instructions are tailored to the particular issue, which may enhance the efficiency of the rework process.

[0097] In a further embodiment, the computer vision system may integrate with a documentation tool, such as WIZ GEN, allowing for the automatic inclusion of precise images and annotations in the rework instructions. This integration may facilitate a more comprehensive understanding of the necessary corrective actions by the assembly workers.

[0098] In still another embodiment, the utilization of computer vision technology in combination with documentation tools may improve the accuracy and efficiency of the entire assembly process by providing real-time feedback and detailed visual instructions for error correction.Automatic verification of (near) real-time step results engineering instruction in production and maintenance

[0099] Real time identification of composite parts, materials, tools and jigs through video cameras in production line - WIZ - SHIELD. The YOLO (You Only Look Once) network is a specialized system designed for the detection and classification of multiple objects within an image or a live stream of images, complete with the marking of bounding boxes.

[0100] YOLO employs a grid-based approach, predicting a multitude of bounding boxes across the image grid, each associated with a confidence value. Subsequently, it predicts the probable class for the bounding boxes at every grid cell. Through non-max suppression, the system retains the single best bounding box for each object in the image, streamlining the detection and classification process with remarkable speed and accuracy. We began incorporating this method for augmenting identification of 3D CAD models by extraction of a multitude of 2Dviews, that catch the model features nearly as much as possible considering the dimensionality reduction. WizShield of the invention is configured to use modules such as YOLO trained on large datasets of of real component and assembly line pictures in order to correctly identify items during assembly via images captured at the assembly line.

[0101] In some embodiments, live images are collected from within the assembly line using WizShield or elsewhere with WizField. The optimal data collection includes a 360° video scanning on the item by rotating camera or base. Such data allows classification by 3D-model based classifiers, PointNet, PointCNN and CADNet. Alternatively, some workstations at the assembly may include three or more cameras for extracting sufficient features of the assembly without stopping it. A minimal set of images is sufficient for identification of relatively simple parts, using both view-oriented and real-life image classification networks such as CADNet and YOLO. During the assembly process parts get joined / connected to other parts thus changing states towards the final produced product. At each step, the there exists a new temporary composite state part which is a combination of its components features. The new temporary composite part might look completely different than the two parts together thus, for separating complex items by their composite state, for example finding a screw fasten in place or not, a classification algorithm will run further after initial image identification to classify the images by creating a bounding box at the region of interest or the "delta" where previous taken images differ from the current significantly in high probability, and limiting available classes to only those connected with the part. Generally all composite parts will have all of the following states and possibly others, the initial "open"-state, in-progress or suboptimally assembled, and final correct state.

[0102] Narrowing candidate classes for classification by prior knowledge of available components further aids in increasing performance accuracy and speeding up fine-tune learning and inference. Constraining the model, mostly easily by taking all irrelevant weights (impossible classes in the output) to zero, is dictated by one or more of the following factors: available parts in the assembly line or in the GEN generated instruction and BOM; the assembly-line and station within, which further specifies available parts tools and techniques; procedure, FOS, step, giving higher priors to allowed and alternative components, and highest to possible elements of the specific step (including faulty part selections). Finally, the classifier algorithm will keep a small history or memory of the last few classified parts and tools, which, dependingon their occurrence and relations, may indicate the possible mechanical parts to be identified and used.

[0103] In one embodiment, narrowing candidate classes for classification may involve utilizing a lookup table to filter out unavailable components based on their occurrence in the assembly line or in the GEN generated instruction and BOM. The lookup table may dynamically update based on real-time data received from the assembly process.

[0104] In yet another embodiment, the classifier algorithm may employ a machine learning model that adapts by incrementally learning from previous classification results. This adaptive learning can refine the model's ability to predict relevant parts and tools by integrating historical data about the parts and tools used in the previous assembly steps.

[0105] In a further embodiment, the model may incorporate contextual factors such as the specific station within the assembly line. For instance, the availability of certain parts and tools may be contingent on the station's designated function and capabilities, thereby informing the constraint mechanisms for setting irrelevant weights to zero.

[0106] In another embodiment, higher priors may be assigned to components that are deemed allowed or alternative according to the procedural guidelines (FOS) of the assembly process. This hierarchical prioritization can enhance the classifier's accuracy in predicting pertinent components while minimizing errors related to faulty part selections.

[0107] In still another embodiment, the classifier algorithm may integrate an exception handling mechanism to account for anomalies such as unexpected faults or deviations in the assembly process. This mechanism may adjust the model's output by incorporating the potential occurrences of faulty part selections and providing viable alternative components.

[0108] shows the confusion matrix for YOLO classifying mechanical parts from 3D CAD model 2D views. Each row summarizes predictions of each class, while each column represents the ground truth class (from dataset labels). Evidently, the diagonal shows all but one category is correctly Fig. 7 classified for >85% of view images.

[0109] Fig. 8 shows examples, with ground truth labels on the left and predictions on the right, detailing the highest occurring misclassification - of ring as washer (orange rectangle here and at the confusion matrix, green in ground truth). This is easily explained by the side view angle which loses most details and dimensions making it a highly uninformative view.

[0110] YOLOv8 classification performance is quite satisfactory, though some classes suffer from low precision. Therefore a novel kind of experiment was constructed, that runs validation onimages of all classes but in a single identical view. By analyzing performance of the trained network in each view, the best views for classification tasks can be determined, and even pruning the worst from the training database, and aiming to improve results. Here, the mean average precision for each view on each class is shown, highlighting the worst values. Also attached are confusion matrices and image classification examples for some of the worst views.Real time identification of symptoms and faults in maintenance tasks WIZ- FIELD

[0111]

[0112] Equipment requires periodic maintenance and repairs collectively called technical support, during its life cycle. These two processes are typically performed by field engineers and technicians preferably on premises, that is at the customer's location that may also included inconvenient locations such as outdoors in a remote area. Maintenance procedures are commonly created by means of printed or electronic spreadsheets, text documents or any other web forms (that require internet availability). The checklists are list of steps that are not connected to the real model or its documentation. Fault isolation reveals a worse situation.They require the identification of possible faults and the definition of decision trees for each one of the identified faults. Due to the complexity of the process, many organizations do not integrate fault isolation in their support but when fault appears they assign a technician or field engineer that briefly documents what faults he identified and what he replaced. The process of assessing the fault may not documented or assessed, relying only of the successful result of eliminating the fault. In order to minimize the costs of technical support, companies provide customers with series of steps to perform before calling for technical support. However, in heavy complicated equipment such as defense, robotics, medical, aviation, and more with hundreds or thousands of electromechanical parts, checklists and fault isolation are very hard to create and maintain, due to recurring updates of obsolete parts, process improvement or technological innovations. Using the WIZGEN Vision module of the invention, enables integrating computer vision capabilities and automation to procure automatic generation of digital maintenance checklists and fault isolation.

[0113] In an embodiment, the WIZGEN Vision module may utilize computer vision algorithms to scan and identify the components of the equipment requiring maintenance or fault isolation. This module may capture real-time images or video feed of the equipment, which can then beanalyzed to automatically recognize parts, determine their status, and generate appropriate maintenance steps or fault isolation procedures.

[0114] In another embodiment, the WIZGEN Vision module may interface with a central database containing the detailed schematics and maintenance history of various equipment models. By cross-referencing the scanned images with this database, the module can ensure that the generated checklists are accurate and tailored to the specific model, accounting for any updates or obsolete parts.

[0115] In yet another embodiment, the module may be equipped with machine learning capabilities to improve its accuracy over time. By learning from previous maintenance and fault isolation procedures, as well as technician input, the WIZGEN Vision module may refine its algorithms to more effectively identify faults and suggest corrective actions, thereby reducing the need for manual intervention.

[0116] In a further embodiment, the WIZGEN Vision module may include an augmented reality (AR) interface. Technicians may use AR glasses or mobile devices to view the equipment, with the module overlaying digital checklists, part labels, and step-by-step fault isolation procedures directly onto the field of view of the user, enhancing efficiency and accuracy during maintenance or repair activities.

[0117] In a different embodiment, the WIZGEN Vision module may support remote collaboration, allowing field engineers and technicians to share real-time visual data with experts located elsewhere. This feature may facilitate remote diagnostics and assistance, reducing the need for on-site visits and enabling quicker resolution of faults.

[0118] Fault isolation involves summarizing observable symptoms and eliciting additional information to uncover the root cause of a malfunction. The system can gather user input through open-field textual descriptions of faults and identify occurrence of previously learned symptoms. Utilizing prior training on the historical data of known faults and their systems, the system is able to recommend the most probable malfunction and the optimal method for identification and resolution. This streamlines the fault isolation decision tree, reducing the time for identification and minimizing errors associated with a simplistic fault isolation procedure and decision tree.

[0119] The supporting neural network for fault isolation undergoes training on all past malfunctions in the assembly line and similar assemblies (both in-house and out-house). Its purpose is to predict the underlying fault causing a variety of symptoms. This neural networkemploys images to extrapolate reported symptoms for future and cross-station applications.Additionally, sensors measuring temperature, vibration, pressure, torque, and audio are referenced for symptom and fault identification.

[0120] Checklists are a major need for supporting multisite, defense, medical equipment companies that require standard checklists procedures. It aims at suggesting highest probable and most efficient user interactions for identifying and isolating fault sources. The WIZ GEN Intelligent Maintenance and fault isolation generation module of the invention fill a huge gap providing support for multisite, defense, medical equipment companies that require standard checklists procedures. Choosing fault discovery suggestions even in a statistically uncertain situation, and pointing the user to fastest fault isolation for short maintenance time and fewest rework instances requires integration of CV predictions, product tree complexity and user interaction logs to optimize the process.

[0121] In an embodiment, the system may utilize a machine learning module that dynamically updates its fault isolation procedures based on newly reported malfunctions. This module may incorporate continuous feedback from user inputs, allowing for real-time adjustments to symptom analysis and fault recommendations, thereby enhancing the adaptability and accuracy of the fault isolation process.

[0122] In another embodiment, the neural network may interface with an augmented reality (AR) system to provide visual guides and interactive checklists for maintenance personnel. This AR system may overlay relevant diagnostic information directly onto the physical components under examination, supporting the fault isolation process by visually presenting the probable fault areas and suggested actions based on sensor data and historical fault records.

[0123] In yet another embodiment, the system may incorporate an advanced analytics dashboard that aggregates data from all connected sensors, user logs, and historical fault records. This dashboard may present real-time metrics and predictive analytics on fault trends, helping maintenance teams prioritize high-risk areas and allocate resources more efficiently. The integration of such a dashboard may enhance decision-making processes by providing comprehensive insights into potential fault symptoms and their underlying causes.

[0124] In a further embodiment, the system may deploy natural language processing (NLP) techniques to interpret user-inputted symptoms with greater nuance. This approach can enhance the system's ability to correlate disparate symptom descriptions with known faults.Step result real time verification using cameras WIZ SHIELD / WIZ FIELD

[0125] Each Illustrated Part Catalog Item ( I PCI) comprises multiple flows-of-steps (FOSs), where the execution of each is contingent upon the successful completion and verification of the preceding one. This verification process can be conducted manually or may involve testing and step successful confirmation by the system.

[0126] The validation method of the invention relates to assisting error free manufacturing through the step by step result validation via a CV algorithm. The system ensures the existence and proper assembly of all components by automatically comparing the assembly step result to the desired previously documented or generated final state. Additional tests can be conducted, with confirmation provided by the CV algorithm.

[0127] This allows faster and safer generation of procedures, while reducing costly and timeconsuming rework rounds while saving in MRB and replacing parts.

[0128] The technological challenge in this task is to be able to classify correctly step images in real time. It requires multiple cameras that are able to catch, extract and compare the step result image with the expected one. The difference between a faulty and a correct result may be subtle. The system performs error-checking in real time. This requires extensive training on images of faulty assemblies very similar to current ones, using data augmentation and hybrid models of various architectures and data formats, including pruning techniques to maximize accuracy.

[0129] In an embodiment, the system may utilize a plurality of cameras positioned at strategic points around the work area to capture images of the assembly process from multiple angles. These images are transmitted to a central processing unit where the CV algorithm performs realtime analysis, comparing each captured image with the expected image of the desired final state.

[0130] In another embodiment, the CV algorithm can use machine learning techniques to differentiate between subtle differences in faulty and correct results. This training process may employ a hybrid model encompassing various neural network architectures, along with data augmentation methods that generate a diverse range of training images, thus enhancing the model's ability to recognize errors accurately.

[0131] In yet another embodiment, the system integrates a feedback mechanism wherein realtime error detection triggers notifications to human operators or automated correction systems.This feedback loop allows for immediate intervention, minimizing the impact of errors and reducing the need for extensive rework rounds, thereby saving time and resources.

[0132] In an embodiment, the system may employ pruning techniques to streamline the image classification process. By selectively removing less significant neural network parameters, the system can achieve higher processing speeds without compromising accuracy, thereby facilitating real-time error checking and validation.

[0133] In another embodiment, additional validation tests may be conducted at each assembly step. These tests can be coordinated through the CV algorithm, which provides confirmation of correct assembly prior to advancing to the next flow-of-steps, thus ensuring comprehensive verification at each stage of the manufacturing process.WIZSFIELD Vision Maintenance and Faults isolationConnecting symptoms and faults with decision trees, clustering, classification and predicting fault chain by pattern recognition - Bayes networks - WIZ -OPT

[0134] By classifying related symptoms and faults the system makes use of a clustering algorithm in the WizOpt module for suggesting nearest neighbor fault candidates during inference, as well as creating a standardized procedure to focus fault isolation on the most probable cluster, target the correct case and instruct the worker for its treatment. Then, by relating both the clustering results and the decision-trees to the Al logger, a Bayes network learns to predict a recurring fault by user actions before it is tackled upon.

[0135] The specialized Al logger, observing user actions during assembly through the WizShield module, captures the action chain that ultimately leads to a fault. The log entries with highest correlation to fault in the next steps and above a set threshold will notify the user of a probable fault predicted, and show the related action as a top candidate for fault cause when revealing the malfunction. This data, combined with a CV algorithm tracking assembly steps and user actions, is processed by a Bayesian network to predict the resulting fault through pattern recognition.

[0136] The solution is made by atomic step assessment that should identify the error as it happens thus corrections should be faster and shorter than if they are discovered at the end of the assembly or even at a later stage. Assembly step and state identification is a non-trivial task, taking into account both the worker actions and visual (and by other sensors) assessment ofassembled state of mechanical parts. Since the difference of an assembly between adjacent sub steps may be minimal, e.g. a single screw fastened in an already similar round site on an apparatus large and complex by several orders of magnitude, any available CV classification solution would hardly separate its novel state from the previous. The system applies specialized training of several deep convolutional neural networks (foremost YOLO, but also CADNET, PointCNN and PointNet, especially for final step results), which learn all probable and possible states of assemblies from many various angular views that the station-bound cameras may receive as input for inference of state and assemblance correctness, then pruned and fine-tuned to accumulate the chosen product and procedure, to create both a generic and a specific assembly directed classification algorithm with maximal accuracy and minimal runtime.

[0137] In an embodiment, the Al logger, in conjunction with the WizOpt module, may capture and analyze user actions by utilizing a combination of clustering algorithms and decision-trees. The clustering algorithms, which may include k-means or hierarchical clustering, group related symptoms and faults to streamline fault isolation, directing the assembly worker to focus on the most probable fault cluster. Decision-trees assist in providing structured guidance based on historical data and user interactions stored in the Al logger.

[0138] In another embodiment, the system may utilize a Bayesian network that integrates data from both the Al logger and a CV algorithm. The Bayesian network may use probability distributions to predict recurring faults based on observed user actions and assembly step sequences. The CV algorithm can track and assess the mechanical assembly state, leveraging convolutional neural networks like YOLO, CADNET, PointCNN, and PointNet to differentiate between minimal but critical changes in the assembly process.

[0139] In yet another embodiment, the assembly step and state identification process can be enhanced through the use of multiple sensors, including visual cameras and possibly other types of sensors such as torque, pressure, or temperature sensors. These sensors provide comprehensive data capturing minute differences between assembly sub-steps. The deep convolutional neural networks, after specialized training and fine-tuning, analyze this multisensor data to ensure precise classification of each assembly state, thus facilitating accurate fault prediction and faster correction.

[0140] In a further embodiment, specialized training of the convolutional neural networks may involve data augmentation techniques to simulate various angular perspectives the stationbound cameras might encounter. This augmented training dataset allows the neural networks tolearn and classify all possible assembly states with higher precision. The pruned and fine-tuned neural networks then operate with optimized runtime performance while maintaining high accuracy, effectively supporting real-time fault detection and correction during the assembly process.Evaluating production efficiency and principal faulty assembly factors and manufacturing bottlenecks. Building hybrid models including vision for yield improvement. Life cycle optimization models based on multiple objective functions (such as cost, production time, errors (MTB), guality, yield (WIZ-OPT)

[0141] Optimization models are a key tool for enhancing yields and product quality, as well as minimizing costs, production time and occurrence of error during manufacturing. Such hybrid models are trained on various types of data, including temporal information (yield, MTB, time of steps completion, etc.), per-step, workstation and employee statistics, and in parallel, visual images on the assembly line and finished products. The last is considered causally related to all its priors. By identifying manufacturing bottlenecks, sources of errors and delays, and general performance of each individual in each according station and at each attempted step, a widely general and profoundly specific analysis of assembly shortcomings can offer a thorough optimization plan. The optimization itself, as previously mentioned, takes into account different aspects which it needs to weigh out, thus allowing flexibility to line planners and managers to improve outcomes of their assembly line. These aspects are weighed first by default standards, then adjusted by user input. They include time of steps, number of assemblies completed, cost of parts tools materials power and manpower, malfunctions in completed assemblies and quality of resulting components and products, itself a measure to be designed by various measures open to configuration by product engineers and endline users.

[0142] The default standard by which the optimization model measures costs at start are total worker hours, which translates to both production time and production cost. By comparing time costs and contrasting resulting yields, the algorithm thrives to achieve near-saturation cost minimization and yield improvement. Notwithstanding, the system may become super sensitive to the occurrence of faults, maintenance issues and rework sessions, since these are discrete events that cause unpredictable delays and additional costs. The hybrid optimization model thus attempts to maximize production profits and find the best production practices for ultimate assembly management and work habits.

[0143] Managers using the WizShield module have comprehensive access to efficiency assessments and time-to-complete analyses for previous tasks and procedures, both recent and historical, specific to stations and operators. Beyond elementary information, the algorithms of the invention can predict potential delays or errors in assembly. They suggest reassignments to other stations or recommend simpler tasks, taking into account the skills of each employee based on their task performance history. This includes hidden factors that an ML algorithm can discern, which may elude the awareness of work managers. This holistic management approach ensures not only efficiency but also anticipates and mitigates potential challenges in real-time.

[0144] In an embodiment, the hybrid optimization model may include a module specifically dedicated to monitoring visual images on the assembly line and finished products. This module can leverage image recognition technologies to identify defects, track the progression of assemblies, and correlate visual data with temporal information, workstation statistics, and employee performance. This correlation may help in pinpointing specific visual cues that are indicative of potential bottlenecks or quality control issues.

[0145] In another embodiment, the optimization model may incorporate a subroutine for realtime adjustment of the default standards based on user-defined parameters. This subroutine allows the system to dynamically reconfigure its weighing metrics, facilitating immediate adaptation to changing production requirements or unexpected disruptions. The user input may come from line planners, managers, or automated sensors detecting variances in production performance metrics.

[0146] In yet another embodiment, the optimization system may utilize machine learning algorithms to develop predictive models for potential maintenance issues and rework sessions. These models could analyze historical data on machine performance, error rates, and employee interactions to forecast the likelihood of faults or downtimes. By providing advance warnings and suggested mitigation strategies, the system helps minimize unscheduled disruptions and optimize maintenance schedules.

[0147] In a further embodiment, the WizShield module may incorporate enhanced analytics capabilities that allow managers to drill down into specific performance metrics of individual employees and workstations. These analytics can provide insights into hidden variables, such as ergonomic factors or subtle inefficiencies, which might not be immediately apparent through traditional monitoring techniques. This detailed analysis may assist in devising more targetedimprovement plans and reallocating tasks in a manner that aligns with each employee's strengths and weaknesses.

[0148] In yet another embodiment, the system may include a feature that correlates the temporal information of completed steps with the quality of resulting components. Through this correlation, the model can identify time patterns that consistently yield high-quality products and suggest adjustments to the workflow to replicate these conditions. This feature can play a crucial role in iterative optimization cycles aimed at continuously refining the assembly process.Additional machine learning implementations for manufacturingAutomatic extraction import and integration of models, images, instructions and manufacturing data from existing instruction documents, and into the WIZ production portfolio generation system as a first draft. (WIZ-GEN)

[0149] Automatic extraction of models, images, instructions and manufacturing data from existing instruction documents is done by a custom algorithm of the invention, isolating media data from a flexible host of formats and document structures. The algorithm connects each item to a label from its description by adjacency or line-connectedness. Then it analyses the extracted label to match with absolute mechanical parts or compounds, and to relay it to the database accordingly, including import to a new or ongoing WizGen manufacturing profile.

[0150] In an embodiment, the algorithm may utilize machine learning techniques to enhance its ability to isolate media data from various formats and document structures. Machine learning models such as convolutional neural networks (CNNs) may be employed to improve accuracy in recognizing and extracting images and models from instruction documents.

[0151] In another embodiment, the algorithm may incorporate natural language processing (NLP) methods to better analyze and match extracted labels with absolute mechanical parts or compounds. These methods may involve tokenization, part-of-speech tagging, and named entity recognition, which facilitate the identification and categorization of relevant terms within the extracted labels.

[0152] In yet another embodiment, the algorithm may employ a multi-stage processing approach where the initial stage focuses on the simple extraction of media data and labels, while subsequent stages refine the matches and associations with the absolute mechanical partsor compounds. This may include iterative feedback loops where the results of the matching process are used to adjust and improve the extraction and analysis algorithms.

[0153] In a further embodiment, the database to which extracted information is relayed may be a distributed database system, allowing for enhanced scalability and fault tolerance. The distributed nature of the database may enable simultaneous processing of multiple documents and efficient handling of large volumes of extracted data.

[0154] In an additional embodiment, the WizGen manufacturing profile, to which data is imported, may support version control mechanisms. This allows for tracking changes in the manufacturing data over time and enables rollback to previous versions if needed. The version control may be integrated with the database to maintain consistency across different stages of the manufacturing process.Building optimization models for assembly workstations order allocation improvement (WIZ-PLAN)

[0155] Recommendations for production line structure are generated, based on procedure / step professions needed. This is made possible by learning the running statistics of the assembly line, what stations take longer to execute their steps or make more errors creating a bottleneck. It is also constrained and directed by the available and allowed changes to the production, some will be instinctively discovered by the workers, while others will be completely occluded by the sheer number of combinations. The possibilities are weighed by projected time and efficiency measures, and by tendency to be found best in similar previous assemblies. The interface with the WizPlan module offers great flexibility together with highly informative interactions by the recommendation algorithm, statistics and best recommendations will appear on screen with the WizPlan user interface.

[0156] In an embodiment, the building optimization models for assembly workstation order allocation improvement may utilize a machine learning algorithm to analyze historical data from the assembly line. The machine learning algorithm may identify patterns and trends in workstation performance, such as which stations experience delays or higher error rates. Using this information, the algorithm may suggest structural changes to the production line to minimize bottlenecks and improve overall efficiency.

[0157] In another embodiment, the optimization model may incorporate real-time data collection from the assembly line. Sensors and data collection devices may be employed at eachworkstation to monitor performance metrics such as time taken for each step and error rates This real-time data may be fed into the WizPlan module, which may dynamically adjust recommendations based on current assembly line conditions.

[0158] In yet another embodiment, the WizPlan module interface may present recommendations using a color-coded system to highlight areas of concern. Workstations that frequently cause delays or errors may be displayed in one color, for example, red, while those operating efficiently may be shown in another color, for example, green. This visual representation may help workers and managers quickly identify and address problem areas.

[0159] In a further embodiment, the recommendations generated by the WizPlan module may be adjustable by users based on their expertise and knowledge. Users may have the ability to input additional constraints or preferences into the system, such as prioritizing certain workstations or steps due to resource availability or strategic considerations. The system may then recalibrate and provide recommendations that align with the newly input parameters or any combination thereof.

[0160] Table 2a shows the mean average precision for each view on each class, highlighting the worst values (<70% or worst classes with gradient)

[0161] Table 2a

[0162] Table 2b - class dictionaryClamp, fork joint and hinge get low precision values (<60%) throughout all views, in some worse than others down to 15%. Any view with elevation of 0 degrees has 3-4 additional classes with precision <70% (most commonly disc, hook and rivet). Finally, views with elevation of 90 degrees and azimuth 0 degrees (bottom row) seem to achieve the lowest precision (though still almost perfect in some classes) with 8 classes scoring average precision <70%. Removing the worst of these problematic uninformative views from the training dataset may improve performance for these classes as well as overall.Following are some examples.

Claims

Claims1. A system for generating and managing a manufacturing file comprising a plurality of standard engineering instructions, the system comprising:a rule-based user interface configured to guide engineers and documentors during the creation of a production tree and step-by-step varied engineering procedures;a manual validation module configured to guide assembly workers on a production line through validation of a bill of materials (BOM) components and step-by-step assembly results;a checklist module configured to guide technicians through single-step checklist manual validation^ hybrid model comprising artificial intelligence (Al) and machine vision algorithms, wherein the hybrid model is configured to automate the validation and verification of engineering instructions at all phases from engineering to production to maintenance.

2. The system of claim 1, wherein the rule-based user interface comprises a graphical user interface (GUI) displaying engineering procedures and allowing for manual input and adjustments.

3. The system of claim 1, wherein the manual validation module is configured to provide real-time feedback on BOM component validation and assembly results.

4. The system of claim 1, wherein the checklist module presents a series of verification steps that must be acknowledged and validated by the technician.

5. The system of claim 1, wherein the hybrid model comprising Al and machine vision algorithms is configured to identify discrepancies within the engineering instructions and flag potential errors for review.

6. A method for generating and managing a manufacturing file comprising a plurality of standard engineering instructions, the method comprising:guiding engineers and documentors in the creation of a production tree and step-by-step varied engineering procedures via a rule-based user interface;guiding assembly workers at the production line via manual validation of a bill of materials (BOM) components and step-by-step assembly results;guiding technicians through a single-step checklist manual validation process;applying hybrid models of Al and machine vision algorithms to automate the validation and verification of engineering instructions at all phases from engineering to production to maintenance.

7. The method of claim 5, further comprising displaying the engineering procedures through a graphical user interface (GUI).

8. The method of claim 5, further comprising providing real-time feedback to assembly workers on the validation of BOM components and assembly results.

9. The method of claim 5, further comprising presenting technicians with a series of verification steps requiring acknowledgment and validation.

10. The method of claim 5, further comprising applying the Al and machine vision algorithms to identify and flag discrepancies and potential errors in the engineering instructions for review.

11. A system for generating and managing manufacturing and maintenance processes, comprising:a computer vision module configured to identify composite parts, materials, tools, and jigs;a machine learning module configured to automate the generation and verification of engineering instructions for assembly, maintenance, and fault diagnosis;a user interface for manual adjustments and oversight of the generated instructions^ data integration module for importing models, images, and manufacturing data from existing documents.

12. The system of claim 11, wherein the computer vision module utilizes convolutional neural networks for identifying components in 2D and 3D images.

13. The system of claim 12, wherein the computer vision module employs a hybrid model comprising PointNet, PointCNN for 3D CAD models, and CADNet, YOLO for 2D views and real images.

14. The system of claim 11, wherein the machine learning module includes a deep learning algorithm configured to classify and label parts, tools, materials, and jigs.

15. The system of claim 14, wherein the deep learning algorithm uses a "winner takes all" selection process for classification accuracy.

16. The system of claim 11, further comprising a real-time identification module utilizing video cameras for component verification during assembly and maintenance tasks.

17. The system of claim 16, wherein the real-time identification module uses YOLO-based networks combined with PointNet and PointCNN classifiers.

18. The system of claim 11, further comprising a module for real-time verification of step-by-step assembly results using computer vision algorithms.

19. The system of claim 11, further comprising a generative engineering visual language model to assist in production planning, instruction generation, and statistical reporting.

20. A method for managing manufacturing and maintenance processes, comprising:identifying composite parts, materials, tools, and jigs using a computer vision module;generating step-by- step visual engineering instructions using a machine learning module;verifying the correctnessof the instructions through a user interface;integrating models, images, and manufacturing data from existing documents into the instruction sets.

21. The method of claim 20, wherein the computer vision module utilizes CNNs for image classification in 2D and 3D formats.

22. The method of claim 21, wherein the CNNs include a hybrid model comprising PointNet, PointCNN, CADNet, and YOLO.

23. The method of claim 20, further comprising automating the classification and labeling of parts, tools, materials, and jigs using a deep learning algorithm.

24. The method of claim 23, wherein the deep learning algorithm employs a "winner takes all" selection process.

25. The method of claim 20, further comprising real-time verification of assembly results using video cameras and computer vision algorithms.

26. The method of claim 25, wherein the real-time verification utilizes YOLO-based networks and hybrid models with PointNet and PointCNN classifiers.

27. The method of claim 20, further comprising generating production plans and reports using a generative engineering visual language model.

28. A system for real-time fault diagnosis and maintenance in manufacturing, comprising:a computer vision module configured to detect symptoms and faults in machinery;a machine learning module configured to predict faults based on detected symptoms;a user interface enabling technicians to view and verify faults detected in real-time;a data integration module for incorporating historical maintenance data.

29. The system of claim 28, wherein the computer vision module employs CNNs and deep learning algorithms for symptom detection.

30. The system of claim 28, further comprising a module for optimizing maintenance processes based on clustering, decision trees, and Bayesian networks.

31. A method for optimizing assembly processes in manufacturing, comprising:utilizing Al algorithms to analyze production line statistics;generating recommendations for workstation order allocation based on past assembly performance;identifying bottlenecks and optimizing workflow using a decision-tree or graph structure;verifying the process integrity through realtime data analysis.

32. The method of claim 31, wherein the Al algorithms include clustering algorithms, decision trees, and Bayesian networks.

33. The method of claim 31, further comprising automatic integration of manufacturing data from existing documents into the workflow optimization process.

Images