Assembly error correction for assembly lines

Abstract

A computational model is provided that uses machine learning to detect errors during a manual assembly process and to determine a sequence of steps to complete the manual assembly process to mitigate the detected errors. The method evaluates a target object at a step in the assembly process where an error is detected against a nominal object, thereby obtaining a comparison, and based on the comparison, obtains a sequence of steps to complete the assembly process for the target object, and based on the sequence of steps, adjusts assembly instructions to produce the target object.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation-in-part of U.S. application Ser. No. 16 / 587,366, entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," filed on September 30, 2019, which is a continuation-in-part of U.S. application Ser. No. 16 / 587,366, entitled "DYNAMIC TRAINING FOR ASSEMBLY LINES," filed on February 28, 2019, now U.S. Patent No. 10,481,579. This application further claims the benefit of U.S. Provisional Application No. 62 / 836,192, entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed April 19, 2019, U.S. Provisional Application No. 62 / 931,448, entitled "A COMPUTATION MODEL FOR DECISION-MAKING AND ASSEMBLY OPTIMIZATION IN MANUFACTURING," filed November 6, 2019, and U.S. Provisional Application No. 62 / 932,063, entitled "DEEP LEARNING QUALITY PREDICTOR FOR MANUAL ASSEMBLY VIDEO ANALYSIS," filed November 7, 2019. The entire contents of the above applications and patents are incorporated herein by reference.

[0002] The subject technology provides improvements to assembly line workflow, and in particular encompasses systems and methods for adaptively updating assembly line worker instructions based on feedback and feedforward error propagation predictions made using machine learning models. As discussed in more detail below, some aspects of the technology encompass systems and methods for automatically adapting guidance videos provided at one or more worker stations based on inferences made regarding manufacturing or assembly deviations. [Background technology]

[0003] Traditional assembly line workflows require human (operator) oversight and expertise to detect manufacturing errors and to determine how the errors can be eliminated through correction in downstream processes. Note that assembly and manufacturing, and assembly line and production line are used interchangeably herein. Because of the reliance on humans to detect assembly errors, there is a high probability that errors will be overlooked (or unreported) and propagated downstream in the assembly process. Furthermore, many assembly workers are only trained to perform a narrow set of tasks and therefore may not know how to modify their own workflow to best correct errors made upstream in the assembly workflow.

[0004] In traditional manufacturing workflows, the repair of human errors as part of the manual inspection process is often handled by taking corrective action on the human node. If there is a continuing problem with the human, that person is often replaced by another person who is vulnerable to many of the same limitations as all humans are. It is difficult to constantly repeat an action for years without making an error, and taking corrective action is outside the rights of most assemblers. Even when these rights are given, there are inconsistencies and they are only informed by the experience the human has in the application of that single process node. Furthermore, there is no mechanism to learn from any mistakes, let alone any positive corrective actions. Summary of the Invention [Problem to be solved by the invention]

[0005] Furthermore, electronic monitoring of assembly lines has limitations and does not include robust mechanisms for providing on-the-fly adjustments to downstream steps in the assembly line to compensate for errors made in upstream steps. Additionally, new mechanisms are needed to evaluate how changes in worker motion and / or changes in assembly patterns affect the resulting manufactured product and to provide corrective actions to improve the performance and / or characteristics of the manufactured product. [Means for solving the problem]

[0006] In some aspects, the disclosed technology relates to a method for optimizing a workflow in an assembly line, the method including detecting an error in an assembly of a target object at a step of the assembly of the target object, evaluating the target object and a nominal object at that step of the assembly process to thereby obtain a comparison, and determining a sequence of steps required to minimize deviations between the target object and the nominal object based on the comparison. In some aspects, the method may further include a step for adjusting assembly instructions for the target object based on the sequence of steps.

[0007] In another aspect, the disclosed technology encompasses a system for optimizing a workflow in an assembly line, the system including a plurality of image capture devices, each of the image capture devices being positioned at a different position to capture a motion of a worker during an assembly process of a target object, and an assembly instruction module coupled to the plurality of imaging devices, the assembly instruction module being configured to automatically modify guidance and instructions provided to the worker. The assembly instruction module can be configured to perform an operation including receiving motion data from the plurality of image capture devices, the motion data corresponding to a worker's execution of a set of steps for assembling the target object, and determining an error in the assembly of the target object based on the motion data at a step of the set of steps. In some implementations, the assembly instruction module can be further configured to perform an operation to evaluate a target object and a nominal object at a step of the set of steps, thereby obtaining a comparison, and based on the comparison, determine a series of steps required to minimize deviations between the target object and the nominal object, and adjust the assembly instructions provided to the worker based on the series of steps. The form of the revised assembly instructions may include, but is not limited to, generated or edited video of the motion data, text-based instructions from natural language processing (NLP) of identified deviations, or other feedback mechanisms to the worker.

[0008] In yet another aspect, the disclosed technology relates to a non-transitory computer-readable medium including instructions stored thereon that, when executed by one or more processors, cause the processor to execute instructions including performing the steps of detecting errors in assembly of the target object at a step of an assembly process of the target object, evaluating the target object and a nominal object at said step of the assembly process thereby obtaining a comparison, and determining, based on the comparison, a series of steps required to minimize deviations between the target object and the nominal object. In some implementations, the instructions can be further configured to cause the processor to perform operations to adjust assembly instructions for the target object based on said series of steps.

[0009] Particular features of the subject technology are set forth in the appended claims. To provide a further understanding, however, the accompanying drawings are included which illustrate disclosed embodiments and, together with the description, serve to explain the principles of the subject technology. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 conceptually illustrates a flow chart of an exemplary production line deployment, in accordance with certain aspects of the disclosed technology.

[0011] [Diagram 2] FIG. 1 illustrates an example process for performing assembly error correction at a given worker station in accordance with certain aspects of the disclosed technology.

[0012] [Diagram 3] FIG. 1 illustrates an example of an electronic system that can be used to implement some aspects of the subject technology. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] The detailed description set forth below is intended to be a description of various configurations of the subject technology, and is not intended to represent the only configurations in which the subject technology may be practiced. The accompanying drawings are incorporated in this specification and form a part of the detailed description. The detailed description includes specific details to provide a more thorough understanding of the subject technology. However, it will be clear and apparent that the subject technology is not limited to the specific details set forth herein, and may be practiced without these details. In some instances, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject technology.

[0014] Aspects of the disclosed technology address the above limitations of traditional assembly line process flows by providing a method for tracking, training, and progressively improving production line assemblies and the resulting manufactured products. Improvements are achieved by providing dynamic visual or other feedback and instructions to individual assemblers, and in some embodiments, the operator feedback is based on errors, including but not limited to assembly errors, non-functional processes and / or motions, and inferior products detected at one or more points in the production line.

[0015] Implementation of the disclosed technology can significantly improve the speed of error correction relative to the speed of manual implementation methods by quickly revising and modifying the reference / instruction information provided at each station (or all stations), for example, based on near real-time error detection. Although some embodiments described herein contemplate the use of reference / instruction information in the form of video, other formats are contemplated. For example, assembly / manufacturing instructions can be provided to an assembler as audio, visual, textual and / or tactile cues, or as other forms of reference. As an example, audio instruction information can include verbal instructions or other audio indicators. Visual assembly instruction information can include video or animated formats, such as the use of an augmented reality (A / R) or virtual reality (V / R) system. In some aspects, visual assembly instructions can be provided as videos that provide examples of how a worker should handle a workpiece (or tool) at a given station of an assembly line. Additionally, in some aspects, assembly / manufacturing instructions can include machine instructions that can be received and implemented, for example, by a robotic or mechanical assembler. The term worker as used herein can refer to a human, robot, or machine using motions to assemble a manufactured product. Additionally, the term worker encompasses human-assisted manufacturing implementations, such as in instances where a human worker works with or is assisted by a robot or mechanical implement.

[0016] In instances where assembly / manufacturing instructions are provided as reference / instruction videos, such videos are sometimes referred to as standard operating protocols (SOPs).The systems of the disclosed technology can be efficiently deployed due to minimal hardware requirements, such as the use of video cameras and displays for individual workers, while machine learning training, updates and error propagation can be performed on centralized computing resources, such as in a computing cluster or cloud environment.

[0017] In some embodiments, video instructional information can be provided to one or more workers as part of an augmented reality display. That is, instructions or deviations from standard assembly / manufacturing methods can be conveyed to the workers using augmented reality, with the display provided as a mix of enhanced video, animated graphics, and / or video data representing recorded scenarios. As an example, the augmented reality display can provide instructions or guidance provided as an animated or graphical overlay on a real-time feed of the workpiece being assembled and / or the tools being used in the assembly / manufacturing process.

[0018] In some implementations, the system of the disclosed technology includes one or more video capture or motion capture devices positioned at various worker stations of a production line. The capture devices are configured to record the worker's motion / interaction with parts, devices, materials or other tools ("components") at that particular station. In some aspects, the worker motion can be captured using video recording, although other motion capture formats are contemplated, such as 3-D point clouds representing the worker motion and / or the worker's interaction with tools or manufactured products. Additionally, a reference video for each station can be created by recording the motion of one or more skilled workers for a particular station and their interaction with the components at that station. This video can be created from a single example or from multiple examples of the skilled worker's actions. A motion path for each skilled worker can be derived, and in embodiments where several skilled workers or several examples are used, calculations can be performed on a set (e.g., an average) of derived motion paths to create a reference video for a particular station. The reference video can be in the form of a digital or animated representation of the motion path to be performed at that particular station. It should be noted that skilled person can mean someone with special skills or knowledge regarding the particular assembly step for which guidance is being provided.

[0019] In some embodiments, video capture or motion capture devices placed at various worker stations in the production line can also capture attributes of the workpieces / components / tools at each station (e.g., quality, tensile strength, number of defects) that can be used to calculate assembly errors.

[0020] By capturing the interactions of the workers at each station, worker errors can be detected by comparing the captured interactions with a baseline (ground truth) model that represents an ideal / expert worker interaction / workflow. That is, worker deviations from the idealized interaction model can be used to calculate assembly errors that can be repaired at various locations in the assembly chain, for example, by revising the worker instructions / guidance provided to different stations. Additionally, the quality of components at individual stations can also be captured and compared against a baseline component for that station. Deviations of components from the baseline component can also be used to assign a quality grade to the component at a particular station or to calculate worker / assembly errors that can be repaired by revising the worker instructions / guidance provided to various stations.

[0021] Assembly corrections can be implemented in a variety of ways depending on the desired implementation. In some embodiments, operator variations / errors can be used to classify parts into quality grades (e.g., A, B, C, etc.) and subsequently perform classification by directing these parts to the appropriate production line. In other embodiments, detected assembly errors can be used to revise the process at a given station, thereby improving quality and reducing variation. That is, detected assembly errors can be used to automatically provide instructions or guidance to the same station, for example, to correct errors introduced at that station (e.g., in-station rework). NLP can be used to process instructions or guidance for the operator. For example, NLP can be used to convert verbal instructions to text form or vice versa.

[0022] For example, assembly error detection can be used to drive updates / changes to worker instructions or videos provided to a given station where an error is known to occur. As an example, once an error / deviation is identified as being due to a first worker working at a first station, the assembly instructions provided to that first worker, e.g., via a display device at the first station, can be revised, thereby reducing the error variance associated with manufactured articles leaving the first station.

[0023] In another aspect, the detected assembly errors can be used to revise subsequent station assembly, thereby overcoming station variance. That is, error detection can be used to automatically trigger downstream propagation of new / updated assembly instructions based on errors caused by an upstream operator. For example, error variance for a motion performed by a first operator can be used to adjust assembly instructions provided from the first station to a second operator associated with a second downstream station.

[0024] In yet another aspect, the detected error distribution across all stations can be propagated forward, thereby ensuring that full or partial rework can be performed across the remaining downstream assembly chain progression, i.e., by adjusting the assembly instructions provided to one (1) or more downstream workers, errors generated across one or more stations can be repaired / reduced. In one example, an error distribution in a manufactured article caused by a first worker at a first station can be repaired through work subsequently performed by a second worker at a second station and by a third worker at a third station, i.e., by adjusting the assembly instructions provided to the second and third stations.

[0025] In another example, an error variance accumulated across multiple stations can be reduced by one or more subsequent stations, such that an error variance in a manufactured article accumulated across a first station and a second station can be subsequently repaired, for example, by adjusting the assembly instructions provided to a third station and a fourth station (e.g., a third worker and a fourth worker, respectively).

[0026] By treating individual workers / worker stations in the assembly flow as network nodes, machine learning models can be used to optimize the assembly process by minimizing errors through reduction of assembly variance at individual nodes (stations). By minimizing individual node (worker) variance and also implementing real-time updates to mitigate forward error propagation, the disclosed technology system can significantly reduce manufacturing variance to the final product. Furthermore, by accurately quantifying and tracking error contributions from specific segments in the assembly workflow, products can be graded and classified by product quality or deviation amount. Thus, for example, depending on product quality, products of a particular quality classification can be directed to different manufacturing processes or to different customers.

[0027] Machine learning / artificial intelligence (AI) models can be used to perform error detection and / or make the necessary corrections to optimize station assembly variations. By way of example, the machine learning models can be trained using multiple sources of training data including, but not limited to, end product ratings, end product variation statistics, desired end product characteristics (e.g., assembly time, amount of materials used, physical properties, number of defects, etc.), station specific component ratings, station specific component variations, desired station component characteristics. Additionally, the deployed machine learning models can be trained or initialized based on input provided by a human expert or "master designer" such that formal knowledge can be represented in an idealized model used to perform error detection and error quantification calculations.

[0028] As will be appreciated by those skilled in the art, machine learning based classification techniques can be modified according to the desired implementation without deviating from the disclosed technology. For example, the machine learning classification scheme can utilize one or more of Hidden Markov Model, Recurrent Neural Network, Convolutional Neural Network (CNN), Reinforcement Learning, Deep Learning, Bayesian Symbolic Model, Generative Adversarial Network (GAN), Support Vector Machine, Image Registration, Adaptive Rule-Based System, and / or any other suitable artificial intelligence algorithms, alone or in combination. If regression algorithms are used, they can include, but are not limited to, Stochastic Gradient Descent Regressor and / or Passive Aggressive Regressor, etc.

[0029] The machine learning classification model may also be based on a clustering algorithm (e.g., Mini-batch K-means clustering algorithm), a recommendation algorithm (e.g., Miniwise Hashing algorithm or Euclidean Locality-Sensitive Hashing (LSH) algorithm), and / or anomaly detection algorithm such as local outlier factor methods. Additionally, the machine learning model may use dimensionality reduction techniques such as one or more of the Mini-batch Dictionary Learning algorithm, the Incremental Principal Component Analysis (PCA) algorithm, the Latent Dirichlet Allocation algorithm, and / or the Mini-batch K-means algorithm, etc.

[0030] In some implementations, multiple different types of machine learning training / artificial intelligence models can be deployed. As an example, a common form of machine learning can be used to dynamically adjust an assembly line process to optimize a manufactured product. As will be appreciated by those skilled in the art, the selected machine learning / artificial intelligence model does not simply include a set of assembly / production instructions, but rather provides feedback on the entire assembly line process and its impact on the resulting manufactured product, and is a method for providing dynamic adjustments to downstream worker stations in the assembly line to compensate for actions occurring at upstream worker stations. This type of artificial intelligence based feedback and feedforward model is referred to herein as Artificial Intelligence Process Control (AIPC). In some embodiments, the machine learning can be based on a deep learning model in a simulated environment that utilizes targeted gated recurrent unit (GRU) model-based learning and Hausdorff distance minimization to effectively explore the space of possible recovery paths, thereby finding the optimal path for correcting errors during the assembly process. In yet other embodiments, for example, the machine learning algorithm can be based on a Long Short-Term Memory model, which analyzes video inputs of the assembly process and estimates the final quality. Metrics (output) Additionally, machine learning models can be used in the form of NLP algorithms to tailor feedback to worker stations, such as text to speech or speech to text, to maximize worker compliance and understanding of tailored instructions.

[0031] In some embodiments, computational models that utilize machine learning can be used to correct errors during the manual assembly process.

[0032] A target object can be assembled through a sequence of steps defined by a processing procedure. During the process, irreversible errors may occur at a particular step k that would require all remaining operations to be revised to obtain the final configuration of the nominal object. In some embodiments, a technique for correcting the errors may include comparing a defective target object at step k against a nominal object at the same step k, or comparing a defective target object at step k against a nominal object in its final configuration. These comparisons may be used to determine the sequence of steps required to minimize the deviation between the final configuration of the defective target object and the final configuration of the nominal object. In some embodiments, a quality metric of the target object may also be used to guide the correction of the defective target object.

[0033] A common numerical method can solve the problem by using the Hausdorff distance algorithm, which determines how similar a sequence of k steps for assembling a flawed target object is to a sequence of steps for assembling a nominal object into its final configuration. Some methods for computationally minimizing the Hausdorff distance between the sequence of k steps in the flawed assembled object and the sequence of steps in the assembled final nominal object are to optimize a Markov Decision Process (MDP) via an instantaneous reward formulation, a multiple reward formulation, or a delayed reward formulation. However, the search space associated with these formulations requires a significant amount of computational resources.

[0034] Alternatively, the machine learning framework can be developed with a delayed reward policy agent using reinforcement learning. The reinforcement learning framework can be designed to have the policy agent determine the appropriate steps required to correct the errors of the flawed target object to obtain a final configuration with a performance metric that matches that of the nominal object. The reward given to the policy agent is delayed, and the policy agent is only paid a reward when the final step is executed.

[0035] In some embodiments, a design for an optimal / desired manufactured product can be selected, and skilled workers can be deployed to perform the individual steps performed at individual worker stations to assemble the manufactured product according to the selected design. Optimal can be based on the desired performance and / or characteristics of the resulting manufactured product that minimizes errors in the resulting manufactured product, or some other criteria (e.g., if the manufactured product is a paper airplane, the optimal paper airplane would be the paper airplane that achieves the desired flight objective). Multiple imaging devices can be used to capture the worker's motion and interaction with the manufactured product they are assembling, thereby generating video, image and / or 3D point cloud data. The captured data can provide detailed information such as the worker's hand coordinates in relation to the manufactured product as it is being assembled, the relationship of one hand to the other, and the relationship of the fingers (and in some embodiments, the knuckles) to the manufactured product as it is being assembled. The data collected from the skilled workers can be used as a ground truth for assembling the optimal / desired manufactured product. This ground truth from a single example can be sufficient to be used in the creation of an initial machine learning model, or additional data can be collected. For example, to understand how variations or errors in worker motions may affect the resulting manufactured product, many workers can be deployed to perform one or more steps of assembling an optimal manufactured product. This may be performed at each worker station in an assembly line. Both the resulting final products and their respective assembly processes can be compared against each other and against a ground truth to determine how errors and / or variations in worker motions may affect the characteristics and / or performance of the manufactured product (e.g., worker speed may result in a poorer quality airplane).Data collected based on operators during an actual assembly process (i.e., a process in which a human, robot, or machine is performing motions at one or more stations) will be referred to herein as "actual training data." The actual training data can be supplemented with simulated data to obtain a richer data set and to provide additional variations to achieve an optimal manufactured product. It should be noted that the terms "optimal" and "desired" will be used interchangeably herein.

[0036] In some embodiments, the different AI / machine learning / deep learning models discussed herein can be deployed in a particular order, as described below, to achieve Artificial Intelligence Process Control (AIPC) to optimize the assembly of a manufactured article. An example process in which an AIPC deep learning model can be implemented is discussed in further detail in conjunction with Figure 1 (e.g., in conjunction with the AIPC deep learning model 112) and in conjunction with Figure 2. Examples of hardware systems and / or devices that can be used to implement the AIPC deep learning model are provided in Figure 3, and corresponding descriptions are also provided below.

[0037] First, CNNs can be used in assembly line processes to classify the hand characteristics of workers and the characteristics of manufactured items of different configurations at individual worker stations.

[0038] Second, Reinforcement Learning (RL) and RL agents can be used and rewarded, both to achieve the desired outcome from CNN classification and for predefined desired outcomes. The RL agents can be supervised or unsupervised.

[0039] Third, Generative Adversarial Networks (GANs) can be used to choose between opposing RL agents. GANs can include minimal human oversight, relying only on humans to select which RL agents to input into the GAN as nodes.

[0040] Fourth, RNNs can take winning RLs as input nodes to create feedback and feedforward systems, thus making learning continuous and unsupervised.

[0041] Implementations of these four AI / machine learning models are discussed in more detail below.

[0042] In some embodiments, actual training data can be input into a CNN to classify relevant data in the assembly process, such as the fingers / hands used in each assembly step per worker station, the parts of the product being assembled that the worker's fingers touch at any point in time and space, and the shape or configuration of the manufactured product being assembled at any point in time and space.

[0043] In yet other embodiments, hand motion is not tracked, but data may be collected that represents various variations in the assembly pattern of the manufactured product (e.g., if the manufactured product is a folded paper airplane, data may be collected based on changing the folding sequence, implementing folding variations, and / or introducing potential errors; if the manufactured product is a clothing article, data may be collected based on, for example, the sewing sequence, implementing sewing variations, and / or introducing potential errors). This data may be simulated and / or collected from actual training data. The resulting manufactured products and their respective assembly processes may be compared to determine how the errors or variations in the assembly pattern affect the characteristics and / or performance of the manufactured product.

[0044] In some embodiments, the captured data (e.g., video and hand tracking of the assembly process, etc.) is used to predict the quality of the final output. This quality prediction allows the captured data to be used to classify products into quality bins without having to manually inspect the quality of the products during the manufacturing process, and allows corrective actions to be taken downstream.

[0045] In some embodiments, the system can focus on manual assembly of a target object, where the assembly process includes several discrete steps in which a worker performs different tasks on a target object per a set of instructions. The system can be built using a machine learning framework (across all worker actions) using a deep learning model that establishes a correlation between the time series of the worker's hand positions and the final quality of the target object. In some embodiments, the model can consist of two neural networks, a first neural network is used to extract the worker's hand position data in the 3D environment and a second neural network is used to filter out unnecessary elements from the hand position data to correlate with the final quality of the target object's performance.

[0046] In some embodiments, the first neural network can use a video collection system to record videos of the worker's hands during the assembly process at different node videos corresponding to individual discrete steps the worker performs in connection with the assembly of the target object. For example, the worker can perform the assembly process using several cameras located at different locations and configured to simultaneously record the assembly process. These cameras can be used to capture videos at predefined positions of the worker's hands multiple times. These videos can then be processed to extract several images or landmark frames representative of the entire assembly process of the target object. These landmark frames can be used to extract hand tracking information that facilitates the definition of the positions or key points of the worker's hands and fingers during the assembly process.

[0047] In some embodiments, a bounding box prediction algorithm and a hand keypoint detector algorithm can be applied to extract hand tracking information. In particular, the bounding box prediction algorithm can include using threshold image segmentation to process landmark frames from the assembly process, thereby obtaining mask images for the worker's hands. The hands can be located on the mask using blob detection. The bounding box prediction uses the mask image to form a box around each of the worker's hands such that the box includes at least the highest point of the shape of the hand position up to the wrist point of the hand. The bounding boxes and their corresponding landmark frames are then fed into the hand keypoint detector algorithm.

[0048] The hand keypoint detector algorithm can include a machine learning model that can detect specific keypoints on the worker's hands. The hand keypoint detector algorithm can detect not only keypoints visible in the landmark frame, but also keypoints visible in the frame for joints, viewpoint, object, and hand interactions. Concealment (occlusion) Different hand positions can predict different key points. Concealment (occlusion) To generate points in different frames, a specific Concealment (occlusion) The point is Concealment (occlusion) Hand keypoint detectors cannot be used to detect hand movements with a certain level of certainty. Concealment (occlusion) However, it predicts the location of the keypoints. Concealment (occlusion) Predictions of keypoint locations that are made may result in the same keypoint locations being recorded for different hand positions. During a manual assembly process step, the hand keypoints defining the operator's hands are then provided to a second neural network.

[0049] In some embodiments, a second neural network is used to predict the quality of the final state of the assembled object. In some embodiments, the neural network can be based on a Long Short-Term Memory (LSTM) model. The LSTM has several cells that are arranged in a sequence and together represent the entire assembly process of the final object. The input to the LSTM cell can be manual keypoint data corresponding to the operator's actions at the particular step of the assembly process represented by the LSTM cell. Each cell in the LSTM decides whether to store information from a previous cell, chooses a value to update, implements the update to the cell, chooses a value to output, and then filters the values ​​so that the cell only outputs the chosen value. The LSTM can be a sequence to one model trained using an Adam optimizer or other adaptive learning rate optimization algorithm. The neural network uses the LSTM framework to correlate the input data extracted from the manual assembly process, thereby determining a quality measurement of the final product.

[0050] In some embodiments, video and hand tracking information representative of the assembly process for target objects, i.e., the input data used to train the model, can be collected from multiple workers performing the assembly process to assemble multiple target objects using a single set of assembly instructions. The target objects assembled by the workers can be used in a controlled environment to collect corresponding quality measures for the performance of the assembled objects, i.e., the output data required to train the model.

[0051] In some aspects, the training data used to generate the machine learning model may be from simulated data, real training data, and / or ground truth recordings of experts, either in combination or individually. In some embodiments, the simulated data results may be used to build the machine learning model, such as, but not limited to, a Reinforcement Learning (RL) agent. In other embodiments, the real training data may be used to build the machine learning model, such as, but not limited to, a Reinforcement Learning (RL) agent. The RL agent is rewarded for achieving good / desired results and punished for bad results.

[0052] In some instances, many RL agents (some based on actual training data and some based on simulated data) can be deployed to work in tandem and configured to maximize a cumulative award, e.g., assembling a manufactured product with the smallest deviation from an ideal model / example. Exemplary outcomes for which RL agents can be rewarded include completing a complete manufactured product in as few steps as possible, reducing the amount of material or time required to achieve the manufactured product. RL agents based on simulated data and RL agents based on actual training data can be used to determine optimal motion patterns and / or optimal assembly patterns that result in an optimal / desired manufactured article.

[0053] These two groups of RL agents (e.g., RL agents created based on real training data and RL agents created based on simulated data) can now work together and even compete, since they are both paid for the act of creating the optimal / desired manufacturing product. In some embodiments, data obtained from the simulated-based RL agents that yielded optimal assembly patterns for the optimal manufacturing product can be used to reduce the possibility space for the real training data set. For example, the simulated RL agents can be used to determine the optimal assembly patterns, and then real training data can be collected only for the optimal assembly patterns and not for the non-optimal assembly patterns. By focusing only on collecting real training data or only on the optimal assembly patterns, less training data can be collected and / or a greater capacity for collecting more real training data can be achieved, but only for the optimal assembly patterns.

[0054] Relying solely on reinforcement learning to optimize an assembly line has limitations because rewards can sometimes be in conflict. For example, in assembling a product, some RL agents can be rewarded for making the fewest spurious moves (e.g., folding and immediately undoing the fold, or adding a stitch and immediately removing the stitch), while other RL agents can be rewarded for speed. The RL agents that are rewarded for speed can determine that more spurious moves result in faster assembly times because fewer downstream corrections are required in the assembly process. Making such implementation trade-off decisions is not something that humans can easily determine. Even with experience and a large number of examples, humans still lack the computational ability to understand how subtle it is to understand the end results of different workers working in different ways.

[0055] To resolve these conflicting RL agent optimizations, a GAN can be deployed to act as a decision maker. The conflicts may be between RL agents based on real training data, between RL agents based on simulated data, and / or between RL agents based on real training data and RL agents based on simulated data.

[0056] In some embodiments, the GAN can test each of the RL agents and store the results to create more robust neural networks. The GAN works by taking RL agents and using a model that results in winners and losers in a zero-sum game. There is a "generator" and a "discriminator" in the GAN. The generator, in this case, will store the reward data from the opposing RL agents, and the discriminator will evaluate which of them is most relevant to the task of creating the desired manufacturing product. The GAN uses a deep network of nodes (or neurons) to determine how to weight the nodes. Since each RL agent believes that it has already made the optimal decision, it is the GAN's role to determine which of the opposing RL agents actually makes the most relevant choice, and the discriminator adjusts the weights accordingly. Once the zero-sum game between the opposing RL agents is started, a group of winners between the opposing RL agents will be generated, and only these winners will be used for the machine learning model used to optimize the workflow in the assembly line. Although a large amount of data may end up being generated to determine the winning RL agent, the results are much poorer than that used to create and find these winners, which are used as input nodes.

[0057] Once the RL agents that survived the GAN war and were properly rewarded have been determined, in some embodiments, they can be fed into another AI system called a Recurrent Neural Network (RNN). The RNN has many similarities to a CNN in that it is a Deep Learning Neural Network, and the end result is optimized through various forms of weighting of the input data. One difference is that unlike a CNN, which is a linear process from input to output, an RNN is a loop that feeds back the resulting output, and even the internal nodes, as new training information. The RNN is both a feedback and feedforward system, such as a GRU.

[0058] In some embodiments, the machine learning framework can be built using targeted GRU model-based learning. The GRU model can be chosen instead of reinforcement learning because of its predictive power and relatively short training time. The GRU is used in the RNN to distinguish between observations that should be stored in memory or update the state and those that should be forgotten or reset the state.

[0059] In some embodiments, the GRU model can be composed of several GRU cells corresponding to the number of assembly steps required to build the target object. Each GRU cell representing one of the number of assembly steps can have several input parameters and hidden state outputs. The GRU cell representing the final step in the assembly process will output the target object. The output of the model is the deviation of the target object from the nominal object. This deviation can be calculated using a stepwise Hausdorff distance from the target object to the nominal object and a performance metric of the final configuration of the nominal object. The individual GRU cells are defined by reset, update and new gates. The GRU neural network is trained iteratively, thereby biasing the GRU neural network towards solving a particular sub-problem and identifying a set of weights for the GRU. For example, at each iteration, several predictions are generated to complete the assembly process in the subsequent steps (one prediction for each possible error in a particular step). In addition, a corresponding predicted distance measure of the modified assembly process can be generated. These predicted assembly process completions can be depicted in a virtual representation system and their stepwise Hausdorff distances calculated to obtain a "ground truth" distance measure. The difference between the "ground truth" and the predicted distance measure can be calculated and fed back into the model, and its network weights are adjusted via backpropagation to generate the next iteration. This process can continue until the set of weights for the GRU is identified. In some embodiments, a stochastic gradient descent method can be used to correct flawed target objects and derive the steps required to obtain a satisfactory final configuration.

[0060] In some embodiments, simulations such as parametric computer-aided design and drafting (CAD) models of the target object in process can be generated to develop and validate the machine learning model. The CAD system can use a local coordinate frame corresponding to the current state of the target object in process and input parameters representing the individual assembly steps. Using the local coordinate frame and input parameters of the target object in process, the CAD system can determine dimensional information for each assembly step. The CAD system can then generate a three-dimensional CAD model representing the configuration of the output at each step. The CAD system can continue this process until all steps in the assembly process have been performed and can output a CAD model of the final configuration of the assembled object. By providing the CAD system with various input parameters, CAD models of different configurations can be generated. To obtain a set of CAD models with a particular range of input criteria, such as length or width, a statistical sample of the input criteria can be provided to the CAD system to generate the set of CAD models.

[0061] The CAD models may be of variable levels of detail and intelligence, but the trained models and systems are specifically designed to work with lower detail CAD systems, which can generate a vast number of examples in a non-computationally expensive manner, and provide sufficient surface morphology detail for trained models and profiling. In some embodiments, the referenced CAD system can be paired with a Finite Element Analysis (FEA) or basic surface modeling tool, which can generate a structural analysis of the surface. This data can be used as an additional quality score for model training and analysis.

[0062] In some embodiments, the CAD system can be incorporated into model training such that additional surface models can be generated on demand for examples from the model, or as additional exploration data is required. This approach pairs with physical observation and allows the development of pre-trained models without requiring vast amounts of physical samples of space.

[0063] In some embodiments, a CAD model of the final configuration of the assembled object can be used in the simulation to generate the performance metrics. By using a CAD model of the final configuration of the assembled object, the simulation can utilize numerical and computational methods to generate the performance metrics.

[0064] Real-world applications of Artificial Intelligence Process Control (AIPC) include providing feedback to workers on an assembly line who have already completed their tasks (e.g., by automatically modifying video instructions), as well as providing instructions to workers downstream on the assembly line who have not yet completed their tasks (also, for example, by automatically modifying video instructions) ("feedforward"). This feedback-feedforward system, or AIPC, can be achieved using the AI ​​techniques described herein, in some embodiments in the specific order described herein, such that workers on the assembly line can make choices that optimize the resulting manufactured product without the need for additional human oversight.

[0065] In some embodiments, this involves simply compressing the system into an RNN and observing every move in two ways, success or failure, during the process of creating one or more manufactured products. Each move acts as training. If the output node of the RNN is not optimal, the network can feed back to the actual individuals on the assembly line to make a different choice, and in the path through the many nodes and layers of the RNN, the weights can be reweighted and the output will be labeled as either success or failure. As the process iterates, the weights improve their accuracy on their own. Furthermore, the network can learn what is working and what is not working even when the individuals performing the assembly are not working. This increases the training set. It also allows adjustments to be made at different stages of the assembly process. In some cases, it may be found that the best way to produce a manufactured article with certain characteristics at any given moment is not to go back to the starting point, but to adjust the instructions as the process progresses. In this case, the RNN is constantly optimizing for the best manufactured product and learning to provide feedback to individual workers at worker stations on the production line who have already performed their tasks, and also to feed forward information to workers at worker stations on the production line who have not yet performed their tasks.

[0066] FIG 1 conceptually illustrates a flow chart of an exemplary process 100 for implementing production line deployment in accordance with some aspects of the disclosed technology. The process of FIG 1 begins at step 102 with production deployment being initiated. The exemplary production line workflow typically includes multiple worker stations (nodes) where workpieces (products) are assembled or otherwise manufactured. The various nodes may be organized in a sequential manner such that work at each subsequent node begins only after work at a preceding node is completed.

[0067] At step 104, one or more reference videos are generated and / or updated. The videos described above can be used to provide manufacturing / assembly instructions to specific nodes (also referred to herein as worker stations). That is, individual nodes in a workflow can be equipped with reference videos that deliver guidance on how to complete a step in the manufacturing workflow that corresponds to that particular node.

[0068] At step 106, each of the videos generated at step 104 is deployed to a respective station / node. As an example, a given workflow may include 10 nodes, each of which has its own different / unique reference video. In other implementations, the number of videos may be less than the total number of nodes. Depending on the implementation, the reference videos deployed to the various stations / nodes may be unique or may provide similar guidance / instructions. As discussed in more detail below, the content of the reference videos may be dynamic and may be constantly updated / augmented.

[0069] At step 108, a continuous recording of motion is captured at each station / node. The motion data obtained from the motion recording can describe the worker's interaction with the workpiece / component / tool ​​at those nodes / stations in the workflow. That is, the motion data captured at each node can represent one or more worker actions corresponding to a particular part of the product assembly or manufacturing, and can correspond to instructions provided by a reference video associated with that node. In some instances, the motion capture can include the capture of video data, i.e., the recording of all or a portion of the worker's actions at that station. In other embodiments, the motion capture can include, for example, the recording of a 3D point cloud, where motion is recorded relative to one or more specific points in the field of view of the image capture device. Both the worker's actions as well as the component's attributes (e.g., component quality, tensile strength, number of defects) can be captured at each node / station in the workflow.

[0070] At step 110, a process recipe analysis deviation can be calculated, where the motion data captured for one or more of the stations at step 108 can be analyzed to, for example, identify any deviations from a comparison model that includes (or represents) an idealized motion profile of the corresponding station. As illustrated in FIG. 1, step 110 can utilize an AIPC deep learning model that can be configured to, for example, identify / classify motion deviations from the comparison model and make inferences regarding the extent to which the assembly or manufacturing process may have been affected (step 112). The comparison can be performed at a per station level and / or at an overall process level. The analysis can also take into account the attributes of components at individual stations, or deviations of components from a baseline, and the extent to which station motion deviations affect component quality.

[0071] The AIPC deep learning model referred to in step 112 can be based on the collection of various types of training data, which can include, for example, examples of ideal or quality-controlled assembly / manufacturing interactions for a given station / node. The AIPC deep learning model can also be augmented (or adjusted) using data provided by domain / industry information 115, by customer feedback regarding a particular product manufactured using process 100 (step 111), and by feedback from quality control inspection of a particular product manufactured using process 100 (step 113). It should be appreciated that the AIPC deep learning model can be realized using various computational systems including distributed hardware and / or software modules. As an example, the AIPC deep learning model can be realized using a distributed system including multiple image capture and display devices deployed on an assembly line and coupled to one or more systems configured to realize various AI / machine learning models and / or classifiers.

[0072] If deviations from the comparison model are detected / identified in step 110, automatic adjustments can be generated in step 114 using the AIPC deep learning model 112. As discussed above, the video adjustments can be aimed at improving manufacturing / assembly quality at one or more stations in the workflow. For example, the video adjustments can be applied at a given node / station where an error is known (or predicted) to occur, e.g., to change the instructions or guidance provided to the worker in a way that reduces the error or repairs the error at the time it occurs. In other embodiments, the video adjustments can be applied downstream from the station where the error occurred, e.g., to correct the error before the manufacturing workflow is completed. In still other embodiments, once the workflow is completed, the entire workflow can be analyzed and adjustments can be implemented for one or more stations in the workflow.

[0073] In some embodiments, the adjustments are made in real time immediately after an error is detected, while in other embodiments, the adjustments are made at regular intervals or after a workflow is completed.

[0074] In some aspects, the automated adjustments determined in step 114 may be summarized and / or provided as a product quality report in step 117. For example, the adjustments provided by the analysis of motion deviations (step 110) may be used to generate one or more quality reports that describe various quality aspects of the workpiece based on the identified deviations from an idealized model of the assembly / manufacturing process.

[0075] FIG. 2 illustrates an example process 200 for performing error detection analysis that can be used to facilitate assembly error correction, in accordance with some aspects of the present technique.

[0076] Beginning at step 210, a process for improving manufacturing / assembly using idealized video guidance can be implemented. At step 215, video tracking of one or more assembly stations is performed. Video tracking can include recording of a human worker at a given station / node. In some embodiments, video tracking can further include capturing component attributes at a given station / node.

[0077] At steps 220-224, processing is performed to analyze the recorded video from the assembly station. For example, in some embodiments, background extraction may be performed to isolate motion / components in the recorded video. In some aspects, once background extraction is complete, the processed video contains only motion / video data associated with the assembler (step 224) and the included components are used for the corresponding assembly step (step 220). At step 220, additional processing may be performed to isolate parts / components. As illustrated by the diagram of process 200, step 220 may include additional processing operations including anomaly detection (step 221), surface change detection (222) and part classification and / or quality scoring (step 223). It should be appreciated that any of the video processing steps may be performed using a variety of signal and / or image processing techniques, including but not limited to the use of one or more AI / machine learning algorithms and / or classifiers, to perform, for example, anomaly detection (221), surface change detection (222), and / or scoring / classification (step 223).

[0078] After processing steps 220-224 are completed, process 200 may proceed to step 226 where a motion comparison is performed. The motion comparison (step 226) may include a comparison of process assembly station video data including one (1) or multiple station operators at one or more stations / nodes with corresponding idealized video / motion data. Motion comparisons performed across multiple stations / nodes may be used to infer / predict changes in resulting part / component quality.

[0079] Variance / quality classification for the various parts / components may be performed at step 228. As an example, the parts / components may be classified into different quality tiers and / or identified for removal or repair depending on their associated classification / variance.

[0080] After the classifications / discrepancies are determined, process 200 may proceed to step 230 where an analysis of the entire process / workflow is performed based on the classifications / discrepancies per station / node determined, for example, in steps 226 and 228. By analyzing the entire workflow, automatic adjustments can be performed on the video to address detected deviations / defects, as discussed above.

[0081] FIG. 3 illustrates an exemplary processing device that can be used to implement the system of the disclosed technology. Processing device 300 includes a master central processing unit (CPU) 362, an interface 368, and a bus 315 (e.g., a PCI bus). Operating under the control of appropriate software or firmware, CPU 362 is responsible for implementing the various error detection monitoring and process adjustment steps of the disclosed technology. CPU 362 preferably accomplishes all these functions under the control of software, including an operating system and any appropriate application software. CPU 362 can include one or more processors 363, such as a processor from the Motorola family of microprocessors or from the MIPS family of microprocessors. In an alternative embodiment, processor 363 is hardware specifically designed to control the operation of AIPC system 310. In a particular embodiment, memory 361 (such as non-volatile RAM and / or ROM) also forms part of CPU 462. However, there are many different ways in which memory can be coupled to the system.

[0082] In some embodiments, the processing device 310 can include or be coupled to an image processing system 370. The image processing system 370 can include various image capture devices, such as video cameras, that can monitor the movements of the worker and generate motion data. As an example, the image processing system 370 can be configured to capture video data and / or output / generate a 3D point cloud.

[0083] The interfaces 368 are typically provided as interface cards (sometimes called "line cards"). Generally, they control the transmission and reception of data packets over the network and sometimes support other peripheral devices used with the router. Interfaces that may be provided include, among others, Ethernet interfaces, frame relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, and so on. In addition, various very high speed interfaces may be provided, such as high speed token ring interfaces, wireless interfaces, Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, and so on. Generally, these interfaces may include ports suitable for communication with an appropriate medium. In some cases, they may also include an independent processor, and in some instances may include volatile RAM. The independent processor may control such communication intensive tasks as packet switching, media control and management. By providing separate processors for communication intensive tasks, these interfaces may allow the master microprocessor 362 to efficiently perform routing calculations, network diagnostics, security functions, and so on.

[0084] The system shown in Figure 3 is one particular processing device of the present invention, but is by no means the only network device architecture in which the present invention can be implemented. For example, architectures having a single processor that handles communications as well as routing computations, etc. are often used. Additionally, other types of interfaces and media may also be used.

[0085] Regardless of the configuration of the network device, the network device may employ one or more memories or memory modules (including memory 361) configured to store program instructions for general network operation, and mechanisms for roaming, route optimization, and routing functions described herein. The program instructions may control the operation of an operating system and / or one or more applications, for example. The one or more memories may also be configured to store tables, such as mobility binding, registration and association tables, and the like.

[0086] The logical operations of the various embodiments are realized as (1) a series of computer-implemented steps, operations or procedures running on a programmable circuit in a general-purpose computer, (2) a series of computer-implemented steps, operations or procedures running on a specific application programmable circuit, and / or (3) a program engine in an interconnected machine module or programmable circuit. The system 300 can practice all or a portion of the described methods, can be part of the described systems, and / or can operate according to instructions in the described non-transitory computer-readable storage media. Such logical operations can be realized as modules configured to control the processor 363 to perform certain functions according to the programming of that module.

[0087] It should be understood that any particular order or hierarchy of steps in the disclosed processes is illustrative of example approaches. It should be understood that the particular order or hierarchy of steps in the processes can be rearranged or only some of the steps illustrated can be performed based on design preferences. Some of the steps can be performed simultaneously. For example, in certain situations, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the program components and systems described can generally be integrated together in a single software product or packaged in multiple software products.

[0088] The above description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Accordingly, the claims are not intended to be limited to the aspects shown herein, and the claims are to be consistent with the full scope of the claims consistent with the language, and references to singular elements are intended to mean "one or more," rather than "one and only one," unless expressly stated to the contrary.

[0089] A phrase such as "an embodiment" does not imply that such an embodiment is essential to the subject technology or that such an embodiment applies to all configurations of the subject technology. Disclosure relating to an embodiment can apply to all configurations, or can apply to one or more configurations. A phrase such as an embodiment can mean one or more embodiments, and one or more embodiments can mean one embodiment. A phrase such as "a configuration" does not imply that such an embodiment is essential to the subject technology or that such an embodiment applies to all configurations of the subject technology. Disclosure relating to a configuration can apply to all configurations, or can apply to one or more configurations. A phrase such as an embodiment can mean one or more configurations, and one or more configurations can mean one configuration.

[0090] The word "exemplary" is used herein to mean "serving as an example or illustration." Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Disclosure statement

[0091] statement 1: A method for optimizing a workflow in an assembly line, the method comprising the steps of detecting an error in the assembly of a target object at a step in an assembly process of the target object, evaluating the target object and a nominal object at that step in the assembly process, thereby obtaining a comparison, determining, based on the comparison, a series of steps required to minimize deviations between the target object and the nominal object, and adjusting assembly instructions for the target object based on the series of steps.

[0092] statement 2: statement 1, in which the target object is evaluated against a nominal object in a preceding step of the assembly process.

[0093] statement 3: statement In any of the methods of 1 to 2, the target object is evaluated against the final configuration of the nominal object.

[0094] statement 4: statement Any of the methods of 1 to 3, wherein the sequence of steps is determined using a machine learning model configured to minimize deviation.

[0095] statement 5: statement Any of the methods of 1 to 4, wherein the deviation is determined based on a similarity between the above sequence of steps for completing the assembly process of the target object and another sequence of steps for completing the assembly process of a nominal object.

[0096] statement 6: statement Any of the methods 1 to 5, wherein the deviation is minimized using a Markov Decision Process (MDP) via a reward formulation.

[0097] statement 7: statement Any of the methods from 1 to 6, in which the above sequence of steps for completing the assembly process of the target object is derived using a stochastic gradient descent method.

[0098] statement8: A system for optimizing a workflow in an assembly line, the system including: a plurality of image capture devices, each of the image capture devices being positioned at a different position to capture a motion of a worker during an assembly process of a target object; and an assembly instruction module coupled to the plurality of image capture devices, the assembly instruction module being configured to automatically modify guidance and instructions provided to a worker, the assembly instruction module being configured to perform operations including: receiving, by the assembly instruction module, motion data from the plurality of image capture devices, the motion data corresponding to a performance by a worker of a set of steps for assembling a target object; determining an error in the assembly of the target object based on the motion data at a step of the set of steps; evaluating a target object and a nominal object at the step of the set of steps, thereby obtaining a comparison; determining, based on the comparison, a series of steps required to minimize a deviation between the target object and the nominal object; and adjusting the assembly instructions provided to the worker based on the series of steps.

[0099] statement 9: statement 8, wherein the motion data includes a digital recording of the worker's hand movements while assembling the target object.

[0100] statement 10: statement 10. The system of any of claims 8 to 9, wherein the building instruction module is further configured to apply a stochastic gradient descent method to derive the sequence of steps.

[0101] statement 11: statement 11. The system of any of claims 8 to 10, wherein the assembly instruction module is further configured to determine the sequence of steps using a machine learning model, the machine learning model being configured to minimize deviations.

[0102] statement 12: statement 8 to 11, wherein the deviation is determined based on a similarity between the sequence of steps for completing the assembly of a target object and another sequence of steps for completing the assembly of a nominal object.

[0103] statement 13: statement 8 to 12, in which the deviation is minimized using a Markov Decision Process (MDP) via a reward formulation.

[0104] statement 14: statement 14. Any of the systems of 8 to 13, wherein the assembly instruction module is further configured to derive a set of images from the motion data representing assembly of the target object, and to evaluate the set of images to identify performance of the set of steps by a worker to assemble the target object.

[0105] statement 15: A non-transitory computer-readable medium having instructions stored thereon, the instructions being configured, when executed by one or more processors, to cause the processor to execute instructions including: detecting an error in assembly of a target object at a step in an assembly process of the target object; evaluating the target object and a nominal object at said step in the assembly process thereby obtaining a comparison; determining, based on the comparison, a series of steps required to minimize deviations between the target object and the nominal object; and adjusting assembly instructions for the target object based on said series of steps.

[0106] statement 16: statement 15. The non-transitory computer readable medium of claim 15, further comprising instructions configured to cause the processor to derive the sequence of steps using stochastic gradient descent.

[0107] statement 17: statement 17. The non-transitory computer-readable medium of any of 15 to 16, wherein the target object is evaluated against a nominal object in said step of the assembly process.

[0108] statement 18: statement 18. The non-transitory computer-readable medium of any of 15 to 17, wherein a target object is evaluated against a final configuration of a nominal object.

[0109] statement 19: statement 19. The non-transitory computer-readable medium of any of 15 to 18, wherein the instructions are further configured to cause the processor to determine the sequence of steps using a machine learning model configured to minimize deviations.

[0110] statement 20: statement 20. The non-transitory computer readable medium of any of 15 to 19, wherein the deviation is minimized using a Markov Decision Process (MDP) via a reward formulation.

Claims

1. 1. A method for generating a plurality of machine learning models for optimizing an assembly process in an assembly line, comprising: generating a first training data set for training a first machine learning model, the first training data set including simulated video data of a simulated worker on a simulated assembly line performing an assembly process, actual video data of an actual worker on the assembly line performing the assembly process, and corresponding ground truth video data of a skilled worker on the assembly line performing the assembly process; generating a second training data set from the first training data set for training a second machine learning model, the second training data set including ground truth final quality metrics of a manufactured article shown in the simulation video data, the actual video data, and the corresponding ground truth video data, as well as keypoint data extracted from the simulation worker in the simulation video data, the actual worker in the actual video data, and a human expert in the ground truth video data, to predict a final quality metric of an object at any point in the assembly process; training the first machine learning model to detect errors in worker movements in the assembly process of assembling a target object based on the simulation video data, the actual video data, and the corresponding ground truth video data; training the second machine learning model to predict a final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; and optimizing an assembly process using the final quality metric of the target object and the worker movement errors.

2. Generating the second training data set for training the second machine learning model includes: The method of claim 1 , comprising applying a keypoint detector algorithm to the simulated video data and the actual video data to detect the keypoint data.

3. Generating the second training data set for training the second machine learning model includes: identifying a set of keypoints visible within a frame of the simulated video data or the actual video data; The method of claim 1 , further comprising identifying a second set of keypoints that are hidden (occluded) from the frame.

4. Identifying the second set of keypoints that are occluded from the frame includes:

4. The method of claim 3, comprising predicting a location of the second set of keypoint data based on visible keypoint data in a previous frame of the simulated video data or the actual video data.

5. 2. The method of claim 1, wherein the second machine learning model is a Long Short-Term Memory (LSTM) model that includes a plurality of cells that are arranged in an ordered sequence and together represent the assembly process.

6. training the second machine learning model to predict the final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; 6. The method of claim 5, comprising training the second machine learning model to learn what information to store as the second training data set passes through each cell of the plurality of cells.

7. training the second machine learning model to predict the final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; supplementing the second training data set with the simulation video data, the actual video data, and the ground truth video data; The method of claim 1 , comprising training the second machine learning model with the extracted keypoint data, the simulated video data, and the actual video data.

8. A non-transitory computer-readable storage medium comprising one or more sets of instructions that, when executed by one or more processors, cause a computer system to perform operations, the operations including: generating a first training data set for training a first machine learning model, the first training data set including simulated video data of a simulated worker on a simulated assembly line performing an assembly process, actual video data of an actual worker on the assembly line performing the assembly process, and corresponding ground truth video data of a skilled worker on the assembly line performing the assembly process; generating a second training data set from the first training data set for training a second machine learning model, the second training data set including ground truth final quality metrics of a manufactured article shown in the simulation video data, the actual video data, and the corresponding ground truth video data, as well as keypoint data extracted from the simulation worker in the simulation video data, the actual worker in the actual video data, and a human expert in the ground truth video data, to predict a final quality metric of an object at any point in the assembly process; training the first machine learning model to detect errors in worker movements in the assembly process of assembling a target object based on the simulation video data, the actual video data, and the corresponding ground truth video data; training the second machine learning model to predict a final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; A non-transitory computer-readable storage medium that optimizes an assembly process using the final quality metrics of the target object and the worker movement errors.

9. Generating the second training data set for training the second machine learning model includes:

10. The non-transitory computer readable storage medium of claim 8, comprising applying a keypoint detector algorithm to the simulated video data and the actual video data to detect the keypoint data.

10. Generating the second training data set for training the second machine learning model includes: identifying a set of keypoints visible within a frame of the simulated video data or the actual video data; 10. The non-transitory computer readable storage medium of claim 8, further comprising identifying a second data set of keypoints that are hidden (occluded) from the frame.

11. Identifying a second set of keypoints that are occluded from the frame includes:

11. The non-transitory computer readable storage medium of claim 10, comprising predicting a location of the second set of keypoint data based on visible keypoint data in a previous frame of the simulated video data or the actual video data.

12. 10. The non-transitory computer-readable storage medium of claim 8, wherein the second machine learning model is a Long Short-Term Memory (LSTM) model including a plurality of cells arranged in an ordered sequence that together represent the assembly process.

13. training the second machine learning model to predict the final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; 10. The non-transitory computer-readable storage medium of claim 8, further comprising training the second machine learning model to learn what information to store as the second training data set passes through each cell of a plurality of cells.

14. training the second machine learning model to predict the final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; supplementing the second training data set with the simulation video data, the actual video data, and the ground truth video data; 10. The non-transitory computer-readable storage medium of claim 8, further comprising training the second machine learning model with the extracted keypoint data, the simulated video data, and the actual video data.

15. Processor and 1. A system including a memory, comprising: The memory stores programmed instructions that, when executed by the processor, cause the system to perform operations, including: generating a first training data set for training a first machine learning model, the first training data set including simulated video data of a simulated worker on a simulated assembly line performing an assembly process, actual video data of an actual worker on the assembly line performing the assembly process, and corresponding ground truth video data of a skilled worker on the assembly line performing the assembly process; generating a second training data set from the first training data set for training a second machine learning model, the second training data set including ground truth final quality metrics of a manufactured article shown in the simulation video data, the actual video data, and the corresponding ground truth video data, as well as keypoint data extracted from the simulation worker in the simulation video data, the actual worker in the actual video data, and a human expert in the ground truth video data, to predict a final quality metric of an object at any point in the assembly process; training the first machine learning model to detect errors in worker movements in the assembly process of assembling a target object based on the simulation video data, the actual video data, and the corresponding ground truth video data; training the second machine learning model to predict a final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; The system optimizes the assembly process using the final quality metric of the target object and the worker movement errors.

16. Generating the second training data set for training the second machine learning model includes: The system of claim 15 , further comprising applying a keypoint detector algorithm to the simulated video data and the actual video data to detect the keypoint data.

17. Generating the second training data set for training the second machine learning model includes: identifying a set of keypoints visible within frames of the simulation video data and the actual video data; The system of claim 15 further comprising identifying a second set of keypoints that are hidden (occluded) from the frame.

18. Identifying the second set of keypoints that are occluded from the frame includes:

20. The system of claim 17, further comprising predicting a location of the second set of keypoint data based on visible keypoint data in a previous frame of the simulated video data or the actual video data.

19. 16. The system of claim 15, wherein the second machine learning model is a Long Short-Term Memory (LSTM) model that includes a plurality of cells that are arranged in an ordered sequence and that together represent the assembly process.

20. training the second machine learning model to predict the final quality metric of the target object by identifying correlations between the ground truth final quality metric and the extracted keypoint data; 20. The system of claim 19, further comprising training the second machine learning model to learn what information to store as the second training data set passes through each cell of the plurality of cells.

Images