Systems, methods and media for use in manufacturing processes

Abstract

A manufacturing system is disclosed herein. The manufacturing system includes one or more stations, a monitoring platform, and a control module. Each of the one or more stations is configured to perform at least one step in a multi-step manufacturing process of a component. The monitoring platform is configured to monitor the progress of the component throughout the multi-step manufacturing process. The control module is configured to dynamically adjust the processing parameters of each step of the multi-step manufacturing process to achieve a desired final quality metric of the component.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application Serial No. 62 / 931,448 filed on November 6, 2019, U.S. Provisional Application Serial No. 62 / 932,063 filed on November 7, 2019, and U.S. Provisional Application Serial No. 62 / 931,453 filed on November 6, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure generally relates to a system, method, and medium for use in a manufacturing process. Background Art

[0004] Automation has controlled the production of goods since the beginning of the Industrial Revolution in the 18th century. While today’s factories have fully embraced automation as a core principle—robots perform many repeatable tasks in high-production environments—many assembly tasks continue to be performed by humans. These tasks are difficult to automate due to cost, risk of catastrophic failure, or the organizational effort of deploying robotic systems for low-volume production runs. These production lines are supervised by standard process controls and personnel management, such that assemblers are taught to perform certain quality metrics over time, or they are replaced by another operator. This process has remained largely unchanged since the advent of assembly. Summary of the invention

[0005] In some embodiments, a manufacturing system is disclosed herein. The manufacturing system includes a monitoring platform, a control module, and one or more stations. Each station is configured to perform at least one step in a multi-step manufacturing process of a component. The monitoring platform is configured to monitor the progress of the component throughout the multi-step manufacturing process. The control module is configured to dynamically adjust the processing parameters of the steps of the multi-step manufacturing process to achieve the desired final quality metric of the component, and the control module is configured to perform operations. The operation includes receiving image data of a tool at a first station of one or more stations. The operation also includes identifying a set of key points from the image data. The key points correspond to position information during the processing of the tool at the first station. The operation also includes determining the final quality metric of the component based on the key points through a machine learning model. The operation also includes determining that the final quality metric is not within a threshold tolerance of the final quality metric. The operation also includes: based on the determination, updating the processing parameters of subsequent stations in the multi-step manufacturing process.

[0006] In some embodiments, a manufacturing system is disclosed herein. The manufacturing system includes a monitoring platform, a control module, and one or more stations. Each station is configured to perform at least one step in a multi-step manufacturing process of a component. The monitoring platform is configured to monitor the progress of the component throughout the multi-step manufacturing process. The control module is configured to dynamically adjust the processing parameters of the steps of the multi-step manufacturing process to achieve the desired final quality metric of the component, and the control module is configured to perform operations. The operation includes identifying position information corresponding to the component at a first processing station in one or more processing stations. The operation also includes determining that there is an irreversible error based on the position information corresponding to the component. The operation also includes generating an updated instruction set based on the determination to correct the irreversible error, and the updated instruction set is executed by the downstream station. The operation also includes predicting the final quality metric of the component by the machine learning model based on the updated instruction set. The operation also includes providing the updated instruction set to the downstream station based on the predicted final quality metric.

[0007] In some embodiments, a manufacturing system is disclosed herein. The manufacturing system includes a monitoring platform, a control module, and one or more stations. Each station is configured to perform at least one step in a multi-step manufacturing process of a component. The monitoring platform is configured to monitor the progress of the component throughout the multi-step manufacturing process. The control module is configured to dynamically adjust the processing parameters of the steps of the multi-step manufacturing process to achieve the desired final quality metric of the component, and the control module is configured to perform operations. The operation includes receiving image data of a tool of a first station of one or more stations. The operation also includes identifying a set of key points from the image data. The key points correspond to the position information of the tool during processing at the first station. The operation also includes determining the final quality metric of the component based on the key points by a machine learning model. The operation also includes determining that the final quality metric is not within a threshold tolerance of the final quality metric. The operation also includes inferring the position information of the component at the first processing station based on the determination. The operation also includes generating an updated instruction set to be executed by a downstream station based on the determination. The operation also includes predicting the final quality metric of the component by the machine learning model based on the updated instruction set. The operation also includes providing the updated instruction set to the downstream station based on the predicted final quality metric. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to be able to understand in detail the manner in which the above-mentioned features of the present disclosure are achieved, a more specific description of the present disclosure briefly outlined above may be obtained by reference to embodiments (some of which are shown in the accompanying drawings). However, it should be noted that the accompanying drawings only illustrate typical embodiments of the present disclosure and are therefore not to be considered as limiting its scope, as the present disclosure may allow other equally effective embodiments.

[0009] Figure 1 is a block diagram illustrating a manufacturing environment according to an example embodiment.

[0010] Figure 2 is a block diagram illustrating a control module according to an exemplary embodiment.

[0011] Figure 3 is a block diagram illustrating an exemplary architecture of an LSTM model according to an example embodiment.

[0012] Figure 4 is a block diagram that pictorially illustrates the overall process flow for a feedback section of a tool module according to an example embodiment.

[0013] Figure 5 is a block diagram illustrating the architecture of a GRU model according to an example embodiment.

[0014] Figure 6 is a flow chart illustrating a method of calibrating a multi-step manufacturing process according to an example embodiment.

[0015] Figure 7 is a flow chart illustrating a method of calibrating a multi-step manufacturing process according to an example embodiment.

[0016] Fig. 8A A system bus computing system architecture is shown according to an example embodiment.

[0017] Figure 8B A computer system having a chipset architecture according to an example embodiment is shown.

[0018] To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation. DETAILED DESCRIPTION

[0019] The manufacturing process can be complex and include raw materials that are processed by different processing stations (or "stations") until the final component is produced. In some embodiments, each processing station receives input for processing and can output an intermediate output that can be passed to a subsequent (downstream) processing station for additional processing. In some embodiments, the final processing station can receive input for processing and can output a final component or more generally a final output.

[0020] In some embodiments, each station may include one or more tools / equipment that can perform a set of processing steps. Exemplary processing stations may include, but are not limited to, conveyor belts, injection molding presses, cutting machines, die presses, extruders, computer numerical control (CNC) milling machines, grinders, assembly stations, three-dimensional printers, quality control stations, verification stations, etc.

[0021] In some embodiments, the operation of each processing station can be managed by one or more process controllers. In some embodiments, each processing station may include one or more process controllers, and one or more process controllers may be programmed to control the operation of the processing station. In some embodiments, an operator or a control algorithm may provide station controller set points for the station controller, which may represent the expected value or value range of each control value. In some embodiments, the values ​​used for feedback or feedforward in the manufacturing process may be referred to as control values. Exemplary control values ​​may include, but are not limited to: speed, temperature, pressure, vacuum, rotation, current, voltage, power, viscosity, materials / resources used at the station, production rate, interruption time, harmful fumes, etc.

[0022] One or more techniques provided herein improve conventional processes by providing a system through which images and / or video data can be used to predict or forecast the final quality metric of a component. In some embodiments, the system may include a monitoring system configured to capture or record video and / or image data of a tool at each processing node or station of a manufacturing system. Based on the positioning of the tool during the processing step, the system may be configured to predict or forecast the final quality metric of a component. If the predicted final quality metric falls outside the range of acceptable values, the system may generate updated processing instructions and provide updated processing instructions to downstream processing nodes and / or stations in an attempt to correct any errors in the process so that the desired final quality metric can be achieved.

[0023] In some embodiments, the system may include a monitoring system configured to capture or record video and / or image data of components at each processing node or station of the manufacturing system. Based on visual information about the components at the end of the processing node, the system may be configured to predict or forecast a final quality metric of the component. If the predicted final quality metric falls outside a range of acceptable values, the system may generate updated processing instructions and provide updated processing instructions to downstream processing nodes and / or stations in an attempt to correct any errors in the process so that the desired final quality metric can be achieved.

[0024] In this way, the present system is able to predict or forecast the final quality metric of a component at any stage of the manufacturing process without actually testing the component. Such a system is particularly useful for final quality metrics that would otherwise require destructive testing, or for final quality metrics that cannot be assessed until the process is complete.

[0025] Figure 11 is a block diagram illustrating a manufacturing environment 100 according to an example embodiment. The manufacturing environment 100 may include a manufacturing system 102, a monitoring platform 104, and a control module 106. The manufacturing system 102 may broadly represent a multi-step manufacturing system. In some embodiments, the manufacturing system 102 may represent an assembly line system, wherein each processing station may represent a worker. In some embodiments, the manufacturing system 102 may represent a manufacturing system for use in additive manufacturing (e.g., a 3D printing system). In some embodiments, the manufacturing system 102 may represent a manufacturing system for use in subtractive manufacturing (e.g., CNC machining). In some embodiments, the manufacturing system 102 may represent a manufacturing system for use in a combination of additive manufacturing and subtractive manufacturing. More generally, in some embodiments, the manufacturing system 102 may represent a manufacturing system for use in a general manufacturing process.

[0026] Manufacturing system 102 may include one or more stations 1081-108 n (generally, "station 108"). Each station 108 may represent a step and / or station in a multi-step manufacturing process. For example, each station 108 may represent a layer deposition operation in a 3D printing process (e.g., station 1081 may correspond to layer 1, station 1082 may correspond to layer 2, etc.). In another example, each station 108 may correspond to a specific processing station. In another example, each station 108 may correspond to a specific human operator who performs a specific task in an assembly line manufacturing process.

[0027] Each station 108 may include a process controller 114 and control logic 116. Each process controller 1141-114 n may be programmed to control the operation of each respective station 108. In some embodiments, the control module 106 may provide a station controller set point for each process controller 114, which may represent a desired value or range of values ​​for each control value. The control logic 116 may reference properties / parameters associated with the processing steps of the station 108. In operation, the control logic 116 for each station 108 may be dynamically updated by the control module 106 throughout the manufacturing process based on the current trajectory of the final quality metric.

[0028] The monitoring platform 104 may be configured to monitor each station 108 of the manufacturing system 102. In some embodiments, the monitoring platform 104 may be a component of the manufacturing system 102. For example, the monitoring platform 104 may be a component of a 3D printing system. In some embodiments, the monitoring platform 104 may be independent of the manufacturing system 102. For example, the monitoring platform 104 may be retrofitted to an existing manufacturing system 102. In some embodiments, the monitoring platform 104 may represent an imaging device configured to capture images of a component or tool (e.g., a worker or a processing tool) at each step of a multi-step process. For example, the monitoring platform 104 may be configured to capture images of components at each station 108 and / or images of components (e.g., tools, people, etc.) developing components at each station 108. In general, the monitoring platform 104 may be configured to capture information associated with the production of components (e.g., images, voltage readings, speed readings, etc.) and / or tools (e.g., hand positions, tool positions, etc.), and provide the information as input to the control module 106 for evaluation.

[0029] The control module 106 can communicate with the manufacturing system 102 and the monitoring platform 104 via one or more communication channels. In some embodiments, the one or more communication channels may represent a separate connection via the Internet (such as a cellular or Wi-Fi network). In some embodiments, the one or more communication channels may use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, or the like. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , Ambient Backscatter Communication (ABC) protocol, USB, direct connection of WAN or LAN, etc. to connect terminals, services and mobile devices.

[0030] The control module 106 may be configured to control each process controller of the manufacturing system 102. For example, based on information captured by the monitoring platform 104, the control module 106 may be configured to adjust a process control associated with a particular station 108. In some embodiments, the control module 106 may be configured to adjust a process control of a particular station 108 based on a projected final quality metric.

[0031] Figure 2 2 is a block diagram illustrating the control module 106 according to an exemplary embodiment. The control module 106 may include a tool module 202 and a component module 204.

[0032] The tool module 202 can be configured to predict the final quality metric of the specimen at a given production stage based on the image data obtained by the monitoring platform 104. In operation, the control module 106 can receive input from the monitoring platform 104. In some embodiments, such input can take the form of an image or video of a tool performing a subprocess at a given step of a multi-step manufacturing process. For example, the image or video data can include an image or video data of a person's hand when performing a specific subprocess of the multi-step manufacturing process. In another example, the image or video data can include an image or video data of a three-dimensional printer performing a specific layer of a multi-layer manufacturing process. Based on the input, the control module 106 can predict the final quality metric of the part. Based on the predicted final quality metric of the part, the control module 106 can determine one or more actions to be taken in subsequent manufacturing steps in order to achieve a desired or threshold final quality metric. For example, if the predicted final quality metric falls outside the range of acceptable values, the control module 106 can take one or more actions to correct the manufacturing process. In some embodiments, the control module 106 can interface with the station controller in the subsequent station 108 to adjust their corresponding control and / or station parameters. In some embodiments, the control module 106 may provide the human manufacturer with updated instructions to be executed at each downstream processing station in the production line. These adjustments may help correct the manufacturing process so that the final quality metric may be within the range of acceptable quality metrics.

[0033] The component module 204 may be configured to predict the final quality metric of the specimen at a given production stage based on the image data obtained by the monitoring platform 104. In some embodiments, the component module 204 may receive input from the monitoring platform 104. In some embodiments, such input may take the form of an image or video of a component at a given step of a multi-step manufacturing process. In some embodiments, the component module 204 may receive inferred component data at a given step of a multi-step manufacturing process from the tool module 202. For example, the tool module 202 may infer information about the component at a given step of a multi-step manufacturing process based on tool images or video data and provide the inferred information as input to the component module 204. Based on the input, the component module 204 may predict the final quality metric of the component. Based on the predicted final quality metric of the component, the component module 204 may determine one or more actions to be taken in subsequent manufacturing steps in order to achieve a desired or threshold final quality metric. For example, if the predicted final quality metric falls outside the range of acceptable values, the component module 204 may identify one or more actions to correct the manufacturing process. In some embodiments, the control module 106 may interface with the station controllers in the subsequent stations 108 to adjust their respective control and / or station parameters. In some embodiments, the control module 106 may provide the human manufacturer with updated instructions to be executed at each downstream processing station of the production line. These adjustments may help to correct the manufacturing process so that the final quality metric may be within the range of acceptable quality metrics.

[0034] Each of the tool module 202 and the component module 204 may include one or more software modules. One or more software modules may be a collection of codes or instructions stored on a medium (e.g., a memory of a computing system associated with the control module 106), which represents a series of machine instructions (e.g., program code) that implement one or more algorithm steps. Such machine instructions may be actual computer code that a processor interprets to implement instructions, or alternatively, may be a higher-level encoding of instructions that are interpreted to obtain actual computer code. One or more software modules may also include one or more hardware components. One or more aspects of the example algorithm may be performed by a hardware component (e.g., a circuit) itself, rather than as a result of an instruction. In addition, in some embodiments, each of the tool module 202 and the component module 204 may be configured to transmit one or more signals between components. In such embodiments, such signals may not be limited to machine instructions executed by a computing device.

[0035] In some embodiments, tool module 202 and component module 204 can communicate via one or more local area networks. The network can be of any suitable type, including a separate connection via the Internet (such as a cellular or Wi-Fi network). In some embodiments, the network can use a wireless network such as radio frequency identification (RFID), near field communication (NFC), Bluetooth, or the like. TM , Bluetooth Low Energy TM (BLE), Wi-Fi TM 、ZigBee TM , ambient backscatter communication (ABC) protocol, USB, WAN or LAN, etc. to connect terminals, services and mobile devices. Because the information transmitted may be personal or confidential, security issues may dictate that one or more of these types of connections be encrypted or otherwise protected. However, in some embodiments, the information transmitted may be less personal, so a network connection may be selected for convenience rather than security.

[0036] The tool module 202 may include a collection system 206, an extractor module 208, and a prediction module 210. In general, a multi-node or multi-station assembly environment (such as the manufacturing system 102) may be broadly represented as in can represent the state of the component at all i-nodes, and It can represent a set of actions to be performed by the tool on the part at all i nodes. Given a standard or specification quality measure V c , the tool module 202 may be configured to utilize the estimated quality metric Optimize the error in the assembly process so that At the threshold distance V c In some embodiments, the tool module 202 can estimate the state And actions state can be a digital representation of the state of the component at all N nodes, the action Can represent the command or control value at each node.

[0037] The acquisition system 206 may be configured to receive image data of the assembly process at each node N. In some embodiments, the acquisition system 206 may receive image data of the assembly process of the monitoring platform 104. In some embodiments, for each node N, the acquisition system 206 may receive V images, where V may represent the number of cameras of the monitoring platform 104 that can record the assembly process at each node. Thus, each of the V images may capture a different perspective of the component during processing. After receiving the image data, the acquisition system 206 may be configured to extract a subset of the images of the frame. For example, the acquisition system 206 may be configured to extract L images from the received image data. The extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion. The extracted images may include certain landmarks containing the component. Those images or frames where And can represent the entire manufacturing process of the component.

[0038] Both the minimum motion (e.g., "landmark") frames and the maximum motion (e.g., "high motion") frames for a given operator may contain useful information for a classifier that attempts to correlate finger-hand-arm data with flight performance data in a robust manner across multiple operators. In some embodiments, an optical flow algorithm may be used to measure the amount of motion in any given frame. The acquisition system 206 may select those frames that contain the most motion.

[0039] The extractor module 208 may be configured to extract key points from the L images. For example, the extractor module 208 may be configured to extract key points for each landmark I i Extract K key points (ie, (x, y) pairs). In other words, the extractor module 208 can be configured to extract K key points (ie, (x, y) pairs) for a given input. (where l∈[0,L-1]), outputting K key points. As output, the extractor module 208 may generate a single vector This vector can include K (x, y) pairs of landmarks, which can be represented by Represents, where i∈[0,L-1].

[0040] In some embodiments, to generate The extractor module 208 may implement two separate algorithms: (1) bounding box estimation; and (2) keypoint detection.

[0041] Regarding bounding box estimation, given Each landmark frame may be processed with threshold image segmentation to generate a mask image for each tool component. For example, in an embodiment where the tool is a person, the extractor module 208 may generate a mask image for each hand of the user. In some embodiments, the extractor module 208 may implement blob detection to locate the components of the tool. Using a person as an example, the extractor module 208 may assume that the image always contains the left and right hands of the user. When a frame fails to contain both hands, the extractor module 208 may assign an arbitrary constant value c to the value.

[0042] Regarding keypoint detection, the extractor module 208 can identify the keypoints of the tool based on the estimated bounding box. For example, given an input, the extractor module 208 can estimate K points Together with its confidence value In some embodiments, the extractor module 208 may estimate not only points that are visible on the frame, but also points that may be occluded in the frame due to one or more of articulation, viewpoint, object, or tool interaction. Since the goal may be to use tracked keypoints to predict quality measures, non-estimated occluded points may be unique and important features that represent the assembly process. Therefore, the occlusion threshold t o We can select c from the occluded points in a small randomly selected subset of the landmark frames. i The observation of the value is obtained using t o , the extractor module 208 can filter out c o <t o For those filtered points, the extractor module 208 may assign an arbitrary constant value c to them. Regardless of the visibility of the tool or keypoint on the frame, the output of the extractor module 208 may include L*K(x,y) pairs for each component.

[0043] In other words, the extractor module 208 can be configured to assign a default base confidence level to the occluded parts, thereby estimating those key points so that the full set of key points can be used to measure the difference between the observed test tool part position and the specification tool part position. In some embodiments, the tool module 202 can draw inferences about the state of the part at a given point in time (e.g., station i), and then output recommended follow-up actions corrected according to the specification to correct the measured difference.

[0044] The prediction module 210 may be configured to predict the final quality metric For example, given tool tracking information at L time points The prediction module 210 may be configured to predict the final quality metric in, can be collected from each processing station 108. In some embodiments, the prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric. The LSTM model may allow the prediction module 210 to overcome the vanishing gradient problem common in conventional recurrent neural networks. The vanishing gradient problem is a situation where the model fails to relate early inputs to later inputs for long sequences of inputs. The LSTM model eliminates this problem.

[0045] Figure 3 300. As shown, the LSTM model 300 may include three layers 3021-3023 (generally layer 302). As shown, each layer 302 may include one or more units 304. In some embodiments, the input of each unit 304 may be p i , where i∈[0,L-1]. Each unit in LSTM can be defined as:

[0046] Equation 1: f i =σ(W f *[h i-1 ,p i ]+b f )

[0047] Equation 2: I_i=σ(W_I*[h_(i-1),p_i]+b_I)

[0048] Equation 3: c = f i *c t-1 +I i *tanh(W c *[h i-1 ,p i ]+b c )

[0049] Equation 4: o i =σ(W o *[h i-1 ,p i ]+b o )

[0050] Equation 5: h i =o i *tanh c i

[0051] Among them, equation 1 determines whether to keep the information from the previous unit; equation (2) determines which values ​​to update; equation (3) updates the unit; and equation (4) determines which part to output. Equation (5) can filter the output part so that the LSTM model 300 only outputs what it is programmed to output.

[0052] In some embodiments, the LSTM model 300 may include three layers 302 with a hidden size of 30. In some embodiments, the LSTM model 300 may be a sequence-to-one LSTM model. For training, an Adam optimizer may be used to minimize the L1, mean absolute error (MAE), loss function:

[0053]

[0054] In some embodiments, MAE may be used regardless of the direction of the error, since the goal may be to minimize or reduce the magnitude of the error.

[0055] Return to see Figure 2 , you can use the canonical instruction set and a plurality of components M to train the prediction module 210. For example, using 10 node videos of M data, the input to the prediction module 210 can be constructed by first preprocessing through the acquisition system 206 and the extractor module 208. In some embodiments, each component involved in the training can be verified by a verification algorithm that checks its composition (e.g., shape). In some embodiments, the verification algorithm can calculate a similarity index about the component at the end of each node by comparing the component to the canonical component. Therefore, the components used for training are roughly similar to the canonical components.

[0056] In some embodiments, for the output, the corresponding physical component can be tested in a controlled environment to measure its quality metric. Using the prepared input data with the corresponding output data, the prediction module 210 can be trained, for example, where a portion of M is training data and another portion of M is validation data. Once trained, the prediction module 210 is able to predict the quality measurement of the component at a given processing step based on the image data of the tool.

[0057] Referring now to the component module 204, the component module 204 may include a stochastic gradient descent (SGD) module 212, a gated recurrent unit (GRU) model 214, and a simulation module 216. For the purposes of this discussion, a partial construction of a component may be defined as Where step k introduces an irreversible error in the manufacturing process, and steps k+1, ..., N are not yet defined. The component module 204 can be configured to identify the optimal correction sequence of the remaining actions [{r k+1 ,t k+1},...,{r N ,t N}], where r and t may correspond to a specific operation performed on the component at each subsequent processing station (k+1 to N) of the manufacturing system 102. More generally, any component can be defined as the sequence of all operations performed at each processing station 1...N of the manufacturing system. Mathematically, At each manufacturing step i=1,...,10, the virtual representation system can convert the components in the Euclidean space (e.g., ) is represented as a set of connected surfaces and a set of connection points evenly distributed along the outer contour of each surface. In some embodiments, the virtual representation functions that generate these representations can be respectively referred to as and In some embodiments, the component module 204 can be configured to target specific specification components. Correction

[0058] The simulation module 216 may be configured to simulate or generate a given component For example, the simulation module 216 may receive tool information from the tool module 202. Based on the key points generated by the tool module 202, the simulation module 216 may be configured to generate a surface model representing the component at a particular process step i. In some embodiments, the surface model can be represented as In some embodiments, the simulation module 216 may be further configured to generate or estimate the component Based on the surface model, the simulation module 216 can be configured to generate a Point model with specific coordinates For example, according to the surface model The simulation module 216 can be Create a point model by placing multiple points evenly spaced around the boundary contour of each surface In some embodiments, Can be used to simulate artifacts performance.

[0059] The SGD module 212 may receive the point model from the simulation module 216 The SGD module 212 can be used to transform the point model With standard parts The canonical point model The comparison is performed to determine whether an irreversible error k has occurred. An irreversible error can be defined as a measurably significant structural deviation from the canonical component at step k. The SGD module 212 can be configured to detect irreversible errors by taking the Hausdorff distance. For example, the SGD module 212 can separate the current component from the corresponding Euclidean point set. Processing steps and standard components Mathematically, the SGD module 212 can be configured to calculate and (where i∈i=1,...,N). For example,

[0060]

[0061] where d(x,y) can be the Euclidean distance between x and y, and the undirected Hausdorff distance can be:

[0062] H(X,Y)=max{h(X,Y),h(Y,X)

[0063] In the current part With standard parts When the Hausdorff distance between exceeds a certain threshold tolerance, there may be an irreversible error. For example, the SGD module 212 may determine that an error occurs at step k when:

[0064]

[0065] Among them, τ H is some appropriately defined tolerance threshold.

[0066] Assuming there is an error, the SGD module 212 can be configured to construct a set of updated actions [{r k+1 ,t k+1},...,{r N ,t N}], assuming that it is a set of actions until the error point (irreversible error k). In some embodiments, the set of updated actions can be called x tail The sequence of steps or actions before and including error step k can be called x head .x tail and x head Together we can define components Based on x head , SGD module 212 can use stochastic gradient descent to solve x tail .

[0067] The GRU model 214 can be configured based on To predict parts The final quality metric is The final quality metric generated by the GRU model 214 can be compared with the canonical final quality metric to determine x tail For example, suppose x tail and x headIf the combination of produces a final quality metric that falls outside the range of acceptable values, GRU model 214 may instruct SGD module 212 to generate an updated sequence of updated actions for further evaluation.

[0068] Figure 5 2 is a block diagram showing the architecture of the GRU model 214 according to an example embodiment. As shown in the figure, the GRU model 214 may include N GRU units 5021-502 N (generally GRU units 502), where each GRU unit 502 corresponds to a corresponding processing station 108. Each GRU unit 502 may include an input pair (r i ,t i ) and hidden state output These input pairs (r i ,t i ) can be used together to define a given component In some embodiments, each GRU unit 502 may be defined by:

[0069]

[0070]

[0071]

[0072]

[0073] in, can be the hidden state at time t, can be the input at time t, and and can represent the reset, update and new gates at time t respectively.

[0074] The GRU model 214 may be trained to generate weights corresponding thereto. For example, the GRU model 214 may be iteratively trained to bias the GRU model 214 toward solving a particular subproblem. In some embodiments, during a first iteration, the GRU model 214 may generate multiple (e.g., hundreds to thousands) best predictions (the best prediction for each possible error at step k) to complete the given component at steps k+1, ..., N, along with a corresponding prediction quality metric corresponding to each prediction.

[0075] In some embodiments, all predicted aircraft completions may be rendered in a virtual representation system, their stepwise Hausdorff distances calculated, and their rendered surfaces simulated to obtain a distance measure between the generated predictions and the specification. In some embodiments, a loss value between the specification quality metric and the predicted quality metric may be calculated and fed back into the GRU model 214, and the weights of the GRU model may be adjusted via backpropagation, resulting in a second iteration. This process may continue until a desired error threshold is obtained.

[0076] Figure 4 is a block diagram that graphically illustrates the overall process flow of the feedback section 400 for the tool module 202 according to an example embodiment.

[0077] As shown, the feedback section 400 may include a first portion 402, a second portion 404, and a third portion 406. During the first portion 402, the tool module 202 may perform an acquisition process, such as an acquisition process performed at least in part by the acquisition system 206.

[0078] As shown, at block 408, the manufacturing system 102 may receive processing instructions for processing the component. In some embodiments, the manufacturing system 102 may receive processing instructions on a per-station 108 basis. For example, each station 108 of the manufacturing system 102 may receive independent processing instructions. In some embodiments, the processing instructions may include control values ​​that define properties (e.g., temperature, pressure, etc.) of the station 108 for manufacturing. In some embodiments, the processing instructions may include a video or image that graphically illustrates to a human operator how to perform a particular processing step at a processing node of the manufacturing process.

[0079] At box 410, the acquisition system 206 may receive image data from the monitoring platform 104. In some embodiments, for each node N, the acquisition system 206 may receive V images, where V may represent the number of cameras of the monitoring platform 104 that can record the assembly process at each node. After receiving the image data, at box 412, the acquisition system 206 may be configured to extract a subset of the images of the frame. For example, the acquisition system 206 may be configured to extract L images from the received image data. The extracted images may be referred to as landmark frames. The extracted images may include those images or frames that contain certain landmarks and may represent the entire manufacturing process of the component.

[0080] The second portion 404 may correspond to operations performed by the extractor module 208. As shown, the extractor module 208 may receive at least an extracted image from the acquisition system 206. The extractor module 208 may be configured to extract key points from the L images. For example, the extractor module 208 may be configured to extract key points for each landmark I. iExtract K keypoints (i.e., (x, y) pairs).

[0081] At block 414, the extractor module 208 may perform bounding box estimation. Each landmark frame may be processed with threshold image segmentation to generate a mask image for each tool component.For example, as shown, in an embodiment where the tool is a person, the extractor module 208 may generate a mask image for each hand of the user.

[0082] At block 416, given the bounding box estimate, the extractor module 208 may perform keypoint detection. For example, given the input, the extractor module 208 may estimate K points Together with its confidence value In some embodiments, the extractor module 208 may estimate not only points that are visible on the frame, but also points that may be occluded in the frame due to one or more of articulation, viewpoint, object, or tool interaction.

[0083] The third portion 406 may correspond to operations performed by the prediction module 210. As shown, at block 418, the prediction module 210 may receive keypoint information from the extractor module 208 and may be configured to predict a final quality metric In some embodiments, the prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.

[0084] Figure 6 6 is a flow chart illustrating a method 600 of calibrating a multi-step manufacturing process according to an example embodiment. The method 600 may begin at step 602 .

[0085] At step 602, an instruction set may be provided to the manufacturing system 102. The instruction set may represent a set of instructions for a manufacturing process performed by the manufacturing system 102. In some embodiments, the instruction set may be provided to each station 108. For example, each canonical instruction set provided to each respective station 108 may define processing parameters for a particular manufacturing step. In another example, each canonical instruction set may be a video of discrete steps to be performed by a human actor at a particular processing node or station 108.

[0086] At step 604, the control module 106 may receive image data of a tool (e.g., a station 108) from the monitoring platform 104. For example, the acquisition system 206 may receive image data of an assembly process at a corresponding processing node. In some embodiments, the acquisition system 206 may receive V images, where V may represent the number of cameras of the monitoring platform 104 that may record the assembly process at a particular processing station 108. Thus, each of the V images may capture a different perspective of the tool during processing.

[0087] At step 606, the control module 106 may extract a subset of images from the acquired image data. For example, after receiving the image data, the acquisition system 206 may be configured to extract a subset of images of frames. For example, the acquisition system 206 may be configured to extract L images, i.e., landmark frames, from the received image data. The extracted images may be referred to as landmark frames. Landmark frames may be those image frames that are in high motion. The extracted images may include certain landmarks that contain parts. Those images or frames where And can represent the entire manufacturing process of the component.

[0088] At step 608, the control module 106 may extract one or more key points of the tool from the landmark frame. For example, the extractor module 208 may extract key points from the L images. The extractor module 208 may identify or extract a given input As output, the extractor module 208 can generate a single vector This vector can include K (x, y) pairs of landmark representations, which can be represented by Represents, where i∈[0,L-1].

[0089] At step 610, the control module 106 may predict a final quality metric of the component based at least on the identified key points. In some embodiments, the prediction module 210 may implement a long short-term memory (LSTM) model to output the final quality metric.

[0090] At step 612, the control module 106 may compare the final quality metric to the desired quality metric. If, at step 610, the control module 106 determines that the final quality metric is a threshold tolerance of the desired quality metric, the manufacturing process may proceed to the next processing station or node (e.g., step 616) according to the original instruction set. However, if, at step 612, the control module 106 determines that the final quality metric is not within the threshold tolerance of the desired quality metric, then at step 614, the control module 106 may adjust the processing parameters of the downstream processing station. In some embodiments, the control module 106 may interface with the station controllers in the subsequent stations 108 to adjust their respective control and / or station parameters. In some embodiments, the control module 106 may provide the human manufacturer with updated instructions to be executed at each downstream processing station of the production line. These adjustments may help to correct the manufacturing process so that the final quality metric may be within the range of acceptable quality metrics.

[0091] Figure 7 is a flow chart illustrating a method 700 of calibrating a multi-step manufacturing process according to an example embodiment. The method 700 may begin at step 702 .

[0092] At step 702, an instruction set may be provided to the manufacturing system 102. The instruction set may represent a set of instructions for a manufacturing process performed by the manufacturing system 102. In some embodiments, the instruction set may be provided to each station 108. For example, each canonical instruction set provided to each respective station 108 may define processing parameters for a particular manufacturing step. In another example, each canonical instruction set may be a video of discrete steps to be performed by a human actor at a particular processing node or station 108.

[0093] At step 704, control module 106 may identify information corresponding to the component at the corresponding processing node. In some embodiments, simulation module 216 may receive tool information from tool module 202. Based on the key points generated by tool module 202, simulation module 216 may be configured to generate information representing the component at a particular processing step i. In some embodiments, the surface model can be represented as In some embodiments, the simulation module 216 may be further configured to generate or estimate the component Based on the surface model, the simulation module 216 can be configured to generate a representation of the component Point model with specific coordinates

[0094] At step 706, the control module 106 may determine whether an irreversible error has occurred. For example, the SGD module 212 may receive the point model from the simulation module 216 The SGD module 212 can be used to transform the point model With standard parts The canonical point model The comparison is made to determine whether an irreversible error k has occurred. The SGD module 212 can be configured to detect irreversible errors by taking the Hausdorff distance between the point model and the corresponding canonical point model. With standard parts When the Hausdorff distance between them exceeds a certain threshold tolerance, there may be irreversible errors.

[0095] If at step 706, the control module 106 determines that no irreversible error has occurred, then according to the original instruction set, the manufacturing process can proceed to the next processing station or node (step 705). However, if at step 706, the control module 106 determines that an irreversible error has occurred, the method 700 proceeds to step 708.

[0096] At step 708, the control module 106 may generate an updated set of actions to correct the irreversible error. The SGD module 212 may construct an updated set of actions [{r k+1 ,t k+1},...,{r N ,t N}], assuming that it is a set of actions until the error point (irreversible error k). In some embodiments, the set of updated actions can be called x tail The sequence of steps or actions before and including error step k can be called x head .x tail and x head Together we can define components Based on x head , SGD module 212 can use stochastic gradient descent to solve x tail The sequence of steps or actions before and including error step k can be called x head .x tail and x head Together we can define components Based on x head , SGD module 212 can use stochastic gradient descent to solve x tail .

[0097] At step 710, the control module 106 may generate a predicted final quality metric for the component based on a set of actions generated by the SGD module 212. For example, the GRU model 214 may be configured to To predict parts The final quality metric is Can represent the vector concatenation operator.

[0098] At step 712, the control module 106 may determine whether the predicted final quality metric is within a threshold tolerance of the canonical final quality metric. For example, the final quality metric generated by the GRU model 214 may be compared to the canonical final quality metric to determine whether x tail At step 712, if the control module 106 determines that the predicted quality metric is within the threshold tolerance, then at step 714, the control module 106 may adjust the processing parameters of the downstream processing station. In some embodiments, the control module 106 may interface with the station controllers in the subsequent stations 108 to adjust their respective control and / or station parameters. In some embodiments, the control module 106 may provide the human manufacturer with updated instructions to be executed at each downstream processing station of the production line. These adjustments may help to correct the manufacturing process so that the final quality metric may be within the range of acceptable quality metrics.

[0099] However, if at step 712, the control module 106 determines that the predicted quality metric is not within the threshold tolerance, the GRU model 214 may prompt the SGD module 212 to generate a new x tail For example, the method 700 may return to step 708 to cause the SGD module 212 to construct a new set of updated actions.

[0100] Fig. 8A A system bus computing system architecture 800 according to an example embodiment is shown. One or more components of the system 800 may communicate with each other electrically using a bus 805. The system 800 may include a processor (e.g., one or more CPUs, GPUs, or other types of processors) 810 and a system bus 805 that couples various system components including a system memory 815 (such as a read-only memory (ROM) 820 and a random access memory (RAM) 825) to the processor 810. The system 800 may include a cache of high-speed memory that is directly connected to, closely adjacent to, or integrated as part of the processor 810. The system 800 may copy data from the memory 815 and / or the storage device 830 to the cache 812 for fast access by the processor 810. In this way, the cache 812 may provide a performance improvement that avoids delays in the processor 810 when waiting for data. These and other modules may control or be configured to control the processor 810 to perform various actions. Other system memories 815 are also available. The memory 815 may include a plurality of different types of memories with different performance characteristics. Processor 810 may represent a single processor or multiple processors. Processor 810 may include one or more general purpose processors or hardware modules or software modules, such as service 1 832, service 2 834, and service 3 836 stored in storage device 830, one or more of which are configured to control processor 810, and a dedicated processor in which software instructions are incorporated into the actual processor design. Processor 810 may be a completely independent computing system in nature, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0101] To enable user interaction with the computing device 800, the input device 845 can be any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphic input, a keyboard, a mouse, motion input, voice, etc. The output device 835 can also be one or more of a plurality of output mechanisms known to those skilled in the art. In some instances, a multimodal system can enable a user to provide multiple types of input to communicate with the computing device 800. The communication interface 840 can generally control and manage user input and system output. There is no limitation on operation on any particular hardware arrangement, and therefore, the basic features herein can be easily replaced with improved hardware or firmware arrangements as they are developed.

[0102] The storage device 830 may be a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a random access memory (RAM) 825, a read-only memory (ROM) 820, and hybrids thereof.

[0103] The storage device 830 may include services 832, 834, and 836 for controlling the processor 810. Other hardware or software modules are contemplated. The storage device 830 may be connected to the system bus 805. In one aspect, a hardware module that performs a particular function may include a software component stored in a computer-readable medium that interfaces with the necessary hardware components (such as the processor 810, bus 805, display 835, etc.) to perform the function.

[0104] Figure 8BA computer system 850 with a chipset architecture according to an example embodiment is shown. The computer system 850 may be an example of computer hardware, software, and firmware that can be used to implement the disclosed technology. The system 850 may include one or more processors 855, which represent any number of physically and / or logically different resources capable of executing software, firmware, and hardware, which are configured to perform the identified calculations. The one or more processors 855 may communicate with a chipset 860, which may control input to and output from one or more processors 855. In this example, the chipset 860 outputs information to an output 865 (e.g., a display), and may read and write information to a storage device 870, which may include, for example, magnetic media and solid-state media. The chipset 860 may also read data from and write data to a RAM 875. A bridge 880 for connecting to various user interface components 885 may be provided for connecting to the chipset 860. Such user interface components 885 may include a keyboard, a microphone, touch detection and processing circuitry, a pointing device such as a mouse, etc. In general, input to the system 850 may come from any of a variety of sources, whether machine-generated and / or manually-generated.

[0105] The chipset 860 may also be connected to one or more communication interfaces 890 which may have different physical interfaces. Such communication interfaces may include interfaces for wired and wireless local area networks, for broadband wireless networks, and personal area networks. Some applications of the methods disclosed herein for generating, displaying, and using GUIs may include receiving a sorted data set through a physical interface, or generated by the machine itself by analyzing data stored in a storage device 870 or 875 through one or more processors 855. Further, the machine may receive input from a user through a user interface component 885, and perform appropriate functions, such as a browsing function, by interpreting these inputs using one or more processors 855.

[0106] It will be appreciated that example systems 800 and 850 may have more than one processor 810 or be part of a group or cluster of computing devices networked together to provide greater processing capabilities.

[0107] Although the foregoing is directed to the embodiments described herein, other and further embodiments may be devised without departing from the basic scope thereof. For example, aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. One embodiment described herein may be implemented as a program product for use with a computer system. The program of the program product defines the functions of the embodiments (including the methods described herein) and may be contained on various computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) a non-writable storage medium on which information is permanently stored (e.g., a read-only memory (ROM) device within a computer, such as a CD-ROM disk that can be read by a CD-ROM drive, a flash memory, a ROM chip, or any type of solid-state non-volatile memory); and (ii) a writable storage medium on which variable information is stored (e.g., a floppy disk or hard disk drive within a disk drive or any type of solid-state random access memory). Such a computer-readable storage medium is an embodiment of the present disclosure when it carries computer-readable instructions that direct the functions of the disclosed embodiments.

[0108] It will be appreciated by those skilled in the art that the foregoing examples are illustrative and not restrictive. All permutations, enhancements, equivalents, and improvements thereof will be apparent to those skilled in the art upon reading the specification and studying the drawings and are included within the true spirit and scope of the present disclosure. Therefore, the appended claims are intended to include all such modifications, permutations, and equivalents that fall within the true spirit and scope of these teachings.

Claims

1. A manufacturing system comprising: one or more stations, each station configured to perform at least one step in a multi-step manufacturing process for a component; a monitoring platform configured to monitor the progress of the component throughout the multi-step manufacturing process; as well as a control module configured to dynamically adjust processing parameters of steps of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations comprising: receiving image data of a processing tool of a first station of the one or more stations; identifying a set of key points from the image data, the set of key points corresponding to position information of a processing tool at the first station during processing at the first station; predicting, by a machine learning model, a final quality metric of the component based on the set of key points, wherein the final quality metric is a metric of the component that is not measured until processing of the component is complete; determining that the final quality metric is not within a threshold tolerance of the final quality metric; and Based on the determination, the processing parameters of subsequent stations of the multi-step manufacturing process are updated.

2. The manufacturing system according to claim 1, wherein: The image data includes a plurality of images, each image corresponding to a respective camera.

3. The manufacturing system according to claim 1, wherein: The operations also include: An image subset is extracted from the image data, wherein each image in the image subset includes a processing tool of the first station.

4. The manufacturing system of claim 1 , wherein identifying the set of key points from the image data comprises: Blob detection is applied to the image data to identify the location of a processing tool in the image data.

5. The manufacturing system according to claim 4, further comprising: A plurality of points corresponding to the processing tool in the image data are generated.

6. The manufacturing system according to claim 1, wherein: The machine learning model is a long short-term memory model.

7. A manufacturing system comprising: one or more processing stations, each processing station configured to perform at least one step in a multi-step manufacturing process for a component; a monitoring platform configured to monitor the progress of the component throughout the multi-step manufacturing process; as well as a control module configured to dynamically adjust processing parameters of steps of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations comprising: identifying, at a first processing station of the one or more processing stations, location information corresponding to the component; determining the presence of an irreversible error based on the position information corresponding to the component; generating an updated set of instructions based on the determination to correct the irreversible error, the updated set of instructions being executed by a downstream processing station; predicting, by a machine learning model, a final quality metric of the component based on the updated instruction set; wherein the final quality metric is a metric of the component that is not measured until processing of the component is complete; and The updated set of instructions is provided to the downstream processing station based on the predicted final quality metric.

8. The manufacturing system according to claim 7, wherein: Identifying the location information corresponding to the component includes: A set of points corresponding to coordinates of the component is received.

9. The manufacturing system according to claim 8, wherein: Determining the presence of the irreversible error based on the position information corresponding to the component includes: A set of points corresponding to the coordinates of the component are compared to a set of points corresponding to the specification of the specification component.

10. The manufacturing system according to claim 9, wherein: Comparing a set of points corresponding to the coordinates of the component with a set of points corresponding to the specification of the specification component includes: A Hausdorff distance between the set of points and the canonical set of points is generated.

11. The manufacturing system according to claim 7, wherein: Generating the updated set of instructions to correct the irreversible error includes: Apply stochastic gradient descent to the set of actions that occurred before the irreversible error.

12. The manufacturing system according to claim 11, wherein: The set of actions occurring prior to the irreversible error includes a respective set of actions for each upstream system.

13. The manufacturing system according to claim 7, wherein: Providing the updated instruction set to the downstream processing station comprises: The control parameters of the downstream processing station are adjusted.

14. A manufacturing system comprising: one or more stations, each station configured to perform at least one step in a multi-step manufacturing process for a component; a monitoring platform configured to monitor the progress of the component throughout the multi-step manufacturing process; as well as a control module configured to dynamically adjust processing parameters of steps of the multi-step manufacturing process to achieve a desired final quality metric for the component, the control module configured to perform operations comprising: receiving image data of a processing tool of a first station of the one or more stations; identifying a set of key points from the image data, the set of key points corresponding to position information of the processing tool during processing at the first station; predicting, by a machine learning model, a final quality metric of the component based on the set of key points; wherein the final quality metric is a metric of the component that is not measured until processing of the component is complete; determining that the final quality metric is not within a threshold tolerance of the final quality metric; inferring location information corresponding to the component at the first station based on the determining; based on the determination, generating an updated set of instructions to be executed by the downstream station; predicting, by the machine learning model, a second final quality metric of the component based on the updated instruction set; wherein the second final quality metric is another metric of the component that is not measured until processing of the component is complete; and The updated set of instructions is provided to the downstream station based on the predicted second final quality metric.

15. The manufacturing system according to claim 14, further comprising: The presence of an irreversible error is determined based on position information corresponding to the component.

16. The manufacturing system according to claim 15, further comprising: A set of points corresponding to the coordinates of the component are compared to a set of points corresponding to the specification of the specification component.

17. The manufacturing system according to claim 15, wherein: Generating the updated set of instructions to correct the irreversible error includes: Apply stochastic gradient descent to the set of actions that occurred before the irreversible error.

Images