System and method for online optimization of sensor fusion model

Abstract

A system and method for collecting data regarding an operation of a robot using, at least in part, a response from a first operational model to an input of sensed data from a plurality of sensors. The collected data may be used to optimize the first operational model to generate a second operational model. While the first operational model is optimized, a training data driven model using an end-to-end learning method may be generated, the training data driven model based at least in part on the collected data. Both the second operational model and the training data driven model may be evaluated, and a determination may be made based on such evaluation as to whether the training data driven model is reliable. Further, based on a comparison of these models, one of a second operational model and a training data driven model may be selected for verification and, if verified, used in the operation of the robot.

Description

Technical Field

[0001] This invention relates to the optimization of robot calibration, and more particularly, to a system and method for combining training data-driven models utilizing end-to-end learning methods with model-based learning to optimize sensor fusion. Background Technology

[0002] Various operations can be performed during the Final Finishing and Assembly (FTA) phase of automobile assembly, including, for example, door assembly, cockpit assembly, seat assembly, and other types of assembly. However, for various reasons, typically only a relatively small number of FTA tasks are automated. For example, often during the FTA phase, as operators perform FTA operations, the vehicles undergoing the FTA are transported along multiple lines in a relatively continuous stop-and-go manner. However, this continuous stop-and-go movement of the vehicles introduces or generates irregularities in the movement and / or position of the vehicles and / or the parts of the vehicles involved in the FTA. Furthermore, this stop-and-go movement causes the vehicles to suffer from movement irregularities, vibrations, and balance problems during the FTA, which prevents or hinders the accurate modeling or prediction of the position of specific parts, sections, or areas of the vehicle directly involved in the FTA. Further, as each subsequent vehicle and / or component passes along the same area of ​​the assembly line, this movement irregularity prevents the FTA from having a consistent degree of repeatability in terms of the movement and / or positioning of each vehicle or its associated components. Therefore, such differences and concerns about repeatability often hinder the use of robot motion control based on traditional teaching and repeated positions in FTA operations.

[0003] Therefore, although various robot control systems are currently available on the market, it is possible to make further improvements to provide systems and devices for calibrating robot control systems to adapt to such movement irregularities. Summary of the Invention

[0004] One aspect of an embodiment of this application is a method comprising collecting data regarding a robot's manipulation of a workpiece, the robot's manipulation being at least partially based on a response from a first manipulation model to input sensing data from a plurality of sensors of the robot. The method may further comprise using at least a portion of the collected data to optimize the first manipulation model to generate a second manipulation model. Additionally, while optimizing the first manipulation model, a training data-driven model may be generated, which utilizes an end-to-end learning method and is at least partially based on the collected data. Further, both the second manipulation model and the training data-driven model may be evaluated, and one of the second manipulation model and the training data-driven model may be selected based on the evaluation results. The method may further comprise using at least a portion of the collected data to validate the selected second manipulation model and the training data-driven model for use in the robot's manipulation.

[0005] Another aspect of embodiments of this application is a system comprising a robot having a plurality of sensors and a controller configured to operate the robot at least in part based on one or more responses to inputs of sensed data from the plurality of sensors from a first operating model. The system may further include one or more databases communicatively connected to the robot, the databases being configured to collect data regarding the robot's manipulation of a workpiece. Additionally, the system may include one or more computational components communicatively connected to the one or more databases and the robot. The one or more computational components may be configured to generate a second operating model based on an optimization of the first operating model using at least a portion of the collected data. Additionally, the one or more computational components may be configured to generate a training data-driven model in parallel with the generation of the second operating model, the training data-driven model being based on an end-to-end learning method utilizing at least a portion of the collected data. Further, the one or more computational components may be configured to evaluate both the second operating model and the training data-driven model, select one of the second operating model and the training data-driven model based on the evaluation results, and validate the selected second operating model and the training data-driven model using at least a portion of the collected data for use in the operation of the robot.

[0006] These and other aspects of the invention will be better understood in view of the accompanying drawings and the following detailed description. Attached Figure Description

[0007] The description in this article refers to the accompanying drawings, which are used throughout several views, and similar reference numerals refer to similar parts.

[0008] Figure 1The illustration shows a schematic representation of at least a portion of an exemplary robot system according to an illustrative embodiment of the present application.

[0009] Figure 2 The illustration shows a schematic representation of an exemplary robotic station through which a vehicle moves via an automated or automated guided vehicle (AGV), and the robotic station includes a robot mounted to a robot base that can move along or via a track.

[0010] Figure 3 An exemplary process is illustrated for optimizing sensor fusion online using a combination of training data-driven models that leverage end-to-end learning methods and model-based learning.

[0011] The foregoing overview and the following detailed description of certain embodiments of this application will be better understood when read in conjunction with the accompanying drawings. Certain embodiments are illustrated in the drawings for the purposes of illustrating this application. However, it should be understood that this application is not limited to the arrangements and means shown in the drawings. Furthermore, similar figures in the corresponding drawings indicate similar or comparable parts. Detailed Implementation

[0012] Certain terms used in the foregoing description are for convenience only and are not intended to be restrictive. Words such as “upper,” “lower,” “top,” “bottom,” “first,” and “second” indicate direction in the referenced figures. This terminology includes the words specifically mentioned above, their derivatives, and words with similar meanings. Additionally, unless specifically mentioned, the words “a” and “an” are defined to include one or more of the referenced items. The phrase “at least one” followed by a list of two or more items (such as “A, B, or C”) means any single one of A, B, or C, and any combination thereof.

[0013] Figure 1 The illustration depicts at least a portion of an exemplary robotic system 100, which includes at least one robotic station 102 communicatively connected to at least one management system 104 (e.g., via a communication network or link 118). The management system 104 may be local or remote relative to the robotic station 102. Further, according to some embodiments, the management system 104 may be cloud-based. Further, according to some embodiments, the robotic station 102 may also include one or more supplementary database systems 105, or be operatively communicating with the one or more supplementary database systems via a communication network or link 118. The supplementary database systems 105 may have various different configurations. For example, according to the illustrated embodiment, the supplementary database systems 105 may be, but are not limited to, cloud-based databases.

[0014] According to some embodiments, robot station 102 includes one or more robots 106 having one or more degrees of freedom. For example, according to some embodiments, robot 106 may have, for example, six degrees of freedom. According to some embodiments, an end effector 108 may be coupled to or mounted to robot 106. End effector 108 may be a tool, part, and / or component of a wrist or arm 110 mounted to robot 106. Further, at least a portion of wrist or arm 110 and / or end effector 108 may be movable relative to other parts of robot 106 via operating robot 106 and / or end effector 108 (such as, for example, via an operator of management system 104 and / or via a program executed to operate robot 106).

[0015] Robot 106 can be operated to position and / or orient end effector 108 within its envelope or workspace, which provides space for robot 106 to perform work using end effector 108, including, for example, grasping and holding one or more parts, components, packages, equipment, components, or products, and other items (collectively, “parts”). Robot 106 can utilize various types of end effectors 108, including tools that can grasp, hold, or otherwise selectively hold and release parts used in final trim and assembly (FTA) operations during vehicle assembly and other types of operations.

[0016] Robot 106 may include or be electrically connected to one or more robot controllers 112. For example, according to some embodiments, robot 106 may include and / or be electrically connected to one or more controllers 112, which may or may not be discrete processing units, such as, for example, a single controller or any number of controllers. Controller 112 may be configured to provide various functions, including, for example, being used to selectively deliver power to robot 106, control the movement and / or operation of robot 106, and / or control the operation of other equipment mounted to robot 106 (including, for example, end effector 108), and / or the operation of equipment not mounted to robot 106 but essential to the operation of equipment constituting robot 106 and / or associated with the operation and / or movement of robot 106. Furthermore, according to some embodiments, the controller 112 may be configured to dynamically control both the movement of the robot 106 itself and the movement of other devices to which the robot 106 is mounted or coupled, including, among other devices, the robot 106 moving along or alternatively via track 130 or a mobile platform (such as an AGV to which the robot 106 is mounted via robot base 142, e.g.) Figure 2 The movement (as shown in the image).

[0017] Controller 112 can take various forms and can be configured to execute program instructions to perform tasks associated with operating robot 106, including operating robot 106 to perform various functions (such as, for example, but not limited to, the tasks described herein and others). In one form, controller(s) 112 are microprocessor-based, and the program instructions are in software form stored in one or more memories. Alternatively, one or more controllers 112 and the program instructions executed therefrom can be in any combination of software, firmware, and hardware (including state machines) and can reflect the output of discrete devices and / or integrated circuits that may be co-located at a particular location or distributed across more than one location, including any digital and / or analog devices configured to achieve the same or similar results as a processor-based controller executing software- or firmware-based instructions. Operations, instructions, and / or commands determined and / or transmitted from controller 112 can be based on one or more models stored in controller 112, a non-transitory computer-readable medium, another computer, and / or accessible or electrically communicative memories.

[0018] According to the illustrated embodiment, controller 112 includes a data interface that can accept motion commands and provide actual motion data. For example, according to some embodiments, controller 112 can be communicatively coupled to a teach pendant (such as, for example, a teaching pendant) that can be used to control at least some operations of robot 106 and / or end effector 108.

[0019] Robot station 102 and / or robot 106 may also include one or more sensors 132. Sensors 132 may include various types of sensors and / or combinations of different types of sensors, including but not limited to vision system 114, force sensor 134, motion sensor, accelerometer and / or depth sensor, and other types of sensors. Furthermore, information provided by at least some of these sensors 132 (including, for example, via the use of algorithms) may be integrated so that operations and / or movements and other tasks performed by robot 106 can be guided at least via sensor fusion. Thus, as by at least Figure 1 and Figure 2 As shown, the controller 120 and / or computing component 124 of the management system 104 can process information provided by the one or more sensors 132 (such as, for example, vision system 114 and force sensor 134 and other sensors 132), such that the information provided by the different sensors 132 can be combined or integrated in a way that can reduce the uncertainty in the movement and / or task execution performed by the robot 106.

[0020] According to the illustrated embodiment, the vision system 114 may include one or more vision devices 114a that can be used to observe at least a portion of the robot station 102, including but not limited to observing parts, components, and / or vehicles, as well as other devices or components that can be positioned in or are moving through or past at least a portion of the robot station 102. For example, according to some embodiments, the vision system 114 may extract information about various types of visual features positioned or placed in the robot station 102 (such as, for example, on a vehicle and / or on an automated guided vehicle (AGV) moving the vehicle through the robot station 102, and other locations), and use this information, along with other information, to at least help guide the movement of the robot 106, the robot 106 along track 130, or the moving platform (such as an AGV in the robot station 102). Figure 2 The movement of the robot 106 and / or the movement of the end effector 108. Further, according to some embodiments, the vision system 114 may be configured to obtain and / or provide information about the position, orientation, and / or orientation of one or more calibration features, which may be used to calibrate the sensors 132 of the robot 106.

[0021] According to some embodiments, the vision system 114 may have data processing capabilities capable of processing data or information obtained from the vision device 114a, which can be transmitted to the controller 112. Alternatively, according to some embodiments, the vision system 114 may not have data processing capabilities. Instead, according to some embodiments, the vision system 114 may be electrically connected to the computing component 116 of the robot station 102, which is adapted to process data or information output from the vision system 114. Additionally, according to some embodiments, the vision system 114 may be operatively connected to a communication network or link 118, such that the controller 120 and / or computing component 124 of the management system 104 can process the information output by the vision system 114, as discussed below.

[0022] Examples of the vision device 114a of the vision system 114 may include, but are not limited to, one or more imaging capture devices (such as, for example, one or more two-dimensional, three-dimensional, and / or RGB cameras), which may be mounted within the robot station 102, including, for example, typically mounted above the work area of ​​the robot 106, mounted to the robot 106, and / or mounted on the end effector 108 of the robot 106, and other locations. Further, according to some embodiments, the vision system 114 may be a position-based or image-based vision system. Additionally, according to some embodiments, the vision system 114 may utilize kinematic control or dynamic control.

[0023] According to the illustrated embodiment, in addition to the vision system 114, the sensor 132 also includes one or more force sensors 134. The force sensors 134 may be configured, for example, to sense contact forces during the assembly process, such as contact forces between the robot 106, the end effector 108, and / or components held by the robot 106 and other components or structures within the vehicle 136 and / or the robot station 102. This information from the force sensors 134 may be combined or integrated with information provided by the vision system 114 to guide the movement of the robot 106 during the assembly of the vehicle 136, at least in part, through sensor fusion.

[0024] according to Figure 1 As illustrated in the exemplary embodiments, the management system 104 may include at least one controller 120, a database 122, a computing component 124, and / or one or more input / output (I / O) devices 126. According to some embodiments, the management system 104 may be configured to provide an operator with direct control of the robot 106, and to provide at least some programming or other information to the robot station 102 and / or to provide operation for the robot 106. Furthermore, the management system 104 may be structured to receive commands or other input information from an operator of the robot station 102 or the management system 104, including commands generated, for example, via operating or selectively engaging the input / output devices 126. Such commands implemented via the use of the input / output devices 126 may include, but are not limited to, commands provided by engaging or using a microphone, keyboard, touchscreen, joystick, stylus device, and / or sensing device (each of which can be operated, manipulated, and / or moved by the operator) and other input / output devices. Furthermore, according to some embodiments, the input / output device 126 may include one or more monitors and / or displays that can provide information to an operator, including, for example, information related to commands or instructions provided by the operator of the management system 104, received from / transmitted to the supplementary database system(s) 105 and / or robot station 102, and / or notifications generated when the robot 106 runs (or attempts to run) a program or process. For example, according to some embodiments, the input / output device 126 may display images (whether physical or virtual) such as those obtained, for example, via at least a vision device 114a using the vision system 114.

[0025] According to some embodiments, the management system 104 may include any type of computing device having a controller 120, such as, for example, a laptop computer, desktop computer, personal computer, programmable logic controller (PLC), or mobile electronic device, and other computing devices, including memory and a processor, which are large and operable to store and manipulate the database 122 and one or more applications to communicate with at least the robot station 102 via a communication network or link 118. In some embodiments, the management system 104 may include a connectivity device that can communicate with the communication network or link 118 and / or the robot station 102 via an Ethernet WAN / LAN connection and other types of connections. In some other embodiments, the management system 104 may include a web server or web portal and may use the communication network or link 118 to communicate with the robot station 102 and / or (multiple) supplementary database systems 105 via the Internet.

[0026] The management system 104 can be located in various locations relative to the robot station 102. For example, the management system 104 can be in the same area, the same room, adjacent rooms, the same building, or the same factory location as the robot station 102, or alternatively, in a remote location relative to the robot station 102. Similarly, the (multiple) supplementary database systems 105 (if any) can also be located in various locations relative to the robot station 102 and / or relative to the management system 104. Therefore, the communication network or link 118 can be structured at least in part based on the physical distance (if any) between the locations of the robot station 102, the management system 104, and / or the (multiple) supplementary database systems 105. According to the illustrated embodiment, the communication network or link 118 includes one or more communication links 118 ( Figure 1 Communication links in 1-N Additionally, the system 100 may be operated to maintain a relatively reliable real-time communication link between the robot station 102, the management system 104, and / or / multiple supplementary database systems 105 via the use of a communication network or link 118. Therefore, according to some embodiments, the system 100 may change the parameters of the communication link 118 based on the currently available data rate and / or transmission time of the communication link 118, including, for example, selecting the communication link 118 to be utilized.

[0027] The communication network or link 118 can be structured in various ways. For example, the communication network or link 118 between the robot station 102, the management system 104, and / or / multiple supplementary database systems 105 can be implemented using one or more of various different types of communication technologies, including but not limited to those using fiber optic, radio, cable, or wireless technologies via data protocols of similar or different types and layers. For example, according to some embodiments, the communication network or link 118 can utilize Ethernet facilities with wireless local area networks (WLANs), local area networks (LANs), cellular data networks, Bluetooth, ZigBee, point-to-point radio systems, laser optics systems, and / or satellite communication links, as well as other wireless industrial links or communication protocols.

[0028] The database 122 of the management system 104 and / or one or more databases 128 of the supplementary database system 105 may include various information that can be used to identify elements within the robot station 102 in which the robot 106 operates. For example, as discussed in more detail below, one or more of the databases 122, 128 may include or store information for detecting, interpreting, and / or deciphering images or other information detected by the vision system 114, such as features used for example, regarding the calibration of sensor 132. Additionally or alternatively, such databases 122, 128 may include information relating to the one or more sensors 132, including, for example, information relating to force or force range that would be expected to be detected, at least when the robot 106 is performing work, via the use of the one or more force sensors 134 at one or more different locations in the robot station 102 and / or along the vehicle 136. Additionally, the information in the databases 122, 128 may also include information for at least initially calibrating the one or more sensors 132, including, for example, a first calibration parameter associated with a first calibration feature and a second calibration parameter associated with a second calibration feature.

[0029] The database 122 of the management system 104 and / or one or more databases 128 of the supplementary database system 105 may also include information that can help identify other features within the robot station 102. For example, images captured by the one or more vision devices 114a of the vision system 114 can be used to identify FTA components (including FTA components in the picking bin) and other components that can be used by the robot 106 to perform FTAs ​​within the robot station 102 by using information from the database 122.

[0030] Figure 2The illustration shows a schematic representation of an exemplary robotic station 102, through which a vehicle 136 moves via an automated or automated guided vehicle (AGV) 138, and the robotic station includes a robot 106 mounted to a robot base 142, which is movable along or via a track 130 or a mobile platform (such as an AGV). While for at least illustrative purposes, Figure 2 The exemplary robotic station 102 depicted is shown as having or being close to vehicle 136 and associated AGV 138, but robotic station 102 may have various other arrangements and components and may be used for various other manufacturing, assembly and / or automation processes. Furthermore, while the depicted robotic station 102 may be associated with the initial setup of robot 106, station 102 may also be associated with the use of robot 106 in assembly and / or production processes.

[0031] Additionally, while the example depicted in this figure illustrates a single robotic station 102, according to other embodiments, robotic station 102 may include multiple robotic stations 102, each having one or more robots 106. The illustrated robotic station 102 may also include or operate with respect to one or more AGVs 138, supply lines or conveyors, inductive conveyors, and / or one or more sorting conveyors. According to the illustrated embodiment, AGVs 138 may be positioned and manipulated relative to the one or more robotic stations 102 to transport, for example, vehicles 136 that can be received or otherwise assembled, or one or more components comprising vehicle(s) 136, including, for example, door assemblies, cockpit assemblies, and seat assemblies, as well as other types of components and parts. Similarly, according to the illustrated embodiment, tracks 130 may be positioned and manipulated relative to the one or more robots 106 to facilitate the assembly of parts by the robots(s) 106 into vehicle(s) 136 that are moving via AGVs 138. Furthermore, the track 130 or mobile platform (such as an AGV), robot base 142, and / or robot can be operated such that robot 106 moves in a manner that at least substantially follows the movement of AGV 138 and thus follows the movement of vehicles(s) 136 on AGV 138. Further, as previously mentioned, such movement of robot 106 may also include movement at least partially guided by information provided by the one or more force sensors 134.

[0032] Figure 3An exemplary process 200 for online optimization of sensor fusion using a combination of a training data-driven model leveraging end-to-end learning methods and model-based learning is illustrated. Unless explicitly stated otherwise, all operations illustrated herein are to be understood as illustrative only, and operations may be combined or divided and added or removed, and may be reordered wholly or partially. Furthermore, according to certain embodiments, the process 200 discussed herein may be utilized at various different time periods during the lifespan of robot 106 and / or during its operational phases, and / or in various different settings. As demonstrated below, the illustrated process 200 can use multi-sensor input guidance to provide self-sufficient optimization for automated systems.

[0033] At step 202, the robot 106 of the robot station 102 can be operated using information from at least sensors 132, which have been calibrated using initial parameters. While the initial parameters can be used to calibrate the sensors 132 at various time periods, according to the illustrated embodiment, this initial parameter-based calibration can occur in conjunction with preparing or programming the robot 106 for introduction or integration into a specific assembly operation, such as an FTA operation, for which the robot 106 will be operated. Thus, for example, with respect to force sensor 134, force sensor 134 can be initially calibrated such that the force detected by force sensor(s)134 and associated with the robot 106, end effector 108, or component attached thereto when in contact with a workpiece (such as, for example, vehicle 136) will be within a force range and / or threshold satisfying the initial force parameters. However, other types of sensors can be calibrated in different ways.

[0034] The sensor fusion model can utilize information provided by one or more of the aforementioned calibration sensors 132, instructing the robot 106 how it should react, such as, move or position, at least in response to the information provided by the calibration sensors 132. Therefore, according to at least some embodiments, the sensor fusion model can be based at least in part on initial parameters used to calibrate the sensors 132. Thus, such a sensor fusion model can be configured to move or position the robot 106 in a manner that allows the robot 106 to perform tasks or operations (such as, for example, performing FTA assembly operations) at least accurately and / or timely, at least during the initial production phase of step 202.

[0035] According to some embodiments, after calibrating at least sensor 132 using initial parameters, robot 106 can be introduced or integrated into the assembly process, allowing robot 106 to continue performing operations or tasks programmed to be performed, while also utilizing a sensor fusion model. Regarding robot 106 performing these tasks, at step 204, data or information generated or otherwise associated with the operation of robot 106 can be collected, recorded, and / or stored via the use of online monitoring tools and optimization functions. For example, such information and data can be collected and stored in database 122 of management system 104 and / or one or more databases 128 of supplementary database system 105, again, these databases may be, for example, cloud-based databases. Furthermore, information and data can be collected at different intervals or at different times at step 204. For example, regarding… Figure 2 The exemplary embodiment depicted herein allows information and data to appear at step 204 whenever robot 106 performs a task for each vehicle 136 that passes through robot station 102 along AGV 138.

[0036] The type of information and data collected and stored can vary and may include, for example, data sensed or detected by one or more of sensors 132, including, for example, but not limited to, information and data detected by vision system 114 and force sensors(s)134. Additionally, such data and information may also include robot motion data, including but not limited to robot motion response data, which may include information relating to the robot 106's response to motion commands and / or instructions. Additionally, according to some embodiments, the collected data or information may, for example, include information relating to system performance, including but not limited to the performance of robot 106 in performing one or more (if not all) robot tasks and other tasks that robot 106 will perform in relation to assembly operations or procedures. For example, the collected data and information can provide indications of the accuracy, duration, and / or responsiveness of robot 106 regarding: robot 106 identifying parts to be grasped by robot 106 for use in the assembly process; robot 106 being moved and / or positioned to grasp parts; robot 106 grasping parts; robot 106 positioning the grasped parts on the workpiece to be assembled; and robot 106 being moved and / or positioned to secure the parts at the positioned workpiece location, as well as other possible tasks and operations. The collected data and information can be used to progressively form the model discussed below and may also include, for example, information related to path compensation (path compensation can be related to deviations or variations in the path taken by robot 106 in performing its associated assembly operations or tasks), and / or may include information regarding delay compensation.

[0037] The collected data and information can indicate changes (if any) in the operation and / or movement of robot station 102 and / or robot 106. For example, regarding Figure 2 The exemplary embodiments depicted herein, wherein the data and information collected at step 204 may reflect changes in lighting in robot station 102, and thus reflect the following associated changes: changes in the ability of vision system 114 to accurately detect certain features or images, changes in the speed of AGV 138 operation and / or changes in the speed of movement of vehicle 136 as it passes through robot station 102, and / or changes in the degree of vibration of vehicle 138 as it is tracked or operably engaged by robot 106 during assembly operations, and other changes. Additionally, such data and information may provide an indication of performance drift in one or more of sensors 132. For at least accuracy purposes, such changes may require alterations to the sensor fusion model, and in particular, alterations or tuning related to the parameters initially used to derive the sensor fusion model. Such indicated changes may also be communicated to the operator of robot station 106 as notification of the potential need for preventative maintenance. Conversely, in the absence of such changes, or in cases where such changes are relatively minimal, such changes may be unfounded and / or unnecessary, such as when robot station 102 and associated assembly processes or operations are functioning normally. Therefore, at least under normal operating conditions, robot 106 can usually continue to operate using the initial sensor fusion model.

[0038] At step 206, using the data and information collected at step 204, the sensor fusion model used in step 202 can be optimized by changing or adjusting at least the parameters initially used to create the sensor fusion model. This refinement of the sensor fusion model can lead to the generation of an optimized sensor fusion model that more accurately reflects the actual conditions detected or experienced in robot station 102. Furthermore, this refinement of the parameters based on the information and data collected from step 204 can lead to the generation of an optimized sensor fusion model that improves the accuracy, reliability, and / or performance of robot 106. Similar to the collection of information and data at step 204, according to some embodiments, this refinement of the sensor fusion model at step 206 can occur at a location remote from robot station 102 (e.g., cloud-based) to avoid increasing the computational and / or communication load at robot station 102.

[0039] In parallel or concurrent with the optimization of the sensor fusion model occurring at step 206, at step 208, a training data-driven model can be progressively formed using the information and data collected at step 204. This training data-driven model utilizes end-to-end deep learning and / or reinforcement learning, as well as (various) other types of learning-based methods, to assess the movement and / or localization of robot 106 regarding the previously discussed operations or tasks that robot 106 will perform. Similar to the optimization of the sensor fusion model, the training data-driven model can be progressively formed to account for variations occurring in robot station 102, including but not limited to variations related to lighting, motion irregularities, vibration, and performance drift of one or more of sensors 132 (particularly if sensors 132 have not been calibrated for an extended period), as well as other variations. Further, according to some embodiments, this training data-driven model can utilize neural networks to cluster and classify layers of collected and stored data, which may include the data collected at step 204. Using this method, training a data-driven model can, for example, gradually form a machine-based deep and / or reinforcement learning that can identify the correlation between certain input information and the best results that can be obtained through responsive actions or executions by the robot 106, such as, for example, the best movement or positioning of the robot 106 in response to input or sensed information.

[0040] The training data-driven model approach can also be based on information and data collected in the cloud database system(s) 105 at step 204, and on building layers of the collected and stored data using cloud-based computing and / or communication, so as not to increase the computational and / or communication load at the robot station 102. Furthermore, the use of cloud-based computing, communication, and / or evaluation implemented through step 208 and other steps of process 200 can allow the various steps of process 200 to occur without interrupting the production or assembly operations being performed by robot 106, while simultaneously minimizing the need for human input in process 200.

[0041] At step 210 (which, according to some embodiments, may be performed using cloud-based, edge-based, or local computing and / or communication, as well as other computing and communication methods), the optimized sensor fusion model output from step 206 is evaluated relative to the training data-driven model output from step 208, which is derived from an end-to-end deep and / or reinforcement learning-based approach. Such comparisons at step 210 between the models output from steps 206 and 208 of process 200 may be based, for example, on one or both of statistical and quantitative evaluations and / or analyses of each of these models. Further, according to some embodiments, such analysis may be based on the use of theoretical models or simulations that, when applied to the models output at steps 206 and 208, can provide estimates or predictions of the expected behavior of robot 106, including, for example, the expected accuracy and / or responsiveness of robot 106's movement, localization, and / or decision-making when utilizing each of these models. Furthermore, such evaluation or analysis may include a comparison of the estimated or expected performance levels that can be obtained by the robot 106 when performing one or more operations or tasks that the robot 106 will perform (when used in an assembly process), utilizing each of the optimized sensor fusion model and the training data-driven model.

[0042] The comparison or evaluation performed at step 210 may further include characterizing or rating the results obtained by using the training data-driven model output from step 208 relative to the results obtained by using the optimized sensor fusion model output from step 206. Furthermore, according to some embodiments, such evaluation may include determining whether the performance results of robot 106 expected or anticipated by using the training data-driven model are close to, significantly lower than, or exceed the performance results of robot 106 expected or anticipated by using the optimized sensor fusion model. This characterization may be based on various criteria, such as whether at least some of the results obtained in the evaluation of the training data-driven model are within a specific or predetermined numerical or statistical range of the results obtained in the evaluation of the optimized sensor fusion model. Further, according to some embodiments, such evaluation may involve ranking the results obtained from the evaluations of both the training data-driven model and the optimized sensor fusion model, determining the degree of difference between those ranked and / or associated results (including, for example, statistical or numerical results), and determining whether those differences are within a specific or predetermined range or whether they meet some other threshold or critical value.

[0043] Additionally, according to some embodiments, the evaluation performed at step 210 may include a Key Performance Indicator (KPI) evaluation. Such an evaluation may include assessing one or more cycle times, such as, for example, the cycles required to advance the vehicle 136 through various workstations and / or the cycles required for the robot 106 to grip a workpiece, move it to the vehicle 136, install the workpiece, and return to the starting position, as well as other cycle times. Such KPIs may also include other metrics, including but not limited to the contact force associated with assembling the workpiece into the vehicle 136, and the assembly success rate.

[0044] If, based on the evaluation at step 210, it is determined that the performance of the trained data-driven model is relatively poor compared to the performance of the optimized sensor fusion model (e.g., the generated results are not within a predetermined range or threshold of the results obtained in the evaluation of the optimized sensor fusion model), then the trained data-driven model is not selected for potential use in the operation of robot 106. However, in this case, the optimized sensor fusion model may still be considered for use in the operation of robot 106.

[0045] Since process 200 can be continuous, the results of the evaluation at step 210 can be expected to lead to the selection of an optimized sensor fusion model for at least some initial time period, at least until the expected performance of the training data-driven model reaches a level indicating that the training data-driven model is reliable. This gradual formation of a reliable training data-driven model can be consistent with the continuous collection of data and information related to the actual operation of the robot 106 and / or the continuous utilization of process 200 described herein, which can also lead to further refinement of the training data-driven model.

[0046] Therefore, if the evaluation at step 210 favors the optimized sensor fusion model and / or indicates that the training data-driven model is unreliable at least at this point, process 200 can then proceed to step 212, where the performance of the optimized sensor fusion model output at step 206 can be verified. According to some embodiments, such verification of the optimized sensor fusion model may include, for example, analyzing the performance of the optimized sensor fusion model in response to actual data and information obtained during the operation of robot 106 (including, for example, actual data and information obtained from sensor 132). Further, such verification may involve iteratively analyzing the performance of the optimized sensor fusion model in response to different actual data obtained from the operation of robot 106. Such data used to verify the optimized sensor fusion model may or may not be the same as or similar to the data collected or continuously collected at step 204. Additionally, this verification at step 212 may include, for example, but not limited to, assessing the accuracy of the robot 106's expected guided movement or positioning, and / or the expected degree of error associated with the robot 106's performance of the task or operation (if the robot 106 is to use an optimized sensor fusion model). This verification may further require that the expected performance obtained by using the optimized sensor fusion model meets predetermined standards and / or thresholds.

[0047] If the optimized sensor fusion model is validated at step 212, it can replace the sensor fusion model currently used in the operation of robot 106 (such as, for example, the initial sensor fusion model currently used at step 202). Otherwise, if the optimized sensor fusion model is not validated at step 212, robot 106 can continue to operate without replacing the existing sensor fusion model and other models currently being used by robot 106.

[0048] Conversely, if the performance of the trained data-driven model is determined to be relatively better than that of the optimized sensor fusion model based on the evaluation at step 210 (e.g., the generated results exceed or are within a predetermined range or threshold of the results obtained in the evaluation of the optimized sensor fusion model), then the trained data-driven model, rather than the optimized sensor fusion model, can be selected for possible use in the operation of robot 106. In this case, at step 214, the trained data-driven model may undergo validation, which can be performed in a manner similar to that discussed above regarding the validation of the optimized sensor fusion model at step 212. Furthermore, at step 214, the performance of the trained data-driven model in response to actual data and information obtained during the operation of robot 106 (including, for example, actual data and information obtained from sensor 132) can be evaluated. Again, such analysis may include, for example, but not limited to, the accuracy of the robot 106's expected guided movement or positioning, and / or the expected degree of error associated with the robot 106's performance of a task or operation (if robot 106 is to use the trained data-driven model). This validation can further require that the expected performance obtained by using the training data to drive the model meets predetermined standards and / or thresholds.

[0049] If the training data-driven model is validated at step 214, it can replace the sensor fusion model currently being used in the operation of robot 106 (such as, for example, the initial sensor fusion model currently being used at step 202, and other models). Otherwise, if the training data-driven model is not validated at step 214, robot 106 can continue to operate without replacing the existing model (such as, for example, without replacing the sensor fusion model that robot 106 is currently actually using).

[0050] As previously discussed, the illustrated process 200 can be continuous, allowing the data-driven model to become more reliable over time and as more information and data are collected than sensor fusion models and / or previously evolved optimized sensor fusion models(s). Furthermore, the data-driven model (whether based on end-to-end deep learning or reinforcement learning) can also be continuously optimized. Thus, for example, embodiments of the subject matter application provide a process 200 for self-sufficient optimization of automated systems. Further, the process 200 can be used as an online monitoring tool and optimization function utilizing cloud-based computing, communication, and / or storage, which can prevent or minimize interference with the actual assembly operations or tasks that the robot 106 is performing or will perform, thereby reducing or preventing any associated downtime. The refinement and optimization generated by the process 200 can also lead to process outputs of preventative maintenance recommendations, while also improving operational robustness in potentially changing manufacturing environments, as discussed above.

[0051] While the invention has been described with respect to embodiments currently considered most practical and preferred, it is to be understood that the invention is not limited to the disclosed embodiments(s), but rather, the invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, the scope of which will be interpreted in the broadest possible sense to cover all such modifications and equivalent structures permitted under law. Furthermore, it should be understood that while the use of the words preferred, preferably, or preferred in the above description indicates that the features described so far may be more desirable, it may not be necessary, and any embodiment lacking these words may be contemplated as being within the scope of the invention, as defined by the appended claims. When reading the claims, it is intended that when words such as “a,” “an,” “at least one,” and “at least a portion” are used, there is no intention to limit the claims to only one item unless expressly stated otherwise in the claims. Further, when the language “at least a portion” and / or “a portion” is used, an item may include a portion and / or the entire item unless expressly stated otherwise.

Claims

1. A method comprising: Data is collected regarding the robot's operation on a workpiece, the robot's operation being at least in part based on a response to inputs of sensing data from multiple sensors of the robot from a first operating model; The first operational model is optimized using at least a portion of the collected data to generate a second operational model; While optimizing the first operational model, a training data-driven model is generated, which utilizes an end-to-end learning approach and is at least partially based on the collected data. Evaluate both the second operational model and the training data-driven model; Based on the results of the evaluation, one of the second operational model and the training data-driven model shall be selected; as well as At least a portion of the collected data is used to validate the selected one of the second operational model and the training data-driven model for use in the operation of the robot.

2. The method of claim 1, wherein the collected data is stored in a cloud-based database, and wherein at least the following steps are performed by a cloud-based computing system: optimizing the first operational model, generating the training data-driven model, and evaluating the second operational model and the training data-driven model.

3. The method of claim 1, wherein the evaluation comprises: The expected accuracy of the training data-driven model is compared with the expected accuracy of the second operational model.

4. The method of claim 3, wherein the comparison comprises: Compare the results of at least one of the statistical evaluation, quantitative evaluation, and simulation of each of the second operational model and the training data-driven model.

5. The method according to claim 1, wherein the end-to-end learning method is at least one of an end-to-end deep learning method and an end-to-end reinforcement learning method.

6. The method of claim 1, wherein the first operating model is at least partially based on a first set of sensor parameters, and wherein the second operating model is based on a second set of sensor parameters, at least some of the second set of sensor parameters being modifications of at least some of the first set of sensor parameters.

7. The method of claim 6, wherein the modification of at least some of the first set of sensor parameters is based at least in part on at least one of a change in the robot station in which the robot operates and a change in the movement of the workpiece.

8. The method of claim 6, wherein the modification of at least some of the first set of sensor parameters is at least partially based on sensor drift of at least one of the plurality of sensors of the robot.

9. The method of claim 1, further comprising the following steps: The robot is operated at least partially using either the second operating model or the validated training data-driven model.

10. The method of claim 1, wherein the robot's operation is a final trim assembly operation of the vehicle, and wherein the step of collecting the data includes: For each vehicle for which the robot performs the final decoration assembly operation, data is collected from the robot.

11. The method of claim 1, wherein the collected data comprises: Data from the multiple sensors, motion data of the robot, and data related to the robot's performance in performing assembly tasks.

12. A system comprising: A robot having multiple sensors and a controller, the controller being configured to operate the robot based at least in part on one or more responses from a first operating model to inputs of sensed data from the multiple sensors; One or more databases are communicatively connected to the robot, and the one or more databases are configured to collect data about the robot's operations on the workpiece; as well as One or more computing components are communicatively connected to the one or more databases and the robot, and the one or more computing components are configured to: A second operational model is generated based on the optimization of the first operational model using at least a portion of the collected data; A training data-driven model is generated in parallel with the generation of the second operational model, the training data-driven model being based on an end-to-end learning method utilizing at least a portion of the collected data. Evaluate both the second operational model and the training data-driven model; Based on the results of the evaluation, one of the second operational model and the training data-driven model shall be selected; as well as At least a portion of the collected data is used to validate the selected one of the second operational model and the training data-driven model for use in the operation of the robot.

13. The system of claim 12, wherein the one or more databases include cloud-based databases.

14. The system of claim 13, wherein the one or more computing components include cloud-based computing components.

15. The system of claim 12, wherein the first operating model is a first sensor fusion model based at least in part on a first set of parameters.

16. The system of claim 15, wherein the second operating model is a second sensor fusion model based on a second set of parameters, the second set of parameters being at least partially a modification of at least a portion of the first set of parameters, the modification being based on data collected from the one or more databases.

17. The system of claim 15, wherein the second operating model is a second sensor fusion model based on a second set of parameters, the second set of parameters being at least partially a modification of the first set of parameters, the modification being at least partially based on sensor drift of at least one of the plurality of sensors of the robot.

18. The system of claim 15, wherein the training data-driven model is based on at least one of an end-to-end deep learning method and an end-to-end reinforcement learning method.

19. The system of claim 12, wherein the controller is further configured to: Replace the first operational model with the validated one of the second operational model and the training data-driven model; and The robot is operated at least in part based on one or more responses from the input of the sensed data from the plurality of sensors, which are verified from the second operating model and the training data-driven model.

20. The system of claim 12, wherein the operation performed by the robot is a final trim assembly operation of a vehicle, and wherein the one or more databases are configured to collect data from the plurality of sensors, motion data of the robot, and data relating to the performance of the robot in performing the final trim assembly operation.

Images