Machine learning logic-based adjustment techniques for robots
Machine learning logic adapts to the unique operational characteristics of manufacturing robots, addressing calibration inaccuracies and inconsistencies to enhance welding precision and efficiency.
Patent Information
- Application Number
- JP2024529489
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-19
- Filing Date
- 2022-11-17
- Publication Date
- 2024-11-19
AI Technical Summary
Conventional robots face challenges in manufacturing tasks due to unique operational characteristics, such as calibration-related inaccuracies and variations in mechanical and electrical components, leading to inconsistent manufacturing outputs and potential collisions, which complicate the determination of welding parameters and increase installation complexity.
Implementing machine learning logic, specifically artificial neural networks, to model and adjust manufacturing processes by learning from the operational characteristics of manufacturing robots, allowing for adaptive tuning of welding parameters and trajectories based on real-time data feedback.
Improves the quality and efficiency of manufacturing outputs by reducing calibration errors and inconsistencies, enhancing the precision of welding operations, and minimizing user intervention through adaptive learning techniques.
Smart Images

Figure 2024543091000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 281,573, entitled "ADAPTIVE MACHINE LEARNING LOGIC-BASED ADJUSTMENT TECHNIQUES FOR MANUFACTURING ROBOTS," filed on November 19, 2021, which is expressly incorporated by reference in its entirety.
[0002] Technical Field Aspects of the present disclosure relate generally to the field of manufacturing robots, and more specifically, to machine learning logic based coordination techniques for manufacturing robots, such as, but not limited to, welding robots. [Background technology]
[0003] Introduction Conventional robots are generally operable to perform any one or more manufacturing operations, including, but not limited to, painting, assembly, welding, brazing, or joining operations to join or adhere separate objects, surfaces, seams, voids, or spaces together. For example, a robot, such as a manufacturing robot having one or more electrical or mechanical components, may be configured to accomplish a manufacturing task (e.g., welding) to produce a manufacturing output, such as a welded part. Illustratively, the robot (e.g., software, program, method, or algorithm) may use a kinematic model of the robot to generate a trajectory for the robot to follow to accomplish the manufacturing task. The trajectory is determined for use in moving or displacing a portion of the robot, such as a welding head or welding tip, to one or more specific points, positions, or poses.
[0004] Robotic manufacturing faces several challenges due to the complexity of the robots used to accomplish manufacturing tasks. For example, each robot or manufacturing robot environment includes one or more behavior characteristics specific to the robot or manufacturing robot environment. The one or more behavior characteristics may also be related to or based on mechanical or electrical components included in the robot, which affect or contribute to the robot's behavior to produce manufacturing output. The one or more behavior characteristics may include capabilities, quality, specifications, tolerances, limitations, variations compared to another robot or environment, calibration-related tolerances / inaccuracies / imprecision, or combinations thereof. Calibration-related tolerances, inaccuracies, imprecisions are also referred to herein as "calibration-related errors" or "calibration errors," which may refer to a robot that is not perfectly matched / identical to a mathematical model (e.g., a kinematic model of the robot) used to generate a trajectory that the robot should follow to accomplish a manufacturing task. If the kinematic model does not precisely match the robot's physical hardware, the robot may not properly navigate or move into one or more desired poses, conditions, or locations, resulting in a manufacturing task not being accomplished satisfactorily (e.g., with respect to welding, a poor quality welded part with a claw or misaligned weld may be produced), a collision with the robot, or another adverse event. Additionally or alternatively, the manufacture of the robot itself, such as how tightly a bolt is tightened, variations in electrical components or systems, etc., may affect or contribute to one or more operating characteristics of the robot, the production of manufacturing output by the robot, or a combination thereof. In some circumstances, one or more operating characteristics may be considered an imperfection or limitation that may be inherent in the construction or design of the robot.
[0005] Variations in the assembly and / or manufacture of individual robots, including manufacturing tools and sensors, etc., can also lead to different manufacturing outputs and discrepancies between the outputs of two manufacturing robots designed to perform the same manufacturing function (e.g., welding). Thus, welding parameters such as current, voltage, and feed rate may need to be determined or adapted for each robot, thereby increasing the complexity of installing and operating the robot. However, determining appropriate welding parameters for a robot can be a time-consuming and challenging task, as one or more operating characteristics are often difficult to generalize across multiple similar manufacturing robots. Summary of the Invention
[0006] A quick overview of some examples The following presents a summary of some aspects of the disclosure in order to provide a basic understanding of the described technology. This summary is not an extensive overview of all contemplated features of the disclosure, and is not intended to identify key or critical elements of all aspects of the disclosure or to delineate the scope of any or all aspects of the disclosure. Its sole purpose is to present some concepts of one or more aspects of the disclosure in summary form as a prelude to the more detailed description that is presented later.
[0007] The present disclosure relates to an apparatus, system, and method that provides training, execution, or updating of machine learning logic, such as an artificial neural network, that models or adjusts a manufacturing process performed in a manufacturing robotic environment. For example, the machine learning logic may be trained and executed to learn from or make adjustments based on one or more behavioral characteristics associated with the manufacturing robotic environment. As another example, the machine learning logic, such as a trained neural network, may be executed in a semi-autonomous or autonomous manufacturing robotic environment to model a manufacturing process and generate manufacturing results. Illustratively, the machine learning logic may be executed to perform a welding operation or a process for welding. As another example, the machine learning logic, such as a trained neural network, may be updated based on data that is captured and associated with the manufacturing results. In some implementations, the machine learning logic may be specific to a manufacturing robot. In other implementations, it may not be robot specific. For example, the machine learning logic may be trained on multiple manufacturing robots, such as two or more similar manufacturing robots or two or more robots in a robotic system. By training and / or updating the machine learning logic, the quality of subsequent manufacturing results generated based on the updated machine learning logic may be improved compared to previous manufacturing results generated based on the (pre-updated) machine learning logic. In at least this sense, machine learning logic, such as a trained neural network, may be considered inherently adaptive.
[0008] In some implementations, machine learning logic associated with a manufacturing robotic environment (including a manufacturing robot) may begin the training process as untrained logic, such as an untrained neural network. The machine learning logic may receive one or more inputs based on the manufacturing robotic environment during operation of the manufacturing robot to learn from the operation of the manufacturing robot in the manufacturing robotic environment. In some implementations, the one or more inputs may relate to one or more operating characteristics, such as a calibration error or an intrinsic error, associated with the manufacturing robot. In some implementations, the one or more inputs may relate to a manufacturing process (e.g., welding) and an outcome of the manufacturing process (e.g., a weld).
[0009] In some implementations, the machine learning logic may be trained or updated using delta inputs such that the machine learning logic is configured to function as a nonlinear regression. In one such implementation, the machine learning logic may receive one or more delta inputs (e.g., one or more delta parameters) associated with a manufacturing robot in a manufacturing environment to train or update the machine learning logic. To train the machine learning logic, a threshold number of inputs may be provided to the machine learning logic as training data. After the machine learning logic is trained, the machine learning logic (or an output of the machine learning logic) may be used to perform one or more operations by the manufacturing robot. Based on the one or more operations, one or more additional inputs may be provided or fed back to the machine learning logic to update the machine learning logic. The one or more additional inputs may include or correspond to image data, input parameter data, or other data associated with the manufacturing results of the manufacturing robot.
[0010] In some implementations, the machine learning logic may receive a set of information related to the manufacturing output produced by the robot. For example, the set of information may include a seam location, a trajectory (e.g., a path plan, a motion plan, a work angle, a tip position, etc.), one or more welding parameters (e.g., a voltage, a current, a feed rate, etc.), a reference welding profile, or a combination thereof. The reference welding profile may include or indicate one or more features of a geometric representation of a weld of the seam. For example, a welding profile such as a reference welding profile may include or indicate a geometry or shape of a weld, a part or a portion of a part (being welded), or a combination thereof. In some implementations, the reference welding profile includes a manufacturing specification (e.g., a desired output) of the manufacturing output. Additionally or alternatively, the reference welding profile may be a welding profile of a weld from the robot.
[0011] The machine learning logic may generate or simulate a first weld profile based on the received trajectory, one or more welding parameters, or a combination thereof. The machine learning logic may perform a first comparison based on the first weld profile and the reference weld profile and determine a first difference based on the first comparison. Based on the first difference, the machine learning logic may update the trajectory, the one or more welding parameters, or a combination thereof. The robot may perform a manufacturing operation based on the updated trajectory, the updated one or more welding parameters, or a combination thereof to produce a first manufacturing output. One or more sensors associated with the manufacturing robot environment including the robot may capture information related to the robot, the manufacturing robot environment, the first manufacturing output, or a combination thereof. In some implementations, the information may include image data, temperature data, audio data, position data, electrical data, a shape of the weld bead after it has been welded, a shape of the welded part that may deform due to the welding process, a heat affected zone visible on the surface, porosity observed on the surface of the weld, a thermal signature associated with the weld head or the weld or the area surrounding the weld, an acoustic fingerprint of the weld, any other information that may predict the subsurface structure or quality of the weld, or a combination thereof. The machine learning logic may receive the captured data and determine a welding profile, such as a second welding profile, of the first manufacturing output based on the captured data. The machine learning logic may perform a second comparison based on the second welding profile and the reference welding profile and determine a second difference based on the second comparison. Based on the second difference, the received captured data, or a combination thereof, the machine learning logic may update the trajectory, one or more welding parameters, or a combination thereof, for use during a subsequent manufacturing operation. In some implementations, after each manufacturing operation, the machine learning logic may receive the captured data and update the trajectory, one or more welding parameters, or a combination thereof, based on the received captured data.
[0012] Certain implementations of the subject matter described in this disclosure may be implemented to realize one or more of the following potential advantages or benefits: In some aspects, the present disclosure provides techniques for training or updating machine learning logic. Training and updating machine learning models may improve the quality of manufacturing output, improve the efficiency of a robot or robotic system, reduce wasted time and materials, or a combination thereof. In embodiments, training or updating the machine learning model may be performed using reinforcement learning.
[0013] In one aspect of the disclosure, a computer-implemented method for operating a manufacturing robot includes receiving information about a first manufacturing output from one or more sensors. The first manufacturing output is generated by the manufacturing robot based on a first welding profile, a first trajectory, a first one or more manufacturing parameters, or a combination thereof. The computer-implemented method also includes processing the information about the first manufacturing output and the first profile, the first trajectory, the first one or more manufacturing parameters, or a combination thereof, by a processor to generate a second trajectory or a second one or more manufacturing parameters. The computer-implemented method further includes initiating generation of a second manufacturing output by the manufacturing robot based on the second trajectory, the second one or more manufacturing parameters, or a combination thereof.
[0014] In a further aspect of the disclosure, a computer-implemented method of operating a welding robot includes estimating a first weld profile based on one or more welding parameters, a reference weld profile, a seam location, a trajectory, or a combination thereof using machine learning logic. The computer-implemented method also includes generating one or more updated weld parameters based on a differential profile generated based on a comparison based on the one or more welding parameters, the first weld profile, and the reference weld profile. The computer-implemented method further includes determining a second weld profile based on sensor data received from the one or more sensors. The sensor data is associated with a weld formed by the welding robot based on the one or more updated welding parameters. The computer-implemented method also includes updating the machine learning logic based on the first weld profile, the second weld profile, the differential profile, the trajectory, or a combination thereof.
[0015] In a further aspect of the disclosure, a computer-implemented method for operating a welding robot includes generating a first differential profile based on a reference welding profile and a first welding profile of a first weld formed by the welding robot, and generating at least one updated welding parameter based on the first differential profile, one or more input parameters, or a combination thereof. The computer-implemented method further includes determining a second welding profile of a second weld formed by the welding robot based on the at least one updated welding parameter. The second welding profile is determined based on first sensor data received from one or more sensors. The computer-implemented method also includes generating a second differential profile based on the reference welding profile and the second welding profile, and updating the machine learning logic based on the first differential profile, the one or more input parameters, the second differential profile, and the at least one updated welding parameter.
[0016] The foregoing has outlined rather broadly the features and technical advantages of examples according to the present disclosure so that the detailed description that follows may be better understood. Additional features and advantages are described below. The concepts and examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent structures do not depart from the scope of the appended claims. The features of the concepts disclosed herein, both their organization and method of operation, together with associated advantages, will be better understood when the following description is considered in conjunction with the accompanying drawings. Each of the figures is provided for the purpose of illustration and description, and not as a definition of the limits of the claims. [Brief description of the drawings]
[0017] A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the accompanying drawings, similar components or features may have the same reference label. For purposes of brevity and clarity, not all features of a given structure may be labeled in every figure in which the structure appears. Identical reference numbers do not necessarily refer to identical structures. Rather, the same reference numbers may be used to refer to similar features or features with similar functionality, as well as non-identical reference numbers.
[0018] [Figure 1] FIG. 1 is a block diagram illustrating a system configured to execute machine learning logic in a robotic manufacturing environment according to one or more aspects.
[0019] [Diagram 2] FIG. 2 is a block diagram illustrating another system configured to execute machine learning logic in a robotic manufacturing environment according to one or more aspects.
[0020] [Diagram 3] FIG. 3 is a schematic diagram of a graph search method according to one or more aspects.
[0021] [Figure 4] FIG. 4 is a flow diagram illustrating an example process for operating machine learning logic in a robotic manufacturing environment according to one or more aspects.
[0022] [Diagram 5] FIG. 5 illustrates an example of updating a location according to one or more aspects.
[0023] [Figure 6] FIG. 6 is an example of a graph illustrating modification of a weld profile according to one or more aspects.
[0024] [Figure 7]FIG. 7 is a flow diagram illustrating another example process for operating machine learning logic in a robotic manufacturing environment according to one or more aspects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] Detailed Description The detailed description set forth below in conjunction with the accompanying drawings is intended as a description of various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the present invention. Those skilled in the art will understand that these specific details are not required in every instance, and that in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.
[0026] The present disclosure relates to apparatus, systems, and methods that provide for training, execution, or updating of machine learning logic, such as artificial neural networks, to model (and in part control or modify) manufacturing processes implemented in a manufacturing robotic environment. For example, the machine learning logic may be trained and executed to learn from or make adjustments based on one or more operating characteristics associated with the manufacturing robotic environment.
[0027] Particular implementations of the subject matter described in this disclosure may be implemented to realize one or more of the following potential advantages or benefits: In some aspects, the present disclosure provides techniques for training or updating machine learning logic. Training and updating machine learning models may improve the quality of manufacturing output, improve the efficiency of a robot or robotic system, reduce wasted time and materials, or a combination thereof.
[0028] FIG. 1 is a block diagram illustrating a system 100 configured to execute machine learning logic in a robotic manufacturing environment according to one or more aspects. In some implementations, the system 100 may include a processor-based system, an assembly robot system, or a combination thereof. The system 100 of FIG. 1 is configured to train or tune machine learning logic for one or more robots (e.g., manufacturing robots) operating in a semi-autonomous or autonomous manufacturing environment. In some implementations, the system 100 is configured to support or implement a machine learning logic-based tuning technique for tuning the machine learning logic.
[0029] A manufacturing environment or robot, such as a semi-autonomous or autonomous welding environment or a semi-autonomous or autonomous welding robot, may include one or more sensors that scan a number of parts, one or more algorithms in the form of software configured to recognize the seam to be welded, and one or more algorithms in the form of software to program the robot's movements, the operator's controls, and any other devices, such as motorized fixtures, in order to accurately or desirably weld the identified seam without collision. Additionally or alternatively, a semi-autonomous or autonomous manufacturing environment or robot may also include one or more sensors that scan the part(s), one or more algorithms in the form of software to recognize, locate, or align a given model(s) of the part(s) when the seam is detected using one or more sensors or perhaps is already displayed in some way on the given model itself, and one or more algorithms in the form of software to program the robot's movements, the operator's controls, and any other devices, such as motorized fixtures, in order to accurately or desirably weld the identified seam without collision. It should be noted that a semi-autonomous or autonomous welding robot may have some of these capabilities and may require user-supplied or selected parameters or other user (e.g., operator) involvement.
[0030] The system 100 includes a control system 110, a robot 120 (e.g., a manufacturing robot), and a manufacturing workspace 130 (also referred to herein as "workspace 130"). In some implementations, the system 100 may include or correspond to an assembly robot system. The system 100 may be configured to join a first object 135 (e.g., a first part) and a second object 136 (e.g., a second part). For example, the first object 135 and the second object 136 may be designed to form a seam 144 between the first object 135 and the second object 136. Each of the first object 135 and the second object 136 may be, but is not limited to, any part, component, subcomponent, or combination of parts or components, or the like.
[0031] The robot 120, also referred to herein as "robot 120," may be configured to perform manufacturing operations, such as welding operations, on one or more parts, such as a first object 135 and a second object 136. In some implementations, the robot 120 may be a robot having multiple degrees of freedom in that the arm may be a six-axis robot having attachment points. The robot 120 may include one or more components, such as motors, servos, hydraulics, or combinations thereof, as illustrative and non-limiting examples.
[0032] In some implementations, the attachment points may attach a welding head (e.g., a manufacturing tool) to the robot 120. The robot 120 may include any suitable tool 121, such as a manufacturing tool. In some implementations, the first robot 120 includes a robot arm, such as a robot arm manufactured by YASKAWA®, ABB® IRB, KUKA®, or Universal Robots®. In addition to the attached tool 121, the robot 120 may be configured to perform arc welding, resistance welding, spot welding, TIG welding, MAG welding, laser welding, plasma welding, and / or combinations thereof, and the like. The robot 120 may be responsible for moving, rotating, translating, feeding, and / or positioning the welding head, the sensor(s), the part(s), and / or combinations thereof. In some implementations, the welding head may be mounted, coupled, or otherwise attached to the robot 120.
[0033] In some implementations, robot 120 may be coupled to or include one or more tools. For example, based on the function the robot performs, a robot arm may be coupled to a tool configured to enable (e.g., perform at least a portion of) that function. Illustratively, a tool such as tool 121 may be coupled to an end of robot 120. In some implementations, robot 120 may be coupled to or include multiple tools, such as manufacturing tools (e.g., welding tools), sensors, picker or holder tools, or combinations thereof. In some implementations, robot 120 may be configured to operate with another device, such as another robotic device, as further described herein.
[0034] Tool 121 may include one or more tools. For example, tool 121 may include a manufacturing tool (e.g., a welding tool), a sensor (e.g., 109), a picker or holder tool, or a combination thereof. As shown, tool 121 is a picker or holder tool configured to selectively couple to a first set of one or more objects, such as a first set of one or more objects including first object 135. In some implementations, the picker or holder tool may include or correspond to a gripper, a clamp, a magnet, or a vacuum, as illustrative and non-limiting examples. For example, tool 121 may include a three-fingered gripper, such as one manufactured by OnRobot®.
[0035] In some implementations, the robot 120, the tool 121, or a combination thereof may be configured to change (e.g., adjust or manipulate) the pose of the first object 135 while the first object 135 is coupled to the tool 121. For example, the configuration of the robot 120 may be modified to change the pose of the first object 135. Additionally or alternatively, the tool 121 may be adjusted (e.g., rotated or tilted) relative to the robot 120 to change the pose of the first object 135.
[0036] The manufacturing tool 126 may be included in the system 100 and configured to perform one or more manufacturing tasks or operations. The one or more manufacturing tasks or operations may include, as illustrative and non-limiting examples, welding, brazing, soldering, riveting, cutting, drilling, and the like. In some implementations, the manufacturing tool 126 is a welding tool configured to bond two or more objects together. For example, the welding tool may be configured to weld two or more objects together, such as welding a first object 135 to a second object 136. By way of example, the welding tool may be configured to apply a weld metal along a seam formed between the first object 135 and the second object 136. Additionally or alternatively, the welding tool may be configured to fuse the first object 135 and the second object 136 together, such as fusing a seam formed between the first object 135 and the second object 136 to bond the first object 135 and the second object 136 together. In some implementations, manufacturing tool 126 may be configured to perform one or more manufacturing tasks or operations in response to a manufacturing command, such as a welding command. Although manufacturing tool 126 is shown as separate from robot 120, in other implementations, manufacturing tool 126 may be coupled to robot 120 or to another robot.
[0037] The workspace 130 may also be referred to as a manufacturing workspace. In some implementations, the workspace 130 may be any suitable welding area designed with appropriate safety measures for welding. For example, the workspace 130 may be a welding area located in a workshop, a job site, a manufacturing plant, an assembly shop, or the like. In some implementations, at least a portion of the system 100 is positioned with the workspace 130. For example, the workspace 130 may be an area or space within which one or more robotic devices (e.g., robotic arm(s)) are configured to operate on one or more objects (or parts). The one or more objects may be positioned on, coupled to, stored in, or otherwise supported by one or more pedestals, containers, bins, racks, holders, or positioners. The one or more objects (e.g., 135 or 136) may be held, positioned, and / or manipulated within the workspace 130 using fixtures and / or clamps (collectively referred to as "fixtures"). The fixtures may be configured to securely hold one or more parts. In some implementations, the fixture can be adjustable. In some implementations, the fixture can be motorized. For example, the fixture may include a motorized positioner that can be used to change the orientation of the part (and thus the seam) relative to gravity and / or the robot 120. In some implementations, the robot 120 and the motorized positioner are programmed to work in coordination with one another to change the orientation of the part and thus the seam.
[0038] The control system 110 is configured to operate and control the robot 120 to perform manufacturing functions within the workspace 130. For example, the control system 110 may operate and / or control the robot 120 (e.g., a welding robot) to perform a welding operation on one or more parts. Although described herein with respect to a welding environment, the manufacturing environment may include one or more of any of a variety of environments, such as assembly, painting, packaging, etc. In some implementations, the workspace 130 may include one or more parts (e.g., 135 or 136) to be welded. The one or more parts may be formed from one or more different parts. For example, the one or more parts may include a first part (e.g., 135) and a second part (e.g., 136), where the first part and the second part form a seam (e.g., 144) at their interface. In some implementations, the first part and the second part may be held together using a tack weld. In other embodiments, the first and second parts may not be welded, and the robot 120 may simply tack weld the seam between the first and second parts to lightly join the parts together. Additionally or alternatively, following the formation of the tack weld, the robot 120 may weld additional portions of the seam to securely join the parts together.
[0039] In some implementations, the control system 110 may be implemented external to the robot 120. For example, the control system 110 may include a server system, a personal computer system, a notebook computer system, a tablet system, or a smartphone system to provide control of the robot 120, such as a semi-autonomous or autonomous welding robot. Although the control system 110 is shown as separate from the robot 120, a portion or the entirety of the control system 110 may be implemented within the robot 120. For example, the portion of the control system 110 that is within the robot 120 may be included as a robot control unit, an electronic control unit, or an on-board computer, and may be configured to provide control of the robot 120, such as a semi-autonomous or autonomous welding robot.
[0040] The control system 110 implemented inside or outside the robot 120 may be collectively referred to herein as the "robot controller 110." The robot controller 110 may include or be coupled to a seam identification system, a trajectory planning system, a welding simulation system, another system associated with a semi-autonomous or autonomous welding robot, or a combination thereof. It is noted that one or more of the seam identification system, trajectory planning system, welding simulation system, or another system associated with a semi-autonomous or autonomous welding robot may be implemented independent of or external to the control system 110.
[0041] The control system 110 may include one or more components. For example, the control system 110 may include a controller 152, one or more input / output (I / O) and communication adapters 104 (collectively referred to hereinafter as "I / O and communication adapters 104"), one or more user interface and / or display adapters 106 (collectively referred to hereinafter as "user interface and display adapters 106"), and a network interface 106. The controller 152 may include a processor 101 and a memory 102. Although the processor 101 and the memory 102 are both described as being included in the controller 152, in other implementations, the processor 101, the memory 102, or both may be external to the controller 152, such that each of the processor 101 or the memory 102 may be one or more separate components.
[0042] The controller 152 may be any suitable machine specifically and specially configured (e.g., programmed) to perform one or more operations as described herein. In some implementations, the controller 152 is not a general-purpose computer, but is specifically programmed or hardware configured to perform one or more operations as described herein. Additionally or alternatively, the controller 308 is or includes an application specific integrated circuit (ASIC), a central processing unit (CPU), a field programmable gate array (FPGA), or a combination thereof.
[0043] In some implementations, the control system 110 may include a bus (not shown). The bus may be configured to electrically or communicatively couple one or more components of the control system 110. For example, the bus may connect the controller 152, the processor 101, the memory 102, the I / O and communications adapter 104, and the user interface and display adapter 106. Additionally or alternatively, the bus may connect one or more components or portions of the controller 152, the processor 101, the memory 102, the I / O and communications adapter 104, and the user interface and display adapter 106.
[0044] Processor 101 may include a central processing unit (CPU), which may also be referred to herein as a processing unit. Processor 101 may include a general-purpose CPU, such as the CORE family of processors available from Intel Corporation, the ATHLON family of processors available from Advanced Micro Devices, Inc., the POWERPC family of processors available from AIM Alliance, etc. However, the present disclosure is not limited by the architecture of processor 101, so long as processor 101 supports one or more operations as described herein. For example, processor 101 may include one or more special-purpose processors, such as an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a field programmable gate array (FPGA), etc.
[0045] Memory 102 may include storage devices such as random access memory (RAM) (e.g., SRAM, DRAM, SDRAM, etc.), ROM (e.g., PROM, EPROM, EEPROM, etc.), one or more HDDs, flash memory devices, SSDs, other devices configured to store data in a persistent or non-persistent state, or a combination of different memory devices. Memory 102 is configured to store user and system data and programs, which may include some or all of the aforementioned program code for performing the functions of the machine learning logic-based tuning techniques, and data associated therewith.
[0046] The memory 102 includes or is configured to store instructions 103, manufacturing information 158, welding profile 163, and information 164. In one or more aspects, the memory 102 may store instructions 103, such as executable code, that when executed by the processor 101, causes the processor 101 to perform operations according to one or more aspects of the present disclosure, as described herein. In some implementations, the instructions 103 (e.g., executable code) are a single self-contained program. In other implementations, the instructions (e.g., executable code) are programs having one or more function calls to other executable code, which may be stored in a memory device or elsewhere. One or more functions attributed to the execution of the executable code may be implemented by hardware. For example, multiple processors may be used to perform one or more individual tasks of the executable code.
[0047] The instructions 103 may include machine learning logic 107 and path planning logic 105. Additionally or alternatively, the instructions 103 may include other logic, such as recognition logic (not shown). Although shown as separate logic blocks, the path planning logic 105 and the machine learning logic 107 may be part of the memory 102 and may include program code (and associated data) for performing path planning and machine learning functions, respectively. For example, the path planning logic 105 is configured to generate a path for the robot 120 along a seam, including, but not limited to, optimizing the movement of the robot 120 to complete a weld.
[0048] As an illustrative and non-limiting example, the path planning logic 105 may be configured for graph matching or graph search techniques to generate a path or trajectory that fits the identified seam. In the case of welding, the task of welding with a welding head coupled to a robot arm may be specified in five degrees of freedom. The hardware capabilities of the system (the five of the robot arm and the degrees of freedom provided by the positioner) exceed five degrees of freedom. In some implementations, the path planning logic 105 may perform a search using more than five degrees of freedom, for example when considering collision avoidance. There are several ways to avoid this redundancy, first by specifying the task in higher dimensions, suppressing over-actuated systems, or using redundancy to explore multiple options. Traditionally, path planning is generally posed as a graph search problem. It may be considered as over-actuated planning, and in some implementations, the redundant degrees of freedom may be discretized and each sample may be treated as a unique node in constructing the graph. The resulting structure of the graph may enable fast graph search algorithms. Each point on a seam can be considered as a layer in the graph, similar to the rungs of a ladder. Due to the nature of the path planning problem, the robot must always transition between these rungs in a forward direction. This is the first aspect of the problem that makes graph exploration simpler. The path planning logic 105 generates multiple joint space solutions for each point on the seam. All solutions for a given point belong to the same layer. There are no points for the robot to transition between different solutions of the same rung, and therefore, in the graph, these nodes are not connected. This puts a further restriction on the structure of the graph.
[0049] 3 is a schematic diagram 300 of a graph search method according to one or more aspects. In some implementations, the schematic diagram represents a graph search method that may determine a path plan for the robot 120 (e.g., the controller 152).
[0050] In some implementations, as another example of the schematic diagram 300, each circle in the diagram 300 represents a different state of the robot 120, such as a configuration of joints (robot and positioner) that meets the welding requirements. Each arrow is a path the robot can take to move along the seam. To illustrate, each circle may be a specific location of the robot 120 in the workspace 130 (e.g., the location of the welding head of the robot 120 in 3D space) and a different configuration of the arms of the robot 120, as well as the location or configuration of fixtures that support the parts, such as positioners, clamps, etc. Each row 302, 306, and 310 represents a different point along the seam to be welded. Thus, for the seam point corresponding to row 302, the robot 120 can be in any of states 304A-304D. Similarly, for the seam point corresponding to row 306, the robot 120 can be in any of states 308A-308D. Similarly, for a seam point corresponding to column 310, the robot 120 may be in any of states 312A-312D. For example, if the robot 120 is in state 304A when at a seam point corresponding to column 302, the robot 120 may then transition to any of states 308A-308D for the next seam point corresponding to column 306. Similarly, upon entering states 308A-308D, the robot 120 may subsequently transition to any of states 312A-312D for the next seam point corresponding to column 310, and so on. In some instances, entering a particular state may preclude entering other states. For example, entering state 304A may allow the possibility of subsequently entering states 308A-308C, but not 308D, while entering state 304B may allow the possibility of subsequently entering states 308C and 308D, but not 308A-308B. The scope of the present disclosure is not limited to any particular number of seam points or to any particular number of robot states.
[0051] In some examples, to determine a path plan for the robot 120 using a graph search method (e.g., according to the technique shown in diagram 300), the controller 152 may determine the shortest path from states 304A-304D to a state corresponding to the seam point N (e.g., states 312A-312D). By assigning a cost to each state and each transition between states, the user or the controller 152 can design an objective function. The controller 152 finds the path that results in the smallest cost value of the objective function. With multiple start and end points at your disposal, a graph search method such as Dijkstra's algorithm or A* may be performed. In some examples, a brute force method may be useful to determine a suitable path plan. A brute force method requires the control system 110 (e.g., the controller 152 or the processor 101) to calculate all possible paths (e.g., through diagram 300) and select the shortest path (e.g., by minimizing or maximizing an objective function). In simple terms, the brute force method calculates all possible paths through this graph and selects the shortest path. The complexity of brute force can be taken as O(E), where E is the number of edges in the graph. Assume there are N points at the seam and M choices per point. Between any two layers, there are M*M edges. Hence, considering all layers, there are N*M*M edges. The time complexity is O(NM 2 ), i.e. O(E).
[0052] 1 , the machine learning logic 107 is configured to learn from and adapt to the operating characteristics of the robot 120 (e.g., calibration errors, inherent errors specific to the robot, etc.). Machine learning logic (e.g., machine learning logic 107) is provided to the robot 120, such as a welding robot, to model or perform one or more manufacturing functions (e.g., welding, painting, etc.) during or based on operation of the system 100. The machine learning logic 107 is configured to be updated based on one or more captured data from a previous manufacturing result to improve the quality of a subsequent manufacturing result.
[0053] The machine learning logic 107 may be trained using, for example, but not limited to, delta inputs, which may enable the machine learning logic 107 to function as a nonlinear regression. The delta inputs allow for normal approximation (a normal central limit type or outcome, i.e., distribution convergence to a normal distribution) of a continuously differentiable function of a sequence. Stated differently, the delta method (e.g., using delta inputs) allows for characterization of changes in the input and the derivatives of the input.
[0054] In some implementations, the machine learning logic 107 may begin the training process as an untrained logic (e.g., a neural network). Over time, as the robot 120 is used to perform manufacturing activities (e.g., welding), the robot 120 provides training data, such as one or more delta inputs (e.g., delta parameters), to the machine learning logic 107 to train the machine learning logic 107 to learn from the operating characteristics (e.g., calibration error, intrinsic error) of the robot 120. After providing a finite amount of training data, the machine learning logic 107 may be considered "trained." In an embodiment, some data (e.g., image data and / or input parameter data and / or any associated data) related to the manufacturing results (e.g., welded parts) of the robot 120 may be utilized to update the trained machine learning logic 107. This feedback process may be used to adapt or further train the trained machine learning logic 107. Examples of the operation of one or more components of the system 100 for training or updating the machine learning logic 107 are further described herein with reference to at least FIG. 4 and FIG. 7.
[0055] By way of example, and not limitation, the machine learning logic 107 (e.g., an artificial neural network) may be trained and updated to adapt to operating characteristics associated with a robot 120, such as a welding robot, thereby improving the quality (e.g., weld quality) of the output (e.g., weld) of the robot 120. In some implementations of such machine learning logic 107 associated with a robot 120, a set of information, such as manufacturing information 158, may be provided to or determined by the robot 120. The set of information may include a location of the seam (e.g., 159), a trajectory 160 to be followed by the robot 120 (e.g., path plan, motion plan, work angle, tip position, etc.), one or more welding parameters 161 (e.g., voltage, current, feed rate, etc.), and a reference weld profile 162 (e.g., a geometric representation of the weld of the identified seam).
[0056] In some implementations of such machine learning logic 107, some or all of the above information may be provided to the machine learning logic 107 to generate a simulated or estimated weld profile (e.g., 163). In some implementations, the simulated or estimated weld profile (e.g., 163) may then be compared to a reference weld profile 162 to generate a “differential weld profile” (which may be stored in memory 102 as part of the weld profile 163). The differential weld profile may include a difference between the reference weld profile 162 and the simulated or estimated weld profile. In some implementations, the differential weld profile may be an average weld profile or an average error between two or more weld profiles. The robot 120 may then determine one or more derivatives of the differential weld profile with respect to one or more input parameters to output updated weld parameters 161 (e.g., delta of the weld parameters), updated trajectory 160, or a combination thereof. In some implementations, the robot 120 may perform a welding function (e.g., welding) on a part, such as the first object 135, the second object 136, or both, using the updated welding parameters or the updated trajectory. Before moving on to the next weld, the robot 120 may capture data (e.g., image data and / or input parameter data and / or any data related to manufacturing results), such as sensor data 180, from the previous weld to determine a profile of that weld (referred to herein as the “actual weld profile”) and store it in memory 102 as part of the weld profile 163. In some implementations, the actual weld profile and / or one or more of other information including the input parameters, the initial path plan, the updated parameters (e.g., 161), and the updated trajectory 160 may then be provided as feedback information to the machine learning logic 107 to build a better learned logic that better maps the welding parameters 161 and trajectory 160 to the shape of the weld profile (e.g., 162 or 163).Updating the machine learning logic 107 based on the captured data improves the quality of subsequent welds.
[0057] Additionally or alternatively, in some implementations of the machine learning logic 107, a set of information (e.g., 158) is provided to or determined by the robot 120 (e.g., a welding robot). The set of information may include a location of the seam (e.g., 159), a weld profile of a previous weld (e.g., a geometric configuration of a weld previously made by the robot 120), a reference weld profile 162 of a reference weld, one or more weld parameters 161, a trajectory 160 (e.g., a path plan, a motion plan, a work angle, a tip position, etc.) that the robot 120 is to follow. In some implementations, the previous weld profile (e.g., 163) and the reference weld profile 162 may be processed to generate a differential weld profile (or residual weld profile). In some implementations, the differential weld profile may be provided to the machine learning logic 107 to generate delta weld parameters that include a proposed delta in the weld parameters (relative to the weld parameters originally provided to or determined by the robot 120). In other implementations, the generated parameters may include updated weld parameters (e.g., a new parameter set including the proposed delta in parameters). The robot 120 may perform the weld using the updated weld parameters (e.g., 161) or the updated trajectory (e.g., 160). Before moving on to the next weld, the robot 120 or the sensor 109 may capture data (e.g., image data or input parameter data, or any data related to the manufacturing result) from the weld to determine a weld profile (e.g., the geometric configuration of the weld) of the weld performed (referred to herein as the "actual weld profile"), which may be stored in the memory 102 as part of the weld profile 163. The robot 120 or the control system 110 may then be configured to compare the desired weld profile to the actual weld profile to generate another differential profile.In some implementations, one or more of the other difference profiles may be provided as feedback information to the machine learning logic 107 along with information related to the difference profile, such as the initial parameters, the initial path plan, the updated parameters (or deltas of the parameters), and the updated trajectory 160. As described above, the feedback information may be provided to the machine learning logic 107 to build a better trained logic.
[0058] Manufacturing information 158 may include location 159 (e.g., of a seam), trajectory 160, one or more welding parameters 161 (also referred to herein as “welding parameters 161”), and reference weld profile 162. Location 159 may include or indicate a location of a seam, such as seam 144. Trajectory 160 may include a path plan, motion plan, work angle, tip position, or other information related to the movement of robot 120. Welding parameters 161 may include or indicate voltage, current, feed rate, or other information related to the welding operation. Reference weld profile 162 may include or indicate one or more features of a geometric representation of a weld of a seam. For example, reference weld profile 162 may include or indicate a geometry or shape of a weld, a part or portion of a part (being welded), or a combination thereof. In some implementations, reference weld profile 162 includes a manufacturing specification for a manufacturing output (e.g., a desired output). Additionally or alternatively, the reference weld profile may be a weld profile of a weld by a robot, such as robot 120 or another robot. The trajectory 160, the weld parameters 161, or a combination thereof, may be used to generate control information 182 provided by control system 110 to robot 120 to cause the robot to perform one or more operations. For example, control system 110 (e.g., controller 152) may be configured to interpret the weld profile 162 and generate the trajectory 160, the weld parameters 161, the control information 182, or a combination thereof. Additionally or alternatively, control system 110 (e.g., controller 152) may send the control information 182 to robot 120 as a weld command.
[0059] The weld profile 163 may include or represent a simulated, estimated, or determined weld profile. The weld profile may include or represent one or more features of a geometric representation of a weld at a seam. Stated differently, the weld profile may include or represent the geometry or shape of a weld, a part or a portion of a part (being welded), or a combination thereof. The weld profile 163 may be simulated or determined based on the reference weld profile 162, the trajectory 160, the weld parameters, the sensor data 165 or 180 (e.g., data from the sensor 109), or a combination thereof. In some implementations, the weld profile 162 may include or represent a differential weld profile, such as an average weld profile between two or more weld profiles or an average error between two or more weld profiles.
[0060] Information 164 may include or be indicative of sensor data 165, pose information 166, or system information 168. Sensor data 165 may include or correspond to sensor data 180 received by controller 152. Pose information 166 may include or correspond to the pose of first object 135, second object 136, or a combination thereof. System information 168 may include information related to one or more devices (e.g., robot 120, tool 121, manufacturing tool 126, or sensor 109). Illustratively, system information 168 may include, by way of illustrative and non-limiting example, ID information, a communication address, one or more parameters, or a combination thereof.
[0061] The communication adapter 104 is configured to connect the control system 110 to a network (e.g., a cellular communication network, a LAN, a WAN, the Internet, etc.). The embodiment communication adapter 104 may include, for example, a WiFi network adapter, a Bluetooth interface, a cellular communication interface, a mesh network interface (e.g., ZigBee, Z-Wave, etc.), a network interface card (NIC), etc. The user interface and display adapter 106 of the example embodiment may be utilized to facilitate a user interface with the control system 110. For example, the user interface and display adapter 106 may connect one or more user input devices (e.g., a keyboard, a pointing device, a touchpad, a microphone, etc.) to the control system 110 to facilitate user input when desired (e.g., when collecting information regarding one or more welding parameters).
[0062] In some implementations, the I / O and communications adapter 104 may also connect sensor(s) 109 (e.g., global sensors, local sensors, etc.) to the processor 101 and memory 102, such as for use with the system to detect or otherwise determine seam locations. The I / O and communications adapter 104 may additionally or alternatively provide connections for various other devices, such as printers (e.g., dot matrix printers, laser printers, inkjet printers, thermal printers, etc.) to facilitate desired functions (e.g., to enable the system to print paper copies of information, such as planned trajectories, results of learned movements, and / or other information and documents).
[0063] The user interface and display adapter 106 may be configured to connect one or more user output devices (e.g., flat panel display, touch screen, heads-up display, holographic projector, etc.) to the control system 110 to facilitate user output (e.g., welding simulation) when desired. It should be understood that various of the aforementioned functional aspects of the control system 110 may be included or omitted as desired or determined to be appropriate depending on the particular implementation of a particular instance of the system 100.
[0064] The user interface and display adapter 106 is configured to be connected to a storage device 108, a sensor 109, another device, or a combination thereof. The storage device 108 may include one or more of a hard drive, an optical drive, a solid state drive, or one or more databases. The storage device 108 may be configured to be connected to the controller 152, the processor 101, or the memory 102, such as for exchanging program code and data associated therewith to perform the functions of the machine learning logic-based tuning technique.
[0065] In some implementations, the storage device 108 may store, for example, program code (e.g., program logic) of the machine learning logic, data used by the machine learning logic, for example, historical parameter information, attributes of the welding environment, etc.
[0066] The sensor 109 may be configured to capture data (e.g., image data) of the workspace 130. The sensor 109 may include an image sensor, such as a camera, a scanner, a laser scanner, a camera with an integrated laser sensor, and the like. The sensor 109 may be configured to capture 2D and / or 3D images of one or more parts or fixtures. In some cases, the sensor 109 may be configured to image or monitor a weld applied by the robot 120 before, during, or after welding. Stated differently, the sensor 109 may be configured to generate one or more images capturing a geometric configuration of a weld applied by the robot 120. The geometric configuration may be anything including 3D point cloud information, a mesh, an image of a slice of a weld, a point cloud of a slice of a weld, and the like. The aforementioned technique is referred to herein as "weld profile generation." In some implementations, the sensor 109 may be mounted on the robot 120 or may be otherwise integrated into the workspace 130. For example, the sensor 109 may be attached to the robot 120 or may be part of the robot instead. In some implementations, one or more sensors 109 may be mounted on a separate robot (not shown in FIG. 1 ) positioned within workspace 130. For example, the robot may be operable to move (e.g., rotate and translate) such that sensor 109 can capture image data of workspace 130 and / or one or more parts (e.g., 135 and 136) and / or fixtures from various angles.
[0067] The sensor 109 may be configured to capture image data of the workspace 130 from various positions and angles. More specifically, the sensor 109 may capture image data of one or more parts and fixtures from multiple angles or perspectives. In some implementations, the one or more parts and fixtures may be disposed on or may include a positioner having rotational or translational motion. For example, the one or more parts or fixtures may be disposed on a welding table that can rotate about any suitable axis or translate in the x-, y-, and / or z-directions. As another example, the one or more parts may be positioned on a positioner to coordinate its rotational or translational motion in coordination with the robot 120. The sensor 109 may capture and collect image data (e.g., of one or more parts or fixtures disposed on the welding table) as the table moves and / or spins.
[0068] Additionally or alternatively, the sensor 109 may be coupled to a fixture or holder configured to rotate or translate. For example, the sensor 109 may be mounted to a holder including a base and side clamps. The sensor 109 may be disposed on the base and clamped between the side clamps. In some implementations, the base may be configured to rotate along at least two axes to capture image data of the one or more parts and fixtures. In some implementations, the base may be configured to rotate 360 degrees about any number of axes to capture image data of the one or more parts and fixtures from multiple angles. In some implementations, the sensor 109 may be stationary, while a fixture holding one or more parts (e.g., 135 and 136) may be configured to rotate or translate. For example, a fixture holding one or more parts may be configured to rotate about any suitable axis and / or translate in the x-, y-, and / or z-directions. As another example, one or more sensors 109 may be coupled to a robot 120 or a positioner system configured to directly manipulate parts. The sensor 109 may capture and collect image data as the sensor fixture moves or rotates.
[0069] In some implementations, the sensor 109 may generate a point cloud representation of or related to the workspace 130. A point cloud may be a collection of points each representing a location in three-dimensional space of a point on one or more parts or fixture surfaces. In some embodiments, one or more images (e.g., image data captured by the sensor 109 at a particular orientation relative to one or more parts) may be overlaid on one another to reconstruct and generate three-dimensional image data of the workspace 130. The three-dimensional image data may be collated such that a point cloud generated from the data may have six degrees of freedom. For example, each point on the point cloud may represent an infinitesimal location in three-dimensional space. As described above, the sensor 109 may capture multiple images of a point from various angles. These multiple images may be collated to determine an average image pixel for the point. The averaged image pixel may be attached to the point. For example, if the sensor 109 is a color camera with red, green, and blue channels, the six degrees of freedom may be {x position, y position, z position, red intensity, green intensity, and blue intensity}. For example, if the sensor 109 is a black and white camera with a black and white channel, four degrees of freedom can be generated.
[0070] As described above, the sensor 109 can be communicatively connected to the processor 101. The processor 101, controller 152, or control system 110 can operate to process data from the sensor 109 to assemble two-dimensional data, data from one or more sensors 109 at various positions relative to one or more parts, or data from the sensor 109 as the sensor moves relative to the part for further processing. The control system 110, such as the processor 101, can generate a point cloud by overlaying and / or stitching images to reconstruct and generate three-dimensional image data. The three-dimensional image data can be collated to generate a point cloud along with associated image data for at least some points in the point cloud. The control system 110 can be configured to operate and control the robot 120. In some cases, control parameters of the robot 120 can be determined or informed by data from the point cloud.
[0071] In some implementations, the control system 110, such as the memory 102, may include recognition logic (not shown in FIG. 1). The recognition logic may include one or more neural networks (e.g., convolutional neural networks, deep neural networks, etc.) configured to perform pixel-by-pixel classification on image data captured by the sensor 109. Additionally or alternatively, the one or more neural networks may perform pixel-by-pixel classification on a combination of image data and spatial information such as depth data. In some implementations, the recognition logic may include one or more neural networks that can perform point-by-point classification on a point cloud that may or may not include image data. The one or more neural networks may be configured to perform pixel-by-pixel (e.g., from the image data) or point-by-point (e.g., from the point cloud data) classification to identify and classify fixtures and clamps in the image or point cloud. If pixel-by-pixel classification is performed, the results may be projected onto three-dimensional point cloud data or a meshed version of the point cloud data, thereby providing information regarding the location of the fixtures or clamps in space. If the input data is image data (e.g., color images), spatial information such as depth information may be included along with the color data to perform pixel-by-pixel segmentation. In some implementations, a pixel-by-pixel classification may be performed to identify one or more seams and locate the seams relative to the part. In some implementations, the recognition logic may be configured to perform classification, recognition, or segmentation on the image data. The image data may include one or more of a point cloud, a mesh (e.g., a collection of vertices and faces), or both.
[0072] In some implementations, the one or more neural networks in the recognition logic may be any suitable neural network capable of performing point-by-point or pixel-by-pixel classification, such as a U-net model. The neural network may be trained on image data, point cloud data, spatial information data, or a combination thereof. Since the point cloud or image data includes information captured from various angles, the neural network (e.g., the U-net model) may operate to classify the fixture or clamp, or one or more seams from multiple angles or perspectives. In some implementations, the neural network may be trained to operate directly on the point set, such as, for example, a dynamic graph convolutional neural network, and may be implemented to analyze unorganized points on the point cloud. In some implementations, a first neural network may be trained on the point cloud data to perform point-by-point classification, and a second neural network may be trained on the image data to perform pixel-by-pixel classification. The first neural network and the second neural network may individually identify and locate one or more seams. The outputs from the first neural network and the second neural network can be combined as a final output to determine the location and orientation of the seam relative to the part.
[0073] In some implementations, the recognition logic may identify one or more seams on a part without prior information about the part (e.g., without a CAD model of the part) and locate them relative to the part. Each seam in the one or more seams may be identified as a subset of a set of points on the part (e.g., from a point cloud). For example, if the neural network(s) perform a pixel-by-pixel classification to identify one of the one or more seams, the identified pixels may be projected into a three-dimensional space to generate a three-dimensional point set. Alternatively, if the neural network(s) classifies points from a point cloud, the identified points will be a three-dimensional point set. For the recognition logic, these three-dimensional points may be an unordered set of points. Furthermore, it may be possible that at least some of these three-dimensional points are clumped together. More specifically, in areas closer to the seam, the three-dimensional points may be clumped across the width of the seam. To eliminate this noise and generate a continuous, unbroken subset of points representing the seam, a Manifold Blurring and Mean Shift (MBMS) technique or similar method may be applied. This technique can cluster points and eliminate noise. A clustering method can then be applied to decompose one or more seams into individual seams. Following the clustering, a spline can be fitted to each individual point subset. Thus, each individual point subset can become an individual seam.
[0074] In some implementations, the control system 110 may be configured to identify potential locations of seams based on prior information provided about the part(s). Similarly, a CAD model of a part may include annotations of seams to be welded in the part. If a CAD model or other information about the part is available, the control system 110 may be configured to parse or analyze the model to identify seam locations. Based on the annotations, the control system 110 may use object matching to determine expected locations and expected orientations of the seams relative to the part. For example, the control system 110 may match components on the part with representations of those components in the CAD model. Thus, the control system 110 may determine expected locations and expected orientations of the seams.
[0075] In the following description, the seam determined by the control system 110 using the CAD model is referred to as the first seam, and the recognition logic determines the location and orientation of the seam (e.g., using a neural network). This may be referred to as the second seam. The first seam and the second seam may be in approximately the same location if the CAD model or the recognition logic accurately predicts the location of the seam. Alternatively, the first seam and the second seam may partially overlap if the CAD model or the recognition engine is partially accurate. In yet another example, the first seam and the second seam may be in different locations, for example, when the CAD model or the recognition engine locates the candidate seam inaccurately. In one of the above examples, the control system 110 may perform a comparison between the first seam and the second seam. This comparison between the first seam and the second seam may be based in part on the shape of both seams and their relative location in space. In some implementations, one of the first seam or the second seam may be deformed relative to the other. For example, if the control system 110 may determine that a first seam is closer to the ground truth, then the second seam may be deformed to match the first seam. Alternatively, if the control system 110 may determine that the second seam is closer to the ground truth, then the first seam may be deformed to match the second seam. Once the seams are identified and finalized, the path planning logic 105 is configured to generate a path for the robot 120 along the seams, including, but not limited to, determining or optimizing the movements of the robot 120 to complete the weld.
[0076] Because the recognition logic is configured to classify the fixture or clamp, or one or more seams, from various angles resulting from multiple viewpoints in the image data or point cloud, the verification logic can be configured to further verify whether the seam is in fact a seam. For example, after the recognition logic identifies and locates a seam, the verification logic can determine whether the seam is an actual seam, thereby eliminating false positives.
[0077]
[0023] Referring to Figure 2, Figure 2 is a block diagram illustrating another system 200 configured to execute machine learning logic in a robotic manufacturing environment according to one or more aspects. System 200 may include or correspond to system 100 of Figure 1.
[0078] Compared with the system 100 of Fig. 1, the system 200 includes a plurality of robots. Illustratively, the plurality of robots includes four robots, a first robot (e.g., 120), a second robot 212, a third robot 214, and a fourth robot 216. Furthermore, the sensor 109 includes a plurality of sensors, such as a first sensor 234 and a second sensor 236. In addition to the first tool (e.g., 121), the system 200 also includes a positioner 240 (e.g., a positioning device), a structure 242, and a second tool 222.
[0079] The workspace 130 of the system 200 may include one or more devices or components of the system 200. As shown, the workspace 130 includes the first robot 120, the first tool 121, the second robot 212, the second tool 222, the first sensor 234, and the manufacturing tool 126. In other implementations, the workspace 130 may include fewer or more components or devices than those shown in FIG. 2. For example, the workspace 130 may include the third robot 214, the fourth robot 216, the second sensor 236, the positioner 240, the structure 242, the control system 110, or a combination thereof.
[0080] In some implementations, the plurality of robotic devices may include or correspond to the robot 120. For example, at least one of the plurality of robotic devices (e.g., 120, 212, 214, 216) may include a robotic arm providing, by way of non-limiting example, six degrees of freedom. In implementations, the robotic arm may be manufactured by YASKAWA®, ABB® IRB, KUKA®, Universal Robots®. Additionally or alternatively, the robotic arm may be configured to couple to one or more tools.
[0081] The second robot 212 may include a second robotic arm. The second tool 222 may be coupled to an end of the second robotic arm. In some implementations, the second tool 222 may include or correspond to the first tool 121. For example, the second tool 222 may be configured to selectively couple to a second set of one or more objects including the second object 136. The second set of one or more objects may be the same as or different from the first set of objects to which the first tool 121 is configured to couple.
[0082] The third robot 214 may include a third robotic arm. The first sensor 234 may be coupled to an end of the third robotic arm. In some implementations, the first sensor 234 is configured to generate first sensor data (e.g., 180). For example, the first sensor 234 is configured to capture an image of one or more of the first object 135, the second object 136, or a combination thereof.
[0083] The fourth robot 216 includes a fourth robot arm. A manufacturing tool 126 (e.g., a welding tool) is coupled to the end of the fourth robot arm.
[0084] The second sensor 236 is configured to generate second sensor data (e.g., 180). For example, the second sensor 236 is configured to capture an image of one or more of the first object 135, the second object 136, or a combination thereof. In some implementations, the second sensor 236 is positioned on or coupled to the structure 242. The structure 242, such as a frame or weldment, can be dynamic or static. In a dynamic or static configuration of the structure 242, the second sensor 236 may be configured to be dynamic or static with respect to the structure 242, for example, if the second sensor 236 is dynamic, the second sensor may be configured to rotate (e.g., pan) or tilt.
[0085] The positioner 240 may be configured to host (e.g., hold or securely hold) the first object 135, the second object 136, or a combination thereof. By way of example, the positioner 240 may be dynamic or adjustable manually by a user or automatically by a motor. For example, the positioner 240 may dynamically adjust its pose (e.g., position, orientation, or other physical configuration) before or during a welding process based on instructions received from the control system 110. In some implementations, the positioner 240 is configured to adjust the pose or location of an object hosted by the positioner 240 while the object is hosted by the positioner 240. For example, the positioner 240 may be configured to hold, position, or manipulate the first object 135, the second object 136, or a combination thereof. In some implementations, the positioner 240 may be configured to hold, position, or attach one or more objects, such as by using a clamp or other type of fastener. Although FIG. 2 illustrates a single positioner, in other implementations, the robotic system may include multiple positioners. In some implementations, the positioner 240 may be essentially static in that the positioner 240 may be a bin-like storage unit that can host objects (e.g., the positioner 240 may not be configured to adjust the pose or location of the object). In some such implementations, the object may be positioned randomly (e.g., in a random pose) within the unit.
[0086] During operation of system 200 , control system 110 may perform one or more operations as described with reference to system 100 .
[0087] In some implementations, the machine learning logic 107 may be robot specific. For example, assume that a first machine learning logic is configured to be trained for a first manufacturing robot and a second machine learning logic is configured to be trained for a second manufacturing robot. In some implementations, the first machine learning logic and the second machine learning logic may begin the training process in the same or similar state (e.g., untrained state), and over time, i.e., as the first manufacturing robot and the second manufacturing robot are used individually to perform their respective manufacturing activities (e.g., welding and painting), the first machine learning logic and the second machine learning logic may learn to adapt to operating characteristics specific to the respective manufacturing robots (e.g., calibration errors, inherent errors specific to the manufacturing robots). In some other implementations, the first machine learning logic may begin the training process in a first state (e.g., untrained state) and may be trained to a second state (e.g., trained state). The first machine learning logic may be further trained from the second state to a third state (e.g., trained state). The second machine learning logic may begin the training process using the first machine learning logic in the second state and may be trained to a fourth state (e.g., a trained state) that is different from the third state. The first machine learning logic and the second machine learning logic may, in some implementations, be trained using a delta method in which delta values of one or more parameters associated with the machine learning logic (e.g., a neural network) are used to train the machine learning logic. For example, the delta method determines a derivative (e.g., delta) of one or more delta parameters (e.g., differential profile, differential weld parameters) to generate updated parameters that can be used to perform a subsequent welding operation. This is more clearly illustrated in block 410 of FIG. 4.
[0088] In some implementations, the machine learning logic 107 may not be robot specific. In such implementations, the machine learning logic 107 may be trained to approximate (e.g., estimate) the behavioral characteristics of multiple (e.g., two or more) similar manufacturing robots. To illustrate, the machine learning logic 107 may be trained to estimate behavioral characteristics associated with both a first manufacturing robot and a second manufacturing robot. In this sense, the machine learning logic 107 may accommodate the behavioral characteristics of both the first manufacturing robot and the second manufacturing robot. In other words, the first manufacturing robot and the second manufacturing robot may be structurally similar (e.g., they may be built using similar parts, such as motors or robot arms for performing welding, to accomplish the same manufacturing task), and the same machine learning logic 107 may be trained for both the first manufacturing robot and the second manufacturing robot, even though each of the first manufacturing robot and the second manufacturing robot has different behavioral characteristics.
[0089] In some embodiments, a computer-implemented method for updating welding instructions for a manufacturing robot, such as a welding robot (e.g., robot 120), is provided. The computer-implemented method includes performing a first manufacturing task (e.g., welding) using the manufacturing robot on a first part positioned within the workspace 130. The first manufacturing task results in a first manufacturing output (e.g., a weld) on the first part. The first manufacturing task is based on a first profile (e.g., 162 or 161), a first trajectory (e.g., 160), a first one or more manufacturing parameters (e.g., 161), or a combination thereof. The method further includes receiving, by one or more sensors 109, information 164 related to the first manufacturing output. The method also includes processing, by the processor 101, the first profile, the first trajectory, the first one or more manufacturing parameters, and the information related to the first manufacturing output to generate a second trajectory (e.g., 160) or a second one or more manufacturing parameters (e.g., 161). The method includes performing a second manufacturing task using the manufacturing robot on a second part using one or more of the second trajectory and the second one or more manufacturing parameters.
[0090] Referring now to FIG. 4, FIG. 4 is a flow diagram illustrating an example process 400 for operating machine learning logic in a robotic manufacturing environment according to one or more aspects. In some implementations, the machine learning logic may include or correspond to machine learning logic in a robotic manufacturing environment (e.g., an environment using robot 120). Process 400 may be performed by robot 120 including machine learning logic 107. Assume that one or more parts (e.g., 135 or 136) on which a manufacturing task (e.g., welding) is to be performed are positioned or fixed on another positioner in manufacturing workspace 130 using a fixture. For example, the positioner may include or correspond to positioner 240. After placement of one or more parts, process 400 may begin.
[0091] At 402, process 400 includes scanning one or more parts. The scanning may be performed by one or more sensors 109. In some implementations, control system 110 may be configured to identify one or more seams on the one or more parts using image data acquired from sensor 109 or a point cloud derived from the image or sensor data. Seams, such as seam 144, may be identified based on one of the techniques described with respect to FIG.
[0092] Once the seam or seam location is determined, the path planning logic 105 of the control system 110 plots an initial path for the robot 120 along the identified seam, including determining or optimizing the movement of the robot 120 to complete the weld. The path or trajectory may be planned based on one of the path planning techniques described with respect to FIG. 1. In some implementations, relevant information regarding the seam, such as the location of the seam, the shape of the seam identified as a seam, etc., may also be provided to the machine learning logic for subsequent processing at 404.
[0093] Additionally or alternatively, in some implementations, one or more weld parameters 161 may also be provided at 403 for subsequent processing. For example, after scanning the part to identify the seam, the control system 110 may provide one or more weld parameters to the operator for selection via a user interface. The operator may select one or more weld parameters via the user interface, and the selected weld parameters may then be provided at 404 to the machine learning logic. In other implementations, the control system 110 may prompt the operator after identifying the seam to physically input one or more weld parameters for use by the machine learning logic 404 in subsequent processing. Alternatively, a user may not be involved and the control system 110 may operate autonomously. In some implementations, a reference weld profile 162, which may be stored in the memory 102 of the control system 110, may also be provided at 403 to the machine learning logic 404 for subsequent processing.
[0094] In some implementations, the machine learning logic at 404 may then receive one or more of the information that may be provided to the robot or determined by the robot 120. The one or more of the information may include the location of the identified seam (e.g., location 159), the trajectory 160 that the robot 120 will follow, one or more welding parameters 161, and the reference weld profile 162. Additionally or alternatively, in some implementations, a second piece of information or pieces of information may be provided to or received by the machine learning logic at 404. This second piece of information or pieces of information may include a thermal signature associated with the weld head or the weld or the area surrounding the weld. The second piece of information or pieces of information may also include an acoustic fingerprint of the weld or other details regarding the weld.
[0095] At 406, the process 400 includes estimating a first weld profile by the machine learning logic using one or more of the information provided or determined by the machine learning logic. The first weld profile may reflect a weld profile that the control system 110 estimates the robot 120 may output based on a current state of the manufacturing robot. The current state of the manufacturing robot may reflect a current training state of the machine learning logic and errors that the manufacturing robot plans to correct based on learnings from one or more previous welds, identified seam locations, trajectories, one or more welding parameters, a baseline weld profile, or a combination thereof.
[0096] At 408, the process 400 includes comparing the estimated weld profile to the reference weld profile 162 (provided at 403) to generate a difference profile. The difference profile may reflect (or may include) differences with respect to coordinates (e.g., polar, Cartesian, etc.) or images.
[0097] At 410, the process 400 includes performing backpropagation on the difference profile (e.g., the difference profile reflecting the error between the desired profile and the actual profile). By way of example, the backpropagation may be performed by the control system 110, such as the processor 101 or the machine learning logic 107. Backpropagation may be considered to be "backpropagation of error," and this technique includes algorithms for learning using gradient descent. For example, backpropagation calculates the gradient of an error function with respect to the weights of the neural network (or machine learning logic). Backpropagation may also be considered to determine a derivative of the difference profile with respect to one or more input parameters, such as the one or more weld parameters 161 provided at 403 and the reference weld profile 162. In some implementations, the one or more operations performed at 408 may be considered to generate a first derivative of the weld profile (e.g., generating a difference profile), and the one or more operations performed at 410 may be considered to generate a second derivative of the estimated weld profile.
[0098] In some embodiments, the process 400 may include, at 412, the control system 110 generating updated welding parameters based at least on the determined derivatives. Additionally or alternatively, in some embodiments, the control system 110 may also generate an updated trajectory based at least on the determined derivatives. The updated trajectory may be similar (or different) to the initial trajectory in one or more aspects, such as path planning, motion planning, work angle, tip position, etc. In other embodiments, the updated trajectory may require the controller 110 to re-plan the trajectory based on the updated parameters. In some embodiments, if the difference between the initial trajectory and the updated trajectory is relatively small, the control system 110 may directly update the initial trajectory to include the recommended changes in the updated trajectory. An example of such a scenario is shown with reference to FIG. 5.
[0099] Referring to FIG. 5, which illustrates an example of position updating according to one or more aspects, as shown in FIG. 5, the initial tip position or weld angle of the welding head is updated based on the updated trajectory, while the rest of the details from the initial trajectory remain the same.
[0100] 4 , at 414, the robot 120 may perform a manufacturing task, such as welding, using the updated welding parameters or trajectory. At 416, before moving to the next weld, the robot 120 or the sensor 109 may capture data (e.g., image data) from the weld performed at 414 to determine a profile (e.g., weld profile 163) of the weld performed at 414. The data may be used by the control system 110 to generate an actual weld profile (e.g., weld profile 163) for the weld performed at 414. In some implementations, the actual weld profile (e.g., weld profile 163) or some information related to the actual weld profile, including one or more input parameters, the initial path plan, the updated one or more parameters, and the updated trajectory, may be provided as feedback information to the machine learning logic, as indicated by the arrow from 416 to 404.
[0101] The process of training the machine learning algorithm used in process 400 will now be described. Referring to FIG. 6, an example graph 600 illustrating the change in weld profile according to one or more aspects is shown. Given a collection of scans of various weld parameters 161, trajectories 160, joint geometries, and weld profiles 163, the goal is to generate machine learning logic (e.g., 108) that receives a set of shapes (e.g., shape deltas or differences) and outputs a set of parameters (e.g., updated parameters). The desired weld profile S i Let S be the scanned welding profile at a given waypoint. k The change required to transform the shape ΔS is represented by ΔW. The shape change ΔS is imposed by a change in a parameter (e.g., welding parameters 161) represented by ΔW. The machine learning logic 107 may then learn how to encode the change in shape and output the necessary parameter updates (e.g., updated parameters or deltas in one or more initial input parameters) that produce such a change.
[0102] The delta shape may be represented in various ways, such as polar or Cartesian coordinates or images. In some implementations, a convolution operation may be performed on the delta shape. The loss function may be either L1 or L2. The machine learning logic 107 thus trained may not require manual labeling, since the ground truth delta parameters are created from the parameters specified in the command for the weld slice or the complete weld. Training the machine learning logic 107 may require scanned data of a large number of welds (e.g., a finite number of previous welds made by a manufacturing robot) along with all the parameters associated with each of those welds. This data set may be referred to as the auxiliary data set.
[0103] Training the machine learning logic 107 may include initially scanning the part with a calibrated system, such as a fully or substantially fully calibrated system. The first scan made with the calibrated system may be referred to as a reference weld, i.e., it may be considered the gold standard for corrections when the robot 120 is not calibrated. During subsequent scans, parameters from the first scan may be used. During the third scan, updated parameters from the second scan may be used. During scan n, updated parameters from scan (n-1) may be used. In this training approach, the weights of the machine learning logic (e.g., neural network) encode the relationship between the shape of the weld profile and its associated parameters. Because this relationship may be different for different cells, at different times, the robot 120 may be continually trained using scans acquired with the manufacturing robot before and after welding to update its machine learning logic. This approach may handle calibration drift when the manufacturing robot is actively in use.
[0104] In some embodiments, a computer-implemented method for updating welding instructions for a manufacturing robot, such as a welding robot (e.g., robot 120), is provided. The computer-implemented method includes receiving, by a processor 101, a reference weld profile 162, one or more desired weld parameters 161, a seam location (e.g., 159), or an initially planned trajectory. The method also includes estimating, by machine learning logic 107 executed by the processor 101, a first weld profile based on the seam 144 location (e.g., 159), the initial trajectory (e.g., 160), the reference weld profile 162, and the one or more desired parameters (e.g., 161). The method then includes comparing, by the processor 101, the first weld profile to the reference weld profile 162 to generate a differential profile (or residual profile). The method then includes processing, by the processor 101, the differential weld profile to determine a derivative of the delta profile with respect to the one or more input parameters. The method further includes generating updated welding parameters (e.g., 161) or trajectory (e.g., 160) based on the determined derivative. The method includes welding the seam using the updated welding parameters, by the robot 120, and receiving, by the processor 101, a second welding profile of the weld made with the updated welding parameters. The method then includes updating the machine learning logic 107 based on at least the first welding profile and the second welding profile, other details associated with the first welding profile and the second welding profile, or a combination thereof.
[0105]
[0036] Referring now to Figure 7, Figure 7 is a flow diagram illustrating another example process 700 for operating machine learning logic in a robotic manufacturing environment according to one or more aspects. For example, process 700 may be performed by a robot (e.g., 120), a control system 110, machine learning logic 107, or a combination thereof. As with the discussion of Figure 4, it is assumed that process 700 may begin following placement or fixation of one or more parts (e.g., 135 and 136) onto a positioner in manufacturing workspace 130. The positioner may include or correspond to positioner 240.
[0106] At 702, process 700 includes scanning one or more parts. The scanning may be performed by one or more sensors 109. The control system 110 may be configured to identify one or more seams on one or more parts (e.g., 135, 136) using image data acquired from the sensors 109 or a point cloud derived from the image or sensor data 180. The seams (e.g., 144) may be identified based on one of the techniques described with respect to FIG.
[0107] After the seam or seam location is determined, the path planning logic 105 of the control system 110 plots an initial path for the robot 120 along the identified seam, including determining or optimizing the movements of the robot 120 to complete the weld. In some implementations, relevant information about the seam, such as seam location, seam identified seam geometry, etc., may be used by the control system 110 for subsequent processing.
[0108] In some implementations, information may be provided as input to the control system 110 at 703 before, during, or after the scanning at 702. The information may include a previous weld profile (e.g., 163), a reference weld profile 162, one or more weld parameters 161, or a combination thereof. Additionally or alternatively, in some implementations, the one or more weld parameters 161 may also be utilized by the control system 110 for subsequent processing. For example, after scanning the part to identify a seam, the control system 110 may provide the operator with one or more weld parameters 161 for selection via a user interface, and the selected one or more weld parameters 161 may then be used by the control system 110 for further processing. In some implementations, after scanning the part to identify a seam, the control system 110 may determine one or more parameters based on one or more information captured during seam identification. For example, the control system 110 may determine one or more parameters based on one or more of the characteristics of the part, such as the material or material type, material thickness, joint geometry and material cross-sectional geometry, welding wire, welding gas present in the machine, the capabilities of the welding power source, the achievable position of the part relative to gravity given the motorized positioner or lack thereof on which the part is mounted, or the space available for the robot to operate without collision. In other embodiments, the control system 110 may prompt the operator to physically input one or more welding parameters for use by the control system 110 after identifying the seam. In some embodiments, the reference weld profile 162 and the weld profile 163 of one or more previous welds, which may all be stored in the memory 102 of the control system 110, may also be used by the control system 110 for subsequent processing.
[0109] After capturing the above information, at 704, the process 700 may compare the previous weld profile 163 to the reference weld profile 162 to generate a difference profile. The difference profile may reflect (or include) differences in terms of coordinates (e.g., polar, Cartesian, etc.) or images. One or more of the information provided, processed, or determined at 702 or 704 may be provided to machine learning logic at 706.
[0110] At 708, process 700 may include generating updated parameters or trajectories. In some embodiments, the updated parameters or trajectories may be used by the robot 120 in performing a welding function (e.g., welding).
[0111] At 710, process 700 may include the robot 120 capturing or receiving data (e.g., image data) from the weld made at 708 using sensors 109 to determine a profile of the weld (the "actual weld profile"). In some embodiments, at 712, the actual weld profile (e.g., 163) may be compared again to the reference weld profile 162 to generate another difference profile. In some embodiments, this other difference profile, or some information related to that other weld profile, including the one or more input parameters, the initial path plan, the updated one or more parameters, the actual weld, the reference weld, or the updated trajectory, may be provided as feedback information to the machine learning logic, as indicated by the arrow from 712 to 706.
[0112] A process for training the machine learning algorithm used in process 700 will now be described. Given the collection of scans of various welding parameters, trajectories, joint geometries, and weld profiles, the goal is to generate machine learning logic that receives a set of parameters and outputs a set of shapes. The machine learning algorithm may be trained with one or more welding parameters and associated trajectories (e.g., initial trajectories) that may include or correspond to trajectory 160. The output may be a weld (or a slice thereof) represented in either polar or Cartesian coordinates as a point cloud. In some implementations, the output may be a weld slice, so that the scanned weld and information captured from the scanned weld can be used as ground truth. This machine learning algorithm may not require manual labeling. Unlike the machine learning algorithm described in process 400, the machine learning algorithm described with respect to FIG. 7 may operate without an auxiliary data set.
[0113] Training the machine learning logic described with reference to FIG. 7 may include scanning the part with a calibrated system, such as a calibrated system or a substantially fully calibrated system (e.g., an initial scan). The scan made with the calibrated system may be referred to as a reference weld profile 162, i.e., it may be considered a gold standard for correction when the manufacturing robot is not calibrated. It may be conventional to directly use the output of the machine learning logic for a particular task. However, the machine learning logic described in FIG. 7 first generates a weld profile, and then the slope of the profile with respect to the parameters (e.g., a delta profile) is calculated to update the parameters, such as the weld parameters 161.
[0114] Let w be the set of parameters that need to be updated, S(w) be the predicted shape of w, and S(w) be the desired shape. D A prediction of the values of these parameters that will result in a desired weld is calculated as follows: TIFF2024543091000002.tif7128
[0115] The gradient of the shape with respect to the parameters may be calculated by the backpropagation process or technique described above. In this approach, the machine learning logic weights encode the welding process by predicting the weld profile given the parameters; in a sense, this is a welding simulator. Because the relationship between parameters and weld profile may be different for different cells and at different times, the robot 120 continually trains and updates itself using scans taken of cells before and after welding.
[0116] In any of the machine learning logics described above, the torch and travel angles may be among the parameters that the machine learning logic 107 may update via the techniques described above. In such an embodiment, the machine learning logic 107 may directly use those values as inputs. To better normalize the input values, a local coordinate frame created from the torch and travel angles may be defined. At each waypoint, the sampled weld profile and joint shape points may be described locally using the angle specified in the command. As previously mentioned, the points may be in polar or Cartesian coordinates. To represent the points in polar coordinates, the weld profile may be sampled at equal angular or equal geodesic distance intervals. For each point, a radius r from that point and the origin may be calculated. The angle α is calculated based on the base e of the frame. o is the angle between and the vector from the origin to the point.
[0117] In some embodiments, a computer-implemented method for updating welding instructions for a manufacturing robot, such as a welding robot (e.g., robot 120), is provided. The computer-implemented method includes receiving a previous weld profile (e.g., 163), a reference weld profile 162, a seam location (e.g., 159), one or more input parameters, an initially planned trajectory, or a combination thereof. The method further includes processing the previous weld profile (e.g., 163) and the reference weld profile 162 to generate a first differential profile. The method also includes processing the differential weld profile to generate at least updated weld parameters 161. The method then includes welding at the seam location using the updated weld parameters. The method includes receiving an observed weld profile of a weld made with the updated weld parameters 161 based on the weld. The method includes processing the observed weld profile (e.g., 163) and the reference weld profile 162 to generate a second difference profile, and updating the machine learning logic 107 based on the first difference profile, the one or more input parameters, the second difference profile, the updated parameters (e.g., 161), or a combination thereof.
[0118] Although portions of the disclosure have been described as using weld profile data to train machine learning logic 107 and generate one or more updated parameters and trajectories, machine learning logic 107 may additionally or alternatively be trained using the shape of the weld bead after it has been welded, the shape of the welded part that may deform due to the welding process, the heat affected zone visible on the surface, porosity observed on the surface of the weld, a thermal signature associated with the weld head or the weld or the area surrounding the weld, an acoustic fingerprint of the weld, or any other information that may predict the subsurface structure or quality of the weld. In any scenario, one or more of the following may be used as inputs (as reference parameters, absolute values, or deltas) to the trained machine learning logic 107: the shape of the weld bead after it has been welded, the shape of the welded part that may deform due to the welding process, the heat affected zone visible on the surface, porosity observed on the surface of the weld, a thermal signature associated with the weld head or the weld or the area surrounding the weld, an acoustic fingerprint of the weld, or any other information that may predict the subsurface structure or quality of the weld.
[0119] The above-mentioned embodiments generally relate to robots configured to autonomously perform assembly tasks. Semi-autonomous or autonomous welding robots typically have capabilities including, but not limited to, including sensors to scan some given object, algorithms in the form of software to automatically identify and position the object to be welded, and algorithms in the form of machine-readable code to control the movement of the robot(s) to pick the object, and algorithms in the form of machine-readable code to control the movement of the robot(s) to perform welding. The algorithms in the form of machine-readable code may also be configured to control any other device, such as a motorized positioner, to change the pose of the object to perform welding. Additionally or alternatively, a semi-autonomous or autonomous welding robot may also have capabilities including, but not limited to, sensors that scan some given object, algorithms in the form of software that automatically recognize, locate and register a given model of object(s), which may possibly already be represented (or annotated) in some way on the given model itself (e.g., may be annotated with tack weld locations), as well as algorithms in the form of software that automatically program the robot(s), movements, picking robots, welding robots, and any other devices, such as motorized positioners, to weld the seam accurately or as desired.
[0120] It should be noted that a semi-autonomous or autonomous robot may only have some of these above-mentioned capabilities and may require some user-imposed or selected parameters or may require user involvement in other ways, such as placing objects in a positioner(s) or providing annotations on a model (e.g., a computer-aided design (CAD) model).
[0121] Although the aspects and advantages of the present application have been described in detail, it should be understood that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the present disclosure as defined by the appended claims. Moreover, the scope of the present application is not intended to be limited to the particular embodiments of the processes, machines, manufacture, compositions, means, methods and steps described herein. As one skilled in the art will readily appreciate from the above disclosure, existing or future developed processes, machines, manufacture, compositions, means, methods or steps may be utilized that perform substantially the same function or achieve substantially the same result as the corresponding embodiments described herein. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions, means, methods or steps.
[0122] The above specification provides a complete description of the structure and use of the exemplary configurations. Although certain configurations have been described above with a certain degree of particularity or with reference to one or more individual configurations, those skilled in the art may make numerous modifications to the disclosed aspects without departing from the scope of the present invention. Thus, the various exemplary configurations of the method and system are not intended to be limited to the particular forms disclosed. Rather, they include all modifications and alternatives that fall within the scope of the claims, and configurations other than those shown may include some or all of the features of the illustrated configurations. For example, elements may be omitted or combined into a unitary structure, connections may be replaced, or both. Furthermore, where appropriate, aspects of any of the above-described examples may be combined with any of the other examples described to form further examples having equivalent or different characteristics and / or functions and addressing the same or different problems. Similarly, it will be understood that the benefits and advantages described above may relate to one configuration or to several configurations. Thus, any single embodiment described herein should not be construed as limiting, and embodiments of the present disclosure may be combined as appropriate without departing from the teachings of the present disclosure.
[0123] While various embodiments have been described above, it should be understood that these have been presented by way of example only, and not limitation. Although various embodiments have been described as having particular combinations of features and / or components, combinations of any feature and / or component from any of the examples, as well as other embodiments having additional features and / or components, are possible, where appropriate.
[0124] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Where the methods described above indicate that certain events are performed in a specific order, the order of the certain events may be modified. Furthermore, some of the events may not only be performed sequentially as described above, but may also be performed simultaneously as a parallel process where possible.
[0125] Similarly, although operations are illustrated in a particular order in the figures, this should not be understood as requiring that such operations be performed in the particular order or sequence illustrated, or that all of the illustrated operations be performed, to achieve desirable results. Furthermore, the figures may generally illustrate another exemplary process in the form of a flow diagram. However, other operations not illustrated may be incorporated into the generally illustrated exemplary process. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the illustrated operations. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may generally be integrated into a single software product or packaged into multiple software products. Furthermore, the following claims include several other embodiments. In some cases, the operations recited in the claims may be performed in a different order and still achieve desirable results.
[0126] Those skilled in the art will appreciate that information, messages, and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, and signals that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.
[0127] The components, functional blocks, and modules illustrated and described herein include, among other things, processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software code, firmware code, or any combination thereof. Software shall be broadly construed to mean, among other things, instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise. Additionally, features discussed herein may be implemented via dedicated processor circuitry, executable instructions, or combinations thereof.
[0128] Various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functions. Various exemplary logic, logic blocks, modules, circuits, and algorithmic processes described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. In one or more aspects, the described functions may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. The embodiments of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions, encoded on a computer storage medium for execution by or for controlling the operation of a data processing device.
[0129] The hardware and data processing devices used to implement the various exemplary logics, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, certain processes and methods may be performed by circuitry specific to a given function.
[0130] If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. The processes of the methods or algorithms disclosed herein may be implemented in processor-executable software modules that may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that may enable a computer program to be transferred from one place to another. A storage medium may be any available medium that may be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store desired program code in the form of instructions or data structures and that may be accessed by a computer. Any connection may also be properly referred to as a "computer-readable medium." Additionally, the operations of a method or algorithm may also be present as one or any combination or set of machine-readable media and codes and instructions on computer-readable media, which may be incorporated into a computer program product.
[0131] Some embodiments described herein relate to methods or processing events. It should be understood that such methods or processing events can be computer-implemented. That is, when methods or other events are described herein, it should be understood that they can be performed by a computing device having a processor and memory. The methods described herein can be performed locally, e.g., on a computing device physically co-located with the robot or on a local computer / controller associated with the robot, and / or remotely, e.g., on a server and / or in the "cloud."
[0132] The memory of a computing device is also called a non-transitory computer-readable medium and can include instructions or computer code for performing various computer-executed operations. The computer-readable medium (or processor-readable medium) is non-transitory in the sense that it does not include a transitory propagating signal itself (e.g., a propagating electromagnetic wave that carries information over a transmission medium such as space or a cable). The medium and computer code (which can also be called code) may be designed and constructed for a specific purpose. Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tapes, optical storage media such as compact disks / digital video disks (CD / DVDs), compact disk read-only memory (CD-ROM), and holographic devices, magneto-optical storage media such as optical disks, carrier wave signal processing modules, read-only memory (ROM), random access memory (RAM), and the like. One or more processors can be communicatively connected to the memory and can operate to execute the code stored on the non-transitory processor-readable medium. Examples of processors include general purpose processors (e.g., CPUs), graphical processing units, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), etc. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those generated by a compiler, code used to generate web services, and files containing higher level instructions that are executed by a computer using an interpreter. By way of illustration, examples may be implemented using imperative programming languages (e.g., C, Fortran, etc.), functional programming languages (Haskell, Erlang, etc.), logic programming languages (e.g., Prolog), object-oriented programming languages (e.g., Java, C++, etc.), or other suitable programming languages and / or development tools.Further examples of computer code include, but are not limited to, control signals, encryption code, and compression code.
[0133] As used herein, various terms are merely for the purpose of describing particular embodiments, and are not intended to limit the embodiments. For example, as used herein, an ordinal term (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, component, operation, etc., does not in itself indicate a priority or order of that element relative to another element, but merely distinguishes that element from another element having the same name (except for the use of the ordinal term). The term "coupled" is defined as connected, although not necessarily directly and not necessarily mechanically, and two things that are "coupled" may be integral with each other. The terms "a" and "an" are defined as one or more, unless the disclosure expressly requires otherwise.
[0134] The term "about" as used herein allows for some variation in value or range, e.g., within 10%, 5%, or 1% of the stated limits of the stated value or range, and includes the exact stated value or range. The term "substantially" is defined as most, but not necessarily all, of what is specified, as understood by those skilled in the art (and includes what is specified, e.g., substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel). In any disclosed embodiment, the term "substantially" may be replaced with "within [a percentage] of" what is specified, with percentages including 0.1, 1, 5, or 10%, and the term "approximately" may be replaced with "within 10% of" what is specified. The statement "substantially X to Y" is synonymous with "substantially X to substantially Y" unless otherwise specified. Similarly, the statement "substantially X, Y, or substantially Z" is synonymous with "substantially X, substantially Y, or substantially Z" unless otherwise specified. Unless otherwise indicated, the word "or" as used herein is an inclusive "or" and is interchangeable with "and / or," such that when "or" is used in a list of two or more items, it means that any one of the listed items can be used alone, or any combination of two or more of the listed items can be used. By way of example, A, B, and / or C includes A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. Similarly, the phrase "A, B, C, or any combination thereof" or "A, B, C, or any combination thereof" includes A alone, B alone, C alone, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C.
[0135] Throughout this specification, values expressed in range format should be interpreted flexibly to include not only the numerical values expressly recited as the limits of the range, but also all individual numerical values or subranges subsumed within that range, as if each numerical value and subrange were expressly recited. For example, the range "about 0.1% to about 5%" or "about 0.1% to 5%" should be interpreted to include not only about 0.1% to about 5%, but also the individual values (e.g., 1%, 2%, 3%, and 4%) and subranges (e.g., 0.1% to 0.5%, 1.1% to 2.2%, 3.3% to 4.4%) within the recited range.
[0136] The term "comprise" (and any form of comprise, e.g., "comprises" and "comprising"), the term "have" (and any form of have, e.g., "has" and "having"), the term "include" (and any form of include, e.g., "includes" and "including"), and the term "contain" (and any form of contain, e.g., "contains" and "containing"). Consequently, an apparatus that "comprises", "has", "includes" or "contains" one or more elements possesses those one or more elements, but is not limited to possessing only those one or more elements. Similarly, a method that "comprises", "has" or "includes" one or more steps possesses those one or more steps, but is not limited to having only those one or more steps.
[0137] Any embodiment of any of the systems, methods, and articles of manufacture can consist of or consist essentially of any of the described steps, elements, or features, rather than including / having / comprising any of the described steps, elements, and / or features. Thus, in any of the claims, the terms "consisting of" or "consisting essentially of" can be used in place of the open-ended linking verbs listed above to modify the scope of a given claim from that which would otherwise use an open-ended linking verb. Additionally, the term "wherein" can be used interchangeably with "where."
[0138] Moreover, a device or system configured in a particular way is configured in at least that way, but may also be configured in other ways other than as specifically described. One or more features of one embodiment may be applied to other embodiments even if not described or illustrated, unless expressly prohibited by the nature of this disclosure or that embodiment.
[0139] The claims are not intended to, and should not be construed as including, means-plus-function or step-plus-function limitations, unless such limitations are expressly recited in a given claim using the phrase(s) "means for" or "step for," respectively.
[0140] The preceding description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure and the following claims are not intended to be limited to the examples and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. 1. A computer-implemented method for operating a manufacturing robot, comprising: receiving information from one or more sensors regarding a first manufacturing output, the first manufacturing output generated by the manufacturing robot based on a first profile, a first trajectory, a first one or more manufacturing parameters, or a combination thereof; processing, by a processor, the information regarding the first manufacturing output and the first profile, the first trajectory, the first one or more manufacturing parameters, or a combination thereof, to generate a second trajectory or second one or more manufacturing parameters; and Initiating production of a second manufacturing output by the manufacturing robot based on the second trajectory, the second one or more manufacturing parameters, or a combination thereof. A method comprising:
2. Receiving the first profile, the first trajectory, the first one or more manufacturing parameters, or a combination thereof.
2. The computer implemented method of claim 1, further comprising:
3. generating first control information based on the first profile, the first trajectory, the first one or more manufacturing parameters, or a combination thereof; 3. The computer implemented method of claim 2, further comprising:
4. The computer-implemented method of claim 3 , wherein the first control information comprises one or more welding instructions.
5. sending the first control information to the manufacturing robot to initiate a first manufacturing task by the manufacturing robot on a first part positioned within a manufacturing workspace to generate the first manufacturing output; 4. The computer implemented method of claim 3, further comprising:
6. Generating the second trajectory, the second one or more manufacturing parameters, or a combination thereof.
2. The computer implemented method of claim 1, further comprising:
7. generating second control information based on the second trajectory, the second one or more manufacturing parameters, or a combination thereof; and sending the second control information to the manufacturing robot to cause the manufacturing robot to initiate a second manufacturing task on a second part positioned within a manufacturing workspace to generate the second manufacturing output; 2. The computer implemented method of claim 1, further comprising:
8. 1. A computer-implemented method for operating a welding robot, comprising: estimating a first weld profile based on one or more weld parameters, a reference weld profile, a seam location, a trajectory, or a combination thereof using machine learning logic; generating one or more updated welding parameters based on a difference profile generated based on a comparison of the one or more welding parameters, the first welding profile and the reference welding profile; determining a second weld profile based on sensor data received from one or more sensors, the sensor data relating to a weld formed by the welding robot based on the one or more updated welding parameters; and updating the machine learning logic based on the first weld profile, the second weld profile, the difference profile, the trajectory, or a combination thereof. A method comprising:
9. 10. The computer-implemented method of claim 8, wherein the machine learning logic is configured to be executed by a process that performs one or more operations.
10. receiving data including or indicative of the one or more welding parameters, the reference weld profile, the seam location, the trajectory, the reference weld profile, or a combination thereof; 10. The computer implemented method of claim 8, further comprising:
11. The receiving process is receiving image data of the parts to be welded from the one or more sensors; and determining the location based on the image data; 12. The computer implemented method of claim 11, comprising:
12. The receiving process is receiving at least one welding parameter of the one or more welding parameters via a user interface; and using path planning logic to determine the trajectory a robot is configured to follow to weld the seam.
13. The computer implemented method of claim 12, comprising:
13. The method further comprising: performing the comparison based on the first weld profile and the reference weld profile; and generating said difference profile based on said comparison Further comprising: generating the one or more updated welding parameters includes processing the difference profile.
9. The computer implemented method of claim 8.
14. sending control information to the welding robot instructing the welding robot to perform a welding operation on the seam to form a weld, the control information indicating the one or more updated welding parameters; and receiving the sensor data from the one or more sensors.
10. The computer implemented method of claim 8, further comprising:
15. 10. The computer-implemented method of claim 8, further comprising providing the machine learning logic with learned behavior characteristics specific to the welding robot.
16. 10. The computer-implemented method of claim 8, further comprising providing the machine learning logic with learned operating characteristics of a plurality of welding robots, including the welding robot.
17. 1. A computer-implemented method for operating a welding robot, comprising: generating a first differential profile based on a reference welding profile and a first welding profile of a first weld formed by the welding robot; generating at least one updated welding parameter based on the first differential profile, one or more input parameters, or a combination thereof; determining a second welding profile of a second weld formed by the welding robot based on the at least one updated welding parameter, the second welding profile determined based on first sensor data received from one or more sensors; generating a second difference profile based on the reference weld profile and the second weld profile; and updating machine learning logic based on the first difference profile, the one or more input parameters, the second difference profile, and the at least one updated welding parameter. A method comprising:
18. receiving first information indicative of the first weld profile, the reference weld profile, a location of a seam to be welded, the one or more input parameters, a planned trajectory, or a combination thereof; and sending control information to the welding robot instructing the welding robot to perform a welding operation to form the second weld, the control information indicating the at least one updated welding parameter; 20. The computer implemented method of claim 17, further comprising:
19. receiving the first sensor data from the one or more sensors, the first sensor data generated based on the second weld; 20. The computer implemented method of claim 18, further comprising:
20. Prior to the step of generating the first difference profile, using the machine learning logic to estimate a third weld profile based on one or more welding parameters, the reference weld profile, a location of the welded seam, a planned trajectory, or a combination thereof; generating one or more updated welding parameters based on the one or more welding parameters, a third difference profile generated based on a comparison based on the third welding profile and the reference welding profile; sending control information to the welding robot instructing the welding robot to perform a welding operation to form a third weld, the control information indicating the one or more updated welding parameters; and determining the first weld profile based on second sensor data received from the one or more sensors, the second sensor data relating to the third weld formed by the welding robot based on the one or more updated welding parameters; 20. The computer implemented method of claim 17, further comprising:
