Systems for generating instructions for welding robots and computer-implemented methods for generating instructions for welding robots, each using techniques for multi-pass welding

Abstract

The present disclosure provides systems, methods, and apparatus, including computer programs encoded on computer storage media, that provide welding techniques for manufacturing robots, such as multi-pass welding techniques for welding robots. For example, the welding techniques may enable the generation of welding instructions based on a weld fill plan. The instructions may be generated based on a bead model or table indicating wire feed speed, travel speed, or voltage. As another example, the techniques may enable the generation of welding instructions based on one or more dimensions of the seam. As another example, the techniques may enable the generation of a joint model of a cross-section of the seam to be welded. The joint model may be generated based on a combination of multiple feature components to generate the joint model of the seam. Other aspects and features are also claimed and described.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 392,366, entitled "AUTONOMOUS MULTIPASS WELDING," filed July 26, 2022, which is expressly incorporated herein by reference in its entirety.

[0002] FIELD OF THE INVENTION Aspects of the present disclosure relate generally to the field of manufacturing robots, and more particularly to welding techniques for manufacturing robots, such as, but not limited to, multi-pass welding techniques for welding robots. [Background technology]

[0003] Introduction Conventional robots are generally operable to perform one or more manufacturing operations, including, but not limited to, painting, assembling, welding, brazing, or joining operations to join or adhere separate objects, surfaces, seams, empty gaps, or spaces together. For example, a robot, such as a manufacturing robot having one or more electrical or mechanical components, may be configured to accomplish a manufacturing task (e.g., welding) and produce a manufacturing output, such as a welded part. Illustratively, the robot (e.g., software, program, method, or algorithm) may use a kinematic model of the robot to generate a trajectory for the robot to follow to accomplish the manufacturing task. The trajectory is determined for use in driving or moving a portion of the robot, such as a welding head or welding tip, to one or more specific points, positions, or poses.

[0004] Robotic manufacturing faces several challenges due to the complexity of the robot used to accomplish the manufacturing task, the variations or tolerances of the parts to be welded, or a combination thereof. For example, a welding plan may be generated for welding two parts, such as a first part and a second part, that define a seam to be welded. Typically, welding plans are generated based on a computer-aided design (CAD) model and involve human input and decision-making. Human involvement in generating a welding plan introduces subjective criteria based on personal experience into the welding plan. Such subjective criteria are difficult to quantify, and welding plans generated by two different people to weld the same two parts can vary significantly. Furthermore, welding plans generated based on human involvement often must be individually programmed, resulting in a static welding plan that is executed to manufacture multiple copies of the same product. Furthermore, the CAD model may include or indicate tolerances or ranges associated with one or more acceptable dimensions of the parts to be welded and therefore the seam to be welded. The welding plan is typically determined based on a single set of part dimensions. Thus, a weld plan generated for a single set of dimensions may not produce acceptable results when executed on parts having a different set of dimensions if the parts (and seams) to be welded have a different set of dimensions that are still within the acceptable range of dimensions per the CAD model. For example, in a multi-pass welding operation, a weld plan may be executed on parts having a different set of dimensions, resulting in an error in the first pass compared to the predicted or desired weld, and the error may propagate or increase with each subsequent pass. Summary of the Invention

[0005] The following summarizes some aspects of the present disclosure in order to provide a basic understanding of the described technology. This summary is not an extensive overview of all contemplated features of the present disclosure, and is not intended to identify key or critical elements of all aspects of the present disclosure or to delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in summary form as a prelude to the more detailed description that is presented later.

[0006] The present disclosure relates to apparatus, systems, and methods that provide for generating welding plans or instructions for a welding robot associated with a manufacturing process performed in a manufacturing robotic environment. For example, the welding plans or instructions may be associated with a multi-pass welding operation, such as a multi-pass welding operation performed by a welding robot, to enable welding two or more components together along one or more seams in multiple passes. By way of example, a method, such as a computer-implemented method, may include identifying a seam defined by a plurality of parts and configured to receive welding material to form a weld (e.g., a joint) joining the plurality of parts. The seam may be identified based on a computer-aided design (CAD) model of the part, a scanned representation of the part, or a combination thereof.

[0007] In some implementations, sensors may capture image data from various positions and viewpoints within the workspace. A set of candidate seams associated with one or more parts may be generated, and the locations and orientations of the candidate seams may be indicated. Pixel-by-pixel and / or point-by-point classification techniques may be performed using neural networks to classify and identify each pixel and / or point as a part, a candidate seam on or associated with a part, or a fixture at an interface between multiple parts. Structures identified as non-part and non-candidate seam structures are segmented and separated. After the set of candidate seams is generated, a determination may be made as to whether the candidate seams are, in fact, seams and suitable for welding.

[0008] Based on the identification of the seam, multiple waypoints represent the seam. In some implementations, each waypoint may be associated with a cross section of the seam. For example, the cross section of the seam may be associated with or used to generate a joint template (e.g., a joint model) for the seam. The joint model may represent the topology of the seam, one or more features, one or more feature relationships, feature tolerances, one or more characteristics of the multiple parts, or a combination thereof. In some implementations, the joint model may be generated based on multiple feature components. For example, a joint library may store multiple feature components. Each feature component may include features such as structural features. Multiple feature components of the multiple feature components may be combined to generate the joint model. Based on features of the multiple feature components, one or more feature relationships may be generated that indicate relationships between the features.

[0009] In some implementations, a weld fill plan may be generated that includes or indicates multiple passes for placing weld material associated with the seam. For example, a weld profile may be generated for the waypoints, and a weld fill plan may be generated based on the one or more weld profiles for the one or more waypoints. The weld fill plan may include an ordered sequencing of weld beads within a cross section associated with at least one waypoint of the multiple waypoints.

[0010] In some implementations, to generate a welding profile, the number of bead layers can be determined based on a bead layer height, such as a bead layer height indicated by a user, based on annotated data associated with a CAD model determined by implicit welding criteria and solved via optimization techniques, or a combination thereof. Additionally or alternatively, the number of weld beads included in a bead layer may be determined. In some implementations, one or more welding parameters for each bead in the welding profile of the waypoint. For example, the one or more welding parameters may include or indicate a welding wire size used to form the weld bead, an area or volume of the weld bead, or a combination thereof. Additionally or alternatively, the one or more welding parameters may include a wire feed speed, a travel speed, or a combination thereof. In some implementations, to determine at least one of the one or more welding parameters, a table indicating a wire feed speed, a travel speed, or a combination thereof may be accessed based on a welding wire size or an area or volume of the weld bead. In some other implementations, a trained model, such as a neural network model, a physics-based model, or a combination thereof, may additionally or alternatively be used to determine the at least one welding parameter. To illustrate, the trained model may determine at least one welding parameter based on the welding wire size or the area or volume of the weld bead, illustrative non-limiting examples being wire feed speed, travel speed, or a combination thereof.

[0011] In some other implementations, the weld profile may be determined based on one or more candidate beads populated in the joint model. For example, one or more candidate beads may be placed in the joint model based on one or more features of the joint model. Additionally or alternatively, one or more candidate beads may be placed in the joint template based on the surfaces included in the joint template, the weld cover profile, the unfilled space, or a combination thereof. The weld profile may be generated based on one or more candidate beads (e.g., size characteristics, spatial characteristics, etc.), one or more welding constraints, one or more robot parameters, a bead model, or a combination thereof. In some implementations, the weld profile may be determined based on the height or volume of the weld to be formed. For example, the weld profile may be generated to include multiple layers of equal or individually optimized heights, equal volumes or areas, or a combination thereof. One or more welding parameters, such as wire feed speed (WFS), travel speed (TS), or voltage, may be determined based on one or more spatial characteristics (e.g., area, width, etc. of the weld bead or layer), wire size, or a combination thereof. Additionally or alternatively, a bead model can be used to estimate the distribution of beads placed in one of the passes.

[0012] In some implementations, a weld fill plan is generated based on one or more weld profiles. The weld fill plan may include or indicate the number of layers, the number of beads in each layer, bead size, weld size, cover profile, material cost, average bead size, minimum bead size, maximum bead size, distance of the bead from the construction point, one or more cross sections, voids (e.g., gaps) in the material, access to supporting weld material (e.g., tack), or combinations thereof. In some implementations, the weld fill plan may be validated based on one or more operating characteristics of a welding robot. By way of example, the one or more operating characteristics of a welding robot may define what the welding robot can or cannot do to form a weld. In some implementations, the one or more operating characteristics may include or correspond to a torch impact model and kinematics, and validating the weld fill plan based on the one or more operating characteristics may confirm whether the weld fill plan can be executed by the welding robot. Welding robot instructions may be generated based on the weld fill plan and sent to the welding robot.

[0013] Particular implementations of the subject matter described in this disclosure may be implemented to realize one or more of the following potential advantages or benefits: In some aspects, the present disclosure provides techniques for generating instructions for a welding robot. The instructions may include or indicate multi-pass operations to be performed by the welding robot. The multi-pass operations may be determined using a bead model that estimates or predicts the profile of weld material to be placed in a portion of the seam weld. Additionally or alternatively, generating instructions for multi-pass operations as described herein may improve the quality of manufacturing output, improve the efficiency of the robot or robot system, reduce wasted time and material, or a combination thereof.

[0014] In one aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes identifying a seam to be welded, the seam being defined based on a first part and a second part. The computer-implemented method also includes, for at least one waypoint of a plurality of waypoints along a length of the seam, determining a number of layers and a number of weld beads for a cross section at the at least one waypoint of the joint to fill the seam. The computer-implemented method further includes generating a weld fill plan for the seam based on the cross section at the at least one waypoint and based on the bead model. The computer-implemented method also includes generating instructions for the welding robot to perform one or more weld passes based on the weld fill plan.

[0015] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes determining one or more joint features based on a seam to be welded, the seam being defined based on a first part and a second part. The computer-implemented method also includes determining a weld fill plan for the seam, the weld fill plan including one or more weld beads. Each weld bead of the one or more weld beads includes one or more properties determined based on the one or more joint features. The computer-implemented method further includes generating one or more welding command properties based on the one or more joint features, one or more properties of the one or more weld beads, or a combination thereof.

[0016] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes identifying a seam to be welded via a plurality of weld passes, the seam being defined based on a first part and a second part. The computer-implemented method also includes identifying a weld volume associated with the seam. The computer-implemented method further includes, for at least one waypoint of the plurality of waypoints associated with the seam, generating a weld profile associated with a cross-section of the weld volume at the at least one waypoint. The computer-implemented method also includes generating instructions for the welding robot to perform the plurality of weld passes and apply weld material within the weld volume based on the weld profile.

[0017] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes receiving a weld fill plan for a seam to be welded via multiple weld passes, the seam being defined based on a first part and a second part. The computer-implemented method also includes identifying a weld volume associated with the seam based on scan data received from one or more sensors. The computer-implemented method further includes generating instructions for the welding robot to perform the multiple weld passes to apply weld material within the weld volume. The instructions are generated based on a comparison performed using the weld fill plan and the identified weld volume.

[0018] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes identifying a seam to be welded, the seam being defined based on a first part and a second part. The computer-implemented method also includes generating a plurality of waypoints along a length of the seam. The computer-implemented method further includes generating a joint model of a cross section of the seam for at least one waypoint of the plurality of waypoints based on the plurality of characteristic components.

[0019] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes identifying a seam to be welded, the seam being defined based on a first part and a second part. The computer-implemented method also includes generating a plurality of waypoints along a length of the seam. The computer-implemented method further includes, for at least one waypoint of the plurality of waypoints, generating a weld profile associated with a cross-section of the seam at the at least one waypoint, the weld profile indicating an ordered sequence of weld beads to be formed. The computer-implemented method also includes generating instructions for the welding robot to perform a plurality of weld passes to form the weld bead based on the weld profile.

[0020] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot includes identifying a location of a seam on parts to be joined together via multiple weld passes. The identification of the seam is based on a computer-aided design (CAD) model of the parts and / or a scanned representation of the parts. The parts may include a first part and a second part separate from the first part. Additionally or alternatively, the first part and the second part are positioned such that the first part and the second part form a seam along which the first part and the second part are welded. In some cases, the scanned representation may be generated using one or more images captured by a sensor communicatively coupled to the welding robot. The computer-implemented method further includes identifying a weld volume around the seam, the weld volume including / receiving multiple weld passes. Each weld pass forms a weld layer at least partially inside the identified weld volume. The computer-implemented method also generates instructions for the welding robot to perform the multiple weld passes.

[0021] In an additional aspect of the present disclosure, a computer-implemented method for generating instructions for a welding robot is executed by a controller. The method includes identifying a location of a seam on parts to be joined together by welding. The identification of the seam is based on a computer-aided design (CAD) model of the parts and / or a scanned representation of the parts. The parts may include a first part and a second part separate from the first part. Additionally or alternatively, the first part and the second part are positioned such that the first part and the second part form a seam along which the first part and the second part are welded. In some cases, the scanned representation may be generated using one or more images captured by a sensor communicatively coupled to the welding robot. The computer-implemented method further includes generating a plurality of waypoints representing the seam or identifying a weld volume around the seam. The weld volume is configured to include / receive a plurality of weld layers configured to join the first and second parts together. The computer-implemented method for a welding robot also includes generating a weld profile for the identified weld volume. To generate the weld profile, the method may further include generating a weld fill plan for at least one of the plurality of waypoints. The weld fill plan may include ordered sequencing of weld beads within a cross section associated with the at least one waypoint of the plurality of waypoints. The computer-implemented method may include generating instructions for a welding robot to weld according to the weld fill plan.

[0022] The foregoing has outlined rather broadly the features and technical advantages of examples according to the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages are described below. The concepts and examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent constructions do not depart from the scope of the appended claims. The properties of the concepts disclosed herein, both their construction and methods of operation, together with associated advantages, will be better understood from the following description when considered in connection with the accompanying figures. Each of the figures is provided for the purpose of illustration and description, and not as a definition of the limits of the claims. [Brief explanation of the drawings]

[0023] A further understanding of the nature and advantages of the present disclosure may be realized by reference to the following drawings. In the accompanying drawings, like components or features may have the same reference labels. For purposes of brevity and clarity, every feature of a given structure is not always labeled in every figure in which that structure appears. Identical reference numbers do not necessarily refer to identical structures. Rather, identical reference numbers may be used to refer to similar features or features with similar functionality, as well as non-identical reference numbers. [Figure 1] FIG. 1 is a block diagram illustrating a system configured to enable multi-pass welding in a robotic manufacturing environment, in accordance with one or more embodiments. [Figure 2] FIG. 1 is a block diagram illustrating another system configured to enable multi-pass welding in a robotic manufacturing environment, in accordance with one or more embodiments. [Figure 3] FIG. 1 is a schematic diagram of a graph search technique, according to one or more embodiments. [Figure 4] FIG. 1 is a representation of a robotic arm, according to one or more embodiments. [Figure 5] FIG. 1 is an example point cloud diagram of a part having a weldable seam, according to one or more embodiments. [Figure 6]FIG. 1 is an example point cloud diagram of a part having a weldable seam, according to one or more embodiments. [Figure 7] FIG. 1 is a block diagram illustrating an alignment process flow, according to one or more embodiments. [Figure 8] FIG. 1 is a schematic diagram of an autonomous robotic welding system, according to one or more embodiments. [Figure 9] 1 is a flow diagram illustrating an example process for generating welding instructions for a welding robot, according to one or more embodiments. [Figure 10] 1 is a flow diagram illustrating an example process for generating welding instructions for a welding robot, according to one or more embodiments. [Figure 11] 1 is a flow chart illustrating an example process for operating a welding robot, according to one or more embodiments. [Figure 12] FIG. 1 is a perspective view of an example representation of parts to be welded, according to one or more embodiments. [Figure 13] FIG. 10 is a perspective view of another example representation of parts to be welded, according to one or more embodiments. [Figure 14] FIG. 14 is an example of a cross-sectional projection of the seam of FIGS. 12 and 13, according to one or more embodiments. [Figure 15] FIG. 10 is an illustration of an example cross-sectional projection of a seam, according to one or more embodiments. [Figure 16] FIG. 10 is another example of a cross-sectional projection of a seam, according to one or more embodiments. [Figure 17] 1 includes a graph illustrating a modeled bead, according to one or more embodiments. [Figure 18] FIG. 1 is a diagram of an example weld fill plan for a weld profile, according to one or more embodiments. [Figure 19] FIG. 10 is an illustration of an example weld fill strategy across multiple weld profiles, according to one or more embodiments. [Figure 20] 1 includes an example of a template matched to a recognized feature, according to one or more embodiments. [Figure 21] 1 is an example of a joint template diagram, according to one or more embodiments. [Figure 22]10 includes an example of a joint template diagram with different leg length definitions, according to one or more embodiments. [Figure 23] 1 is another example of a joint template diagram, according to one or more embodiments. [Figure 24] 1 is another example of a joint template diagram, according to one or more embodiments. [Figure 25] 1 is another example of a joint template diagram, according to one or more embodiments. [Figure 26] 1 is an example of generating a fill plan using a joint template diagram, according to one or more embodiments. [Figure 27] 1 is an example of generating a fill plan using a joint template diagram, according to one or more embodiments. [Figure 28] 1 is an example of generating a fill plan using a joint template diagram, according to one or more embodiments. [Figure 29] 1 is an example of generating a fill plan using a joint template diagram, according to one or more embodiments. [Figure 30] 1 is an example of an improved fill plan using a joint template diagram, according to one or more embodiments. [Figure 31A] 1 includes an example of different portions of a multi-pass welding operation, according to one or more embodiments. [Figure 31B] 1 includes an example of different portions of a multi-pass welding operation, according to one or more embodiments. [Figure 31C] 1 includes an example of different portions of a multi-pass welding operation, according to one or more embodiments. [Figure 31D] 1 includes an example of different portions of a multi-pass welding operation, according to one or more embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0024] The detailed description set forth below with reference to the accompanying drawings is intended as an illustration of various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the present invention. Those skilled in the art will appreciate that these specific details are not required in all cases, and in some instances, well-known structures and components are shown in block diagram form for clarity of presentation.

[0025] FIG. 1 illustrates a system 100 configured to enable multi-pass welding in a robotic manufacturing environment, according to one or more embodiments. In some implementations, system 100 may include a processor-based system, an assembly robot system, or a combination thereof. System 100 of FIG. 1 is configured to enable multi-pass welding for one or more robots (e.g., manufacturing robots) functioning in a semi-autonomous or autonomous manufacturing environment. In some implementations, system 100 is configured to support or generate instructions for a welding robot to perform multi-pass welding operations.

[0026] A manufacturing environment or robot, such as a semi-autonomous or autonomous welding environment or semi-autonomous or autonomous welding robot, may include one or more sensors for scanning a number of part(s), one or more algorithms in the form of software configured to recognize a seam to be welded, and one or more algorithms in the form of software that program the robot's movements, operator controls, and any other devices, such as motorized fixtures, so that the identified seam can be welded accurately or without collisions as desired. Additionally or alternatively, a semi-autonomous or autonomous manufacturing environment or robot may also include one or more sensors for scanning a number of part(s), one or more algorithms in the form of software that recognize, locate, or align a given model of a part(s), where the seam is detected using one or more sensors or perhaps already indicated in some way in the given model itself, and one or more algorithms in the form of software that program the robot's movements, operator controls, and any other devices, such as motorized fixtures, so that the identified seam can be welded accurately or without collisions as desired. It should be noted that a semi-autonomous or autonomous welding robot may partially have these capabilities, some user-given or selected parameters may be required, or user (e.g., operator) involvement may be required in other ways.

[0027] System 100 includes a control system 110, a robot 120 (e.g., a manufacturing robot), and a manufacturing workspace 130 (also referred to herein as "workspace 130"). In some implementations, system 100 may include or correspond to an assembly robot system. System 100 may be configured to join one or more parts, such as a first part 135 (e.g., a first part) and a second part 136 (e.g., a second part). For example, first part 135 and second part 136 may be designed to form a seam 144 between first part 135 and second part 136. Each of first part 135 and second part 136 may be, but is not limited to, any part, component, sub-component, combination of parts or components, etc.

[0028] The terms "position" and "orientation" are described as separate entities in the above disclosure. However, the term "position," when used in the context of a part, means "a particular way in which a part is placed or arranged." The term "position," when used in the context of a seam, means "a particular way in which a seam on a part is positioned or oriented." Thus, the position of a part / seam may inherently take into account the orientation of the part / seam. Thus, "position" can include "orientation." For example, position can include the relative physical location or orientation (e.g., angle) of a part or a candidate seam.

[0029] Robot 120, also referred to herein as "robot 120," may be configured to perform manufacturing operations, such as welding operations, on one or more parts, such as first part 135 and second part 136. In some implementations, robot 120 may be a robot with multiple degrees of freedom, in that it may be a six-axis robot with an arm having an attachment point. Robot 120 may include one or more components, such as motors, servos, hydraulics, or combinations thereof, as illustrative, non-limiting examples.

[0030] In some implementations, the attachment point can attach a welding head (e.g., a manufacturing tool) to the robot 120. The robot 120 can include any suitable tool 121, such as a manufacturing tool. The robot 120 (e.g., the welding head of the robot 120) can be configured to move within the workspace 130 according to a path plan and / or welding plan received from the control system 110 or the controller 152. The robot 120 is further configured to perform one or more suitable manufacturing processes (e.g., welding operations) on one or more parts (e.g., 135, 136) according to received instructions, such as control information 182. In some examples, the robot 120 can be a six-axis robot with a welding arm. In some implementations, robot 120, in addition to attached tool 121, may be any suitable robotic welding equipment, such as a YASKAWA® robot arm, an ABB® IRB robot, a KUKA® robot, etc., and robot 120 may be configured to perform arc welding, resistance welding, spot welding, tungsten inert gas (TIG) welding, metal active gas (MAG) welding, metal inert gas (MIG) welding, laser welding, plasma welding, combinations thereof, and / or the like, as illustrative, non-limiting examples. Robot 120 may be responsible for moving, rotating, translating, feeding, and / or positioning a welding head, sensor(s), part(s), and / or combinations thereof. In some implementations, a welding head may be mounted, coupled, or otherwise attached to robot 120.

[0031] In some implementations, robot 120 may be coupled to or include one or more tools. For example, based on the function the robot performs, a robot arm may be coupled to a tool configured to enable (e.g., perform at least a portion of) the function. Illustratively, a tool such as tool 121 may be coupled to an end of robot 120. In some implementations, robot 120 may be coupled to or include multiple tools, such as manufacturing tools (e.g., welding tools), sensors, picker or holder tools, or combinations thereof. In some implementations, robot 120 may be configured to operate with another device, such as another robotic device, as further described herein.

[0032] Tool 121 may include one or more tools. For example, tool 121 may include a manufacturing tool (e.g., a welding tool), a sensor (e.g., 109), a picker tool or a holder tool, or a combination thereof. As shown, tool 121 is a picker tool or a holder tool configured to selectively couple to a first set of one or more objects, such as a first set of one or more objects including first part 135. In some implementations, the picker tool or the holder tool can include or correspond to a gripper, a clamp, a magnet, or a vacuum, as illustrative, non-limiting examples. For example, tool 121 may include a three-fingered gripper, such as one manufactured by OnRobot®.

[0033] In some implementations, the robot 120, the tool 121, or a combination thereof may be configured to change (e.g., adjust or manipulate) the pose of the first part 135 while the first part 135 is coupled to the tool 121. For example, the configuration of the robot 120 may be modified to change the pose of the first part 135. Additionally or alternatively, the tool 121 may be adjusted (e.g., rotated or tilted) relative to the robot 120 to change the pose of the first part 135.

[0034] Manufacturing tool 126 may be included in system 100 and may be configured to perform one or more manufacturing tasks or operations. The one or more manufacturing tasks or operations may include, by way of illustrative, non-limiting example, welding, brazing, soldering, riveting, cutting, drilling, etc. In some implementations, manufacturing tool 126 is a welding tool configured to join two or more objects together. For example, the welding tool may be configured to weld two or more objects together, such as welding first part 135 to second part 136. By way of example, the welding tool may be configured to place weld metal along a seam formed between first part 135 and second part 136. Additionally or alternatively, the welding tool may be configured to melt first part 135 and second part 136 together, such as melting a seam formed between first part 135 and second part 136 to join first part 135 and second part 136 together. In some implementations, manufacturing tool 126 may be configured to perform one or more manufacturing tasks or operations in response to a manufacturing command, such as a welding command. Although shown as separate from robot 120, in other implementations, manufacturing tool 126 may be coupled to robot 120 or another robot.

[0035] Workspace 130 may also be referred to as a manufacturing workspace. Workspace 130 may be or define an area or enclosure in which robotic arm(s), such as robot 120, operate on one or more parts based on or in conjunction with information from one or more sensors. In some implementations, workspace 130 may be any suitable welding area designed with appropriate safety measures for welding. For example, workspace 130 may be a welding area located in a workshop, shop floor, manufacturing plant, assembly plant, etc. In some implementations, at least a portion of system 100 is located in workspace 130. For example, workspace 130 may be an area or space in which one or more robotic devices (e.g., robotic arm(s)) are configured to operate on one or more objects (or parts). The one or more objects may be positioned on, coupled to, stored in, or supported by one or more platforms, containers, bins, racks, holders, or positioners. One or more objects (e.g., 135 or 136) may be held, positioned, and / or manipulated within workspace 130 using fixtures and / or clamps (collectively referred to as "fixtures" or fixtures 127). In some examples, workspace 130 may include one or more sensors, fixtures 127, and a robot 120 configured to perform welding-type processes, such as welding, brazing, and joining, on one or more parts to be welded (e.g., parts having seams).

[0036] The fixture 127 may be configured to hold, position, and / or manipulate one or more parts (135, 136). In some implementations, the fixture 127 may include or correspond to the tool 121 or the manufacturing tool 126. The fixture may include a clamp, a platform, a positioner, or other type of fixture, as shown by way of non-limiting example. In some examples, the fixture 127 is adjustable manually by a user or automatically by a motor. For example, the fixture 127 may dynamically adjust its position, orientation, or other physical configuration before or during the welding process.

[0037] The control system 110 is configured to operate and control the robot 120 to perform manufacturing functions within the workspace 130. For example, the control system 110 can operate and / or control the robot 120 (e.g., a welding robot) to perform a welding operation on one or more parts. While described herein with reference to a welding environment, a manufacturing environment can include any one or more of a variety of environments, such as assembly, painting, packaging, etc. In some implementations, the workspace 130 can include one or more parts (e.g., 135 or 136) to be welded. The one or more parts can be formed from one or more different parts. For example, the one or more parts can include a first part (e.g., 135) and a second part (e.g., 136), where the first and second parts form a seam (e.g., 144) at their interface. In some implementations, the first and second parts can be held together using a tack weld. In other implementations, the first and second parts may not be welded, and the robot 120 may simply tack weld the seam of the first and second parts to lightly join the parts together. Additionally or alternatively, following the formation of the tack weld, the robot 120 may weld additional portions of the seam to firmly join the parts together. In some implementations, the robot 120 may perform a multi-pass welding operation to place weld material in the seam 144 to form the joint.

[0038] In some implementations, the control system 110 may be implemented externally to the robot 120. For example, the control system 110 may include a server system, a personal computer system, a notebook computer system, a tablet system, or a smartphone system to provide control of the robot 120, such as a semi-autonomous or autonomous welding robot. Although the control system 110 is shown as separate from the robot 120, some or all of the control system 110 may be implemented internal to the robot 120. For example, a portion of the control system 110 internal to the robot 120 may be included as a robot control unit, an electronic control unit, or an on-board computer and configured to provide control of the robot 120, such as a semi-autonomous or autonomous welding robot.

[0039] Control systems 110 implemented internally or externally to robot 120 may be collectively referred to herein as "robot controllers 110." Robot controller 110 may be included in or coupled to a seam identification system, a trajectory planning system, a welding simulation system, another system associated with a semi-autonomous or autonomous welding robot, or a combination thereof. It should be noted that one or more of the seam identification system, trajectory planning system, welding simulation system, or another system associated with a semi-autonomous or autonomous welding robot may be implemented independently of or external to control system 110.

[0040] The control system 110 may include one or more components. For example, the control system 110 may include a controller 152, one or more input / output (I / O) and communication adapters 104 (hereinafter collectively referred to as “I / O and communication adapters 104”), one or more user interface and / or display adapters 106 (hereinafter collectively referred to as “user interface and display adapters 106”), storage device 108, and one or more sensors 109 (hereinafter referred to as “sensors 109”). The controller 152 may include a processor 101 and a memory 102. Although the processor 101 and the memory 102 are both described as being included in the controller 152, in other implementations, the processor 101, the memory 102, or both may be external to the controller 152, such that each of the processor 101 or the memory 102 may be one or more separate components.

[0041] Controller 152 may be any suitable machine specifically and specially configured (e.g., programmed) to perform one or more operations attributed herein to controller 152, or more generally to system 100. In some implementations, controller 152 is not a general-purpose computer, but rather is specially programmed or hardware-configured to perform one or more operations attributed herein to controller 152, or more generally to system 100. Additionally or alternatively, controller 308 is or includes an application-specific integrated circuit (ASIC), a central processing unit (CPU), a field-programmable gate array (FPGA), or a combination thereof. In some implementations, controller 152 includes a memory, such as memory 102, that stores executable code that, when executed by controller 152, causes controller 152 to perform one or more of the operations attributed herein to controller 152, or more generally to system 100. Controller 152 is not limited to the specific examples described herein.

[0042] In some implementations, the controller 152 is configured to control the sensor(s) 109 and the robot 120 within the workspace 130. Additionally or alternatively, the controller 152 is configured to control the fixture(s) 127 within the workspace 130. For example, the controller 152 may control the robot 120 to perform welding operations according to path planning and / or weld planning techniques and move within the workspace 130. The controller 152 may also manipulate the fixture(s) 127, such as positioners (e.g., platforms, clamps, etc.), to rotate, translate, or otherwise move one or more parts within the workspace 130. Additionally or alternatively, the controller 152 may control the sensor(s) 109 to move within the workspace 130 and / or capture images (e.g., 2D or 3D), audio data, and / or EM data.

[0043] In some implementations, the controller 152 may also be configured to control other aspects of the system 100. For example, the controller 152 may further interact with the user interface (UI) and display adapter 106. Illustratively, the controller 152 may provide a graphical interface on the UI and display adapter 106 through which a user can interact with and provide input to the system 100, and through which the controller 152 can interact with a user, such as by providing and / or receiving various types of information from the user (e.g., identified seams that are candidates for welding, possible paths during path planning, welding parameter options or selections, etc.). The UI and display adapter 106 may be any type of interface, including a touchscreen interface, a voice-activated interface, a keypad interface, combinations thereof, etc.

[0044] In some implementations, control system 110 may include a bus (not shown). The bus may be configured to electrically or communicatively couple one or more components of control system 110. For example, the bus may couple controller 152, processor 101, memory 102, I / O and communications adapter 104, and user interface and display adapter 106. Additionally or alternatively, the bus may couple one or more components or portions of controller 152, processor 101, memory 102, I / O and communications adapter 104, and user interface and display adapter 106.

[0045] Processor 101 may include a central processing unit (CPU), sometimes referred to herein as a processing unit. Processor 101 may include a general-purpose CPU, such as a processor from the CORE family of processors available from Intel Corporation, a processor from the ATHLON family of processors available from Advanced Micro Devices, Inc., or a processor from the POWERPC family of processors available from AIM Alliance, although the present disclosure is not limited by the architecture of processor 101 so long as processor 101 supports one or more operations as described herein. For example, processor 101 may include one or more special-purpose processors, such as an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), a field-programmable gate array (FPGA), or the like.

[0046] Memory 102 may include storage devices such as random access memory (RAM) (e.g., SRAM, DRAM, SDRAM, etc.), ROM (e.g., PROM, EPROM, EEPROM, etc.), one or more HDDs, flash memory devices, SSDs, other devices configured to store data in a persistent or non-persistent state, or a combination of different memory devices. Memory 102 is configured to store user and system data and programs, such as may include some or all of the aforementioned program code and associated data for performing the functions of the machine learning logic-based tuning technique.

[0047] The memory 102 includes or is configured to store instructions 103 and information 164. The memory 102 can also store other information or data, such as a design 170, joint model information 171, one or more waypoints 172, a bead model, a cross-sectional weld profile 174, a weld fill plan 175, and welding instructions 176. In one or more aspects, the memory 102 can store instructions 103, such as executable code, that, when executed by the processor 101, causes the processor 101 to perform operations according to one or more aspects of the present disclosure, as described herein. In some implementations, the instructions 103 (e.g., executable code) are a single, self-contained program. In other implementations, the instructions (e.g., executable code) are a program having one or more function calls to other executable code, which may be stored in a memory device or other location. One or more functions resulting from the execution of the executable code may be implemented by hardware. For example, multiple processors may be used to perform one or more separate tasks of the executable code.

[0048] The instructions 103 may include path planning logic 105, machine learning logic 107, multi-pass logic 111, and recommendation logic. Additionally or alternatively, the instructions 103 may include other logic, such as alignment logic as further described herein with reference to at least FIG. 7 . While shown as separate logic blocks, the path planning logic 105, machine learning logic 107, multi-pass logic 111, and / or recommendation logic 114 may be part of the memory 102 and may include program code (and associated data) for performing the functions of path planning machine learning and multi-pass operations, respectively. For example, the path planning logic 105 is configured to generate a path for the robot 120 along the seam, including, but not limited to, optimizing the movement of the robot 120 to complete the weld. Additionally or alternatively, although shown as separate logic blocks, the path planning logic 105, machine learning logic 107, multi-pass logic 111, and recommendation logic 114 may be combined. Additionally, other logic (eg, alignment logic) may be included in or combined with the path planning logic 105, machine learning logic 107, multi-path logic 111, and recommendation logic 114.

[0049] As an illustrative, non-limiting example, the path planning logic 105 may be configured for graph matching or graph search techniques to generate a path or trajectory that fits the identified seam. In the case of welding, the task of welding with a welding head coupled to a robotic arm may be specified in five degrees of freedom. The system's hardware capabilities (the robotic arm's five degrees of freedom) exceed five degrees of freedom. In some implementations, the path planning logic 105 can perform a search using more than five degrees of freedom, such as when considering collision avoidance. There are several ways to avoid this redundancy. First, constraining an overactuated system by specifying the task in higher dimensions or exploiting the redundancy to explore multiple options. Traditionally, path planning has typically been presented as a graph search problem. It may be considered an overactuated plan, and in some implementations, the redundant degrees of freedom(s) can be discretized, and each sample can be treated as a unique node in constructing the graph. The resulting graph structure may enable fast graph search algorithms. Each point on the seam can be considered a layer in the graph, similar to a rung on a ladder. The nature of the path planning problem is that the robot must always transition forward between these rungs. This is the first aspect of the problem that makes graph exploration easier. The path planning logic 105 generates multiple joint space solutions for each point on the seam. All solutions for a given point belong to the same layer. There is no point where the robot transitions between different solutions for the same rung, and therefore, in the graph, these nodes are not connected. This places further constraints on the structure of the graph.

[0050] 3, which is a schematic diagram 300 of a graph search technique, according to one or more aspects. In some implementations, the schematic diagram represents a graph search technique (e.g., controller 152) by which a path plan for robot 120 may be determined. For example, the graph search technique may be performed using path planning logic 105.

[0051] In some implementations, each circle in diagram 300 represents a different state of the robot 120, such as, by way of illustrative, non-limiting example, a configuration of the robot's joints to meet welding requirements. Each arrow represents a path the robot can take to move along the seam. Illustratively, each circle may represent a specific position of the robot 120 within the workspace 130 (e.g., the position of the robot's 120 welding head in 3D space), a different configuration of the robot's 120 arms, and the location or configuration of fixtures supporting parts, such as positioners, clamps, etc. Each row 302, 306, and 310 represents a different point, such as a waypoint, along the seam to be welded. Thus, for a seam point corresponding to row 302, the robot 120 can be in any one of states 304A-304D. Similarly, for a seam point corresponding to row 306, the robot 120 can be in any one of states 308A-308D. Similarly, for the seam point corresponding to column 310, robot 120 can be in any one of states 312A-312D. For example, if robot 120 is in state 304A when at the seam point corresponding to column 306, robot 120 can transition to any one of states 308A-308D for the next seam point corresponding to column 302. Similarly, upon entering states 308A-308D, robot 120 can subsequently transition to any one of states 312A-312D for the next seam point corresponding to column 310, and so on. In some instances, entering a particular state may preclude entering other states. For example, entering state 304A may allow the possibility of subsequently entering states 308A-308C, but not 308D, while entering state 304B may allow the possibility of subsequently entering states 308C and 308D, but not 308A-308B. The scope of the present disclosure is not limited to any particular number of seam points or any particular number of robot states.

[0052] In some examples, to determine a path plan for the robot 120 using a graph search technique (e.g., according to the technique shown in diagram 300), a controller 152, such as the path planning logic 105, can determine the shortest path from states 304A-304D to a state corresponding to seam point N (e.g., states 312A-312D). An objective function can be designed by a user or by the controller 152 by assigning a cost to each state and each transition between states. The controller 152 finds a path that results in the lowest possible cost value for the objective function. With degrees of freedom having multiple start and end points to choose from, the Dijkstra algorithm or A * A graph search method such as may be implemented. In some instances, a brute force method may be useful in determining an appropriate path plan. A brute force technique involves the control system 110 (e.g., controller 152 or processor 101) calculating all possible paths (e.g., through graph 300) and selecting the shortest path (e.g., by minimizing or maximizing an objective function). Simply put, the brute force method calculates all possible paths through this graph and selects the shortest path. The complexity of the brute force method may be O(E), where E is the number of edges in the graph. Assume N points in a seam with M options per point. Between any two layers, there are M * There are M edges. Therefore, considering all layers, there are N * M * There are M edges. The time complexity is O(NM^2) or O(E).

[0053] A controller 152, such as path planning logic 105, can determine whether the state at each seam point is feasible, which means that the controller 152 can determine, at least in part, whether implementing a state chain along a seam's sequence of seam points would cause any collisions between the robot 120 and structures in the workspace 130, or even parts of the robot 120 itself. To this end, the concept of realizing different states at different points in a seam may alternatively be expressed in the context of a seam having multiple waypoints, such as waypoint 172.

[0054] In some implementations, the controller 152 may discretize the identified seam into a series of waypoints. The waypoints can constrain the orientation of a welding head connected to the robot 120 in three (spatial / translational) degrees of freedom. Typically, constraints on the orientation of the welding head of the robot 120 are provided in one or two rotational degrees of freedom around each waypoint with the goal of producing a desired weld of a certain quality, and the constraints are typically related to surface normal vectors emanating from the waypoint and the path of the weld seam. For example, the position of the welding head may be constrained in the x-, y-, and z-axes, and about one or two rotational axes perpendicular to the axis of the welding wire or welder tip, all relative to the waypoint and some nominal coordinate system attached to the waypoint. These constraints may, in some examples, be bounds or tolerances on angles. Those skilled in the art will recognize that the ideal or desired welding angle may vary based on the geometry of the part or seam, the direction of gravity relative to the seam, and other factors. In some examples, the controller 152 may constrain the welding head to a first position or a second position for one or more reasons to ensure that the seam is perpendicular to gravity (e.g., to find a balance between welding and path planning for optimization purposes). Thus, the position of the welding head may be held (constrained) by each waypoint at any suitable orientation relative to the seam. Typically, the welding head is not constrained about an axis of rotation (θ) coaxial with the axis of the welding head. For example, each waypoint may define the position of the welding head of the welding robot 120 such that at each waypoint, the welding head is at a fixed position and orientation relative to the weld seam. In some implementations, the waypoints are discretized fine enough to make the movement of the welding head substantially continuous.

[0055] In some implementations, the controller 152 can divide each waypoint into multiple nodes. Each node can represent a possible orientation of the welding head at that waypoint. As an illustrative, non-limiting example, the welding head can be unconstrained about a rotational axis coaxial with the axis of the welding head so that the welding head can rotate along rotational axis θ (e.g., 360 degrees) at each waypoint. Each waypoint can be divided into 20 nodes such that each node at each waypoint represents the welding head in 18-degree rotational increments. For example, the first waypoint-node pair can represent 0 degrees of welding head rotation, the second waypoint-node pair can represent welding head rotation from 18 degrees, the third waypoint-node pair can represent welding head rotation from 36 degrees, etc. Each waypoint can be divided into 2, 10, 20, 60, 120, 360, or any suitable number of nodes. Subdivision of the nodes can represent division of orientation beyond one degree of freedom. For example, the orientation of the welder tip around a waypoint can be defined by three angles. A welding path can be defined by connecting each waypoint-node pair. Thus, the distance between waypoints and the offset between adjacent waypoint-nodes can represent the amount of translation and rotation of the welding head as it moves between waypoint pairs.

[0056] A controller 152, such as the path planning logic 105, can evaluate each waypoint-node pair for feasibility of welding. For example, if the waypoints are divided into 20 nodes, the controller 152 can evaluate whether the first waypoint-node pair, representing the welding head held at 0 degrees, is feasible. In other words, the controller 152 can evaluate whether the robot 120 will collide or interfere with the parts (135, 136), the fixture 127, or the welding robot itself when placed in the position and orientation defined by that waypoint-node pair. Similarly, the controller 152 can evaluate whether the second waypoint-node pair, the third waypoint-node pair, and so on are feasible. The controller 152 can similarly evaluate each waypoint. In this manner, all feasible nodes for all waypoints can be determined.

[0057] In some examples, the collision analysis described herein may be performed by comparing a 3D model of workspace 130 with a 3D model of robot 120 to determine whether the two models overlap, and optionally whether some or all of the triangles overlap. The 3D model of workspace 130, the 3D model of robot 120, or both may be stored in memory 102 or storage device 108, as illustrative, non-limiting examples. If the two models overlap, controller 152 may determine that a collision is likely. If the two models do not overlap, controller 152 may determine that a collision is unlikely. More specifically, in some examples, controller 152 may compare the models for each of a set of waypoint-node pairs (such as the waypoint-node pairs described above) and determine that the two models overlap for a subset, or possibly all, of the waypoint-node pairs. For a subset of waypoint-node pairs for which model intersections are identified, controller 152 may omit waypoint-node pairs within that subset from the planned path and may identify replacement pairs for those waypoint-node pairs. Controller 152 may repeat this process as necessary until a collision-free path is planned. Controller 152 may use a flexible collision library (FCL) containing various techniques for efficient collision detection and proximity calculations as a tool in the collision avoidance analysis. The FCL may be stored in memory 102 or storage device 108, as an illustrative, non-limiting example. The FCL is useful for performing multiple proximity queries against different model representations and can be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used in conjunction with or in place of the FCL.

[0058] The controller 152 can generate one or more feasible simulated (or estimated, both terms are used interchangeably herein) welding paths, provided they are physically feasible. A welding path can be a path a welding robot (e.g., 120) would take to weld a seam. In some examples, a welding path can include all waypoints of the seam. Alternatively, a welding path can include some, but not all, of the waypoints of the seam. A welding path can include the movement of the robot 120 and the welding head as the welding head moves between each waypoint-node pair. Once a feasible path between a node-waypoint pair is identified, a feasible node-waypoint pair for the next successive waypoint can be identified, if one exists. Those skilled in the art will recognize that many search trees or other strategies can be used to evaluate the space of feasible node-waypoint pairs. Additionally or alternatively, as described herein, a cost parameter can be assigned or calculated for movement from each node-waypoint pair to a subsequent node-waypoint pair. The cost parameters may be related to travel time, amount of travel (e.g., including rotation) between node-waypoint pairs, and / or simulated / expected weld quality produced by the welding head during travel.

[0059] If the nodes suitable for welding for one or more waypoints are not feasible and / or no feasible path exists for traveling between the previous waypoint-node pair and any of the waypoint-node pairs for a particular waypoint, a controller 152, such as the path planning logic 105, can determine alternative welding parameters to make at least some additional waypoint-node pairs suitable for welding. For example, if the controller 152 determines that none of the waypoint-node pairs for a first waypoint are feasible, thereby rendering the first waypoint unweldable, the controller 152 can determine alternative welding parameters, such as alternative weld angles, to make at least some waypoint-node pairs for the first waypoint weldable. For example, the controller 152 can remove or relax constraints on rotation about the x-axis and / or y-axis. Similarly, the controller 152 can allow the weld angle to vary in one or two additional rotational (angular) dimensions. For example, the controller 152 can divide the unweldable waypoint into nodes in two or three dimensions. Each node can then be evaluated for weld feasibility with the welding robot and weld held at various welding angles and rotations. Additional rotational or other degrees of freedom about the x-axis and / or y-axis may make the waypoint accessible to the welding head so that the welding head does not encounter any collisions. In some implementations, the controller 152 can use the degrees of freedom in determining a feasible path between a previous waypoint-node pair and one of the waypoint-node pairs of a particular waypoint if a node suitable for welding at one or more waypoints is not feasible and / or if no feasible path exists for traveling between the previous waypoint-node pair and one of the waypoint-node pairs of the particular waypoint.

[0060] Based on the generated welding path, the controller 152 can optimize the welding path for welding. As used herein, “optimum” and “optimizing” do not refer to determining the absolute best welding path, but generally refer to techniques that can reduce welding time and / or improve weld quality relative to less efficient welding paths. Illustratively, the controller 152 can determine a cost function that seeks local and / or global minima for the movements of the robot 120. Typically, an optimal welding path minimizes welding head rotation, as welding head rotation can increase the time to weld a seam and / or reduce weld quality. Thus, optimizing the welding path can include determining a welding path that passes through the greatest number of waypoints with the least amount of rotation.

[0061] In evaluating the feasibility of welding at each of the divided nodes or node-waypoint pairs, the controller 152 may perform multiple calculations. In some examples, each of the multiple calculations may be mutually exclusive. In some examples, a first calculation may include a kinematic feasibility calculation that calculates whether the arm of the robot 120 of the employed welding robot can mechanically reach (or be at) the state defined by the node or node-waypoint pair. In some examples, in addition to the first calculation, a second calculation, which may be mutually exclusive with the first calculation, may also be performed by the controller 152. The second calculation may include determining whether the arm of the robot 120 will encounter a collision (e.g., collide with the workspace 130 or a structure within the workspace 130) when accessing a portion of the seam (e.g., the node or node-waypoint pair in question).

[0062] A controller 152, such as the path planning logic 105, may perform the first calculation before performing the second calculation. In some examples, the second calculation may be performed only if the result of the first calculation is positive (e.g., if it is determined that the arm of the robot 120 can mechanically reach (or be in) the state defined by the node or node-waypoint pair). In some examples, the second calculation may not be performed if the result of the first calculation is negative (e.g., if it is determined that the arm of the robot 120 cannot mechanically reach (or be in) the state defined by the node or node-waypoint pair).

[0063] Kinematic feasibility may be correlated with the type of robot arm used. In some implementations, the welding robot 120 includes a six-axis robotic welding arm with a spherical wrist. A six-axis robotic arm can have six degrees of freedom: three in the X, Y, and Z Cartesian coordinates, plus three additional degrees of freedom for the wrist-like nature of the robot 120. For example, the wrist-like nature of the robot 120 provides a fourth degree of freedom in the up-down direction of the wrist (e.g., the wrist moves in the +y and -y directions), a fifth degree of freedom in the lateral direction of the wrist (e.g., the wrist moves in the -x and +x directions), and a sixth degree of freedom in rotation. In some examples, a welding torch is attached to the wrist portion of the robot 120.

[0064] To determine whether the arm of the robot 120 being used can mechanically reach (or exist in) a state defined by a node or node-waypoint pair, for example, the robot 120 can be mathematically modeled. An example representation 400 of a robot arm according to one or more embodiments is shown with reference to FIG. 4. In some examples, a controller 152, such as the path planning logic 105, may solve for the first three joint variables based on the wrist position and the other three joint variables based on the wrist orientation. Note that the torch (e.g., welding head) is rigidly attached to the wrist. Therefore, the transformation between the torch tip and the wrist is assumed to be fixed. Referring to FIG. 4, the robot arm representation 400 includes a base, a wrist center, and links B 410, L 404, R 408, S 402, T 412, and U 406, which may be considered joint variables. To find the first three joint variables (eg, variables S, L, U in 402, 404, 406, respectively), geometric methods (eg, the law of cosines) can be employed.

[0065] After the first three joint variables (i.e., S, L, and U) are successfully calculated, the controller 152 can then solve for the last three joint variables (i.e., R, B, and T in 408, 410, and 412, respectively), for example, by considering the wrist orientation as ZYZ Euler angles. The controller 152 may also consider some offsets in the robot 120. These offsets may need to be considered and accounted for due to inconsistencies in the unified robot description format (URDF) file. For example, in some instances, the position values (e.g., the X-axis of the joint) of a joint (e.g., the actual joint of the robot 120) may not match the values described in the URDF file. Such offset values may be provided to the controller 152 in a table, such as data stored in the memory 102 or storage device 108. The controller 152 may, in some instances, consider these offset values while mathematically modeling the robot 120. In some examples, after the robot 120 is mathematically modeled, the controller 152 can determine whether the arm of the robot 120 can mechanically reach (or be at) a state defined by a node or node-waypoint pair.

[0066] As described above, the controller 152 can evaluate whether the robot 120, when placed in the position and orientation defined by that waypoint-node pair, will collide or interfere with one or more parts (135, 136), fixture 127, or anything else in the workspace 130, including the robot 120 itself. Once the controller 152 determines the state the robot arm may be in, the controller 152 may use a second calculation to make the aforementioned evaluation (e.g., as to whether the robot will collide with something in its environment).

[0067] 1 , the machine learning logic 107 is configured to learn from and adapt to results based on one or more welding operations performed by the robot 120. During or based on operation of the system 100, the machine learning logic (e.g., the machine learning logic 107) is provided with sensor data 180 associated with at least a portion of a weld formed by the robot 120. For example, the sensor data 180 can indicate one or more spatial characteristics of the weld. In some implementations, the portion of the weld can include or correspond to one or more passes of a multi-pass welding operation.

[0068] In some implementations, the machine learning logic 107 is configured to update a model, such as the bead model 173 or the weld model, based on the sensor data 180. For example, the bead model 173 may be configured to predict a bead profile, and the weld model may be configured to generate one or more welding instructions (e.g., 176) to achieve the bead profile or weld fill plan (e.g., 175). The controller 152 may generate a first set of welding instructions based on the bead model 173, the weld model, or a combination thereof. After execution of the first set of welding instructions by the robot 120, the controller 152 may receive feedback information (e.g., the sensor data 180). The machine learning logic 107 may update the bead model 173 or the weld model based on the feedback. Examples of updating one or more aspects of the bead model 173 are described further herein with reference to at least FIG. 31D . Updating the bead model 173 or the weld model may involve minimizing an error function that describes the difference between a predicted shape and an observed shape after execution. For example, the machine learning logic 107 may formulate the error as an L2 norm.

[0069] The multi-pass logic 111 is configured to determine a weld fill plan that includes multiple weld passes for the seam. For example, the controller 152 may execute the multi-pass logic 111 to generate one or more weld profiles (e.g., 174), weld fill plans (e.g., 175), one or more weld instructions (e.g., 176), or a combination thereof, as described further herein.

[0070] The recommendation logic 114 is configured to determine a fill plan sequence for a scenario, such as a welded seam. For example, the recommendation logic 114 may be configured to receive a set of target weld profiles representing a desired cross-sectional area to be filled with a weld. The desired cross-sectional area may be determined based on or using sensor data 180, point cloud 169, design 170, joint model information 171, or a combination thereof. One or more target weld profiles in the set of target weld profiles may be received from, generated by, or generated using multi-pass logic 111, design 170, or sensor data (e.g., 180). Additionally or alternatively, the recommendation logic 114 may be configured to receive status information, such as, by way of illustrative, non-limiting example, a minimum bead size, a maximum bead size, or a combination thereof. In some implementations, the status information may be represented as scalar information. Based on the received set of target weld profiles, the status information, or a combination thereof, the recommendation logic 114 can output a set of candidate fill plan sequences. Each fill plan sequence, also referred to herein as a fill key, may represent the number of weld passes in each layer of the fill plan. The set of candidate fill plan sequences may enable or provide a list of fill plans for a particular joint (e.g., seam) that are evaluated to determine and / or select the fill plan to be used for welding.

[0071] Information 164 may include or be indicative of sensor data 165, pose information 166, or system information 168. Sensor data 165 may include or correspond to sensor data 180 received by controller 152. Pose information 166 may include or correspond to the pose of first part 135, second part 136, or a combination thereof. System information 168 may include information associated with one or more devices (e.g., robot 120, tool 121, manufacturing tool 126, or sensor 109). Illustratively, system information 168 may include, by way of illustrative, non-limiting example, ID information, a communication address, one or more parameters, or a combination thereof. Additionally or alternatively, the system information 168 may include or indicate the position 159 (e.g., of the seam 144), path planning, motion planning, work angle, tip position, or other information associated with the movement of the robot 120, voltage, current, feed rate, or other information associated with the welding operation, or combinations thereof.

[0072] The design 170 may include or represent a CAD model of one or more parts. In some implementations, the CAD model may be annotated with or represent one or more weld parameters, weld geometry or shape, dimensions, tolerances, or a combination thereof. The joint model information 171 may include or represent multiple feature components. The multiple feature components may represent a joint model or may be combined to represent a joint model. In some implementations, each feature component of the multiple feature components includes a feature point, a feature point vector, a tolerance, or a combination thereof. The one or more waypoints 172 may include, represent, or correspond to a location along the seam 144.

[0073] The bead model 173 is configured to model the interaction of a bead weld placed on a surface. For example, the bead model 173 can represent a bead profile or cross-sectional area that results from a bead weld placed on a surface. In some implementations, the bead model is a first-order model that models the formation of the bead weld based on changes in the energy source and the shape or profile of the bead weld (e.g., the exposed bead cap).

[0074] In some implementations, the bead model 173 can be configured to represent or associate energy sources or sinks associated with the bead that push or pull the exposed bead cap. The bead model 173 can associate a radius of influence for each energy source or sink relative to one or more points on the exposed bead cap. The bead model 173 can also include a weighting plane that can be applied to its normal based on equalizing the influence of each energy source and sink. Note that movement along that normal can emulate how the area of the bead weld may be redistributed along the surface.

[0075] The bead model 173 may also link the end of the extended bead cap to the surface, e.g., the tow contact angle. To model the tow contact angle, the bead model 173 may consider or factor surface tension, torch angle, aspect ratio, or a combination thereof. Surface tension may be related to the pressure on the bead (on the plate) due to gravity. The torch angle may represent the work angle and, therefore, the arc distribution. The closer the torch is to the surface, the higher the temperature of the weld pool, thereby reducing surface tension in the direction of the torch and increasing the wetting effect. The aspect ratio can represent the effect that voltage can have on the arc cone angle, making wetting more or less pronounced. The bead model 173 may use a first-order system model to control the convergence of the defective cap to the wetted toe point.

[0076] In some implementations, the bead model 173 models the energy source using the following equation:

[0077]

number

[0078]

number

[0079] In some implementations, the bead model 173 may take the form of a parameterized curvature model. Parameterization of the bead model 173 can help maintain a core shape that can be adjusted to appropriately model various properties under different conditions. Additionally, the beam may be modeled or modified based on one or more interactive models so that bead shape profiles can be created with improved accuracy and stability. In some implementations, data may be collected from various tests and experiments to be analyzed and annotated for essential geometric measurements. These measurements can be used in a regression model to relate bead width and bead height or aspect ratio, and area to a set of welding parameters.

[0080] The cross-sectional weld profile 174 (also referred to herein as “weld profile 174”) may include or represent a cross-section of the seam 144, such as a cross-section of the seam 144 including weld material. The weld profile 174 may correspond to one waypoint of the one or more waypoints 172. In some implementations, the weld profile may include or represent a joint model, one or more weld beads or weld bead locations, or a combination thereof. The weld fill plan 175 represents one or more fill parameters, one or more weld bead parameters (e.g., one or more weld bead profiles), or a combination thereof. The one or more fill parameters may include or represent, by way of illustrative, non-limiting examples, a number of beads, a bead sequence, a number of layers, a fill area, a cover profile shape, a weld size, or a combination thereof. The one or more weld bead parameters may include or represent, by way of illustrative, non-limiting examples, a bead size (e.g., height, width, or distribution), a bead spatial characteristic (e.g., bead origin or bead orientation), or a combination thereof. Additionally or alternatively, the weld fill plan 175 may include or indicate one or more welding parameters for forming one or more weld beads. The one or more welding parameters may include or indicate, by way of illustrative, non-limiting example, a wire feed speed, a travel speed, a travel angle, a work angle (e.g., torch angle), a welding mode (e.g., waveform), a welding technique (e.g., TIG or MIG), a voltage or current, a contact tip-to-work distance (CTWD) offset, a weaving or movement parameter (e.g., weaving type, weaving amplitude characteristic, weaving frequency characteristic, or phase lag), a wire characteristic (wire diameter or wire type—composition / material), a gas mixture, a heat input, or a combination thereof.

[0081] The weld fill plan 175 may be generated based on one or more weld profiles 174, one or more bead models 173, one or more context variables, or a combination thereof. The one or more context variables may be associated with or correspond to a joint model. In some implementations, the one or more context variables include or indicate gravity, surface tension, gap, tack, surface features, joint features, part material properties or dimensions, or a combination thereof. The welding instructions 176 may include or indicate one or more actions to be performed by the robot 120. The welding instructions 176 may be generated based on one or more weld profiles 174, weld fill plan 175, or a combination thereof.

[0082] In some implementations, controller 152 is configured to optimize weld fill plan 175, including bead / weld commands (e.g., 176), based on context-specific welding styles in the form of rules formed from application-specific requirements / needs. Additionally or alternatively, controller 152 may be configured to consider or determine weld fill plan 175 based on additional capabilities, including movement capabilities (weaving), additional welding strategies (such as weld tacks), or combinations thereof.

[0083] The communications adapter 104 is configured to couple the control system 110 to a network (e.g., a cellular communications network, a LAN, a WAN, the Internet, etc.). The communications adapter 104 of embodiments may comprise, for example, a WiFi network adapter, a Bluetooth interface, a cellular communications interface, a mesh network interface (e.g., ZigBee, Z-Wave, etc.), a network interface card (NIC), etc. The user interface and display adapter 106 of the illustrated embodiment may be utilized to facilitate user interaction with the control system 110. For example, the user interface and display adapter 106 may couple one or more user input devices (e.g., a keyboard, pointing device, touchpad, microphone, etc.) to the control system 110 to facilitate user input when desired (e.g., when collecting information regarding one or more welding parameters).

[0084] In some implementations, the I / O and communications adapter 104 may also be used in connection with a system that couples sensor(s) 109 (e.g., global sensors, local sensors, etc.) to the processor 101 and memory 102 to detect and otherwise determine seam location. The I / O and communications adapter 104 may additionally or alternatively provide for coupling of various other devices, such as printers (e.g., dot matrix printers, laser printers, inkjet printers, thermal printers, etc.), to facilitate desired functionality (e.g., allowing the system to print paper copies of information such as planned trajectories, results of learned operations, and / or other information and documents).

[0085] User interface and display adapter 106 may be configured to couple one or more user output devices (e.g., flat panel displays, touch screens, heads-up displays, holographic projectors, etc.) to control system 110 to facilitate user output (e.g., welding simulations) when desired. It should be understood that various of the aforementioned functional aspects of control system 110 may be included or omitted as needed or determined to be appropriate depending on the specific implementation of a particular instance of system 100.

[0086] The user interface and display adapter 106 is configured to be coupled to a storage device 108, a sensor 109, another device, or a combination thereof. The storage device 108 may include one or more of a hard drive, an optical drive, a solid-state drive, or one or more databases. The storage device 108 may be coupled to the controller 152, the processor 101, or the memory 102 and configured to exchange program code for executing the techniques described herein, at least with reference to the instructions 103. The storage device 108 may include a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a read-only memory (ROM), a flash memory, or the like. The storage device 108 may include or correspond to the memory 102.

[0087] In some implementations, the storage device 108 includes a database 112 and executable code 113. The controller 152 may interact with the database 112, for example, by storing data in the database 112 and / or retrieving data from the database 112. While the database 112 is described as being within the storage device 108, in other implementations, the database 112 may be stored on a cloud-based platform. The database 112 may store any information useful to the system 100 in performing welding operations. For example, the database 112 may store a CAD model (e.g., 1708) of one or more parts (e.g., 135, 136). Additionally or alternatively, the database 112 may store annotated versions of the CAD models of one or more parts (e.g., 135, 136). The database 112 may also store point clouds (also referred to herein as CAD model point clouds) of one or more parts generated using the CAD models. Similarly, welding instructions (e.g., 176) for one or more parts that are generated based on a 3D representation of the one or more parts and / or user input provided regarding the one or more parts (e.g., which seams in the parts to weld, welding parameters, etc.) may be stored in database 112.

[0088] In some implementations, executable code 113, when executed, may cause controller 152 to perform one or more operations attributed to controller 152 herein, or to system 100 more generally. Executable code 113 may comprise a single, self-contained program. Additionally or alternatively, executable code 113 may be a program having one or more function calls to other executable code, which may be stored in storage device 108 or other locations, such as group storage device or memory 102, as illustrative, non-limiting examples. In some examples, one or more functions resulting from execution of executable code 113 may be implemented by hardware. For example, multiple processors may be useful for performing one or more separate tasks of executable code 113.

[0089] The sensor 109 may include an image sensor, such as a camera, a scanner, a laser scanner, a camera with a built-in laser sensor, or a combination thereof. In an example, the sensor 109 is an image sensor configured to capture visual information (e.g., images) regarding the workspace 130. For example, the sensor 109 may be configured to capture images of one or more parts (135, 136) or fixtures 127. In some implementations, the sensor 109 may include a light detection and ranging (LiDAR) sensor, an audio sensor, an electromagnetic sensor, or a combination thereof. An audio sensor, such as an acoustic navigation and ranging (SONAR) device, may be configured to emit and / or capture sound. An electromagnetic sensor, such as a radio detection and ranging (RADAR) device, may be configured to emit and / or capture electromagnetic (EM) waves. Through visual, audio, electromagnetic, and / or other sensing technologies, the sensor 109 may collect information about physical structures within the workspace 130. Additionally or alternatively, the sensors 109 may be configured to collect static information (eg, stationary structures within the workspace 130), dynamic information (eg, moving structures within the workspace 130), or a combination thereof.

[0090] The sensors 109 may be configured to capture data (e.g., image data) of the workspace 130 from various positions and angles. In some implementations, the sensors 109 may be mounted on the robot 120 or otherwise integrated into the workspace 130. For example, one or more sensors (e.g., 109) may be positioned on the robot 120 (e.g., on a welding head of the robot 120) and configured to collect image data as the robot 120 moves around the workspace 130. Because the robot 120 is movable with multiple degrees of freedom and therefore in multiple dimensions, one or more sensors disposed on the robot 120 can capture images from various perspectives. Additionally or alternatively, one or more sensors may be disposed on an arm of the robot 120 (e.g., on a welding head attached to the arm). In another example, the sensors 109 may be disposed on a movable non-welding robot arm (which may be different from the robot 120). In yet another example, at least one sensor 109 may be disposed on the arm of the robot 120, and another sensor 109 may be disposed on a movable piece of equipment within the workspace 130. In yet another example, at least one sensor 109 may be disposed on the arm of the robot 120, and another sensor 109 may be disposed on a movable non-welding robot arm. In some implementations, the sensor 109 may be attached to another robot (not shown in FIG. 1 ) disposed within the workspace 130. For example, the robot may be operable to move (e.g., rotationally or translationally) so that the sensor 109 can capture image data of the workspace 130, one or more parts (e.g., 135 or 136), and / or fixture 127 from various angles. In some implementations, the sensor 109 may be stationary, but the physical structure being imaged is moved around or within the workspace 130.For example, the part to be imaged (e.g., 135 or 136) may be positioned on a fixture 127, such as a positioner, and the positioner and / or part may rotate, translate (e.g., in the x, y, and / or z directions), or otherwise move within workspace 130, and a stationary sensor 109 (e.g., either coupled to robot 120 or separate from robot 120) captures multiple images of various facets of the part.

[0091] In some implementations, the sensor 109 can collect or generate information, such as images or image data, about one or more physical structures within the workspace 130. In some cases, the sensor 109 may be configured to image or monitor a weld placed by the robot 120 before, during, or after welding. Stated differently, the information may include or correspond to the geometry of a seam, the weld placed by the robot 120, or a combination thereof. The geometry may include, as illustrative, non-limiting examples, 3D point cloud information, a mesh, an image of a slice of the weld, a point cloud of a slice of the weld, or a combination thereof. The sensor 109 may provide the information to another component or device, such as the control system 110, the controller 152, or the processor 101. The other component or device may generate a 3D representation of the physical structures within the workspace 130 based on the information (e.g., the image data).

[0092] The sensor 109 may be communicatively coupled to another device, such as the processor 101, the controller 152, or the control system 110, which may be operable to process data from the sensor 109 and assemble two-dimensional data, data from the sensor 109 at various positions relative to one or more parts, or data from the sensor 109 as the sensor moves relative to the part for further processing. The control system 110, such as the controller 152 or the processor 101, may generate a point cloud by overlaying and / or stitching images to reconstruct and generate three-dimensional image data. The three-dimensional image data may be collated to generate a point cloud having associated image data for at least some points in the point cloud. The control system 110 may be configured to operate and control the robot 120. In some cases, control parameters of the robot 120 may be determined or informed by data from the point cloud.

[0093] In some implementations, the controller 152 is configured to receive information from the sensor 109, such as image or image data, audio data, EM data, or a combination thereof. The controller 152 can generate a 3D representation, such as a point cloud, of one or more structures associated with the received information. For example, the one or more structures may be depicted in an image. A point cloud may be a set of points, each representing a location in 3D space of a point on the surface of the part (e.g., 135 or 136) and / or fixture 127. Examples of points are further described herein with reference to at least FIGS. 5 and 6. In some examples, one or more images (e.g., image data captured by the sensor 109 at a particular orientation relative to the part) may be overlaid and / or stitched together by the controller 152 to reconstruct and generate 3D image data of the workspace 130. The 3D image data may be collated to generate a point cloud having associated image data for at least some of the points in the point cloud.

[0094] In some implementations, the 3D image data can be collated by the controller 152 so that a point cloud generated from the data can have six degrees of freedom. For example, each point in the point cloud can represent an infinitesimally small location in 3D space. As described above, the sensor 109 can capture multiple images of a point from various angles. These multiple images can be collated by the controller 152 to determine an average image pixel for each point. The averaged image pixel can be associated with the point. For example, if the sensor 109 is a color camera with red, green, and blue channels, the six degrees of freedom can be {x position, y position, z position, red intensity, green intensity, and blue intensity}. Alternatively, if the sensor 109 is a monochrome camera with a monochrome channel, four degrees of freedom can be generated.

[0095] In some implementations, to generate a 3D representation of workspace 130, sensor 109 may capture images of physical structures within workspace 130 from various angles. For example, a single 2D image of fixture 127 or a part (e.g., 135 or 136) may be insufficient to generate a 3D representation of that component; similarly, a set of multiple images of fixture 127 or a part from a single angle, view, or plane may be insufficient to generate a 3D representation of that component; however, multiple images captured from multiple angles at various positions within workspace 130 may be sufficient to generate a 3D representation of a component such as fixture 127 or a part. This is because capturing images at multiple orientations provides spatial information about the component in three dimensions, similar in concept to how a plan view of a component, including a front view, a side view, and a top view of the component, provides all the information necessary to generate a 3D representation of the component. Thus, in an embodiment, sensor 109 is configured to move around workspace 130 to capture sufficient information to generate a 3D representation of structures within workspace 130 .

[0096] In some implementations, multiple sensors (e.g., 109) are stationary but present in a sufficient number and at a sufficient variety of positions around the workspace 130 so that sufficient information is captured by the sensors to generate the aforementioned 3D representation. In examples in which the sensors 109 are movable, any suitable structure may be useful to facilitate such movement around the workspace 130. For example, the sensors 109 may be disposed on a motorized track system. The track system itself may be stationary, while the sensors 109 are configured to move around the workspace 130 on the track system. In some other implementations, the sensors 109 are movable on the track system, and the track system itself is movable around the workspace 130. In other implementations, one or more mirrors are disposed within the workspace 130 along with the sensors 109, and the sensors 109 may pivot, swivel, rotate, or translate about and / or along a point or axis such that the sensors 109 are configured to capture images from an initial viewpoint when in a first configuration and to capture images from another viewpoint using the mirrors when in a second configuration. In still other implementations, the sensors 109 may be mounted on arms that may be configured to pivot, swivel, rotate, or translate about and / or along a point or axis, and the sensors 109 may be configured to capture images from various viewpoints as these arms extend through their full range of motion.

[0097] Referring to FIG. 5 , FIG. 5 is an example point cloud 500 of parts having a weldable seam in accordance with one or more embodiments. Point cloud 500 represents a first part 502 and a second part 504. First part 502 and second part 504 may include or correspond to first part 135 and second part 136, respectively. First part 502 and second part 504 may be positioned to define seam 506. Seam 506 may include or correspond to seam 144. First part 502 and second part 504 may be configured to be welded to each other along seam 506. In some implementations, first part 502 and second part 504 may be welded to each other based on a multi-pass welding operation performed by robot 120.

[0098] Referring to FIG. 6 , FIG. 6 is an example point cloud 600 of parts having a weldable seam in accordance with one or more embodiments. Point cloud 600 represents a first part 602 and a second part 604. First part 602 and second part 604 may include or correspond to first part 135 and second part 136, respectively. First part 602 and second part 604 may be positioned to define seam 606. Seam 606 may include or correspond to seam 144. First part 602 and second part 604 may be configured to be welded to each other along seam 606. In some implementations, first part 602 and second part 604 may be welded to each other based on a multi-pass welding operation performed by robot 120.

[0099] 1 , configured controller 152 may be configured to generate 3D point cloud 500 or 600 based on images captured by sensor 109. Controller 152 can then use point cloud 500 or 600, the image data, or a combination thereof to identify and locate a seam, such as seam 506 or 606, plan a welding path along seam 506 or 606, and place welding material along seam 506 or 606 using robot 120 according to the path plan. In some implementations, controller 152 can execute instructions 103 (e.g., path planning logic 105, machine learning logic 107, or multi-pass logic 111), executable code 113, or a combination thereof to perform one or more operations, such as seam identification, path planning, model training or updating, or a combination thereof.

[0100] In some implementations, controller 152 may be configured to perform pixel-by-pixel and / or point-by-point classification using a neural network to identify and classify structures within workspace 130. For example, pixel-by-pixel classification may use or be based on images captured by sensor 109, and point-by-point classification may use one or more point clouds. Illustratively, controller 152, upon execution of instructions 103 or executable code 113, may perform pixel-by-pixel and / or point-by-point classification using a neural network to identify and classify structures within workspace 130. For example, controller 152 may perform pixel-by-pixel and / or point-by-point classification to identify one or more imaged structures within workspace 130 as a part (e.g., 135 or 136), as a seam on a part or at an interface between multiple parts (collectively referred to herein as a candidate seam), as fixture 127, as robot 120, etc.

[0101] In some implementations, the controller 152 may identify and classify pixels and / or points based on a neural network (e.g., a U-net model) trained using appropriate training data. For example, the neural network may be trained on image data, point cloud data, spatial information data, or a combination thereof. In some implementations, the point cloud and / or image data may include information captured from various viewpoints within the workspace 130, and the neural network may be operable to classify fixtures (e.g., 127) or candidate seams on a part from multiple angles and / or viewpoints. In some examples, the neural network may be trained to operate directly on a set of points (e.g., the neural network includes a dynamic graph convolutional neural network), or the neural network may be implemented to analyze unorganized points on a point cloud. In some examples, a first neural network may be trained on point cloud data to perform point-by-point classification, and a second neural network may be trained on image data to perform pixel-by-pixel classification. The first neural network and the second neural network can individually identify one or more candidate seams and locate the one or more candidate seams. The outputs from the first neural network and the second neural network can be combined as a final output to determine the location and orientation of the one or more candidate seams on the part.

[0102] In some examples, when pixel-by-pixel classification is performed, one or more results can be projected onto the 3D point cloud data and / or a meshed version of the point cloud data, thereby providing information about the location of fixture 127 within workspace 130. If the input data is image data (e.g., a color image), spatial information such as depth information can be included along with the color data to perform pixel-by-pixel segmentation. In some examples, pixel-by-pixel classification can be performed to identify candidate seams and locate the candidate seams relative to the part (e.g., 135 or 136).

[0103] In some implementations, the controller 152 can identify and classify pixels and / or points as particular structures within the workspace 130. For example, the controller may identify and classify pixels or points as fixture 127, parts (e.g., 135 or 136), candidate seams in parts, etc. Portions of the image and / or point cloud data classified as non-part and non-candidate seam structures, such as fixture 127, may be segmented (e.g., redacted or otherwise removed) from the data, thereby isolating data identified and classified as corresponding to parts and / or candidate seams associated with parts. In some examples, after identifying candidate seams and segmenting the non-part and non-candidate seam data (or, optionally, prior to such segmentation), a neural network may be configured to analyze each candidate seam to determine the seam type. For example, the neural network may be configured to determine whether the candidate seam is a butt joint, corner joint, edge joint, lap joint, T-joint, etc. A model (e.g., a U-net model) can classify seam types based on data captured from multiple viewpoints within workspace 130.

[0104] If pixel-by-pixel classification is performed using image data, controller 152 may project pixels of interest (e.g., pixels representing one or more parts and one or more candidate seams associated with the one or more parts) onto 3D space to generate a set of 3D points representing the parts and candidate seams. Additionally or alternatively, if point-by-point classification is performed using point cloud data, the points of interest may already exist in 3D space within the point cloud. In either case, from the perspective of controller 152, the 3D points may be an unordered set of points, and at least some of the 3D points may be clumped or clustered together. To eliminate such noise and generate a subset of contiguous, unbroken points to represent the candidate seams, a Manifold Blurring and Mean Shift (MBMS) technique or a similar technique may be applied. Such a technique may cluster the points and eliminate noise. Subsequently, controller 152 may apply a clustering method to decompose one or more candidate seams into individual candidate seams. In other words, instead of having several subsets of points representing multiple candidate seams, clustering can decompose each subset of points into individual candidate seams. Following clustering, controller 152 can fit a spline to each individual subset of points. Thus, each individual subset of points can be an individual candidate seam.

[0105] In some implementations, controller 152 receives image data captured by sensor 109 from various positions and viewpoints within workspace 130. Controller 152 can generate a set of candidate seams associated with one or more parts (e.g., 135 or 136), where the set of candidate seams indicates the locations and orientations of those candidate seams. For example, controller 152 performs pixel-by-pixel and / or point-by-point classification techniques using a neural network to classify and identify each pixel and / or point as a part (e.g., 135 or 136), a candidate seam on or associated with a part, or at an interface between multiple parts, fixture 127, etc. Structures identified as non-part and non-candidate seam structures are segmented, and controller 152 can perform additional processing on the remaining points (e.g., to reduce noise). After the set of candidate seams is generated, controller 152 can determine whether the candidate seams are actually seams and, optionally, can perform additional processing using prior information, such as CAD models of the parts and seams. The resulting data is suitable for use by the controller 152 to plan a path for placing a weld along the identified seam.

[0106] In some implementations, the identified candidate seam may not be a seam (e.g., the identified candidate seam may be a false positive). To determine whether the identified candidate seam is an actual seam, the controller 152 can determine a confidence value based on information from the sensor 109. For example, the controller 152 can use images captured by the sensor 109 from various viewpoints within the workspace 130 to determine the confidence value. The confidence value represents the likelihood that the candidate seam determined from the corresponding viewpoint is an actual seam. The controller 152 can then compare the confidence values for the different viewpoints and eliminate candidate seams that are unlikely to be actual seams. For example, the controller 152 can determine the mean, median, maximum, or any other suitable summary statistic of the candidate values associated with a particular candidate seam. Generally, a candidate seam that corresponds to an actual seam has a consistently high confidence value (e.g., above a threshold) across the various viewpoints used to capture the candidate seam. If the summary statistic of the confidence values for the candidate seam is above a threshold, the controller 152 can designate the candidate seam as an actual seam. Conversely, if the summary statistics of the confidence values for the candidate seam are below the threshold, the candidate seam may be designated as a false positive that is not eligible for welding.

[0107] After identifying a candidate seam that is an actual seam, the controller 152 can perform additional processing referred to herein as alignment. Illustrative examples of alignment are further described herein with reference to at least FIG. 7. In some implementations, the controller can perform alignment using prior information, such as a CAD model (or a point cloud version of the CAD model). For example, there may be differences between seam dimensions associated with a part (e.g., 135 or 136) and the seam dimensions in the CAD model. In some implementations, the CAD model (or a copy of the CAD model) may be transformed (e.g., updated) to account for any such differences. Note that the updated CAD model may be used to perform path planning. Thus, the controller 152 can compare a first seam (e.g., a candidate seam on a part that has been verified as an actual seam) with a second seam (e.g., a seam annotated on the CAD model that corresponds to the first seam) to determine differences between the first seam and the second seam. In some implementations, the annotated seam on the CAD model may have been annotated by an operator / user. If the CAD model and / or controller 152 accurately predict the location of the candidate seam, the first seam and the second seam may be approximately in the same location. Alternatively, if the CAD model and / or controller 152 are only partially accurate, the first seam and the second seam may partially overlap. The controller 152 may perform a comparison between the first seam and the second seam. This comparison between the first seam and the second seam may be based in part on the shape and relative position in space of both seams. If the shapes of the first seam and the second seam are relatively similar and close to each other, the second seam may be identified as the same as the first seam. In this way, the controller 152 may take into account the topography of surfaces on the part that are not accurately represented in the CAD model. In this way, the controller 152 may identify candidate seams and use the CAD model of the part to subselect, refine, or update the candidate seam for the part.Each candidate seam can be an updated set of points that represent the position and orientation of the candidate seam relative to the part.

[0108] 7, which is a block diagram illustrating an alignment process flow 700 in accordance with one or more embodiments. Some or all of the steps of the alignment process flow 700 may be performed by the controller 152. For example, the controller 152 may include instructions 103 or executable code 113 for performing at least a portion or all of the alignment process flow 700.

[0109] Controller 152 can perform coarse registration 702 using CAD model point cloud 704 and scan point cloud 706 formed using images captured by sensor 109. CAD model point cloud 704 and scan point cloud can include or correspond to point cloud 169. CAD model can include or correspond to design 170. CAD model point cloud 704 and scan point cloud 706 can be sampled so that their points have a uniform or near uniform distribution, so that they both have equal or near equal point densities, or a combination thereof.

[0110] In some implementations, the controller 152 downsamples the point clouds 704, 706 by randomly and uniformly selecting and retaining points within the clouds and discarding the remaining unselected points. For example, the controller 152 may use a Poisson Disk Sampling (PDS) downsampling algorithm to downsample the point clouds 704 or 706. The controller 152 can provide as input to the PDS algorithm the boundaries of the point clouds 704 or 706, the minimum distance between samples, and the limit of samples to be selected before they are rejected, or a combination thereof.

[0111] In some implementations, a delta network can be used to deform one model into another during coarse registration 702. The delta network can be a Siamese network that takes a source model and a target model and encodes them into latent vectors. The controller 152 can use the latent vectors to predict point-by-point transformations that transform or update one model into another. Note that a delta network may not require a training dataset. Given a CAD model and a scan point cloud 704, 706, the controller 152 in the context of a delta network spends one or more epochs learning the dissimilarity or similarity between the two. During these epochs, the delta network can learn one or more features that are subsequently useful for registration. In some implementations, the delta network may use skip connections to learn the transformations; in other implementations, skip connections may not be used. In some cases, the CAD model includes surfaces that are not present in the 3D point cloud generated using the captured image (e.g., scan). In such a case, the delta network moves all points corresponding to the missing surface from the CAD model point cloud 704 to some points in the scan point cloud 706 (and updates the scan point cloud 706). Thus, during registration, the controller 152 (e.g., the delta network) may use the learned features to transform (or update) the original CAD model, or may use the learned features to transform the deformed CAD model.

[0112] In some implementations, the delta network may include an encoder network such as a dynamic graph convolutional neural network (DGCNN). After the point cloud is encoded into features, a concatenated vector composed of both the CAD and scan embeddings may be formed. After performing a pooling operation (e.g., max pooling), a decoder may be applied to the resulting vector. In some examples, the decoder may include five convolutional layers with specific filters (e.g., 256, 256, 512, 1024, Nx3 filters). The resulting output may be concatenated with the CAD model and scan embeddings, max pooled, and then provided to the decoder again. The final result may include a point-wise transformation.

[0113] Extraneous data and noise in the data (e.g., the output of coarse alignment 702) can affect the alignment of the parts. For at least this reason, it is desirable to remove as much extraneous data and noise as possible. Illustratively, a bounding box 708 may be used to remove this extraneous data and noise (e.g., fixture 127) to limit the area over which the alignment is performed. Stated differently, data within the bounding box is retained, but all data, 3D or otherwise, from outside the bounding box is discarded. Such a bounding box may be any shape that can surround or encapsulate (e.g., partially or completely) the CAD model itself. For example, the bounding box may be a dilated or enlarged version of the CAD model. Data outside the bounding box may be removed from the final alignment, or may still be included but weighted to reduce its impact.

[0114] During refined registration 710, the controller 152 passes the output of the bounding box 708 as a patch through a set of convolutional layers in a neural network trained as an autoencoder. Specifically, the data may be passed through the encoder section of the autoencoder, and the decoder section of the autoencoder may not be used. The input data may be the XYZ positions of the points of the patch within the shape, e.g., (128, 3). The output may be a vector of length 1024, for example, which is useful for point-wise features.

[0115] A set of corresponding points that best supports a rigid transformation between the CAD point cloud model and the scanned point cloud model should be determined during registration. The correspondence candidates may be stored (e.g., in database 112) as a matrix where each element stores a confidence or probability of a match between two points.

[0116]

number

[0117] The controller 152 can use various techniques to find corresponding points based on this matrix. For example, techniques may include hard correspondence, soft correspondence, product manifold filter, graph clique, covariance, etc. After completion of the improved registration 710, the registration process 700 is complete at 712.

[0118] In some implementations, the actual location of the seam may differ from the seam location as determined by controller 152 using sensor imaging (e.g., using a scan point cloud) and / or as determined by a CAD model (e.g., using a CAD model point cloud). In such cases, a scanning procedure (sometimes referred to herein as a pre-scan) may be performed to correct the determined seam location to more closely or accurately match the actual seam location, such as the location on a part (e.g., 135 or 136). In the scanning procedure, sensor 109 (referred to herein as an on-board sensor) disposed on robot 120 is configured to perform a scan of the seam, such as seam 144. In some cases, this scanning may be performed using an initial movement and / or path plan generated by controller 152 based on the CAD model, the scan, or a combination thereof. For example, sensor 109 may scan any or all areas of workspace 130. During the initial movement and / or path planning, sensor 109 may capture observed images and / or data. The observed images and / or data may be processed by controller 152 to generate seam point cloud data. The controller 152 may use the seam point cloud data when processing the point cloud(s) 704 and / or 706 to correct the seam positions. The controller 152 may also use the seam point cloud data in correcting the path and motion planning.

[0119] In some examples, the alignment techniques described with reference to the alignment process flow 700 may be useful for comparing and matching seams identified by the on-board sensor 109 with seams determined using sensors other than the on-board sensor 109. By matching the seams in this manner, the robot 120 (more specifically, the head of the robot 120) is positioned as desired relative to the actual seam.

[0120] In some examples, the pre-scan trajectory of the robot 120 may be identical to the trajectory planned for welding along the seam. In some such examples, the movements taken for the robot 120 during the pre-scan may be generated separately to better visualize the seam or critical geometry using on-board sensors (e.g., 109) or to scan the geometry around the seam in question to limit the probability of collision or reduce the instances of collision. In some such implementations, the pre-scan trajectory is determined based on a CAD model, a multi-pass weld plan, or a combination thereof.

[0121] In some examples, pre-scanning techniques may involve scanning more than a specific seam or seams, but rather may include scanning other geometries of one or more parts (e.g., 135 or 136). The scan data may be useful for more precise application of any or all of the techniques described herein (e.g., alignment techniques) to find, locate, and detect seams and ensure that the head of robot 120 is positioned and moved along the seams as desired.

[0122] In some examples, scanning techniques (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / weld head) may be useful for identifying gap variability information about the seam, rather than position and orientation information about the seam. For example, scanned images captured by the sensor 109 on the robot 120 during a scanning procedure can be used to identify variability in one or more gaps and adjust the weld trajectory or path plan to account for such gaps. For example, 3D points, 2D image pixels, or a combination thereof may be useful for locating variable gaps between one or more parts to be welded. Illustratively, gaps between parts to be welded together can be located, identified, and measured to determine the size of the gap. In tack weld detection or general weld detection, previous deposits or material deposits in the gaps between parts to be welded can be identified using 3D points and / or 2D image pixels. Any or all such techniques may be useful for optimizing welding, including path planning. In some cases, gap variability can be identified within a 3D point cloud generated using images captured by the sensor 109. In still other cases, gap variability may be identified based on or using scanning techniques (e.g., scanning the actual seam using a sensor / camera mounted on the welding arm / welding head) performed while performing welding operations for the task. In either case, controller 152 may be configured to dynamically adapt one or more welding instructions (e.g., welding voltage) based on the determined gap location and size. For example, dynamically adjusting welding instructions for a welding robot can result in accurate welding of seams with variable gaps. Adjusting the welding instructions may include adjusting welder voltage, welder current, electric pulse duration, electric pulse shape, material feed rate, or a combination thereof.

[0123] In some implementations, the user interface and display adapter 106 may provide the user with an option to view the candidate seams. For example, the user interface and display adapter 106 may provide a graphical representation of the part and / or the candidate seams on the part. Additionally or alternatively, the user interface and display adapter 106 may group or present the candidate seams based on the type of seam. By way of example, the controller 152 may identify the types of seams that may be presented to the user via the user interface and display adapter 106. For example, candidate seams identified as lap seams may be grouped under the label "lap seam" and presented to the user via the user interface and display adapter 106 under the label "lap seam." Similarly, candidate seams identified as edge seams may be grouped under the label "edge seam" and presented to the user via the user interface and display adapter 106 under the label "edge seam."

[0124] The user interface and display adapter 106 may further provide the user with the option of selecting a candidate seam to be welded by the robot 120. For example, each candidate seam on the part may be presented as a selectable option (e.g., a push button) on the user interface and display adapter 106. When the user selects a particular candidate seam, the selection may be transmitted to the controller 152. The controller 152 may generate instructions for the robot 120 to perform a welding operation on the particular candidate seam.

[0125] In some examples, the user may be provided with an option to update the welding parameters. For example, the user interface and display adapter 106 may provide the user with a list of different welding parameters. The user may select a particular parameter to be updated. Changes to the selected parameter may be made using a drop-down menu, via text entry, or the like. This update may be sent to the controller 152 so that the controller 152 can update the instructions for the robot 120.

[0126] In examples where system 100 is not provided with prior information (e.g., a CAD model) of a part (e.g., 135 or 136), sensor 109 can scan the part. A representation of the part can be presented to a user via user interface and display adapter 106. This representation of the part can be a point cloud and / or a mesh of the point cloud including projected 3D data of the scanned image of the part obtained from sensor 109. The user can annotate one or more seams to be welded in the representation via user interface and display adapter 106. Alternatively, controller 152 can identify candidate seams in the representation of the part and present them to the user via user interface and display adapter 106. The user can select a seam to be welded from the candidate seams. User interface and display adapter 106 can annotate the representation based on the user's selection. In some implementations, the annotated representation can be stored in database 112.

[0127] After one or more seams on a part are identified, the controller 152 can plan a path for the robot 120 for the subsequent welding process. In some examples, graph matching and / or graph searching techniques can be useful in planning the path for the robot 120. A particular seam identified as described above can include multiple points, and path planning techniques involve determining different states of the robot 120 for each such point along a given seam. The states of the robot 120 can include, for example, the position of the robot 120 within the workspace 130 and specific configurations of the arms of the robot 120 in any number of degrees of freedom that may be applicable. For example, in the case of a robot 120 having an arm with six degrees of freedom, the state of the robot 120 not only includes the position of the robot 120 in the workspace 130 (e.g., the position of the welding head of the robot 120 in three-dimensional x-y-z space), but also specific sub-states for each of the six degrees of freedom of the robot arm. Furthermore, as the robot 120 transitions from a first state to a second state, the robot may change its position within the workspace 130, and in such cases, the robot 120 will necessarily move a particular path within the workspace 130 (e.g., along the seam being welded). Thus, specifying a series of states for the robot 120 will necessarily involve specifying a path for the robot 120 to move within the workspace 130. The controller 152 may perform a pre-scanning technique or a variant thereof after path planning is complete, and the controller 152 may use information captured during the pre-scanning technique to make any of a variety of appropriate adjustments (e.g., adjustments to the X, Y, and Z axes or coordinate systems used to perform the actual welding along the seam).

[0128] In some implementations, the controller 152 may be configured to determine one or more dimensions of the seam (e.g., 144). For example, the controller 152 may determine the one or more dimensions of the seam based on sensor data 180 from the sensor 109. The one or more dimensions may include a seam depth, a seam width, a seam length, or a combination thereof. Additionally or alternatively, the controller 152 may determine how the one or more dimensions vary along the length of the seam. In some implementations, the controller 152 can determine gap variability information, such as one or more dimensions of the seam, how the one or more dimensions of the seam vary over the length of the seam, or a combination thereof. The controller 152 can determine control information 182 based on the gap variability information. For example, the controller 152 can generate or update control information 182, such as the movement of the robot 120 or one or more welding parameters, based on the gap variability information. In some implementations, to generate or update the control information, the controller 152 can compare the gap variability information to the design 170, waypoints 172, weld profile 174, weld fill plan 175, weld instructions 176, or a combination thereof.

[0129] In some implementations, the robot 120 may be configured to autonomously weld across a seam 144 having one or more varying dimensions, such as varying widths. Thus, in addition to identifying position and orientation information about the seam 144, one or more scanning techniques described herein (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / welding head or scanning the part from a sensor / camera positioned elsewhere in the workspace to identify the seam) may be implemented to identify gap variability information about the seam. Identifying gap variability information may include determining a gap width along the length of the seam or determining a gap profile along the length of the seam (e.g., how the gap along the seam varies). Based on the determined gap variability information, the controller 152 may generate or update waypoints 172 (e.g., waypoint information) or trajectory information associated with the waypoints 172. Updating the trajectory information associated with the waypoints 172 may include generating or updating control information 182, such as welding parameters and robot 120 movement parameters at each waypoint, based on the gap variability information. For example, at each waypoint whose dimensions are equal to or greater than the average gap dimension of seam 144 or the tolerance of seam 144, the welding and / or movement parameters of robot 120 may be generated / updated (e.g., by increasing voltage / current) to melt / deposit more or less metal at the waypoint.

[0130] In some implementations, identifying variable gap information may include or correspond to determining the position and orientation of the seam, such as seam position information, seam orientation information, or a combination thereof. After determining the seam position or orientation, the controller 152 may detect one or more edges that form or define the seam. For example, the controller 152 may use an edge detection technique, such as Canny detection, Kovalevsky detection, another first-order technique, or a second-order technique, to detect the one or more edges. In some implementations, the controller 152 may detect the one or more edges using a supervised or self-supervised neural network. The detected edges may be used to determine the variability of the gap (e.g., one or more dimensions) along the length of the seam. In some cases, the gap variability may be identified within or based on a 3D point cloud generated using images captured by the sensor 109. In some other examples, the gap variability may be identified using a scanning technique performed while performing a welding operation on the seam (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / welding head).

[0131] In some implementations, variable gap information determined using one or more variable gap identification techniques can be used to optimize one or more operations associated with welding the seam, including path planning. For example, controller 152 may be configured to dynamically generate or adapt welding instructions and / or motion parameters (e.g., welding voltage) based on the gap width / size. For example, dynamically adjusted welding instructions for robot 120 can result in accurate welding of seams with variable gaps. Adjusting welding instructions, such as welding instruction 176, may include adjusting welder voltage, welder current, electric pulse duration, electric pulse shape, material feed rate, or a combination thereof. Additionally or alternatively, adjusting motion parameters may include adjusting the motion of the welding head to include different weaving patterns, such as convex weaving, concave weaving, etc., to weld seams with variable gaps.

[0132] In some implementations, the controller 152 can instruct or control the sensor 109 to generate information associated with the seam 144. For example, the controller 152 can instruct or control the sensor 109 to capture one or more images associated with the seam 144. The controller 152 can receive the information associated with the seam 144 and process the information. For example, the controller 152 can perform a segmentation operation using a neural network to remove non-seam information from the information. In some implementations, the controller 152 can process the information based on a design 170, such as a CAD model. The design 170 can include annotated data.

[0133] In some implementations, the controller 152 can identify the seam 144 based on the information. Additionally or alternatively, the controller may determine the location of the seam 144 based on the information. By way of example, the controller 152 can perform seam recognition to identify pixel locations in multiple images and triangulate pixels corresponding to the seam 144 that fall within epipolar constraints. In some implementations, the controller 152 can determine one or more offsets of the information (from the sensor 109) compared to the design 170.

[0134] In some implementations, the controller 152 is configured to generate welding instructions 176 for welding along the seam 144. For example, the welding instructions 176 may be associated with a weld performed in a single pass, i.e., a single pass of welding along the seam 144, or a weld performed in multiple passes. In some implementations, the controller 152 may be configured to enable multi-pass welding, which is a welding technique used by the robot 120 to perform multiple passes on the seam 144. For example, the controller 152 may be configured for Multi-Pass Adaptive Fill (MPAF), which is a framework for determining the optimal number of weld passes and subsequent welding parameters to fill a weld joint. The weld joint may have volumetric variations, and the welding parameters are adapted to create the appropriate fill level.

[0135] To enable multi-pass welding in the robot 120, the controller 152 can identify the seam 144 to be welded and one or more characteristics of the seam 144. For example, the controller 152 may identify the seam 144 based on the design 170, the scan data, or a combination thereof. The one or more characteristics of the seam 144 may include a height of the seam 144, a width of the seam 144, a length of the seam 144, or a volume of the seam 144. Additionally or alternatively, the one or more characteristics of the seam 144 may be associated with a weld joint formed at the seam 144. For example, the one or more characteristics may include a height of the weld joint, a first leg length (S1) of the weld joint, a second leg length (S2) of the weld joint, a capping surface profile of the weld joint, a joint type, or a combination thereof.

[0136] To enable multi-pass welding in the robot 120, the controller 152 can also determine a fill plan and optionally optimize the fill plan. In some implementations, the fill plan can indicate the number of weld layers to fill out the weld joint, the number of target beads to be deposited to fill out the weld joint, one or more target bead profiles, or a combination thereof. The fill plan may be optimized to determine a minimum number of layers, a minimum number of target beads, or a combination thereof. The controller 152 may also generate the fill plan. Generating the fill plan may include determining or prescribing one or more welding parameters for each pass (e.g., each bead) of the fill plan. In some implementations, the one or more welding parameters for a pass may indicate values of one or more welding parameters at each of a plurality of waypoints 172 associated with the seam 144. After the fill plan is generated, the controller 152 can generate a welding instruction 176 based on the fill plan. Additionally or alternatively, the controller 152 can generate control information 182 based on the welding instruction 176. The controller 152 may send welding instructions 176 , control information 182 , or a combination thereof to the robot 120 .

[0137] In some implementations, to enable multi-pass welding, the controller 152 can receive or generate sensor data 180, seam pose information, seam feature information, one or more transformations, joint geometry information, or a combination thereof. The sensor data 180 (or information 164) can include a mesh from a scan performed by the sensor 109. The seam pose information can include or correspond to the point cloud 169 or information 164 (e.g., the sensor data 165 or the pose information 166). In some implementations, the seam pose information can include information based on an alignment, such as the alignment process described with reference to at least FIG. 7 . In some implementations, the alignment process can be a deformation alignment process that includes identifying expected locations and expected orientations of candidate seams on the parts (e.g., 135 or 136) to be welded based on the design 170 (e.g., a computer-aided design (CAD) model of the parts). The deformation alignment process may also include scanning the workspace 130 containing the part to generate a point cloud 169 or a scan point cloud 706 (e.g., a representation of the part), and identifying candidate seams on the part based on the representation of the part and the expected locations and expected orientations of the candidate seams.

[0138] The seam feature information may include or indicate one or more seam features determined based on a seam segmentation process. The seam segmentation process may include converting annotated features (of the design 170) into a set of waypoints 172 and normal information. For example, the seam segmentation process may use a mesh of a part as input and output a set of waypoints and surface normal information that represent the features in a manner suitable for planning. In some implementations, the seam feature information may include or indicate, by way of illustrative, non-limiting examples, an S1 direction (e.g., a vector at the waypoint indicating a first surface tangent), an S2 direction (e.g., a vector at the waypoint indicating a second surface tangent), and a movement direction (e.g., a direction of movement of the welding head perpendicular to a plane associated with a weld profile that passes through the waypoint, such as a sliced mesh at the waypoint). Additionally or alternatively, the seam information may be used or applied to a local coordinate frame associated with the seam 144 and a global frame associated with the workspace 130 to enable the controller 152 to determine a global transformation of the gravity vector.

[0139] The one or more transformations may include a transformation of the part to a real-world coordinate system or may indicate one or more waypoints. For example, the controller 152 may transform each point of the part's features (and corresponding normals) to a real-world coordinate system.

[0140] The joint geometry information may include or indicate, by way of illustrative, non-limiting example, a bevel angle, a root gap size, a wall thickness, a number of sides, a shaft radius, a shape (e.g., round or faceted), or a combination thereof. In some implementations, the joint geometry information may be determined based on the design 170, such as annotated information included in the design 170, or based on a point cloud of the part. Additionally or alternatively, the joint geometry information may include or correspond to the point cloud 169, the design 170 (e.g., a CAD model), the joint model information 171, or a combination thereof. For example, the joint geometry information may include a joint template generated based on the point cloud 169, the design 170 (e.g., a CAD model), the joint model information 171, or a combination thereof.

[0141] In some implementations, to enable multi-pass welding, controller 152 can receive or access multi-pass configuration information or tables. The multi-pass configuration information or tables can include or correspond to information 164 or system information 168. The multi-pass configuration can include or indicate one or more parameters, such as a voltage offset, a contact tip-to-work distance (CTWD) offset, an algorithm-specific parameter, or a combination thereof. Additionally or alternatively, the multi-pass configuration may include or indicate a configuration or serialized string that includes one or more parameters, such as one or more welding parameters, a weld bead spatial relationship to one or more welding parameters, a bead model 173, a welding parameter offset, system information 168, or a combination thereof.

[0142] The table, such as a lookup table, can indicate a wire feed speed (WFS), a travel speed (TS) value, a voltage (V), or a combination thereof. In some implementations, the table can indicate the WFS, TS, and voltage as a function or based on an area, such as a weld bead area. In some such implementations, the table can indicate the WFS, TS, and voltage as a tuple, for example, a (WFS, TS, V) tuple. In some implementations, the controller 152 can query the table for each waypoint. Illustratively, the controller 152 can access the table to determine a wire feed speed (WFS), a travel speed (TS) value, a voltage (V), or a combination thereof, for each weld bead to be placed at the waypoint. Note that the WFS and TS can be used to control the weld bead profile, thereby making the fill “adaptive.”

[0143] In some implementations, the tables may be generated or constructed using a volumetric flow equivalent assumption. Illustratively, the volumetric flow rate of wire leaving the tip of the welder may be equal to the volumetric flow rate of metal deposited at the joint multiplied by a fill factor that accounts for various ways in which the volume of metal may end up outside the actual joint (e.g., splatter). In some implementations, one or more other factors may be used as a set of constraints imposed on the WFS, TS, and leg length for the target weld. Tables may be constructed for a particular wire diameter, and multiple tables (for different wire diameters) may be available or accessible to the controller 152.

[0144] In some implementations, one or more constant parameters and WFS / TS limits for generation of the table include: Vwire=B_eff * Vweld, where Vwire is the volume of the wire, B_eff is the packing efficiency greater than or equal to 0 and less than or equal to 1, and Vwed is the volume of the weld. Further, the one or more constant parameters may include:

[0145] [Table 1]

[0146] The following pseudocode is designed to create a set of potentially valid rows for a table based on the range provided:

[0147] [Table 2]

[0148] Constructing a table based on the pseudocode may result in redundant entries for a single A_weld value. For example, all entries in the table (omitted for brevity) are valid solutions for A_weld=5.067.

[0149] [Table 3]

[0150] The entry containing the WFS value closest to the midpoint of the range is selected as the entry for A_weld=5.067. Since the WFS range is set between 100 and 700, the entry containing the WFS closest to [100+700] / 2=400 is selected. Most A_weld values may not appear explicitly in the table, so specific values of WFS and TS can be linearly interpolated using the entries where A_weld is immediately above and below the target A_weld value.

[0151] In some implementations, to enable multi-pass welding, the controller 152 can determine, for each pass, a phase lag, a start offset, a voltage (or arc length) offset, a contact tip-to-work distance value, or a combination thereof. The phase lag, the start offset, the voltage (or arc length) offset, the contact tip-to-work distance value, or a combination thereof may be constant (e.g., the same) for multiple weld points. Additionally or alternatively, the controller 152 may determine, for each pass and each weld point, a fill area (e.g., converted to a (WFS, TS, V) tuple), a wire feed speed, a travel speed, a voltage or arc length, a wire offset (e.g., tip position), a work angle offset, a weaving amplitude, a weaving angle offset, or a combination thereof. The fill area, the wire feed speed, the travel speed, the voltage or arc length, the wire offset, the work angle offset, the weaving amplitude, the weaving angle offset, or a combination thereof may be the same or different at two different waypoints among the multiple waypoints associated with the seam.

[0152] In some implementations, when a joint is filled with a target bead profile, the height of the target bead may be the same for all passes made within the joint (equal layer height assumption). In some such implementations, all target weld layers within the joint are planned to have the same height. When a weld layer is filled with a bead, each bead within a layer may have approximately the same area (equal bead area within layer assumption). Additionally or alternatively, the generated fill plan (also referred to as a fill plan solution) may represent a valid fill plan with a minimum number of total passes. If multiple candidate fill plan solutions are generated, each with the same number of total passes (e.g., the same number of minimum passes), such as sum([2,2,2])=6 and sum([3,3])=6, the fill plan with the minimum number of weld layers is selected (i.e., len([3,3]) <len([2,2,2])。

[0153] In some implementations, modifying the wire feed speed (WFS) and travel speed (TS) at each waypoint may be sufficient to control the target bead profile, the associated target bead profile area, or a combination thereof. Additionally or alternatively, the target bead profile, the associated target bead profile area, or a combination thereof may be controlled based on a target wire offset or a work angle offset. Note that the wire offset may be defined relative to the root along the S1 and S2 directions, with the root itself having a [0,0] offset in the S1 / S2 coordinates. If the root itself needs to have a non-zero offset within the part frame, the user can manually adjust the root offset via a GUI presented via the UI and display adapter 106. All work angle offsets (e.g., + / - offset directions) may be defined relative to the average normal direction (e.g., a 0 offset is the average normal direction). In some implementations, the pass tolerance may depend on the leg length. The leg length may be indicated and specified in the annotations of the design 170.

[0154] In some implementations, the controller 152 can determine a "crop radius" to identify the target surface profile. For example, the controller 152 may determine the crop radius based on a leg length generated by the controller 152, determined by a user, or indicated in the design 170. The crop radius is the radius around the joint seam to which the input scan is cropped, excluding all data outside this radius. A surface vector describing the tangent vector of the interface around the seam along the direction of travel determines the projection plane for each waypoint.

[0155] In some implementations, the controller 152 can identify a seam, such as seam 144. The controller can generate a plurality of waypoints 172 along the seam 144. To illustrate, refer to FIG. 12 , which is a perspective view of an example of a representation of parts to be welded, according to one or more embodiments. The parts include a first part 135 and a second part 136 that define the seam 144. The controller 152 is generating waypoints 172 disposed along the seam 144.

[0156] In some implementations, the controller 152 can identify a cross-sectional profile to be used to generate the weld profile 174 or joint template. The weld profile 174 or joint template may be aligned with one or more fill vectors associated with the seam 144. To illustrate, with reference to FIG. 13 , FIG. 13 is a perspective view of another example representation of parts to be welded, according to one or more embodiments. The controller 152 can generate multiple cross-sectional profiles, such as representative cross-sectional profile 1301. Each of the cross-sectional profiles can be positioned at a waypoint (e.g., 172). In some implementations, the controller 152 can generate the weld profile 174 based on the point cloud 169, the design 170, the sensor data 165 or 180, the joint model information 171, or a combination thereof. For example, the weld profile 174 can be generated at least in part using an ideal weld profile associated with the joint type / weld type. The ideal weld profile can include or correspond to the joint model information 171. 14 , which is an illustration of an example cross-sectional projection of the seam of FIGS. 12 and 13 , according to one or more embodiments. In some implementations, the controller 152 can determine one or more fill direction vectors of the weld profile 174 or weld fill plan 175. For example, the controller 152 can determine one or more fill direction vectors by aligning the fill-out direction to be collinear with gravity and aligning the fill-in vector to be a function of the fill-out direction and the movement direction. The fill-out direction can be or can indicate the direction in which layers of beads are stacked on top of each other. The fill-in direction can be or can indicate the direction in which beads are distributed along the layers. In some implementations, the controller 152 can determine the fill direction, the movement direction, or a combination thereof based on a gravity vector based on a local coordinate system.

[0157] In some implementations, the controller 152 can generate a weld fill plan 175 based on the weld profile 174. The weld fill plan 174 can include or represent an ordered sequencing of weld beads within a weld profile cross-section (e.g., 174). One or more weld beads may be positioned within the weld profile, and one or more three-dimensional weld characteristics may be mapped to each feature point within the seam 144 based on the multiple weld profiles. In some implementations, a particular sequencing pattern within the cross-section is referred to as a fill key. The fill key may be a one-dimensional numeric array-based representation of how the weld beads are distributed within the joint. The position of each entry in the array represents a layer position within the layer tacking in the fill-out direction. The value at each position in the array represents the number of beads in the layer along the fill-in direction.

[0158] Referring to Figure 15, Figure 15 is an illustration of an example cross-sectional projection of a seam according to one or more embodiments. As shown in Figure 15, layers of a weld fill plan (e.g., 174) for forming a joint in seam 144 are shown. For example, the layers include a first layer (layer 1), a second layer (layer 2), and a third layer (cover layer).

[0159] Referring to FIG. 16 , FIG. 16 is a diagram of another example cross-sectional projection of a seam according to one or more embodiments. As shown in FIG. 16 , an ordered sequence of weld beads to be placed to form a joint within seam 144 is illustrated. For example, the ordered bead sequence indicates the order of eight beads to be placed. For example, the first and second beads are placed as part of the first layer (layer 1), the third, fourth, and fifth beads are placed as part of the second layer (layer 2), and the sixth, seventh, and eighth beads are placed as part of the third layer (cover layer). By stacking the weld beads according to the order between each weld profile, three-dimensional weld characteristics can be mapped back to each feature point within the seam. As shown in FIG. 16 , the bead sequence process has a fill key of [2, 3, 3] because there are two beads on the first layer and three beads on each of the second and third layers.

[0160] In some implementations, to determine the weld fill plan (e.g., 175), the controller 152 may be configured to generate one or more slices (e.g., cross sections) of a seam, such as seam 144. The controller 152 may determine the height of a target weld profile (e.g., a joint profile). For example, the height of the target weld profile may be based on a capping surface of the target weld. The controller 152 may divide a height, such as a maximum height, by a positive integer to determine a layer height. The layer height may be greater than or equal to a minimum layer height, less than or equal to a maximum layer height, or a combination thereof. In some implementations, the controller 152 may compare the layer height to a minimum layer height, a maximum layer height, or a combination thereof. The minimum layer height, the maximum layer height, or a combination thereof may be determined by a user, annotated on the design 170, or a combination thereof. For each layer, the controller 152 may determine the area of the layer. Based on the area of the layer, the controller 152 may determine the number of beads per layer based on the layer area, based on a bead area minimum or a bead area maximum. The bead area minimum, bead area maximum, or combinations thereof may be determined by a user, annotated in the design 170, or combinations thereof. For each bead in the weld fill plan, the controller 152 may determine one or more welding parameters. The one or more welding parameters may include a phase lag, a start offset, a voltage (or arc length) offset, a contact tip-to-work distance value, a fill area, a wire feed speed (WFS), a travel speed (TS), a voltage (V), a wire offset (e.g., tip position), a work angle offset, a weaving amplitude, a weaving angle offset, or combinations thereof. In some implementations, the controller 152 may determine the one or more welding parameters by accessing a table.

[0161] In some implementations, the controller 152 can perform a packing plan selection process to generate one or more candidate packing plans. The packing plan selection process can use a discrete solution space of potential packing plans. The potential packing plans can be discretized based on layer height variations and various combinations in which each layer can be subdivided into a set of valid bead sizes. A set of sample cross sections can be selected to be "filled" or have pre-planned beads for each packing plan in the solution space. A metric of intersection-over-union between the area generated from the combination of beads filled within the cross sections and the target profile area is used to score each packing plan. The packing plan with the highest score is selected to be fully generated.

[0162] In some implementations, based on the selection of a fill plan, each waypoint has its cross section filled according to the fill key. Starting at the origin and proceeding along the fill-in direction, each bead is placed successively after each other until the layer is filled. The fill then moves along the fill-out direction to start a new layer. This process is repeated until the entire fill key is generated.

[0163] In some implementations, the controller 152 can execute the recommendation logic 114 to generate a set of candidate fill plan sequences. For example, the controller 152 can generate or determine a set of target weld profiles associated with the seam 144. The controller 152 may execute the recommendation logic 114 and generate the set of candidate fill plan sequences based on the set of target weld profiles, the status information, or a combination thereof. The controller 152 can select one fill plan sequence from the set of candidate fill plan sequences as the weld fill plan 175 and can generate welding instructions based on the selected weld fill plan 175.

[0164] To select a fill plan sequence from the set of candidate fill plan sequences, the controller 152 may use a cost evaluation algorithm (e.g., an optimized cost evaluation algorithm). The cost evaluation algorithm may include inputs such as one or more weld constraints, one or more heuristics, or a combination thereof, and may apply weights to the inputs. The one or more weld constraints or one or more heuristics may include or be based on, as illustrative, non-limiting examples, overall fill, leg size, throat, convexity, bead sequencing, or a combination thereof. The weights may be determined based on or indicate the priority of the corresponding inputs for producing a high-quality weld with respect to passing visual testing, destructive testing, non-destructive testing, or a combination thereof. In some implementations, the weights may be adjusted over time to reflect changes in welding strategy or to emphasize application-specific requirements, such as a requirement for a concave weld profile or a given value of toe angle. For each candidate fill plan sequence, the controller 152 may use the cost evaluation algorithm to determine a total (e.g., a weighted total) associated with the cost of the candidate fill plan sequence. In some implementations, the controller 152 can select the candidate packing plan sequence with the lowest cost. By way of example, the candidate packing plan sequence with the lowest cost may represent the solution with the highest probability of passing a visual test, a destructive test, a non-destructive test, or a combination thereof. In some implementations, the controller 152 can rank the candidate packing plan sequences based on cost and select the candidate packing plan sequence based on the ranking.

[0165] In some implementations, the recommendation logic 114 (e.g., a cost assessment algorithm) can be configured to receive and / or consider the weld quality feedback to generate candidate fill plan sequences and / or determine the cost of the candidate fill plan sequences. To incorporate the weld quality feedback into the recommendation logic 114, a record of error samples (e.g., fill plans selected by the recommendation logic 114 that failed to produce a quality weld) can be stored (e.g., in an error cache). The recommendation logic 114 can use the error samples to adjust a final prediction of the modeled / generated candidate fill plan sequences. For example, the cost assessment algorithm can have an input corresponding to a given error as indicated in the error sample, and a weight (or cost, e.g., a float value) can be assigned to the observed error. In some implementations, the value of the weight can be selected based on or associated with a particular corrective action or a desired output fill plan sequence. For example, if poor quality welds are identified as having a large number of layers and a small average bead size, the controller 152 (or recommendation logic 114) may assign a high weight value to error inputs based on the large number of layers and small average bead size, thereby selecting a candidate fill plan sequence that corrects for the observed errors in production. Such corrections may be directly reactive, proportionally appropriate, or a combination thereof, to the observed errors.

[0166] Note that the use of error samples can enable faster and more timely adaptation of the selection of a fill plan sequence that avoids identified errors. Illustratively, a model or algorithm can be optimized against a training data set using backpropagation within an empirical risk minimization (ERM) framework, where the true distribution of the data is unknown and assumed to be stationary. In some such implementations, the model or algorithm can be periodically retrained against new offline simulation data generated under updated cost assumptions that can be adjusted to reflect the results of new observed data samples. During model or algorithm updates, the controller 152 can incorporate the use of error samples to adapt to frequent weld quality feedback and account for identified errors.

[0167] In some implementations, the controller 152 can use a bead model 173. The bead model 173 may be modeled from a constant spline generated using control points determined using a user-defined (or controller-generated) arc length parameter, with the estimated bead width giving the target area and work angle. The bead may be shifted along the work angle until its polygonal area (determined from the intersection of the constant spline with the surface profile) is sufficiently close to the desired bead area. Each subsequent bead uses a unified set of the previous bead profile and the target profile to determine its shape and area. This process continues until each bead is produced.

[0168] Referring to FIG. 17 , FIG. 17 includes graphs illustrating a modeled bead in accordance with one or more embodiments. The graphs show a modeled bead 1707 (based on bead model 173). In each graph, the bead 1707 is modeled based on one or more welding parameters, such as an arc length parameter, a work angle, or a combination thereof. As shown in FIG. 17 , a first graph 1701, a second graph 1702, and a third graph 1703 each show the bead 1707 modeled based on a first work angle 1708. A fourth graph 1711, a fifth graph 1712, and a sixth graph 1713 each show the bead 1707 modeled based on a second work angle 1718. A seventh graph 1721, an eighth graph 1722, and a ninth graph 1723 each show the bead 1707 modeled based on a third work angle 1728. Each of the graphs 1701-1703, 1711-1713, and 1721-1723 shows a different arc length parameter. In some implementations, each of the graphs 1701-1703, 1711-1713, and 1721-1723 is associated with a [1,2] filling plan.

[0169] Note that adaptive fill aspects can be specific to how each cross-section is filled, since each cross-section follows the same fill key. The size of each bead can be adjusted to fill the layer that includes the bead. Referring to FIG. 18 , FIG. 18 is an illustration of an example weld fill plan for a weld profile, according to one or more embodiments. As shown in FIG. 18 , each bead is associated with an area, indicating a target area and work angle associated with the bead's deposition. Referring to FIG. 19 , FIG. 19 is an illustration of an example weld fill plan across multiple weld profiles, according to one or more embodiments. For example, FIG. 19 includes a first weld profile 1901, a second weld profile 1902, a third weld profile 1903, and a fourth weld profile 1904. The first weld profile 1901 is associated with a first waypoint, the second weld profile 1902 is associated with a second waypoint, the third weld profile 1903 is associated with a third waypoint, and the fourth weld profile 1904 is associated with a fourth waypoint. For each of the weld profiles 1901-1904, each bead is associated with an area, indicating the target area and work angle associated with the bead deposition.

[0170] In some implementations, the controller 152 can generate a welding control signal, such as a welding instruction 176 or control information 182, based on the weld fill plan 175. The controller 152 may transmit the welding control signal to the robot 120. For example, the controller 152 can generate or transmit the welding control signal based on an empirical and physical interaction relationship. In some implementations, the empirical and physical interaction relationships can be stored in a database table, such as a database table stored in the memory 102 or the database 112. The database table can discretize the welding control signal between boundary extremes determined from experiments for a selected welding wire, welding gas, welding mode, applied joint type, material, or combinations thereof. Bounded welding parameters can include a travel speed (TS), a wire feed speed (WFS), and a voltage (V). These boundaries, along with a minimum and maximum bead size, can be specified as a design space. A weld bead cross-sectional area can be determined based on the specified design space.

[0171] In some implementations, to enable multi-pass welding, the controller 152 can determine the weld profile 174 of the seam 144 based on the sensor data 180, seam pose information, seam feature information, one or more transformations, joint geometry information, joint type, or a combination thereof. In some implementations, the controller 152 may determine the joint type based on the sensor data 180, the design 170 (e.g., annotation information), user-input data, the point cloud 169, the joint geometry information (e.g., a joint template), or a combination thereof. For example, the controller 152 may determine the joint type by matching the joint geometry information with the sensor data 180. Referring to FIG. 20 , FIG. 20 includes an example of templates matched to recognized features, according to one or more aspects. As shown in FIG. 20 , a first template includes a T-joint with a sloped root gap, and a second template includes a V-groove with a sloped root gap. Each of the templates includes one or more feature points, shown as stars. The controller 152 may match the scan data (e.g., 180), the point cloud data 169, or the design 170 based on one or more feature points of the template. In some implementations, the controller 152 may determine one or more recognized features based on the scan data (e.g., 180), the point cloud data 169, or the design 170 and match the one or more recognized features with one or more features of the template. In some implementations, the controller 152 may determine a joint type and select a joint model that matches the data 180, the point cloud data 169, or the design 170 based on the determined joint type.

[0172] In some implementations, the joint model information 171 may include one or more joint templates, one or more joint template components, or a combination thereof. One or more joint template components may be combined to form a joint template. A joint template can describe or define the spatial relationship of a joint. For example, the spatial relationship may include or correspond to the spatial relationship between pairs of feature points of the template. By way of example, a joint template may include or define a set of feature vectors and a set of bounded dimensions. The vectors and their relationships to each other allow a discrete set of feature points to be generated using one or more known constraints. The one or more constraints may be generated from dimensional ranges / bounds describing each of the variables in the variable feature vector. Types of variables that can describe a feature vector may include rotation angle, magnitude, or a combination thereof. Each feature vector may be linked to a parent vector or feature point, so that each feature vector can be well-defined relative to a single origin, which may enable a well-defined yet generalized method of determining important variable dimensions and geometries.

[0173] In some implementations, the variables in each variable feature vector can be driving or driven variables. Referring to FIG. 21 , FIG. 21 is a joint template diagram in accordance with one or more embodiments. As shown in FIG. 21 , the joint template diagram is a T-joint template with a sloped root gap. The bevel vector in FIG. 21 drives the bevel angle, and its magnitude is driven by the bevel thickness vector. In some implementations, a label with a physical distinction that can indicate how the joint is secured can be applied to each feature vector. For example, a base vector can be a part-level vector indicating that the weld bead is constrained to be within the radius of its associated contour point. Similarly, a hidden gap vector can represent an inferred void in the part, which can be associated with missing areas of the weld bead targeted by that feature and thus inform the weld bead to overfill those areas.

[0174] In some implementations, the controller 152 can match the joint template to the identified seam using one or more techniques. The one or more techniques can include a deep learning model or a shape fitting model. In some implementations, the one or more techniques can include a first deep learning model. The first deep learning model can be configured to receive feature points, a nominal 2D contour, an actual 2D contour, or a combination thereof. The first deep learning model can be configured to output the actual feature points, a rotation of the feature points from the nominal, or a combination thereof. In some implementations, the one or more techniques can include a second deep learning model. The second deep learning model can be configured to receive the actual 2D contour. The second deep learning model can be configured to output the actual feature points, a rotation of the feature points from the nominal, or a combination thereof. In some implementations, the one or more techniques can include a shape fitting technique. The shape fitting technique can include statistical and analytical methods that can fit a model template contour with indexed feature point locations to the actual contour to identify the locations of the actual feature points. Shape fitting techniques may include generalized Procrustes analysis or active contour models.

[0175] In some implementations, each joint template may include one or more boundaries (or thresholds) for its variables so that the matching model (or recognition pipeline / network) can be trained. By iterating through different combinations of values for each variable in the template, an exhaustive set of training data and labels can be generated and used in an unsupervised manner. To enable a more robust pipeline, various levels of noise and different types of features, such as tacks and gaps, can be injected into the training data. In some implementations, a user can generate a joint template or specify one or more boundaries for the joint template. For example, a user can generate or specify one or more boundaries via the UI and display adapter 106. After generating the joint template, the joint template can be trained.

[0176] In some implementations, the controller 152 can include or generate labels for one or more faces / points that make up the cross-sectional projections, such as the point cloud 169, the sensor data 165, 180, or the cross-sectional projections of the joint template. Doing so can allow the interaction model to be more robust and adapt to geometric features such as tacks and gaps.

[0177] In some implementations, the controller 152 can determine or identify a capping surface of the weld joint. For example, the capping surface can be determined or identified based on user requirements, engineering information of the joint, the design 170, or historical information. The capping surface may be determined based on the leg length of the weld joint, a convexity connecting two major surfaces, or a combination thereof. The leg length may be associated with or determined based on the design 170 or scan data. Additionally or alternatively, the leg length may include a boundary, which can allow for extension of the capping surface if a solution (e.g., a weld profile or weld fill plan) cannot be found.

[0178] Referring to FIG. 22 , FIG. 22 includes an example of a joint template view having different leg length definitions, according to one or more embodiments. As shown in FIG. 22 , a first joint template view 2201 and a second joint template view 2202 are each T-joint templates with angled root gaps. In the first joint template view 2201, the leg length definition is generalized. For example, the base leg length vector and the bevel leg length vector have the same magnitude. In the second joint template view 2202, the leg length definition is variable. For example, the base leg length vector and the bevel leg length vector have different magnitudes.

[0179]

[0013] Referring to Figure 23, Figure 23 is another example of a joint template diagram according to one or more embodiments. As shown in Figure 23, the joint template diagram is a T-joint template with a sloped root gap, where the target profile of the joint is shown.

[0014] Referring to Figure 24, Figure 24 is another example of a joint template diagram according to one or more embodiments. As shown in Figure 24, the joint template diagram is a T-joint template with a sloped root gap, where the capping surface variation is shown.

[0180] Boundaries can be assigned to one or more variable vectors and one or more leg lengths, so that measurements of each variable can be reported with respect to whether it meets design specifications (e.g., of design 170). Referring to FIG. 25, FIG. 25 is another example of a joint template diagram according to one or more embodiments. As shown in FIG. 24, the joint template diagram is a tee template with a sloped root gap, where the target profile extension is indicated by a dashed line. For example, the dashed line can include or correspond to one or more surfaces of the part, a capping surface of the joint, or a combination thereof.

[0181] Additionally, the joint template may enable the generation of the weld fill plan 175 and / or the welding instructions 176 without the need for tables. For example, the weld profile 174, the weld fill plan 175, one or more welding parameters, the welding instructions 176, or a combination thereof may be determined or generated based on the joint template, which may enable fine-grained control of the weld and continuously improve the weld model without having to rebuild the software interface with the plan.

[0182] In some implementations, to enable multi-pass welding, the controller 152 can use a search space modeled in a continuous region, where the cross section uses primitive geometry to estimate bead size and location. Each bead can be connected to another bead according to adjacent beads in the layer. Additionally or alternatively, one or more beads, or all beads, may be initially generated or placed in the joint template based on an average or median bead size. For example, the design 170 (e.g., annotation information for the design 170) may indicate a minimum bead size, a maximum bead size, or a combination thereof. In some implementations, a fill plan can be generated by populating the joint template with a plurality of one or more candidate beads.

[0183] 26-29 are examples of generating a fill plan using a joint template diagram, according to one or more embodiments. To generate a fill plan, the joint template may be populated with multiple candidate beads, such as one or more beads having the same size. While a single joint template is described, it should be noted that a seam (e.g., 144) may be associated with multiple waypoints, each corresponding to a joint template that is populated to generate the fill plan.

[0184] Referring to FIG. 26 , a first set of beads 2601 are placed at feature points relative to a given geometry of the joint template. Note that if a seam has multiple waypoints, each with a corresponding joint template, each joint template (e.g., cross-section) may have the same number of beads in the first set of beads 2601. Referring to FIG. 27 , a second set of beads 2702 are placed at one or more auxiliary structure points. The auxiliary structure points may include gaps between beads in the first set of beads 2601 (e.g., structural beads) along the boundary of the part, such as along a surface / part location. The placement of the second set of beads 2702 can ensure that the entire part surface has a weld bead substantially close to that surface to provide proper fusion. Referring to FIG. 28 , a third set of beads 2803 are placed to conform to the capping surface boundary to provide a stable cover layer. Referring to FIG. 29 , a fourth set of beads 2904 are placed as filler beads along fill-in lines connecting adjacent boundary beads. It should be noted that to populate the joint template, the controller 152 may populate the joint template with the first set of beads 2601, and optionally the second set of beads 2702, the third set of beads 2803, the fourth set of beads 2904, or a combination thereof, or may not populate any of the second set of beads 2702, the third set of beads 2803, or the fourth set of beads 2904.

[0185] In some implementations, the controller 152 may refine the placement of multiple candidate beads within the joint template. For example, refinement by the controller 152 may include optimizing within a joint template associated with a seam (e.g., 144) and / or across multiple or all joint templates (e.g., cross sections). Illustratively, the controller 152 may determine a maximum number of beads or a minimum number of beads across one or more joint templates (e.g., one or more cross sections). The maximum number of beads and / or the minimum number of beads may provide a bead limit for the weld fill plan 175. In some implementations, the maximum number of beads and / or the minimum number of beads may be determined based on user input, the design 170 (e.g., annotated data included in the design 170), or a combination thereof. Furthermore, the controller 152 may refine the placement of multiple candidate beads such that one or more beads merge into a larger, but valid, bead until the joint template converges to a minimum feasible bead count. Referring to FIG. 30 , FIG. 30 is an example of an improved fill plan using a joint template view according to one or more aspects. As shown in Figure 30, multiple candidate beads have been refined, including a first set of beads 2601, a second set of beads 2702, a third set of beads 2803, and a fourth set of beads 2904. For example, the joint template of Figure 30 includes a first set of beads 3001, a second set of beads 3002, a third set of beads 3003, and a fourth set of beads 3004. The first set of beads 3001 may include or correspond to the first set of beads 2601. The second set of beads 3002 may include or correspond to the second set of beads 2702. The third set of beads 3003 may include or correspond to the third set of beads 2803. The fourth set of beads 3004 may include or correspond to the fourth set of beads 2904. In some implementations, each bead in a cross section may be linked to adjacent cross sections of previous and / or subsequent beads, for example in a tree-like structure. Such linking may allow for refinement and synchronization when beads are updated.For example, the bead may be updated during a multi-pass operation, such as after the first of multiple passes.

[0186] Referring back to FIG. 1 , in some implementations, the controller 152 performs bead prediction and determines spatial relationships based on multiple candidate beads (e.g., a first set of candidate beads), the refined fill plan, or a combination thereof. By considering the initial location and region of the refined fill plan, each bead may be provided with a boundary search radius within which the bead's origin may be located, which may generate a higher likelihood that one or more structural beads will retain their connections to structurally important interfaces and features. Furthermore, by considering the initial location and region of the refined fill plan, since each bead is organized to be contained and controlled from the structural or part bead, the bead location may be more likely to remain close to locations in adjacent slices (different joint templates). In some implementations, the controller 152 may generate a metric that can be associated with the risk of structural compromise. For example, based on the refined fill plan, the controller 152 can identify beads that encompass two structural features that may pose a higher risk. The controller 152 may be configured to generate a notification of the condition to the user and present the notification to the user via the UI and display adapter 106, or the like.

[0187] In some implementations, controller 152 determines a welding control signal, such as welding instructions 176 or control information 182. The welding control signal can be determined from empirical and physical conservative relationships. Additionally or alternatively, the control signal may be determined from an inverse model of bead model 173. By way of example, inputs to the welding model can include spatial characteristics of the bead, and outputs of the welding model can include welding control signals necessary to produce a desired shape of the bead. In some implementations, the welding model can be specific to a welding wire size, a welding wire type, a welding gas composition, a welding mode (pulse / CV), or a combination thereof. Additionally or alternatively, the welding model may receive as inputs a welding wire size, a welding wire type, a welding gas composition, a welding mode (pulse / CV), or a combination thereof.

[0188] In some implementations, the controller 152 can use the global deformation seam to place the first bead, for example, the first bead of a multi-pass welding operation. The controller 152 can assume that the location of the global deformation seam is the root path, and all offsets can be applied according to the global deformation seam. Alternatively, the global deformation seam can be used by the controller 152 to guide the search for the root feature. Once the root feature is identified, the root feature can be used as the origin of the first bead. The offset between the identified root feature and the global deformation seam can then be applied to the offset between the global deformation seam and the local deformation seam. In other words, instead of applying a rigid transformation between the global deformation seam and the local deformation seam, a combination of two rigid transformations can be applied.

[0189] In some implementations, after the weld fill plan 175 is generated, the controller 152 can update the weld fill plan 175 based on one or more actions by the robot 120. Examples of weld fill plans are described further herein with reference to at least FIG. 31A . The controller 152 may update the weld fill plan 175 after one or more passes, or after each pass of multiple passes. Illustratively, after a pass, the last executed bead in the weld fill plan 175 may be updated or replaced with its corresponding local scanner data (e.g., sensor data 180). Examples of scan operations associated with the local scanner data are described further herein with reference to at least FIG. 31B . Replacing the bead shape with the executed bead shape may enable the controller 152 to react to and correct for observed differences between the global scanner or CAD model and the local scanner data. Examples of updating the weld fill plan based on scan operations are described further herein with reference to at least FIG. 31C . Additionally or alternatively, the controller 152 may update the bead model 173 based on the scanning motion, as further described herein with reference to at least FIG. 31D.

[0190] In some implementations, after the robot 120 places a bead, the controller 152 may receive scan data (e.g., sensor data 180) of the bead. The scan data may include image data or laser data. Note that the scan of the bead may occur during a cooling period after the bead is placed. The controller 152 can update one or more beads in the weld fill plan 175 based on the received scan data. For example, the controller 152 may update one or more beads at a time. Based on the scan data, the controller 152 can extract geometric characteristics, such as one or more toe points, bead profiles, or combinations thereof. In some implementations, the controller 152 can update one or more beads placed before the last placed bead. For example, heat from the last placed bead may affect one or more previously placed beads, and the controller 152 can update the weld fill plan to reflect the changes to the previously placed beads.

[0191] After updating the weld fill plan 175 to reflect the placed beads, the controller 152 can update one or more beads in the weld fill plan 175 that have not yet been placed by the robot 120. For example, the controller 152 may update one or more bead profiles, capping layer profiles, or a combination thereof. In some implementations, to update the weld fill plan 175, the controller 152 can generate a new weld fill plan using the joint template and scan data of the placed bead or beads as a starting point. Illustratively, the controller 152 can place one or more candidate beads and refine the one or more candidate beads to generate a new weld fill plan. Based on the updated or new weld fill plan, the controller 152 can generate additional welding instructions (e.g., 176). For example, the controller 152 can generate the additional welding instructions using the bead model 173.

[0192] 31A-31D, which include example diagrams of different portions of a multi-pass welding operation according to one or more embodiments. For example, FIG. 31A shows an example of a nominal fill plan 3100, FIG. 31B shows example results of a scanning operation (e.g., "Deposition Feedback Scan" results 3110) after multiple passes of a multi-pass adaptive fill, FIG. 31C shows an example of a process for updating the weld fill plan based on the scanning operation, and FIG. 31D shows an example of updating the bead model 173 based on the scanning operation.

[0193] Referring to FIG. 31A , the controller 152 is configured to generate a nominal fill plan 3100 based on the bead model 173. For example, the nominal fill plan 3100 may include or correspond to the weld fill plan 175. The purpose of the nominal fill plan 3100 is to establish an appropriate welding strategy (e.g., ideal number of weld passes, bead size, bead sequencing, etc.) for filling a given weld joint. In some implementations, the controller 152 can generate the nominal fill plan 3100 using the multi-pass logic 111, the recommendation logic 114, or a combination thereof. For example, the nominal fill plan may be selected or generated using a neural network as part of an optimization procedure constrained by standard welding requirements and heuristics, or a combination thereof. Once selected, the nominal fill plan 3100 provides (e.g., indicates) one or more inputs for the welding operation. In this manner, the nominal fill plan 3100 constitutes a “feed-forward” approach for the welding operation.

[0194] To address potential issues with feedforward approaches, the controller 152 can implement a feedback approach for the nominal fill plan 3100, such as a learned weld “feedback” solution for the multi-pass adaptive fill logic (e.g., the machine learning logic 107, the multi-pass logic 111, or a combination thereof). The feedback approach may include updating (e.g., iteratively updating) the nominal fill plan 3100 based on the weld surface observed from scans collected during production welding (e.g., the generated weld feedback scan results 3110), as described further herein with reference to at least FIG. 31C. The feedback approach may also provide an effective mechanism for trajectory correction between passes. In some implementations, the feedback method may facilitate the process of annotating or automatically labeling a physical bead model that guides the formation of the fill plan. This annotation may be informed by the weld surface data, thereby creating a self-supervised data collection process. During this process, the modeled profile receives guiding inputs, called supervisory signals, derived from the weld surface profile. Stated another way, feedback methods use surface data from the weld deposit to enable automatic labeling of the physical bead model involved in creating the fill plan, and this self-supervised data collection relies on input of monitoring signals taken from the weld surface profile to provide guidance to the modeled profile. Improved weld quality can be achieved based on feedback techniques that enable the ability to update the trajectory (e.g., welding head / welding tool tip trajectory) during welding. Additionally or alternatively, feedback techniques can improve both the speed and accuracy of the bead modeling process, resulting in a faster and more robust adaptive welding system.

[0195] The weld feedback scan may be performed using the sensor 109 and may include or correspond to the sensor data 109. In some implementations, a local scanner, such as a laser scanner, may be used to perform a local scanning operation to generate the sensor data 180 (e.g., the weld feedback scan results 3110). For example, the local scanner may be coupled to the same robot (e.g., 120) that includes the manufacturing tool 126 (e.g., the welding head). As another example, the local scanner may be located at another location within or adjacent to the workspace 130. In some implementations, the weld feedback scan is performed during or after N passes (i.e., scans are collected during each N welding passes), where N is a positive integer. In other implementations, the weld feedback scan is performed during each welding pass, during a pass completing a bead layer, or a combination thereof. The local scanning operation may capture a profile of the “just-deposited” bead on the weld surface immediately following the arc and may be used to generate the weld feedback scan results 3110. The weld surface at this point includes any remaining exposed joint surface visible within the field of view of the scanner (e.g., 109), as well as the current weld bead and all previous weld beads. The weld feedback scan can use a neural network to segment the subsequent laser line from the image, including the arc flash. Using the segmented laser line, the controller 152 can use additional logic to generate point clouds for each image and aggregate the individual laser line point clouds into a single point cloud of the weld surface. The slice of the weld surface at points along the trajectory is a profile used to facilitate self-supervised learning to improve trajectory correction as well as bead modeling (as further described herein with reference to at least FIG. 31D).

[0196] Referring to Figure 31C, Figure 31C shows an example of a process for updating the weld fill plan based on scan motion. For example, the process may include a "feedback" setup for a multi-pass adaptive fill (MPAF) technique, where the nominal fill plan 3100 provides the initial feedforward commands, but at intervals of N passes (e.g., every 3 passes), rather than all at once, feedback is provided during the interval. Specifically, if the weld feedback scan is k * After collecting N passes (k is a positive integer in a counter used to track the number of feedback scans collected about the weld joint), sampled slice pairs of the modeled weld surface (model) and the corresponding observed weld surface (observation) are fused into a globally consistent frame. In this frame, the observation replaces the model, and after k>1, the origin of the model needs to be reseeded with the observation. This transformation, which serves to reseed the origin of the next pass, is a "spaced trajectory correction" that ensures that the feedforward commands for the k+1 set of N beads are matched to the starting observation. Note that when N=1, this "spaced trajectory correction" becomes an "inter-pass trajectory correction." Replacing the modeled weld surface with the observed weld surface and calculating the transformation needed to seed the next set of passes on the observed weld surface allows the algorithm to quickly react (as opposed to ignoring) to local part variations or welding process errors that cause the system to deviate from the idealized state captured in the nominal fill plan.

[0197] As shown in FIG. 31C , in a first stage 3120, a model deposition surface profile is identified for the first three weld passes based on the nominal fill plan 3100. The first three weld passes are performed based on the nominal fill plan 3100, and a weld feedback scan is performed in conjunction with the third weld pass to determine an observed deposition surface profile. In a second stage 3122, an observed deposition surface profile of the actual weld is determined based on the weld feedback scan. In a third stage 3124, trajectory corrections are identified and performed on the nominal fill plan 3100 based on a comparison between the model deposition surface profile (e.g., shown in the first stage 3120) and the observed deposition surface profile (e.g., shown in the second stage 3122).

[0198] In a fourth step 3126, an updated nominal fill plan for the fourth through sixth weld passes is presented along with the observed deposition surface profile (e.g., determined in the second step 3122). The fourth through sixth weld passes are performed based on the updated nominal fill plan, and a weld feedback scan is performed in association with the sixth weld pass to determine another observed deposition surface profile. In a fifth step 3128, an observed deposition surface profile of the actual weld deposit is determined based on the weld feedback scan performed in association with the sixth weld pass. In a sixth step 3130, trajectory corrections are identified and performed to further update the nominal fill plan based on a comparison between the model deposition surface profile (e.g., shown in the fourth step 3126) and the observed deposition surface profile (e.g., shown in the fifth step 3128). At least one or more of steps 3120-3130 may be repeated as necessary until a weld is formed in accordance with or based on the nominal fill plan 3100.

[0199] 31D , which illustrates an example of updating the bead model 173 based on the scanning operation. For example, the bead model 173 may be updated using the machine learning logic 107. Illustratively, the controller 152 can use or execute the machine learning logic 107 to update the bead model 173. In some implementations, the bead model 172 can include a set of adjustable, physics-driven geometric modeling parameters (e.g., gravity coefficients, surface tension coefficients, etc.) that control how an “idealized” weld bead operates when placed in a particular situation with a solid surface. The controller 152 can train the bead model 173 by using a self-supervised learning paradigm, where the model training leverages input data for supervision.

[0200] In some implementations, for self-supervised learning, the controller 152 can start with a simple “ideal” parametric bead model to create a nominal fill plan (e.g., 3100). In this context, “ideal” implies or conveys that the points that make up a geometric object are represented by a mathematical equation that, when deposited on a given surface, produces a bead profile of appropriate size. In some implementations, the mathematical equation may not consider one or more properties of the fluid from a fluid dynamics perspective. Self-supervised learning can enable an understanding of how a set of welding parameters, when applied in the context of a given surface (geometric context), modifies the shape of the nominal bead.

[0201] Modifications to the nominal bead can be represented by a set of vectors (weld profile motion vectors) assigned to each point of the bead profile. These vectors may encode a direction and magnitude indicating the direction and extent to which each point is adjusted by a set of welding parameters given a geometric context. The weld profile motion vectors may be supervisory signals that, when paired with the profile of the nominal weld bead, form training data for a network-based learning model. Illustratively, the weld profile motion vectors may be formed by subtracting corresponding points of the weld surface profile (e.g., 3122) from the model deposition surface profile (e.g., 3120). Because the weld profile motion vectors are constructed from the weld surface profiles collected during welding, the training data can be considered to be constructed in a self-supervised manner.

[0202] In some implementations, the system 100 (e.g., the controller 152) is configured to enable on-the-fly collection of weld profile motion vectors for each weld pass during production welding. In some such implementations, the process of learning an optimal set of model parameters is achieved using a continuous learning process. The continuous learning process can facilitate the progressive learning of new weld profile motion vectors from current data collected on the fly, while implementing various safeguards to prevent previously learned weights from being overwritten by current learning. Safeguards may include, by way of illustrative, non-limiting example, storing large amounts of pre-trained models for joint optimization of all parameters across all updates (multi-task learning) or selectively adjusting and constraining model parameters and learning coefficients. As more data is collected, predictions of weld profile motion vectors can improve. Furthermore, when learning stops, the result is a slice-based welding simulator fine-tuned to the application-specific welding parameter design space. The slice-based welding simulator can consume weld parameters and geometric context to generate correct weld profile motion vectors. Furthermore, the weld profile motion vectors, when applied to the current bead, produce a final bead shape that best approximates how the bead will deform in geometric space that models the effects of important fluid-mechanical behavior.

[0203] FIG. 31D shows a first example 3150 illustrating the nominal fill plan 3100 from the first stage 3120 and observed deposition surface profiles from the second stage 3122 that can be used to update the modeling parameters used to generate the nominal fill plan 3100. Further, a second example 3152 shows the updated nominal fill plan from the fourth stage 3126 and observed deposition surface profiles from the fifth stage 3128 that can be used to update the modeling parameters used to generate the updated nominal fill plan. Stated another way, the profiles from the “weld feedback scan” (e.g., as shown in the second stage 3122 or the fifth stage 3128) are the monitoring signals that drive the improvement of the bead model 173. Using self-supervised learning, these model parameters are adjusted through observation, and ultimately, with sufficient data, the appropriate parameters are learned to generate accurate weld bead shapes and interactions, taking into account the system context. This approach relies on the assumption that current, minimally tuned weld bead models are sufficiently accurate given the relatively few situations in which they are applied in production. However, to unlock adaptive welding at scale, and especially quality at scale, feedback from observed deposition surface profiles can facilitate true optimization of the bead model 173 during welding.

[0204] Referring back to FIG. 1 , during operation of the system 100, the controller 152 can identify a seam 144 to be welded. For example, the seam 144 can be identified for performing a welding operation via multiple welding passes. In some implementations, the seam 144 is defined based on one or more parts, such as a first part 135 and a second part 136. For example, the first part 135 and the second part 136 are configured to be positioned such that the first part 135 and the second part 136 can define the seam 144 along which the first part 135 and the second part 136 will be welded. The controller 152 may identify the seam 144 based on a computer-aided design (CAD) model (e.g., design 170) of one or more parts, scan data (e.g., sensor data 180) associated with a scanning operation, or a combination thereof. By way of example, the controller 152 can receive scan data from one or more sensors, such as the sensor 109. Scan data may include information such as image data captured by one or more sensors.

[0205] In some implementations, the controller 152 can generate a representation, such as a point cloud 169, of one or more parts. For example, the controller 152 may generate a representation for each part of one or more parts, seams 144, or a combination thereof. The representations can be generated based on the design 170, sensor data, or a combination thereof. For example, the controller 152 can generate a first point cloud (e.g., 704) based on the design 170, such as a CAD model, and generate a second point cloud (e.g., 706) based on the scan data. In some implementations, the controller 152 can segment the representation to remove non-part (or non-seam) information. In some such implementations, after segmenting the representation to remove non-part (or non-seam) information, the representation may be referred to as a joint representation or a joint geometry representation.

[0206] In some implementations, the controller 152 can identify one or more characteristics (e.g., one or more dimensions) of the seam 144. For example, the controller 152 may determine the one or more characteristics (e.g., one or more dimensions) based on the representation or the segmented representation. The one or more characteristics may include or represent a depth, width, length, volume, cross-sectional area, or a combination thereof. In some implementations, the controller 152 can determine a weld volume or cross-sectional area associated with the seam.

[0207] In some implementations, the controller 152 can determine one or more characteristics based on the first point cloud (e.g., 704) and the second point cloud (e.g., 706). For example, the controller 152 may perform a comparison based on the first point cloud (generated based on the design 170) and the second point cloud (generated based on the scan data). Based on the comparison, the controller 152 can determine one or more characteristics including or indicative of gap width variability (e.g., gap width deviation), gap depth variability (e.g., gap depth deviation), weld volume variability (e.g., weld volume deviation), or a combination thereof.

[0208] In some implementations, controller 152 may generate multiple waypoints (e.g., 172) associated with seam 144. For example, controller 152 may generate multiple waypoints along seam 144. Illustratively, the multiple waypoints may be spaced along seam 144, such as evenly spaced along seam 144. For example, the multiple waypoints may be spaced based on a bead size of the weld bead, such as an average bead size or a bead size parameter indicated or annotated on design 170.

[0209] In some implementations, the controller 152 can generate a weld profile 174 associated with a cross-section of the seam 144 at one of the multiple waypoints. Additionally or alternatively, the controller 152 can generate a waypoint weld profile 174 for each waypoint of the multiple waypoints. In some implementations, the controller 152 may determine an area (e.g., a weld volume) of the seam 144 in the waypoint weld profile.

[0210] In some implementations, to determine the weld profile (e.g., 174), the controller 152 can generate a model (e.g., a joint model) of the seam 144. The joint model may include a 2D model or a 3D model. The joint model may be generated based on the seam 144. For example, the joint model of the seam 144 may be generated based on the design 170 (e.g., a CAD model) or a first point cloud, or based on scan data or a second point cloud. Illustratively, the controller 152 can determine a plurality of waypoints for the identified seam 144 and can generate a joint model for at least the waypoints of the plurality of waypoints.

[0211] In some implementations, to generate the joint model, the controller 152 can use or access one or more feature components of the plurality of feature components. The plurality of feature components can include or correspond to the joint model information 171. Each feature component of the plurality of feature components includes a set of feature geometries. As an illustrative, non-limiting example, the set of feature geometries can include points, lines, or curves and geometric constraints with tolerances and their interrelationships, such as modeled by a tree, graph, or other generalized, searchable, and solvable structure, or a combination thereof. The controller 152 can generate the joint model of the seam 144 based on or to include a first feature component of the plurality of feature components and a second feature component of the plurality of feature components. Illustratively, the controller 152 can match a first feature component of the plurality of feature components to a cross-section of the seam 144 (determined based on the design 170 or scan data).

[0212] In some implementations, the controller 152 can determine one or more joint parameters based on the joint model. For example, for each feature point in the joint model, the controller 152 may determine or identify a vector for the feature point, another feature point associated with the feature point based on the vector, or a combination thereof. Additionally or alternatively, the controller 152 may determine a fill-in direction (for multiple passes), a fill-out direction (for multiple passes), an ordered sequence of weld beads to be placed for welding the seam 144, a movement direction (of the robot 120), or a combination thereof based on the joint model. In some implementations, the weld profile 174 includes or indicates an ordered sequence of weld beads to be placed for welding the seam 144, a number of layers, a number of beads, a bead size, or a combination thereof.

[0213] In some implementations, to generate the weld profile, the controller 152 determines the number of bead layers. For example, the controller can determine the number of bead layers based on a bead layer height, such as a bead layer height indicated by a user, based on annotated data associated with the design 170, or based on a combination thereof. Additionally or alternatively, the controller 152 may determine the number of weld beads included in the bead layer. In some implementations, the controller 152 can determine or calculate the volume or area of the bead layer, the weld bead, or a combination thereof. For example, the controller 152 may determine or calculate the volume or area based on the bead layer, the bead layer height, the number of bead layers, a joint model, or a combination thereof.

[0214] In some implementations, controller 152 can determine one or more welding parameters for each bead in the welding profile of the waypoint. For example, the one or more welding parameters may include or indicate a welding wire size used to form the weld bead, an area or volume of the weld bead, or a combination thereof. Additionally or alternatively, the one or more welding parameters may include a wire feed speed, a travel speed, or a combination thereof. In some implementations, controller 152 can use or access a table to determine at least one welding parameter of the one or more welding parameters. For example, controller 152 may access a table and determine a wire feed speed, a travel speed, or a combination thereof based on a welding wire size or an area or volume of the weld bead. In some implementations, controller 152 is configured to determine a weld fill plan 175 based on a plurality of welding profiles.

[0215] In some implementations, to generate the weld profile, controller 152 populates a cross-sectional joint model with multiple candidate weld beads. To populate the joint model, controller 152 can determine one or more design parameters based on design 170 (e.g., annotated data), scan data, user input, or a combination thereof. For example, the one or more design parameters may include or indicate the number of layers, bead size, tolerance, or a combination thereof. In some implementations, each candidate weld bead of the multiple candidate weld beads has a size, a position (e.g., location, orientation, or both) within a bead size range, or a combination thereof. Stated differently, each candidate weld bead of the multiple candidate weld beads can have a size characteristic, a spatial characteristic, or a combination thereof. The bead size range may be based on design 170 (e.g., annotated data), user input, known weld quality parameters, or a combination thereof.

[0216] In some implementations, to populate the joint model of the cross section with the plurality of candidate weld beads, controller 152 places a first set of beads of the plurality of candidate weld beads at one or more structural points of the joint model. Additionally or alternatively, controller 152 may place a second set of beads of the plurality of candidate weld beads at one or more structural points of the joint model, place a third set of beads of the plurality of candidate weld beads based on a cover profile of the joint model, place a fourth set of beads of the plurality of candidate weld beads in one or more unfilled spaces of the joint model, or any combination thereof.

[0217] In some implementations, controller 152 generates a weld fill plan 175 for seam 144. For example, controller 152 can generate weld fill plan 175 based on one or more weld profiles 174. Illustratively, weld fill plan 175 can be generated for each waypoint of a plurality of waypoints based on a welding profile associated with the waypoint. Weld fill plan 175 can include or indicate a wire feed rate, a travel rate, a voltage, or a combination thereof, for each weld bead of weld fill plan 175. In some implementations, weld fill plan 175 includes or indicates a multi-pass welding operation. Each pass of a plurality of weld passes (of a multi-pass welding operation) is performed for at least a portion of a weld layer (of a weld).

[0218] In some implementations, the weld fill plan 175 includes or illustrates multiple fill plan weld beads based on multiple candidate weld beads. The multiple fill plan weld beads may correspond to a single weld profile at a single waypoint or multiple weld profiles at multiple waypoints at multiple waypoints. In some implementations, the multiple fill plan weld beads correspond to all waypoints at multiple waypoints. For each weld bead of the multiple fill plan weld beads, the controller 152 can determine a weld bead size. Additionally or alternatively, for each weld bead of the multiple fill plan weld beads, the controller 152 determines a torch angle, weaving or motion characteristics, or a combination thereof, to form the bead. Additionally or alternatively, for each weld bead of the plurality of fill plan weld beads, the controller 152 may determine a bead profile based on, for each of the plurality of weld profiles, illustrative, non-limiting examples, a bead model, a torch angle, a travel speed, a travel angle, a torch speed, gravity, surface tension, a gas mixture, heat input, a voltage or current, a wire feed speed, wire characteristics (wire diameter or wire type—composition / material), a weaving or movement parameter (e.g., a weaving type, a weaving amplitude characteristic, a weaving frequency characteristic, or a phase lag), a contact tip-to-work distance (CTWD) offset, a welding mode (e.g., a waveform), a welding technique (e.g., TIG or MIG), or a combination thereof. In some implementations, the controller 152 can use machine learning to determine the plurality of fill plan weld beads. For example, for each bead of the plurality of fill plan weld beads, the controller 152 can use machine learning and a bead model to determine the torch angle, the wire feed speed, the gas mixture, the voltage, the torch speed, or a combination thereof.

[0219] In some implementations, controller 152 can generate weld fill plan information based on weld fill plan 175 that indicates, by way of illustrative, non-limiting examples, the number of layers, the number of beads in each layer, the bead size, the weld size, the cover profile, the material cost, the average bead size, the minimum bead size, the maximum bead size, the distance of the bead from the construction point, one or more cross sections, or a combination thereof. In some implementations, weld fill plan 175 can be validated based on one or more operating characteristics of the welding robot. Additionally or alternatively, weld fill plan 175 can be validated or approved by a user of system 100.

[0220] In some implementations, the controller 152 can generate instructions, such as welding instructions 176 or control information 182, to place weld material in the seam 144. For example, the controller 152 may generate the welding instructions 176 based on one or more weld profiles (e.g., 174), weld fill plans 175, or a combination thereof. The instructions (e.g., 176) may correspond to a single-pass welding operation or a multi-pass welding operation. The controller 152 can send control information 182 to the robot 120 that includes or indicates the welding instructions 176. The control information 182 may be provided to the robot 120 to cause the robot to perform one or more operations. Additionally or alternatively, the control system 110 (e.g., the controller 152) may send the control information 182 to the robot 120 as welding commands.

[0221] In some implementations, the robot 120 can receive and execute instructions to weld one or more parts. Additionally or alternatively, the robot 120 can execute instructions to scan the seam 144 before, during, or after depositing weld material associated with the seam 144. Scan data can be generated based on the scan of the seam 144 and transmitted to the controller 152. Based on the scan data, the controller 152 can update or generate additional instructions to deposit additional weld material. For example, the controller 152 can receive scan data before depositing weld material for one or more multi-pass welding operations and compare the scan data with the weld fill plan 175. In some implementations, the controller 152 can determine one or more characteristics based on the comparison, such as, by way of non-limiting example, gap width variability (e.g., gap width deviation), gap depth variability (e.g., gap depth deviation), weld volume variability (e.g., weld volume deviation), tack, or a combination thereof. The controller 152 can update or modify the weld fill plan 175 based on the results of the comparison. In some implementations, updating or modifying the weld fill plan 175 may include updating a welding profile for at least one waypoint of the plurality of waypoints associated with the seam. For example, the controller 152 may update the weld bead, wire feed rate, travel speed, voltage, or a combination thereof indicated by the weld profile. Additionally or alternatively, the controller 152 may generate one or more additional welding instructions 176 for the robot 120 based on the updated or modified weld fill plan 175.

[0222] 2, which is a block diagram illustrating another system 200 configured to implement machine learning logic in a robotic manufacturing environment, according to one or more embodiments. System 200 may include or correspond to system 100 of FIG. 1.

[0223] 1, system 200 includes a plurality of robots. Illustratively, the plurality of robots includes four robots, including a first robot (e.g., 120), a second robot 212, a third robot 214, and a fourth robot 216. Furthermore, sensor 109 includes a plurality of sensors, such as a first sensor 234 and a second sensor 236. System 200 also includes a structure 242 and a second tool 222 in addition to the first tool (e.g., 121).

[0224] Workspace 130 of system 200 may include one or more devices or components of system 200. As shown, workspace 130 includes first robot 120, first tool 121, second robot 212, second tool 222, first sensor 234, and manufacturing tool 126. In other implementations, workspace 130 may include fewer or more components or devices than those shown in FIG. 2. For example, workspace 130 may include third robot 214, fourth robot 216, second sensor 236, structure 242, control system 110, or a combination thereof.

[0225] In some implementations, the plurality of robotic devices may include or correspond to robot 120. For example, at least one of the plurality of robotic devices (e.g., 120, 212, 214, 216) may include a robotic arm providing six degrees of freedom, as non-limiting examples. In implementations, the robotic arm may be manufactured by YASKAWA®, ABB® IRB, KUKA®, Universal Robots®. Additionally or alternatively, the robotic arm may be configured to couple to one or more tools.

[0226] The second robot 212 may include a second robotic arm. The second tool 222 may be coupled to the end of the second robotic arm. In some implementations, the second tool 222 may include or correspond to the first tool 121. For example, the second tool 222 may be configured to selectively couple to a second set of one or more objects including the second part 136. The second set of one or more objects may be the same as or different from the first set of objects to which the first tool 121 is configured to couple.

[0227] The third robot 214 may include a third robotic arm. The first sensor 234 may be coupled to an end of the third robotic arm. In some implementations, the first sensor 234 is configured to generate first sensor data (e.g., 180). For example, the first sensor 234 is configured to capture one or more images of the first part 135, the second part 136, or a combination thereof.

[0228] The fourth robot 216 includes a fourth robot arm. A manufacturing tool 126 (e.g., a welding tool) is coupled to the end of the fourth robot arm.

[0229] The second sensor 236 is configured to generate second sensor data (e.g., 180). For example, the second sensor 236 is configured to capture one or more images of the first part 135, the second part 136, or a combination thereof. In some implementations, the second sensor 236 is disposed on or coupled to a structure 242. The structure 242, such as a frame or weldment, may be dynamic or static. In either the dynamic or static configuration of the structure 242, the second sensor 236 may be configured to be dynamic or static relative to the structure 242; for example, if the second sensor 236 is dynamic, the second sensor may be configured to rotate (e.g., pan) or tilt.

[0230] 8, which is a schematic diagram of an autonomous robotic welding system 800 according to one or more embodiments. System 800 may include or correspond to system 100 or system 200.

[0231] System 800 includes workspace 801. Workspace 801 may include or correspond to workspace 130. In some implementations, workspace 801 includes one or more sensors 802, a robot 810, and one or more fixtures 816. One or more sensors 802 may include or correspond to sensor 109, first sensor 234, or second sensor 236. In some implementations, one or more sensors 802 may include a movable sensor. For example, at least one sensor of one or more sensors 802 may be coupled to or included in robot 810. Robot 810 may include or correspond to robot 120, 212, 214, or 216. One or more figures 816 may include or correspond to fixture 127. System 800 may also include a UI 806 coupled to workspace 801. UI 806 may include or correspond to UI and display adapter 106. Although workspace 801 is described as including one or more sensors 802, robot 810, and one or more fixtures 816, in other implementations workspace 801 may or may not optionally include one or more of sensors 802, robot 810, or fixtures 816. Additionally or alternatively, system 800 may include one or more additional components, such as control system 110 or components thereof.

[0232] The robot 810 may include multiple joints and members (e.g., shoulders, arms, elbows, etc.) that allow the robot 810 to move with any suitable number of degrees of freedom. Additionally or alternatively, the robot 810 includes a welding head 810A that performs a welding operation on a part. For example, the part (e.g., 135, 136, 502, 504, 602, or 604) may be supported by a fixture 816, such as a clamp.

[0233] During operation of system 800, one or more sensors 802 capture one or more images of workspace 801. In some implementations, the one or more images include image data. The one or more sensors 802 can provide the one or more images to a controller (not shown in FIG. 8 ). For example, the controller may include or correspond to controller 152. The controller may generate one or more 3D representations (e.g., one or more point clouds) of workspace 801. For example, the one or more point clouds may include or correspond to one or more fixtures 816, parts supported by the one or more fixtures 816, and / or other structures within workspace 801. The controller can identify a seam, such as seam 144, based on the 3D representation. For example, the seam may include or correspond to a part (e.g., 135 or 136) supported by one or more fixtures 816. Additionally or alternatively, the controller may plan a path for the robot 810 to weld the seam without colliding with structures in the workspace 801 based on the 3D representation, and control the robot 810 to weld the seam.

[0234] 9 , which is a flow diagram illustrating an example process 900 for generating welding instructions for a welding robot according to one or more embodiments. The operations of process 900 may be performed by a control system or controller (collectively referred to as a “controller” with reference to FIG. 9 ), such as control system 110, controller 152, or processor 101. For example, the example operations (also referred to as “blocks”) of process 900 may enable the controller to generate welding instructions for the welding robot. The welding instructions may include or correspond to control information 182. The welding robot may include or correspond to robot 120, 212, 214, 216, or 810.

[0235] In block 902, the controller obtains workspace information. For example, the workspace may include or correspond to workspace 130 or 801. The information may include or correspond to sensor data 180. The information may be received by the controller from one or more sensors, such as sensors 109, 234, 236, or 802. In some implementations, the information includes image data, such as an image of the workspace. Additionally or alternatively, the workspace may include or be positioned within one or more components (e.g., 135, 136, 502, 504, 602, or 604), one or more fixtures (e.g., 127, 816), or a combination thereof. The one or more fixtures, such as clamps, may be configured to securely hold one or more components.

[0236] In some implementations, the controller may be configured to generate a point cloud based on the information. For example, the point cloud may include or correspond to point clouds 169, 500, 600, or 706. Illustratively, the image data (or multiple images) may be overlaid on one another to reconstruct and generate three-dimensional image data. The three-dimensional image data may be collated together to generate the point cloud.

[0237] At block 904, the controller identifies a set of points on the parts to be welded. For example, the controller can identify the set of points based on the information. In some implementations, the set of points can represent a potential seam to be welded. For example, the seam may include or correspond to seam 144, 506, or 606. In some implementations, the controller can perform pixel-wise segmentation on the image data using a neural network to identify the set of points. Additionally or alternatively, one or more fixtures or one or more clamps can be classified based on the image data by the neural network, such as by using one or more image classification techniques. Portions of the image data associated with one or more fixtures and / or one or more clamps can be segmented such that those portions of the image data are not used to identify the set of points. Segmenting one or more portions of the image data can reduce the computational resources required to identify the set of points to be welded by reducing the search space. In some examples, the set of points can be identified from other portions of the image data (e.g., portions of the image data that are not segmented).

[0238] At block 906, the controller identifies candidate seams. For example, the candidate seams may be identified based on the set of points. The candidate seams may include or correspond to seams 144, 506, or 606. For example, a subset of points in the set of points may be identified as candidate seams. In some implementations, the controller may perform image classification and / or depth classification using a neural network to identify the candidate seams. In some examples, the candidate seams may be located relative to the part. Illustratively, the position and orientation of the candidate seams may be determined relative to the part to locate the candidate seams.

[0239] In some implementations, process 900 may include verifying whether the candidate seam is an actual seam. For example, the controller may be configured to verify whether the candidate seam is an actual seam. For example, information, such as image data from one or more sensors, may be based on multiple angles (e.g., fields of view) of the workspace. For each image captured from a different angle, the controller may determine a confidence value representing whether the candidate seam determined from that angle is an actual seam. The seam may be verified as an actual seam based on the confidence value being equal to or greater than a threshold. In some examples, after the candidate seam is identified and verified, a subset of points may be clustered together to form a seamless, continuous seam.

[0240] Additionally or alternatively, process 900 may include classifying the candidate seam (or the determined actual seam) as a seam type. For example, the controller may be configured to classify the candidate seam or the actual seam as one type of seam out of a plurality of types of seams. Illustratively, the controller may use a neural network to determine whether the candidate seam or the actual seam is a butt seam, a corner seam, an edge seam, a lap seam, a T-joint, another type of seam, or a combination thereof. In some implementations, the controller classifies the seam (e.g., the actual seam) after determining that the candidate seam is an actual seam.

[0241] At block 908, the controller generates welding instructions for the welding robot. For example, the controller may generate the welding instructions by tracing a path from one end of the subset of points to the other end of the subset of points. A seam path can be generated by tracing a path from one end of the subset of points to the other end of the subset of points. In other words, the path identified by the controller may correspond to a path of a welding head for welding at the seam. Furthermore, path planning can be performed based on the identified and located candidate seams. For example, path planning can be performed based on seam paths that can be generated from clustering the subset of points. In some implementations, the controller can execute the path planning logic 105 to perform the path planning.

[0242] In some implementations, the welding instructions can be based on the type of seam (e.g., butt joint, corner joint, edge joint, lap joint, T-joint, etc.). Additionally or alternatively, the welding instructions can be updated based on input from a user via a user interface (e.g., user interface and adapter 106 or UI 806). For example, a user can select a candidate seam to be welded from all available candidate seams via the user interface. In some implementations, a user can select a seam from identified actual seams. Path planning can be performed on the selected candidate seam, and welding instructions can be generated for the selected candidate seam. In some examples, a user can update welding parameters via the user interface. The welding instructions can be updated based on the updated welding parameters.

[0243] Based on process 900, a welding robot can be operated and controlled by performing without prior information of one or more parts to be welded. For example, the prior information can include or correspond to a CAD model, such as design 170. As one or more parts can be scanned to generate welding instructions, representations of the scanned image(s) of the one or more parts can be annotated (e.g., via a user interface) with one or more candidate seams. The annotated representations can be used to define a 3D model of the part. The 3D model of the part can be stored in a database for subsequent welding of additional instances of the part. For example, the database can include or correspond to database 112.

[0244] 10 , which is a flow diagram illustrating an example process 1000 for generating welding instructions for a welding robot according to one or more aspects. The operations of process 1000 may be performed by a control system or controller (collectively referred to as a “controller” with reference to FIG. 10 ), such as control system 110, controller 152, or processor 101. For example, the example operations (also referred to as “blocks”) of process 1000 may enable the controller to generate welding instructions for the welding robot. The welding instructions may include or correspond to control information 182. The welding robot may include or correspond to robot 120, 212, 214, 216, or 810.

[0245] In block 1002, the controller identifies expected portions and expected orientations of candidate seams based on the CAD model. For example, the candidate seams may include or correspond to seams 144, 506, or 606. The CAD model may include or correspond to design 170. The CAD model may include or represent one or more parts, such as parts 135, 136, 502, 504, 602, or 604. The expected orientations and expected locations may be determined using annotations included in the CAD model. For example, the annotations may be provided by a user / operator / designer of the CAD model or by a robotic system, such as system 100, 200, or 800. In some implementations, the controller identifies one or more candidate seams based on model geometry. For example, the controller may perform object matching to match components or features on the parts to components or features included in or indicated by the CAD model. In other words, the expected locations and orientations of candidate seams may be identified based on object matching.

[0246] In block 1004, the controller scans the workspace. Illustratively, the controller can command or initiate one or more sensors to scan a portion or the entire workspace. For example, the workspace may include or correspond to workspace 130 or 801. The scan of the workspace can generate information, such as sensor data 180, received by the controller. The information may include or correspond to sensor data 180. The information may be received by the controller from one or more sensors, such as 109, 234, 236, or 802. In some implementations, the information includes image data, such as an image of the workspace. Additionally or alternatively, the workspace may include or be positioned within one or more components (e.g., 135, 136, 502, 504, 602, or 604), one or more fixtures (e.g., 127, 816), or a combination thereof. The one or more fixtures, such as clamps, may be configured to securely hold one or more components.

[0247] In some implementations, the controller may be configured to generate a point cloud based on the information. For example, the point cloud may include or correspond to point clouds 169, 500, 600, or 706. Illustratively, the image data (or multiple images) may be overlaid on one another to reconstruct and generate three-dimensional image data. The three-dimensional image data may be collated together to generate the point cloud.

[0248] In some implementations, to reduce processing time for generating welding instructions, the scan performed by the sensor may be a partial scan of the workspace. Stated differently, instead of scanning the workspace from every angle, the sensor scans a portion of the workspace such that information (e.g., image data) is collected from one or more angles, such as one or more angles at which a candidate seam is expected to be visible. In such an example, the point cloud generated from the image data is a partial point cloud. For example, generating a partial point cloud that does not include portions of a part indicating that the model does not include the seam to be welded can reduce scanning and / or processing time.

[0249] At block 1006, the controller identifies candidate seams. For example, the controller may identify the candidate seams based on information received from sensors, image data, point clouds, and / or subpoint clouds. For example, the seams may include or correspond to seams 144, 506, or 606. By way of example, the candidate seams may be identified as described herein with reference to at least FIGS. 1 and 9.

[0250] In block 1008, the controller determines an actual location and an actual orientation of the candidate seam. For example, the controller may identify a first subset of points and a second subset of points. Illustratively, the controller may identify a first subset of points of the modeled seam in block 1002. The controller may identify the second set of points as the candidate seam in block 1006. In some implementations, the first subset of points and the second subset of points may be compared. Illustratively, the controller may compare the first subset of points and the second subset of points as described with at least reference to FIG. 7 . For example, the first subset of points and the second subset of points may include or correspond to the CAD model point cloud 704 and the scan point cloud 706, respectively. In some implementations, the first subset of points may be allowed to deform to determine the actual location and orientation of the candidate seam.

[0251] In some implementations, a comparison between the first subset of points and the second subset of points can be used to determine a tolerance for the first subset of points (e.g., a predicted location and a predicted orientation of the candidate seam). In some such implementations, the first subset of points can be allowed to deform based on the tolerance to determine an actual location and orientation of the candidate seam. Stated differently, the predicted location and predicted orientation of the candidate seam can be refined (in some examples, based on the tolerance) to determine an actual location and actual orientation of the candidate seam. This deformation / refinement technique can account for surface topography on the part that is not accurately represented in the CAD model of the part.

[0252] In block 1010, the controller generates welding instructions for the welding robot. For example, the welding instructions may be generated based on the actual positions and actual orientations of the candidate seams. In some implementations, the controller can perform path planning based on the actual positions and actual orientations of the candidate seams to determine the welding instructions.

[0253] In some implementations, similar to at least one or more operations described with reference to process 900, once the actual location and actual orientation of the candidate seam are identified, process 1000 may include a controller validating the candidate seam. However, in some other implementations, in contrast to process 900, process 1000 may not include user interaction. By way of example, user interaction may be required because one or more seams to be welded may already be annotated in a CAD model. Thus, in some instances, a welding robot may be operated and controlled by performing process 1000 without any user interaction.

[0254] In some implementations, a controller, such as controller 152, can initiate a scan of the workspace such that one or more sensors (e.g., sensors coupled to a welding robot) scan at least a portion of the workspace. The portion of the workspace may include a part to be welded, and the controller may generate a representation of the part (e.g., a point cloud representation) based on the scan. In some implementations, the controller is provided with or has access to an annotated CAD model of the part. The controller can determine an expected location and an expected orientation of a candidate seam on the part according to (or based on) the CAD model of the part and the representation of the part. For example, the controller can identify the candidate seam based on model geometry; for example, the controller can perform object matching to match components or features (e.g., topographical features) on the representation of the part to components or features in the CAD model. The controller can use the results of the object matching to determine an expected location and an expected orientation of the candidate seam on the part. After the expected location and orientation are determined, the controller can determine an actual location and an actual orientation of the candidate seam based at least in part on the representation of the part.

[0255] 11 , which is a flow diagram illustrating an example process 1100 for operating a welding robot according to one or more aspects. The operations of process 1100 may be performed by a control system or controller (collectively referred to as a “controller” with reference to FIG. 11 ), such as control system 110, controller 152, or processor 101. For example, the example operations (also referred to as “blocks”) of process 1100 may enable the controller to generate welding instructions for the welding robot. The welding instructions may include or correspond to control information 182. The welding robot may include or correspond to robot 120, 212, 214, 216, or 810.

[0256] In some implementations, a welding robot may be configured to perform a manufacturing task (e.g., welding) on one or more parts disposed within a workspace. For example, the workspace may include or correspond to workspace 130 or 801. The workspace may include or be positioned within one or more parts (e.g., 135, 136, 502, 504, 602, or 604), one or more fixtures (e.g., 127, 816), or a combination thereof. The one or more fixtures, such as clamps, may be configured to securely hold the one or more parts.

[0257] In block 1110, the controller scans the part (of the one or more parts). In some implementations, the controller can command or initiate one or more sensors to scan a portion or a part or the entire workspace. For example, the scan of the part (or workspace) can generate information, such as sensor data 180, that is received by the controller. The information can include or correspond to sensor data 180. The information may be received by the controller from one or more sensors, such as 109, 234, 236, or 802. In some implementations, the information includes image data, such as an image of the part or workspace. In some implementations, the one or more scanners can be coupled to or included in the robot.

[0258] In block 1112, the controller determines a position of one or more parts within the workspace. The controller can determine the position based on the information.

[0259] At block 1114, the controller identifies seams, such as candidate seams or actual seams. For example, the controller may use image data acquired from a sensor and / or a point cloud derived from the image or sensor data to identify one or more seams on the part. The point cloud may include or correspond to point clouds 169, 500, 600, or 706.

[0260] In block 1116, the controller determines a path for the manufacturing robot to move along the seam. For example, once the part and seam locations are determined, the controller can plot a path for the manufacturing robot along the identified seam. In some implementations, the controller may plot the path using path planning logic 105. The plotted path includes optimized motion parameters for the manufacturing robot to complete the weld without colliding with itself or anything else in the workspace. In some implementations, no human input is required in generating the optimized motion parameters for the robot to complete the weld.

[0261] Although aspects of the present application and their advantages have been described in detail, it should be understood that various changes, substitutions, and alterations can be made herein without departing from the spirit and scope of the present disclosure, as defined by the appended claims. Moreover, the scope of the present application is not limited to the particular implementations of the processes, machines, manufacture, compositions of matter, means, methods, and steps described herein. As one skilled in the art will readily understand from the above disclosure, existing or later-developed processes, machines, manufacture, compositions of matter, means, methods, or steps can be utilized that perform substantially the same function or achieve substantially the same result as the corresponding implementations described herein. Accordingly, it is intended that the appended claims include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or steps.

[0262] The above specification provides a complete description of the structure and use of the exemplary configurations. While particular configurations have been described above with a certain degree of particularity or with reference to one or more individual configurations, those skilled in the art may make numerous modifications to the disclosed configurations without departing from the scope of the present disclosure. Therefore, the various exemplary configurations of methods and systems are not intended to be limited to the particular forms disclosed. Rather, they include all modifications and alternatives that fall within the scope of the claims, and configurations other than those shown may include some or all of the features of the illustrated configurations. For example, elements may be omitted or combined into a single structure, connections may be substituted, or both. Furthermore, where appropriate, aspects of any of the above-described examples may be combined with aspects of any of the other described examples to form further examples having equivalent or different characteristics and / or functionality and addressing the same or different problems. Similarly, it will be understood that the benefits and advantages discussed above may relate to one configuration or to several configurations. Thus, any single implementation described herein should not be construed as limiting, and implementations of the present disclosure may be combined as appropriate without departing from the teachings of the present disclosure.

[0263] While various implementations have been described above, it should be understood that they have been presented by way of example only, and not by way of limitation. While various implementations have been described as having particular combinations of features and / or components, combinations of any feature and / or component from any of the examples, as well as other implementations having additional features and / or components, are possible, where appropriate.

[0264] Certain features described herein in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Where the methods described above show certain events occurring in a particular order, the order of certain events may be changed. Furthermore, some of the events may be performed simultaneously in parallel processes where possible, or may be performed sequentially as described above.

[0265] Similarly, although acts are depicted in the figures in a particular order, this should not be understood as requiring that such acts be performed in the particular order shown, or in any sequential order, or that all illustrated acts be performed, to achieve desirable results. Furthermore, the figures may generally depict one or more exemplary processes in the form of a flow chart. However, other acts not shown may be incorporated into the generally depicted exemplary process. For example, one or more additional acts may be performed before, after, simultaneously with, or between any of the illustrated acts. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the above implementations should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Furthermore, several other implementations are within the scope of the following claims. In some cases, the acts recited in the claims can be performed in a different order and still achieve desirable results.

[0266] Those skilled in the art will appreciate that information, messages, and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, and signals that may be referred to throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0267] The components, functional blocks, and modules described herein in connection with the figures include, among other examples, processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software code, firmware code, or any combination thereof. Software shall be construed broadly to mean, among other examples, instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, whether referred to as software, firmware, middleware, microcode, hardware description languages, etc. Additionally, features described herein may be implemented via dedicated processor circuitry, via executable instructions, or via combinations thereof.

[0268] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithmic steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and processes have been described above generally in terms of their functionality. The various illustrative logic, logical blocks, modules, circuits, and algorithmic processes described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and their structural equivalents. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by or to control the operation of a data processing apparatus.

[0269] The hardware and data processing devices used to implement the various example logic, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using general-purpose single or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. In some implementations, particular processes and methods may be performed by circuitry specific to a given function.

[0270] If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium. The processes of a method or algorithm disclosed herein may be implemented in a processor-executable software module, which may reside on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that may be enabled to transfer a computer program from one place to another. A storage medium may be any available medium that can be accessed by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection may be properly referred to as a computer-readable medium. Furthermore, the operations of a method or algorithm may reside as one or any combination or set of code and instructions on a machine-readable medium and a computer-readable medium, which may be incorporated into a computer program product.

[0271] Some implementations described herein relate to methods or processing events. It should be understood that such methods or processing events may be computer-implemented. That is, when methods or other events are described herein, it should be understood that they may be performed by a computing device having a processor and memory. The methods described herein may be performed locally, e.g., on a computing device physically co-located with the robot or a local computer / controller associated with the robot, and / or remotely, such as on a server and / or in the "cloud."

[0272] The memory of a computing device, also referred to as a non-transitory computer-readable medium, may contain instructions or computer code for performing various computer-implemented operations. The computer-readable medium (or processor-readable medium) is non-transitory in the sense that it does not include a transitory propagating signal itself (e.g., a propagating electromagnetic wave carrying information over a transmission medium such as space or a cable). The medium and computer code (which may also be referred to as code) may be designed and constructed for one or more specific purposes. Examples of non-transitory computer-readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, and magnetic tape; compact disks / digital video disks (CDs / DVDs), compact disk-read-only memories (CD-ROMs), and holographic devices; magneto-optical storage media such as optical disks; carrier wave signal processing modules; read-only memories (ROMs); and random access memories (RAMs). One or more processors may be communicatively coupled to the memory and operable to execute the code stored on the non-transitory processor-readable medium. Examples of processors include general-purpose processors (e.g., CPUs), graphical processing units, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), etc. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those generated by a compiler, code used to generate web services, and files containing higher-level instructions executed by a computer using an interpreter. By way of illustration, examples may be implemented using imperative programming languages (e.g., C, Fortran, etc.), functional programming languages (Haskell, Erlang, etc.), logic programming languages (e.g., Prolog), object-oriented programming languages (e.g., Java, C++, etc.), or other suitable programming languages and / or development tools. Further examples of computer code include, but are not limited to, control signals, encryption code, and compression code.

[0273] As used herein, various terms are intended to describe particular implementations only and are not intended to limit the implementations. For example, as used herein, ordinal numbers (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not in themselves indicate any priority or order of the element relative to another element, but merely distinguish the element from another element having the same name (except for the use of the ordinal number). The term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically, and two items that are "coupled" may be unitary with one another. The terms "a" and "an" are defined as one or more, unless this disclosure explicitly requires otherwise.

[0274] As used herein, the term "about" allows for some degree of variability in values or ranges, e.g., within 10%, 5%, or 1% of a stated value or a stated range limit, and includes the exactly stated value or range. The term "substantially" is defined as most, but not necessarily all, of what is specified (including what is specified, e.g., substantially 90 degrees includes 90 degrees, and substantially parallel includes parallel), as understood by those skilled in the art. In any disclosed implementation, the term "substantially" can be substituted with "within [percentage]" of what is specified, where percentages include 1, 1, 5, or 10 percent, and the term "approximately" can be substituted with "within 10 percent" of what is specified. The phrase "substantially X to Y" has the same meaning as "substantially X to substantially Y" unless otherwise indicated. Similarly, the phrase "substantially X, Y, or substantially Z" has the same meaning as "substantially X, substantially Y, or substantially Z" unless otherwise indicated. Unless otherwise indicated, the word "or" as used herein is an inclusive "or" or is interchangeable with "and / or," and when "or" is used in a list of two or more items, it means that any one of the listed items can be used alone, or any combination of two or more of the listed items can be used. Illustratively, A, B, and / or C includes A alone, B alone, C alone, A and B in combination, A and C in combination, B and C in combination, or A, B, and C in combination. Similarly, the phrase "A, B, C, or combinations thereof" or "A, B, C, or any combination thereof" includes A alone, B alone, C alone, A and B in combination, A and C in combination, B and C in combination, or A, B, and C in combination.

[0275] It should be understood that throughout this specification, values expressed in range format should be interpreted flexibly to include not only the numerical values explicitly recited as the limits of the range, but also all of the individual numerical values or subranges subsumed within that range, as if each numerical value and subrange were explicitly recited. For example, a range of "about 0.1% to about 5%" or "about 0.1% to 5%" should be interpreted to include not only about 0.1% to about 5%, but also individual values (e.g., 1%, 2%, 3%, and 4%) and subranges (e.g., 0.1% to 0.5%, 1.1% to 2.2%, 3.3% to 4.4%) within the recited range.

[0276] The terms "comprise" (and any form of "comprise", such as "comprises" and "comprising"), "have" (and any form of "have", such as "has" and "having"), "include" (and any form of "include", such as "includes" and "including"), and "contain" (and any form of "contain", such as "containing"). Consequently, an apparatus that "comprises", "has", "includes", or "contains" one or more elements may possess those one or more elements, but is not limited to possessing only those one or more elements. Similarly, a method that "comprises", "has", "includes", or "contains" one or more steps may possess those one or more steps, but is not limited to possessing only those one or more steps.

[0277] Any implementation of any of the systems, methods, and articles of manufacture can consist of, or consist essentially of, any of the described steps, elements, or features, rather than comprising / having / including any of the described steps, elements, or features. Thus, in any of the claims, the terms "consisting of" or "essentially consisting of" can be substituted for any of the open-ended linking verbs listed above to modify the scope of a given claim from one that would otherwise use an open-ended linking verb. Furthermore, the term "wherein" can be used interchangeably with "where."

[0278] Furthermore, a device or system configured in a certain way is configured in at least that way, but may also be configured in a way other than as specifically described. One or more features of one implementation may apply to other implementations even if not described or illustrated, unless expressly prohibited by this disclosure or the nature of the implementation.

[0279] The claims are not intended to, and should not be construed to, include means-plus or step-plus-function limitations unless such limitations are expressly recited in a given claim using the phrase(s) "means for" or "step for," respectively.

[0280] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the scope of the disclosure and the following claims is not intended to be limited to the examples and designs described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. 1. A system for generating instructions for a welding robot, the system comprising: a controller associated with the welding robot, the controller comprising: identifying a seam to be welded, the seam being defined based on the first part and the second part; generating a plurality of waypoints along the length of the seam; generating a joint model of a cross section of the seam for at least one waypoint of the plurality of waypoints based on a plurality of feature components; generating a weld fill plan for the seam based on the cross section at the at least one waypoint and based on a bead model; generating instructions for the welding robot to perform one or more welding passes of the plurality of welding passes based on the weld fill plan; and initiating execution of the instructions to cause the welding robot to perform the one or more welding passes of a plurality of welding passes.

2. a storage device configured to store computer-aided design (CAD) models of the first part and the second part; The system of claim 1 further comprising the welding robot.

3. 2. The system of claim 1, wherein to generate the joint model, the controller is configured to determine one or more joint feature components based on a seam to be welded, the seam being defined based on a first part and a second part.

4. 2. The system of claim 1, wherein to generate the weld fill plan, the controller is configured to determine, for at least one waypoint of a plurality of waypoints along a length of the seam, a number of layers and a number of weld beads for a cross section of a joint at the at least one waypoint to fill the seam.

5. To generate the weld fill plan, the controller: generating a plurality of candidate packing plans; The system of claim 1 , configured to select one candidate fill plan of the plurality of candidate fill plans as the weld fill plan.

6. To generate the weld fill plan, the controller is configured to determine, for each candidate fill plan among the plurality of candidate fill plans, a cost associated with the candidate fill plan; The system of claim 5 , wherein the one candidate filling plan is selected based on the determined costs of the plurality of candidate filling plans.

7. The controller receiving scan data associated with the seam from one or more sensors; Identifying a weld volume associated with the seam based on the scan data; performing a comparison based on the weld fill plan and the identified weld volume; The system of claim 1 , wherein the instructions are generated or updated based on the results of the comparison.

8. further comprising one or more sensors configured to generate sensor data associated with the seam; The system of claim 1 , wherein the sensor data is generated before or after performance of at least one welding pass of the one or more welding passes.

9. The system of claim 8 , wherein the sensor data includes data associated with the formation of a weld bead based on the at least one welding pass, data associated with the formed weld bead, or a combination thereof.

10. The controller determining one or more properties of a weld deposit formed based on the welding pass; updating the weld fill plan based on the determined one or more characteristics; The system of claim 9 , configured to generate additional instructions for the welding robot to perform at least one welding pass based on the updated weld fill plan.

11. The system of claim 10 , wherein the controller is configured to update the bead model based on the determined one or more characteristics.

12. 1. A computer-implemented method for generating instructions for a welding robot, the computer-implemented method comprising: identifying a seam to be welded, the seam being defined based on the first part and the second part; For at least one waypoint of a plurality of waypoints along a length of the seam, determining a number of layers and a number of weld beads for a cross section of a joint at the at least one waypoint to fill the seam; generating a weld fill plan for the seam based on the cross section at the at least one waypoint and based on a bead model; generating instructions for the welding robot to perform one or more welding passes based on the weld fill plan; A computer-implemented method comprising:

13. receiving computer-aided design (CAD) models of a first part and a second part, the seam being defined based on the first part and the second part; determining one or more design parameters based on the CAD model; The computer-implemented method of claim 12 , wherein the one or more design parameters include a number of layers, a bead size, a tolerance, or a combination thereof.

14. generating a joint model of the cross section based on a plurality of feature components; and populating the joint model of the cross section with a plurality of candidate weld beads, each candidate weld bead of the plurality of candidate weld beads comprising: Size within the bead size range, Location, including position, orientation, or both; or 13. The computer implemented method of claim 12, comprising a combination thereof.

15. Populating the joint model of the cross section with a plurality of candidate weld beads includes: The computer-implemented method of claim 14 , comprising placing a first set of beads of the plurality of candidate weld beads at one or more structural points of the joint model.

16. Populating the joint model of the cross section with a plurality of candidate weld beads includes: a second set of beads from the plurality of candidate weld beads at one or more auxiliary structure points of the joint model; a third set of beads of the plurality of candidate weld beads based on a cover profile of the joint model; a fourth set of beads of the plurality of candidate weld beads in one or more unfilled spaces of the joint model; or 16. The computer-implemented method of claim 15, comprising disposing the image data into a plurality of rows and a plurality of columns.

17. Generating the weld fill plan for the seam comprises: determining a plurality of fill plan weld beads based on the plurality of candidate weld beads; and for each weld bead of the plurality of fill plan weld beads, determining a size of the weld bead.

18. Generating the weld fill plan for the seam comprises: For each weld bead of the plurality of fill schedule weld beads: determining a torch angle, weaving or movement characteristics, or a combination thereof, to form the weld bead; determining a bead profile based on the bead model, the torch angle, torch speed, gravity, surface tension, gas mixture, voltage, wire feed speed, weaving motion, contact tip-to-work distance (CTWD) offset, welding mode, or combinations thereof.

19. Generating the weld fill plan for the seam comprises:

20. The computer-implemented method of claim 18, comprising using machine learning to determine the plurality of fill plan weld beads and to determine one or more welding parameters for each bead of the plurality of fill plan weld beads.

20. The computer-implemented method of claim 12 , further comprising validating the weld fill plan based on one or more operating characteristics of the welding robot.

21. transmitting the command to the welding robot; 13. The computer-implemented method of claim 12, further comprising: for each welding pass of a plurality of welding passes, receiving sensor data based on a weld bead formed by the welding pass, wherein the sensor data includes data associated with formation of the weld bead, data associated with the formed weld bead, or a combination thereof.

22. outputting weld fill plan information based on the weld fill plan; 13. The computer-implemented method of claim 12, wherein the weld fill plan information indicates a number of layers, a number of beads in each layer, a bead size, a weld size, a cover profile, a material cost, an average bead size, a minimum bead size, a maximum bead size, a distance of a bead from a construction point, one or more cross sections, or a combination thereof.

23. 1. A computer-implemented method for generating instructions for a welding robot, the computer-implemented method comprising: receiving a weld fill plan for a seam welded via multiple weld passes, the seam being defined based on a first part and a second part; identifying a weld volume associated with the seam based on scan data received from one or more sensors; generating instructions for the welding robot to perform the plurality of welding passes to apply welding material within the weld volume, the instructions being generated based on a comparison performed using the weld fill plan and the identified weld volume; A computer-implemented method comprising:

24. receiving the scan data from the one or more sensors, the scan data including image data associated with one or more images captured by the one or more sensors; generating a representation of the first part, the second part, the seam, or a combination thereof based on the scan data; 24. The computer-implemented method of claim 23, further comprising: segmenting a joint representation associated with the seam from the representation.

25. determining one or more characteristics of the joint representation; 25. The computer-implemented method of claim 24, wherein the one or more characteristics include gap width, gap variability, gap deviation, weld volume deviation, tack, or a combination thereof.

26. updating the weld fill plan based on the determined one or more characteristics; 26. The computer implemented method of claim 25, wherein updating the weld fill plan includes, for at least one waypoint of a plurality of waypoints associated with the seam, updating a weld profile for the at least one waypoint.

27. 27. The computer-implemented method of claim 26, wherein updating the weld profile comprises updating one or more welding parameters for the weld bead indicated by the weld profile.

28. 1. A computer-implemented method for generating instructions for a welding robot, the computer-implemented method comprising: identifying a seam to be welded, the seam being defined based on the first part and the second part; generating a plurality of waypoints along the length of the seam; generating a joint model of a cross section of the seam for at least one waypoint of the plurality of waypoints based on a plurality of feature components; A computer-implemented method comprising:

29. each feature component of the plurality of feature components includes a set of feature geometries; 30. The computer-implemented method of claim 28, wherein the joint model includes a first feature component of the plurality of feature components and a second feature component of the plurality of feature components.

30. determining the cross section of the seam at the at least one waypoint; accessing the plurality of feature components; 30. The computer-implemented method of claim 29, further comprising matching a first feature component of the plurality of feature components to the cross-section of the seam.

31. Regarding the joint model, identifying a joint constraint value based on the set of feature geometries of at least one of the plurality of feature components; or solving a joint feature geometry from the one or more identified joint constraint values; 30. The computer-implemented method of claim 29, further comprising determining a fill-in direction, a fill-out direction, a movement direction, or a combination thereof based on the joint model.

32. 1. A computer-implemented method for generating instructions for a welding robot, the computer-implemented method comprising: determining one or more joint features based on a seam to be welded, the seam being defined based on a first part and a second part; determining a weld fill plan for the seam, the weld fill plan including one or more weld beads, each weld bead of the one or more weld beads including one or more properties determined based on the one or more joint characteristics; generating one or more weld command characteristics based on the one or more joint feature components, the one or more characteristics of the one or more weld beads, or a combination thereof; A computer-implemented method comprising:

33. 33. The computer-implemented method of claim 32, wherein the one or more characteristics of the one or more beads include a sequence order, a size characteristic, a spatial characteristic, or a combination thereof.

Images