Autonomous Welding Robot

Abstract

In some examples, an autonomous robotic welding system includes a workspace including a part having a seam, a sensor configured to capture a plurality of images within the workspace, a robot configured to apply a weld along the seam, and a controller, the controller configured to identify the seam on the part within the workspace based on the plurality of images, plan a path for the robot to follow when welding the seam, the path including a plurality of different configurations of the robot, and instruct the robot to weld the seam according to the planned path. TIFF2024508564000003.tif157129

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 153,109, filed February 24, 2021, entitled "SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT," and U.S. Provisional Patent Application No. 63 / 282,827, filed November 24, 2021, entitled "SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT," the entire contents of each of which are incorporated herein by reference for all purposes. [Background technology]

[0002] background Robotic manufacturing involves the use of robots to perform one or more aspects of a manufacturing process. Robotic welding is one application in the field of robotic manufacturing. In robotic welding, a robot welds two or more components together along one or more seams. Such robots automate processes that would normally be performed by humans or by machines directly controlled by humans, offering significant advantages in production time, reliability, efficiency, and cost. Summary of the Invention

[0003] overview In various examples, a computer-implemented method for generating instructions for a welding robot includes identifying expected locations and expected orientations of candidate seams on a part to be welded based on a computer-aided design (CAD) model of the part, scanning a workspace including the part to generate a representation of the part, identifying candidate seams on the part based on the representation of the part and the expected locations and expected orientations of the candidate seams, determining actual locations and actual orientations of the candidate seams, and generating welding instructions for the welding robot based at least in part on the actual locations and actual orientations of the candidate seams.

[0004] In an example, a computer-implemented method for generating welding instructions for a welding robot includes acquiring, via a sensor, image data of a workspace including parts to be welded, identifying a plurality of points on the parts to be welded based on the image data, identifying candidate seams on the parts to be welded from the plurality of points, and generating welding instructions for the welding robot based at least in part on the identification of the candidate seams. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram of an autonomous robotic welding system, according to various examples. [Figure 2] FIG. 1 is a schematic diagram of an autonomous robotic welding system, according to various examples. [Figure 3] FIG. 1 is a schematic diagram of an autonomous robotic welding system, according to various examples. [Figure 4] 1 is an exemplary point cloud of a part having a weldable seam, in accordance with various examples. [Figure 5] 1 is an exemplary point cloud of a part having a weldable seam, in accordance with various examples. [Figure 6] FIG. 1 is a block diagram illustrating an alignment process flow, according to various examples. [Figure 7] FIG. 1 is a schematic diagram of a graph search technique that may determine a path plan for a robot, in accordance with various examples. [Figure 8] 1 is a mathematical model of a robot arm, with various examples. [Figure 9] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. [Figure 10] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. [Figure 11] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. DETAILED DESCRIPTION OF THE INVENTION

[0006] Detailed Description Traditional welding techniques are tedious, labor-intensive, and inefficient. Traditional welding techniques are also not flexible enough to accommodate irregularities commonly encountered during the manufacturing process, leading to undesirable downtime and inefficiencies. For example, traditional welding techniques require skilled programmers to generate instructions for welding robots to perform welding tasks. These instructions instruct the welding robots on the movements, paths, trajectories, and welding parameters that must be used to perform a particular welding task. The instructions are written for high-volume jobs, where the same welding task is repeated multiple times. Therefore, any deviations encountered during the welding process (e.g., different parts) can result in misplaced welds. Misplaced welds increase inefficiencies, costs, and other negative aspects of mass production.

[0007] In some cases, a computer-aided design (CAD) model of a part can be useful to a welding robot to facilitate the welding operation. For example, a CAD model of the part to be welded can be provided to the welding robot, which can use the CAD model to guide its movements, including the location of the seam to be welded. The seam(s) to be welded are annotated in the CAD model and the welding robot (e.g., annotations including user-selected edges, each edge representing a seam), and sensors are used to locate the seam and then apply the weld according to the annotations. While such approaches may reduce or eliminate the need for skilled programmers or manufacturing engineers, they have limitations. For example, welding is a highly precise operation. Generally, to create an acceptable weld, it is desirable for the welding tip to be located within 1 mm of the target location associated with the seam. When guiding a welding robot based on a CAD model, even if the part fits the CAD model perfectly, the actual seam may be more than 1 mm from the modeled location, which can make it difficult or impossible to accurately position the welding tip to create an acceptable weld. When the CAD model is a simplification of the actual part, which is common in low-volume production, the seam may be shifted by a centimeter or more from the modeled location. Therefore, it can be difficult to precisely locate the seam based on the CAD model using known techniques. Therefore, the welding robot may create an unacceptable weld, thereby resulting in a defective part.

[0008] Furthermore, prior solutions for controlling welding robots require a skilled operator to provide specific instructions to the welding robot (more specifically, the robot arm) to avoid collisions with other components (e.g., parts, sensors, clamps, etc.) as the welding robot (more specifically, the robot arm) moves through the manufacturing workspace along a path from a first point to a second point, such as a seam. Identifying an obstacle- and collision-free path (e.g., a path the robot could follow to weld a seam) is referred to herein as path planning. Requiring a skilled operator to perform path planning for dozens or even hundreds of potential paths for the robot arm is inefficient, tedious, and expensive. Furthermore, conventional welding robots are often programmed to repeatedly follow the same path, the same movements, and the same trajectory. While this repetitive process may be acceptable in high-volume manufacturing situations where the manufacturing process is highly mature, in low- or medium-volume situations, components may be positioned in unexpected locations relative to the welding robot, which can lead to collisions, misaligned parts, poor tolerances, and other problems. Therefore, in certain situations, a skilled operator may be required to facilitate welding.

[0009] The welding techniques described herein are superior to prior welding robots and techniques because they can automatically and dynamically generate instructions useful for a welding robot to accurately and precisely identify and weld seams. Unlike prior systems and techniques, the welding techniques described herein do not necessarily require a CAD model of the parts to be welded (although in some instances, as described below, a CAD model may be useful), nor do they necessarily require other prior information about the parts or the manufacturing workspace. Rather, the welding techniques described herein use movable sensors to map the manufacturing workspace (particularly the parts and seams) in three-dimensional (3D) space and use such maps to locate and weld seams with a high degree of accuracy and precision. The welding techniques described herein include various additional features that further distinguish them from prior, inferior solutions, including, among other features, the ability to identify multiple candidate seams for welding, the ability to interact with a user to select candidate seams for welding, and the ability to dynamically change welding parameters and provide feedback regarding the welding operation. Additionally, the welding techniques described herein are configured to automatically and dynamically perform path planning using sensor-acquired data, i.e., to automatically and dynamically identify, without prior information, one or more paths through a manufacturing workspace along which a robotic arm may travel without colliding with other components. The welding techniques described herein are also configured to dynamically perform path planning and welding using a combination of sensor-acquired data and prior information (e.g., annotated CAD models). These and other examples are now described below with reference to the drawings.

[0010] FIG. 1 is a block diagram of an autonomous robotic welding system 100, according to various examples. System 100 includes a manufacturing workspace 101, a user interface 106, a controller 108, and a storage device 109 storing a database 112. System 100 may include other components or subsystems not explicitly described herein. Manufacturing workspace 101 is an area or enclosure within which a robotic arm(s) operates on one or more parts positioned on, coupled to, or otherwise supported by a pedestal or positioning device, aided by information received by one or more sensors. In examples, workspace 101 can be any suitable welding area designed with appropriate safety measures for welding. For example, workspace 101 can be a welding area located in a workshop, job shop, manufacturing plant, assembly shop, etc. In examples, the manufacturing workspace 101 (or, more generally, the workspace 101) may include a sensor 102, a robot 110 configured to perform a welding-type process, such as welding, brazing, or joining, a part 114 to be welded (e.g., a part having a seam), and a fixture 116. The fixture 116 may hold, position, and / or manipulate the part 114 and may be, for example, a clamp, a table, a positioning device, or another type of fixture. The fixture 116 may be configured to securely hold the part 114. In examples, the fixture 116 is adjustable manually by a user or automatically by a motor. For example, the fixture 116 may dynamically adjust its position, orientation, or other physical configuration before or during the welding process. In some examples, the robot 110 may include one or more sensors 102. For example, the one or more sensors 102 may be positioned on an arm of the robot 110 (e.g., on a welding head attached to the arm). In another example, one or more sensors 102 may be positioned on a movable, non-welding robotic arm (which may be different from robot 110).In yet another example, one of the one or more sensors 102 may be positioned on the arm of the robot 110 and another one of the one or more sensors 102 may be positioned on a movable piece of equipment in the workspace. In yet another example, one of the one or more sensors 102 may be positioned on the arm of the robot 110 and another one of the one or more sensors 102 may be positioned on a movable, non-welding robot arm.

[0011] The sensor 102 is configured to capture information about the workspace 101. In an example, the sensor 102 is an image sensor configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For example, the sensor 102 may include a camera (e.g., a camera with a built-in laser), a scanner (e.g., a laser scanner), etc. The sensor 102 may include a sensor such as a light detection and ranging (LiDAR) sensor. Alternatively or additionally, the sensor 102 may be an acoustic sensor configured to emit and / or capture sound, such as an acoustic navigation and ranging (SONAR) device. Alternatively or additionally, the sensor 102 may be an electromagnetic sensor configured to emit and / or capture electromagnetic (EM) waves, such as a radio detection and ranging (RADAR) device. Through visual, acoustic, electromagnetic, and / or other sensing technologies, the sensor 102 may collect information about physical structures within the workspace 101. In examples, the sensors 102 collect static information (e.g., fixed structures within the workspace 101), in other examples, the sensors 102 collect dynamic information (e.g., moving structures within the workspace 101), and in still other examples, the sensors 102 collect a combination of static and dynamic information. The sensors 102 may collect any suitable combination of any and all such information regarding physical structures within the workspace 101 and may provide such information to other components (e.g., the controller 108) to generate a 3D representation of the physical structures within the workspace 101. As discussed above, the sensors 102 may capture and communicate any of a variety of information types, but this description assumes that the sensors 102 primarily capture visual information (e.g., 2D images) of the workspace 101, which is then used collectively to generate a 3D representation of the workspace 101 as described below.

[0012] To generate a 3D representation of workspace 101, sensor 102 captures 2D images of physical structures within workspace 101 from various angles. For example, a single 2D image of a fixture 116 or part 114 may be inadequate to generate a 3D representation of that component, and similarly, a set of multiple 2D images of a fixture 116 or part 114 from a single angle, view, or plane may also be inadequate to generate a 3D representation of that component, but multiple 2D images captured from multiple angles at various positions within workspace 101 may be sufficient to generate a 3D representation of a component such as fixture 116 or part 114. This is because capturing 2D images at multiple orientations provides spatial information about the component in three dimensions, conceptually similar to how a plan drawing of a component that includes front, side, and top views of the component provides all the information necessary to generate a 3D representation of the component. Thus, in examples, the sensors 102 are configured to move around the workspace 101 to capture sufficient information to generate a 3D representation of structures within the workspace 101. In examples, the sensors are stationary but are present in a sufficient number and at a sufficient variety of locations around the workspace 101 so that sufficient information is captured by the sensors 102 to generate said 3D representation. In examples in which the sensors 102 are mobile, any suitable structure may be useful to facilitate such movement around the workspace 101. For example, one or more sensors 102 may be positioned on a motorized track system. While the sensors 102 are configured to move around the workspace 101 on the track system, the track system itself may be stationary. However, in some examples, the sensors 102 are mobile on the track system, and the track system itself is mobile around the workspace 101. In yet another example, one or more mirrors are arranged in the workspace 101 with the sensor 102 being capable of pivoting, swiveling, rotating, or translating about a point and / or along an axis so as to capture a 2D image from an initial viewpoint when the sensor 102 is in a first configuration, and to use the mirrors to capture a 2D image from another viewpoint when the sensor 102 is in a second configuration.In yet other examples, the sensors 102 may be mounted on arms that may be configured to pivot, swivel, rotate, or translate about a point and / or along an axis, and the sensors 102 may be configured to capture 2D images from various viewpoints as the arms extend through their full range of motion.

[0013] Additionally or alternatively, one or more sensors 102 may be positioned on the robot 110 (e.g., on the welding head of the robot 110) and configured to collect image data as the robot 110 moves around the workspace 101. Because the robot 110 is movable in multiple degrees of freedom and therefore in multiple dimensions, the sensors 102 positioned on the robot 110 may capture 2D images from various perspectives. In yet other examples, the one or more sensors 102 may be stationary while the physical structure being imaged moves around or within the workspace 101. For example, the part 114 to be imaged may be positioned on a fixture 116, such as a positioning device, and the positioning device and / or part 114 may rotate, translate (e.g., in the x, y, and / or z directions), or otherwise move within the workspace 101, while a fixed sensor 102 (e.g., either a sensor coupled to the robot 110 or a sensor detached from the robot 110) captures multiple 2D images of various facets of the part 114.

[0014] Some examples implement some or all of the aforementioned sensor 102 configurations. Other sensor 102 configurations are contemplated and are within the scope of this disclosure.

[0015] With further reference to FIG. 1 , the robot 110 (e.g., a welding head of the robot 110) is configured to move within the workspace 101 according to a path plan received from the controller 108, as described below. The robot 110 is further configured to perform one or more suitable manufacturing processes (e.g., welding operations) on the part 114 according to instructions received from the controller 108. In some examples, the robot 110 may be a six-axis robot having a welding arm. The robot 110 may be any suitable robotic welding equipment, such as a YASKAWA® robot arm, an ABB® IRB robot, a KUKA® robot, or the like. The robot 110 may be configured to perform arc welding, resistance welding, spot welding, tungsten inert gas (TIG) welding, metal active gas (MAG) welding, metal inert gas (MIG) welding, laser welding, plasma welding, combinations thereof, or the like.

[0016] With further reference to FIG. 1 , workspace 101, and specifically the sensor(s) 102 and robot 110 within workspace 101, are coupled to controller 108. Controller 108 is any suitable machine that is specifically and specifically configured (e.g., programmed) to perform operations attributed herein to controller 108, or more generally to system 100. In some examples, controller 108 is not a general-purpose computer, but instead is specifically programmed and / or hardware-configured to perform operations attributed herein to controller 108, or more generally to system 100. In some examples, controller 108 is or includes an application-specific integrated circuit (ASIC) configured to perform operations attributed herein to controller 108, or more generally to system 100. In some examples, controller 108 includes or is a processor, such as a central processing unit (CPU). In some examples, controller 108 is a field-programmable gate array (FPGA). In examples, the controller 108 includes memory storing executable code that, when executed by the controller 108, causes the controller 108 to perform one or more of the operations attributed here to the controller 108, or more generally to the system 100. The controller 108 is not limited to the specific examples described herein.

[0017] The controller 108 controls the sensor(s) 102 and the robot 110 within the workspace 101. In some examples, the controller 108 controls the fixture(s) 116 within the workspace 101. For example, the controller 108 may control the sensor(s) 102 to move within the workspace 101 as described above and / or to capture 2D images, acoustic data, and / or EM data as described above. For example, the controller 108 may control a robot 110, such as those described herein, to perform welding operations and move within the workspace 101 according to path planning techniques as described below. For example, the controller 108 may operate the fixture(s) 116, such as positioning devices (e.g., tables, clamps, etc.), to rotate, translate, or otherwise move one or more parts within the workspace 101. The controller 108 may also control other aspects of the system 100. For example, the controller 108 may further interact with the user interface (UI) 106 by providing a graphical interface thereon through which the user may interact with and provide input to the system 100, and through which the controller 108 may interact with the user by providing and / or receiving various types of information from the user (e.g., identified seams that are candidates for welding, possible paths during path planning, welding parameter options or selections, etc.). The UI 106 may be any type of interface, including a touchscreen interface, a voice-activated interface, a keypad interface, combinations thereof, etc.

[0018] Further, the controller 108 may interact with the database 112, for example, by storing data in and / or retrieving data from the database 112. The database 112 may more generally be stored in any suitable type of storage device 109 configured to store any type of information. In some examples, the database 112 may be stored in a storage device 109 such as a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a flash memory, or the like. In some examples, the database 112 may be stored on a cloud-based platform. The database 112 may store any information useful to the system 100 in performing a welding operation. For example, the database 112 may store a CAD model of the part 114. As another example, the database 112 may store an annotated version of the CAD model of the part 114. The database 112 may also store a point cloud of the part 114 (also referred to herein as a CAD model point cloud) generated using the CAD model. Similarly, welding instructions for part 114, generated based on a 3D representation of part 114 and / or user input provided regarding part 114 (e.g., regarding which seams in part 114 to weld, welding parameters, etc.), may be stored in database 112. In examples, storage device 109 stores executable code 111 that, when executed, causes controller 108 to perform one or more operations attributed here to controller 108, or more generally to system 100. In examples, executable code 111 is a single self-contained program; in other examples, executable code is a program having one or more function calls to other executable code that may be stored in storage device 109 or elsewhere. In some examples, one or more functions resulting from execution of executable code 111 may be implemented by hardware.For example, multiple processors may be useful for performing one or more separate tasks of the executable code 111 .

[0019] FIG. 2 is a schematic diagram of an exemplary autonomous robotic welding system 200, according to various examples. System 200 is an example of system 100 of FIG. 1 , with like numbers referring to like components. For example, system 200 includes workspace 201. Workspace 201 may further include, optionally, a movable sensor 202, a robot 210 (to which one or more sensors 202 (other than movable sensor 202) may be attached), and a fixture 216. Robot 210 includes multiple joints and members (e.g., shoulders, arms, elbows, etc.) that enable robot 210 to move with any suitable number of degrees of freedom. Robot 210 includes a welding head 210A that performs a welding operation on a part, such as a part that may be supported by fixture 216 (e.g., a clamp). System 200 further includes a UI 206 coupled to workspace 201. During operation, sensor 202 collects 2D images of workspace 201 and provides the 2D images to a controller (not explicitly shown in FIG. 2 ). The controller generates a 3D representation (e.g., a point cloud) of workspace 201, including fixture 216, parts supported by fixture 216, and / or other structures within workspace 201. The controller uses the 3D representation to identify seams (e.g., parts supported by fixture 216), plan a path for robot 210 to weld the seams without colliding with structures within workspace 201, and control robot 210 to weld the seams, as described herein.

[0020] FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, according to various examples. System 300 is an example of system 100 of FIG. 1 and system 200 of FIG. 2, with like numbers referring to like components. For example, system 300 includes a workspace 301. Workspace 301 may further include, optionally, a movable sensor 302, a robot 310 (to which one or more sensors 302 may be attached), and a fixture 316 (e.g., a table or positioning device). Robot 310 includes multiple joints and members (e.g., shoulders, arms, elbows, etc.) that enable robot 310 to move with any suitable number of degrees of freedom. Robot 310 includes a welding head 310A that performs a welding operation on a part, such as a part that may be supported by fixture 316. System 300 further includes a UI 306 coupled to workspace 301. During operation, sensor 302 collects a 2D image of workspace 301 and provides the 2D image to a controller (not explicitly shown in FIG. 3 ). The controller generates a 3D representation (e.g., a point cloud) of workspace 301, including fixture 316, parts supported by fixture 316, and / or other structures within workspace 301. The controller uses the 3D representation to identify seams (e.g., parts supported by fixture 316), plan a path for robot 310 to weld the seams without colliding with structures within workspace 301, and control robot 310 to weld the seams, as described herein.

[0021] Referring again to FIG. 1 , as described above, the controller 108 is configured to receive the 2D images (and possibly other data, such as acoustic or EM data) from the sensors 102 and generate a 3D representation of the structure depicted in the 2D images. The 3D representation may also be referred to as a point cloud. A point cloud may be a collection of points, each representing the location in 3D space of a point on the surface of the part 114 and / or fixture 116. In some examples, one or more 2D images (e.g., image data captured by the sensor(s) 102 at a particular orientation relative to the part 114) may be overlaid and / or stitched together by the controller 108 to reconstruct and generate 3D image data of the workspace 101. The 3D image data may be collated to generate a point cloud having associated image data for at least some of the points in the point cloud.

[0022] In an example, the 3D image data can be collated by the controller 108 so that a point cloud generated from the data can have six degrees of freedom. For example, each point in the point cloud can represent an infinitesimally small location in 3D space. As described above, the sensor(s) 102 can capture multiple 2D images of a point from various angles. These multiple 2D images can be collated by the controller 108 to determine an average image pixel for each point. The averaged image pixel can be associated with the point. For example, if the sensor(s) 102 are color cameras with red, green, and blue channels, the six degrees of freedom can be {x position, y position, z position, red intensity, green intensity, and blue intensity}. For example, if the sensor(s) 102 are monochrome cameras with monochrome channels, four degrees of freedom can be generated.

[0023] Figure 4 is an example point cloud 400 of parts having a weldable seam, according to various examples. More specifically, point cloud 400 represents parts 402 and 404 to be welded together along seam 406. Figure 5 is an example point cloud 500 of parts having a weldable seam, according to various examples. More specifically, point cloud 500 represents parts 502 and 504 to be welded together along seam 506. Controller 108 (Figure 1) is configured to generate 3D point clouds 400, 500 based on 2D images captured by sensor 102, as described above. The controller 108 may then use the point clouds 400, 500 (or, in some examples, image data useful for generating the point clouds 400, 500) to identify and locate seams, such as seams 406, 506, plan a welding path along the seams 406, 506, and apply the weld along the seams 406, 506 using the robot 110 ( FIG. 1 ) according to the path plan. The manner in which the controller 108 executes the executable code 111 ( FIG. 1 ) to perform such operations, including seam identification and path planning, will now be described in detail.

[0024] Upon executing the executable code 111, the controller 108 uses a neural network to perform pixel-by-pixel classification (e.g., using images captured by the sensor 102 or based on images captured by the sensor 102) and / or point-by-point classification (e.g., using one or more point clouds) to identify and classify structures within the workspace 101. For example, the controller 108 may perform pixel-by-pixel and / or point-by-point classification to identify each imaged structure within the workspace 101 as, for example, a part 114, an interface seam (referred to herein as a candidate seam) on or between multiple parts 114, a fixture 116, or a robot 110. The controller 108 may, in examples, identify and classify pixels and / or points based on a neural network (e.g., a U-net model) trained using appropriate training data. The neural network may be trained on image data, point cloud data, spatial information data, or a combination thereof. Because the point cloud and / or image data includes information captured from various viewpoints within the workspace 101, the neural network can operate to classify fixtures 116 or candidate seams on the part(s) 114 from multiple angles and / or viewpoints. In some examples, a neural network can be trained to operate directly on the point set, such as a dynamic graph convolutional neural network, or a neural network may be implemented to analyze unorganized points on the point cloud. In some examples, a first neural network can be trained on the point cloud data to perform point-by-point classification, and a second neural network can be trained on the image data to perform pixel-by-pixel classification. The first neural network and the second neural network can individually identify candidate seams and locate the candidate seams. The outputs from the first neural network and the second neural network can be combined as a final output to determine the position and orientation of one or more candidate seams on the part 114.

[0025] In some examples, when pixel-by-pixel classification is performed, the results can be projected onto the 3D point cloud data and / or a meshed version of the point cloud data, thereby providing information about the location of the fixture 116 within the workspace 101. If the input data is image data (e.g., a color image), spatial information such as depth information can be included along with the color data to perform pixel-by-pixel segmentation. In some examples, pixel-by-pixel classification can be performed to identify candidate seams and locate the candidate seams relative to the part 114, as described further below.

[0026] As described above, the controller 108 may identify and classify pixels and / or points as particular structures within the workspace 101, such as the fixture 116, the part 114, candidate seams on the part 114, etc. Portions of the image and / or point cloud data classified as non-part and non-candidate seam structures, such as the fixture 116, may be segmented and separated (e.g., deleted or otherwise removed) from the data, thereby isolating data identified and classified as corresponding to the part 114 and / or candidate seam(s) on the part 114. In some examples, after identifying candidate seams and segmenting the non-part 114 and non-candidate seam data as described above (or optionally prior to such segmentation), a neural network can be configured to analyze each candidate seam to determine a seam type. For example, the neural network can be configured to determine whether the candidate seam is a butt joint, corner joint, edge joint, lap joint, T-joint, etc. A model (e.g., a U-net model) can classify seam types based on data captured from multiple viewpoints within the workspace 101.

[0027] If pixel-by-pixel classification is performed using image data, the controller 108 may project pixels of interest (e.g., pixels representing the part 114 and candidate seams on the part 114) into 3D space to generate a set of 3D points representing the part 114 and candidate seams on the part 114. Alternatively, if point cloud data is used to perform point-by-point classification, the points of interest may already exist in 3D space in the point cloud. In either case, to the controller 108, the 3D points are an unordered set of points, and at least some of the 3D points may be clumped together. To eliminate such noise and generate a subset of contiguous, unbroken points to represent the candidate seams, a Manifold Blurring and Mean Shift (MBMS) technique or a similar technique may be applied. Such a technique may cluster the points and eliminate noise. The controller 108 may then apply a clustering method to decompose the candidate seams into individual candidate seams. In other words, instead of having several subsets of points representing multiple seams, clustering can decompose each subset of points into individual seams. Following clustering, controller 108 can fit a spline to each individual subset of points. Thus, each individual subset of points can be an individual candidate seam.

[0028] In summary, using the techniques described above, but without limitation, the controller 108 receives image data captured by the sensor 102 from various positions and viewpoints within the workspace 101. The controller 108 performs pixel-by-pixel and / or point-by-point classification techniques using neural networks to classify and identify each pixel and / or point as a part 114, a candidate seam on the part 114 or at an interface between multiple parts 114, a fixture 116, etc. Non-part 114 structures and structures identified as non-candidate seam structures are segmented and separated, and the controller 108 may perform additional processing on the remaining points (e.g., to mitigate noise). By performing these operations, the controller 108 may generate a set of candidate seams on the part 114 that indicate the location and orientation of these seams. As described below, the controller 108 may then determine whether the candidate seams are actually seams and, optionally, perform additional processing using prior information, such as CAD models of the parts and seams. The resulting data is suitable for use by the controller 108 to plan a path for applying a weld along the identified seam, as also described below.

[0029] In some cases, an identified candidate seam may not be a seam (i.e., the identified candidate seam may be a false positive). To determine whether an identified candidate seam is actually a seam, the controller 108 determines a confidence value using images captured by the sensor 102 from various viewpoints within the workspace 101. The confidence value represents the likelihood that the candidate seam determined from the corresponding viewpoint is an actual seam. The controller 108 may then compare the confidence values ​​for the different viewpoints and eliminate candidate seams that are unlikely to be actual seams. For example, the controller 108 may determine the mean, median, maximum, or any other suitable summary statistic of the candidate values ​​associated with a particular candidate seam. Generally, a candidate seam that corresponds to an actual seam will have a consistently high (e.g., above a threshold) confidence value across the various viewpoints used to capture the candidate seam. If the summary statistic of the confidence values ​​for the candidate seam is above a threshold, the controller 108 may designate the candidate seam as an actual seam. Conversely, if the summary statistics of the confidence values ​​for the candidate seam are below a threshold, the candidate seam can be designated as a false positive that is not eligible for welding.

[0030] As described above, after identifying a candidate seam that is an actual seam, the controller 108 may use prior information, such as a CAD model (or a point cloud version of the CAD model), to perform additional processing, referred to herein as registration. More specifically, in some cases, differences may exist between the seam dimensions on the part and the seam dimensions in the CAD model, and because the CAD model may later be used to perform path planning as described herein, the CAD model needs to be transformed (e.g., updated) to account for such differences. Thus, the controller 108 compares a first seam (e.g., a candidate seam on the part 114 that has been verified as an actual seam) with a second seam (e.g., a seam annotated (e.g., by an operator / user) on the CAD model that corresponds to the first seam) to determine the difference between the first seam and the second seam. The seam on the CAD model may be annotated as described above. The first seam and the second seam may be in approximately the same location if the CAD model and / or the controller 108 accurately predict the location of the candidate seam. Alternatively, the first seam and the second seam may partially overlap if the CAD model and / or the controller 108 are partially accurate. The controller 108 may perform a comparison between the first seam and the second seam. This comparison between the first seam and the second seam may be based in part on the shape and relative position in space of both seams. If the shapes of the first seam and the second seam are relatively similar and are close to each other, the second seam may be identified as the same as the first seam. In this manner, the controller 108 can take into account surface topography on the part that is not accurately represented in the CAD model. In this manner, the controller 108 can identify candidate seams and subselect, refine, or update the candidate seams relative to the part using the CAD model of the part. Each candidate seam may be an updated point set representing the position and orientation of the candidate seam relative to the part.

[0031] FIG. 6 is a block diagram illustrating an alignment process flow 600 according to various examples. Some or all of the steps in the alignment process flow 600 are performed by the controller 108 ( FIG. 1 ). The controller 108 may first perform a coarse alignment 602 using a CAD model point cloud 604 and a scan point cloud 606 formed using images captured by the sensor 102. The CAD model point cloud 604 and the scan point cloud 606 may be sampled so that their points have uniform or nearly uniform distribution and so that both points have equal or nearly equal point densities. In examples, the controller 108 downsamples the point clouds 604, 606 to randomly select and retain a uniform number of points within the clouds and discard the remaining unselected points. For example, in some examples, a Poisson disk sampling (PDS) downsampling algorithm may be implemented. The controller 108 may provide as inputs to the PDS algorithm the boundaries of the point clouds 604, 606, the minimum distance between samples, and the limit of samples to be selected before they are rejected. In some examples, a delta network may be useful for transforming one model into another during coarse registration 602. The delta network may be a Siamese network that takes a source model and a target model and encodes them into latent vectors. The controller 108 may use the latent vectors to predict point-by-point transformations that transform or update one model into another. A delta network may not require a training dataset. Given a CAD model and scanned point clouds 604, 606, the controller 108 in the context of a delta network spends one or more epochs learning the dissimilarity or similarity between the two point clouds. During these epochs, the delta network learns meaningful features that are useful for later registration. The delta network may use skip connections to learn the transformations, although in some examples, skip connections may not be used. In some cases, the CAD model contains surfaces that are not present in the 3D point cloud generated using the captured image (e.g., scan).In such a case, the delta network moves all points corresponding to the missing surfaces from the CAD model point cloud 604 to some points in the scan point cloud 606 (and updates the scan point cloud 606). Thus, during alignment, the controller 108 (e.g., the delta network) may use the learned features to transform (or update) the original CAD model, or may use the learned features to transform the deformed CAD model. In an example, the delta network may include an encoder network such as a dynamic graph convolutional neural network (DGCNN). After the point cloud is encoded into features, a concatenated vector consisting of both the CAD and scan embeddings may be formed. After performing a pooling operation (e.g., max pooling), a decoder may be applied to the resulting vector. In some examples, the decoder may include five convolutional layers with specific filters (e.g., 256, 256, 512, 1024, Nx3 filters). The resulting output may be concatenated with the CAD model and scan embeddings, max pooled, and then provided to the decoder once again. The final result may include a point-wise transformation.

[0032] Extraneous data and noise in the data (e.g., the output of the coarse alignment 602) can affect the alignment of the parts 114. For at least this reason, it is desirable to remove as much of the extraneous data and noise as possible. The bounding box 608 is useful for removing this extraneous data and noise (e.g., fixtures 116) to limit the area over which the alignment occurs. In other words, data within the bounding box is retained, but all data from outside the bounding box, 3D or otherwise, is discarded. The aforementioned bounding box may be any shape that can surround or encapsulate (e.g., partially or completely) the CAD model itself. For example, the bounding box may be a dilated or scaled version of the CAD model. Data outside the bounding box may be removed from the final alignment, or it may still be included but weighted to reduce its influence.

[0033] 6, during fine alignment 610, the controller 108 passes the output of the bounding box 608 as a patch through a set of convolutional layers in a neural network trained as an autoencoder. More specifically, the data may be passed through the encoder section of the autoencoder, and the decoder section of the autoencoder may not be used. The input data may be the XYZ positions of the points of the patch within the shape, e.g., (128, 3). The output may be a vector of length 1024, for example, which is useful for point-wise features.

[0034] The set of correspondence points that best supports the rigid transformation between the CAD point cloud model and the scanned point cloud model should be determined during registration. The correspondence candidates are represented as a matrix where each element stores the confidence or probability of a match between two points: It may be stored (e.g., in database 112) as TIFF2024508564000002.tif18128.

[0035] Various methods are available for finding corresponding points from this matrix, e.g., hard correspondences, soft correspondences, product manifold filters, graph cliques, covariance, etc. After the fine registration 610 is completed, the registration process is then completed (612).

[0036] As described above, in some cases, the actual location of a seam on the part 114 may differ from the seam location determined by the controller 108 using sensor imaging (e.g., using a scan point cloud) and / or determined by a CAD model (e.g., using a CAD model point cloud). In such cases, a scanning procedure (sometimes referred to herein as a pre-scan) is useful to correct the determined seam location to more closely match the actual seam location on the part 114. In a scanning procedure, a sensor 102 positioned on the robot 110 (referred to herein as an on-board sensor) performs a scan of the seam. In some cases, this scanning may be performed using an initial movement and / or path plan generated by the controller 108 using a CAD model, a scan, or a combination thereof. For example, the sensor 102 may scan any or all areas of the workspace 101. During this initial movement and / or path plan, the sensor 102 may capture observation images and / or data. The observation images and / or data may be processed by the controller 108 to generate seam point cloud data. The controller 108 may use the seam point cloud data to correct seam locations when processing point cloud(s) 604 and / or 606. The controller 108 may also use the seam point cloud data in correcting path and motion planning.

[0037] In some examples, the above-described alignment techniques may be useful for comparing and matching seams determined using sensors 102 other than the on-board sensor 102 with seams identified by the on-board sensor 102. By matching seams in this manner, the robot 110 (more specifically, the head of the robot 110) may be positioned relative to the actual seam as needed.

[0038] In some examples, the pre-scan trajectory of the robot 110 is identical to the trajectory planned for welding along the seam. In some such examples, the movements taken for the robot 110 during the pre-scan may be generated separately to limit the probability of or reduce the instances of collision, to better visualize the seam or critical geometry with the on-board sensor 102, or to scan the geometry around the seam in question.

[0039] In some examples, pre-scanning techniques may include scanning more than a specific seam or seams, but rather may also include scanning other geometries of the part(s) 114. The scan data may be useful for more precise application of any or all of the techniques described herein (e.g., alignment techniques) to find, locate, and detect seams and ensure that the head of the robot 110 is positioned and moved along the seams as needed.

[0040] In some examples, scanning techniques (e.g., scanning the actual seam using a sensor / camera attached to a welding arm / weld head) may be useful for identifying gap variability information about the seam, rather than position and orientation information about the seam. For example, scanned images captured by the sensor(s) 102 on the robot 110 during a scanning procedure may be useful for identifying gap variability and adjusting the weld trajectory or path plan to account for such gaps. For example, 3D points, 2D image pixels, or a combination thereof may be useful for locating variable gaps between the parts 114 to be welded. In some examples, variable gap finding is useful, in which 3D points, 2D image pixels, or a combination thereof are useful for locating, identifying, and measuring the variable sizes of multiple gaps between the parts 114 to be welded together. In tack weld finding or general weld finding, previous welds or material deposits in the gaps between the parts 114 to be welded may be identified using 3D points and / or 2D image pixels. Any or all of these techniques may be useful for optimizing welding, including path planning. In some cases, gap variability may be identified in a 3D point cloud generated using images captured by the sensor 102. In yet other cases, gap variability may be identified through scanning techniques performed during welding operations for a task (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / welding head). In either case, the controller 108 may be configured to dynamically adapt welding instructions (e.g., welding voltage) based on the determined location and size of the gap. For example, dynamically adjusting welding instructions for a welding robot can result in precise welding of seams with variable gaps. Adjusting the welding instructions may include adjusting one or more of the welder voltage, welder current, electric pulse duration, electric pulse shape, and material feed rate.

[0041] In an example, the user interface 106 may provide the user with an option to display candidate seams. For example, the user interface 106 may provide a graphical representation of the part 114 and / or the candidate seams on the part 114. Additionally or alternatively, the user interface 106 may group the candidate seams based on seam type. As described above, the controller 108 may identify the seam type. For example, candidate seams identified as lap seams may be grouped under the label "lap seam" and presented to the user via the user interface 106 under the label "lap seam." Similarly, candidate seams identified as edge seams may be grouped under the label "edge seam" and presented to the user via the user interface 106 under the label "edge seam."

[0042] The user interface 106 may further provide the user with the option to select a candidate seam to be welded by the robot 110. For example, each candidate seam on the part 114 may be presented as a push button on the user interface 106. When the user presses a particular candidate seam, the selection may be transmitted to the controller 108. The controller 108 may generate instructions for the robot 110 to perform a welding operation on that particular candidate seam.

[0043] In some examples, the user may be provided with the option to update the welding parameters. For example, the user interface 106 may provide the user with a list of different welding parameters. The user may select the specific parameter to be updated. Changes to the selected parameter may be made, for example, using a drop-down menu or by text entry. This update may be sent to the controller 108 so that the controller 108 can update the instructions for the robot 110.

[0044] In examples where the system 100 is not provided with prior information (e.g., a CAD model) of the part 114, the sensor(s) 102 can scan the part 114. A representation of the part 114 can be presented to a user via the user interface 106. This representation of the part 114 can be a point cloud and / or a mesh of a point cloud, including projected 3D data of the scanned image of the part 114 obtained from the sensor(s) 102. The user can annotate seams to be welded in the representation via the user interface 106. Alternatively, the controller 108 can identify candidate seams in the representation of the part 114. The candidate seams can be presented to the user via the user interface 106. The user can select the seam to be welded from among the candidate seams. The user interface 106 can annotate the representation based on the user's selection. In some examples, the annotated representation can be stored in the database 112.

[0045] After one or more seams on the part(s) 114 have been identified and, to the extent possible, corrected using the techniques described above (or using other suitable techniques), the controller 108 plans a path for the robot 110 during the subsequent welding process. In some examples, graph matching and / or graph searching techniques may be useful in planning the path of the robot 110. A particular seam identified as described above may include multiple points, and path planning techniques involve determining different states of the robot 110 for each such point along the given seam. The states of the robot 110 may include, for example, the position of the robot 110 within the workspace 101 and specific configurations of the robot 110's arm in any number of degrees of freedom that may be applicable. For example, in the case of a robot 110 having an arm with six degrees of freedom, the state of the robot 110 would not only include the position of the robot 110 within the workspace 101 (e.g., the position of the robot's 110's welding head in three-dimensional x-y-z space), but would also include specific sub-states for each of the robot arm's six degrees of freedom. Additionally, when the robot 110 transitions from a first state to a second state, the robot 110 may change its position within the workspace 101, which will entail the robot 110 traversing a particular path within the workspace 101 (e.g., along the seam being welded). Thus, specifying a series of states for the robot 110 entails specifying a path for the robot 110 to travel within the workspace 101. The controller 108 may perform a pre-scanning technique or a variant thereof after path planning is complete, and the controller 108 may use information captured during the pre-scanning technique to make any of a variety of appropriate adjustments (e.g., adjustments to the X, Y, and Z axes or coordinate systems used to perform the actual welding along the seam).

[0046] FIG. 7 is a schematic diagram 700 of a graph search technique for determining a path plan for the robot 110 (e.g., by the controller 108). Each circle in the diagram 700 represents a different state of the robot 110 (e.g., a specific position of the robot 110 within the workspace 101 (e.g., the position of the welding head of the robot 110 in 3D space), and a different configuration of the robot 110's arms, as well as the location and / or configuration of fixtures supporting the parts, such as positioners, clamps, etc.). Each column 702, 706, and 710 represents a different point along the seam to be welded. Thus, for the seam point corresponding to column 702, the robot 110 can be in any one of states 704A-704D. Similarly, for the seam point corresponding to column 706, the robot 110 can be in any one of states 708A-708D. Similarly, for the seam point corresponding to column 710, the robot 110 can be in any one of states 712A-712D. For example, if the robot 110 is in state 704A when at a seam point corresponding to column 702, the robot 110 may then transition to any of states 708A-708D for the next seam point corresponding to column 706. Similarly, upon entering states 708A-708D, the robot 110 may subsequently transition to any of states 712A-712D for the next seam point corresponding to column 710, and so on. In some examples, entering a particular state may preclude entering other states. For example, entering state 704A may subsequently allow the possibility of entering states 708A-708C, but not 708D, while entering state 704B may subsequently allow the possibility of entering states 708C and 708D, but not states 708A-708B. The scope of the present disclosure is not limited to any particular number of seam points or any particular number of robot 110 states.

[0047] In some examples, to determine a path plan for the robot 110 using a graph search technique (e.g., according to the technique shown in diagram 700), the controller 108 may determine the shortest path from states 704A-704D to the state corresponding to seam point N (e.g., states 712A-712D). By assigning a cost to each state and each transition between states, an objective function can be designed by the controller 108. The controller 108 finds the path that results in the minimum cost value of the objective function. For the freedom of having multiple start and end points to choose from, graph search methods such as Dijkstra's algorithm or A* may be implemented. In some examples, a brute force technique may be useful to determine an appropriate path plan. The brute force technique involves the controller 108 calculating all possible paths (e.g., through diagram 700) and selecting the shortest path.

[0048] The controller 108 may determine whether the state at each seam point is feasible, which means, at least in part, that the controller 108 may determine whether enforcing a chain of states along a series of seam points in the seam would cause a collision between the robot 110 and a structure in the workspace 101, or even with parts of the robot 110 itself. To this end, the concept of realizing different states at different points in the seam may alternatively be expressed in the context of a seam having multiple waypoints. First, the controller 108 may discretize the identified seam into a series of waypoints. The waypoints may constrain the orientation of a welding head connected to the robot 120 with three (spatial / translational) degrees of freedom. Typically, constraints on the orientation of the welding head of the robot 120 are provided in one or two rotational degrees of freedom about each waypoint with the goal of producing some desired weld of some quality, and the constraints are typically relative to the surface normal vectors emanating from the waypoints and the path of the weld seam. For example, the position of the welding head can be constrained in the x-, y-, and z-axes, as well as about one or two rotational axes perpendicular to the axis of the welding machine's welding wire or tip, all relative to the waypoint and some nominal coordinate system attached to the waypoint. In some examples, these constraints may be angular limits or tolerances. Those skilled in the art will recognize that the ideal or desired welding angle can vary based on the part or seam geometry, the direction of gravity relative to the seam, and other factors. In some examples, the controller 108 may constrain the welding position to 1F or 2F to ensure that the seam is perpendicular to gravity for one or more reasons (e.g., to find a balance between welding and path planning for optimization purposes). Thus, the position of the welding head can be held (constrained) by each waypoint at any suitable orientation relative to the seam. Typically, the welding head is not constrained about a rotational axis (θ) coaxial with the axis of the welding head.For example, each waypoint may define a position of the welding head of the welding robot 120 such that at each waypoint the welding head is at a fixed position and orientation relative to the weld seam. In some implementations, the waypoints are discretized fine enough to make the movement of the welding head substantially continuous.

[0049] The controller 108 may divide each waypoint into multiple nodes. Each node may represent a possible orientation of the welding head at that waypoint. As a non-limiting example, the welding head may be unconstrained about a rotational axis coaxial with the axis of the welding head, such that the welding head can rotate (e.g., 360 degrees) along rotational axis θ at each waypoint. Each waypoint may be divided into 20 nodes, such that each node at each waypoint represents the welding head in 18-degree rotational increments. For example, the first waypoint-node pair may represent 0 degrees of welding head rotation, the second waypoint-node pair may represent welding head rotation from 18 degrees, the third waypoint-node pair may represent welding head rotation from 36 degrees, and so on. Each waypoint may be divided into 2, 10, 20, 60, 120, 360, or any suitable number of nodes. Subdivision of the nodes may represent division of orientation beyond one degree of freedom. For example, the orientation of the welder tip around a waypoint may be defined by three angles. A welding path can be defined by connecting each waypoint-node pair, and the distance between waypoints and the offset between adjacent waypoint-nodes can represent the amount of translation and rotation of the welding head as it moves between node-waypoint pairs.

[0050] The controller 108 can evaluate each waypoint-node pair for feasibility of welding. For example, consider a non-limiting example in which the waypoints are divided into 20 nodes. The controller 108 can evaluate whether the first waypoint-node pair, representing the welding head held at 0 degrees, is feasible. In other words, the controller 108 can evaluate whether the robot 110 will collide or interfere with a part, a fixture, or the welding robot itself when placed in the position and orientation defined by that waypoint-node pair. Similarly, the controller 108 can evaluate whether the second waypoint-node pair, the third waypoint-node pair, and so on are feasible. The controller 108 can similarly evaluate each waypoint. In this manner, all feasible nodes for all waypoints can be determined.

[0051] In some examples, collision analysis as described herein may be performed by comparing a 3D model of the workspace 101 with a 3D model of the robot 110 to determine whether the two models overlap, and optionally, whether some or all of the triangles overlap. If the two models overlap, the controller 108 may determine that a collision is likely. If the two models do not overlap, the controller 108 may determine that a collision is unlikely. More specifically, in some examples, the controller 108 may compare the models for each of a set of waypoint-node pairs (such as the waypoint-node pairs described above) and determine that the two models overlap for a subset, or possibly all, of the waypoint-node pairs. For the subset of waypoint-node pairs for which model intersections are identified, the controller 108 may omit the waypoint-node pairs in that subset from the planned path and may identify alternative pairs for those waypoint-node pairs. The controller 108 may repeat this process as necessary until a collision-free path is planned. The controller 108 may use a flexible collision library (FCL), which includes various techniques for efficient collision detection and proximity calculation as a tool for collision avoidance analysis. The FCL is useful for performing multiple proximity queries on different model representations and can be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used in conjunction with or in place of the FCL.

[0052] The controller 108 can generate one or more feasible simulated (or evaluated; both terms are used interchangeably herein) welding paths, provided they are physically feasible. A welding path can be a path a welding robot takes to weld a candidate seam. In some examples, a welding path can include all waypoints of a seam. In some examples, a welding path can include some, but not all, waypoints of a candidate seam. A welding path can include the movement of the robot and welding head as the welding head moves between each waypoint-node pair. Once a feasible path between a node-waypoint pair is identified, a feasible node-waypoint pair for the next consecutive waypoint can be identified, if one exists. Those skilled in the art will recognize that many search trees or other strategies can be employed to evaluate the space of feasible node-waypoint pairs. As discussed in further detail herein, a cost parameter can be assigned or calculated for movement from each node-waypoint pair to the subsequent node-waypoint pair. Cost parameters can be associated with travel time, amount of travel between node-waypoint pairs (e.g., including rotations), and / or simulated / expected weld quality produced by the welding head during travel.

[0053] If nodes are not feasible for welding for one or more waypoints and / or no feasible path exists for traveling between a previous waypoint-node pair and any of the waypoint-node pairs for a particular waypoint, the controller 108 can determine alternative welding parameters such that at least some additional waypoint-node pairs are feasible for welding. For example, if the controller 108 determines that none of the waypoint-node pairs for a first waypoint are feasible, thereby rendering the first waypoint unweldable, the controller 108 can determine alternative welding parameters, such as alternative weld angles, such that at least some waypoint-node pairs for the first waypoint are weldable. For example, the controller 108 can remove or relax constraints on rotation about the x-axis and / or y-axis. Similarly, the controller 108 can allow the weld angle to vary in one or two additional rotational (angular) dimensions. For example, the controller 108 can divide unweldable waypoints into two-dimensional or three-dimensional nodes. Each node can then be evaluated for weld feasibility with the welding robot and weld held at various welding angles and rotations. Additional rotation or other degrees of freedom about the x- and / or y-axes can make the waypoint accessible to the welding head so that the welding head does not encounter any collisions. In some implementations, if a node is not feasible for welding one or more waypoints and / or no feasible path exists for traveling between the previous waypoint-node pair and one of the waypoint-node pairs for the particular waypoint, the controller 108 can use the degrees of freedom provided by the positioning device system in determining a feasible path between the previous waypoint-node pair and one of the waypoint-node pairs for the particular waypoint.

[0054] Based on the generated welding path, the controller 108 can optimize the welding path for welding. (Optimal and optimizing, as used herein, do not refer to determining the absolute best welding path, but generally refer to techniques that can reduce welding time and / or improve weld quality relative to less efficient welding paths.) For example, the controller 108 can determine a cost function that seeks local minima and / or minimums in the movement of the robot 110. Typically, an optimal welding path minimizes welding head rotation, because rotation can increase the time to weld a seam and / or reduce weld quality. Thus, optimizing the welding path can include determining a welding path that passes through the greatest number of waypoints with the least amount of rotation.

[0055] In evaluating the feasibility of welding at each of the partitioned nodes or node-waypoint pairs, the controller 108 may perform multiple calculations. In some examples, each of the multiple calculations may be mutually exclusive of one another. In some examples, a first calculation may include a kinematic feasibility calculation, which calculates whether the arm of the robot 110 of the employed welding robot can mechanically reach (or be at) the state defined by the node or node-waypoint pair. In some examples, in addition to the first calculation, a second calculation, which may be mutually exclusive of the first calculation, may also be performed by the controller 108. The second calculation may include determining whether the arm of the robot 110 will encounter a collision (e.g., collide with the workspace 101 or a structure within the workspace 101) when accessing a portion of the seam (e.g., the node or node-waypoint pair in question).

[0056] The controller 108 may perform the first calculation before performing the second calculation. In some examples, the second calculation may be performed only if the result of the first calculation is positive (e.g., if it is determined that the arm of the robot 110 can mechanically reach (or be in) the state defined by the node or node-waypoint pair). In some examples, the second calculation may not be performed if the result of the first calculation is negative (e.g., if it is determined that the arm of the robot 110 cannot mechanically reach (or be in) the state defined by the node or node-waypoint pair).

[0057] Kinematic feasibility may be correlated with the type of robot arm employed. For purposes of this discussion, we will assume that the welding robot 110 includes a six-axis robotic welding arm with a spherical wrist. A six-axis robotic arm can have six degrees of freedom: three in the X, Y, and Z Cartesian coordinates, plus three additional degrees of freedom due to the wrist-like nature of the robot 110. For example, the wrist-like nature of the robot 110 provides a fourth degree of freedom in the up-down direction of the wrist (e.g., the wrist moves in the +y and -y directions), a fifth degree of freedom in the lateral direction of the wrist (e.g., the wrist moves in the -x and +x directions), and a sixth degree of freedom in rotation. In some examples, a welding torch is attached to the wrist portion of the robot 110.

[0058] To determine whether the employed robot 110 arm can mechanically reach (or reside in) a state defined by a node or node-waypoint pair, i.e., to perform the first calculation, the robot 110 can be mathematically modeled as shown in example model 800 of FIG. 8. In some examples, the controller 108 may solve for the first three joint variables based on the wrist position and the other three joint variables based on the wrist orientation. Note that the torch is rigidly attached to the wrist; therefore, the transformation between the torch tip and the wrist is assumed to be fixed. Geometric techniques (e.g., the law of cosines) may be employed to find the first three joint variables (e.g., variables S, L, and U in 802, 804, and 806, respectively).

[0059] After the first three joint variables (i.e., S, L, and U) are successfully calculated, the controller 108 may then solve for the last three joint variables (i.e., R, B, and T in 808, 810, and 812, respectively), for example, by considering the wrist orientation as ZYZ Euler angles. The controller 108 may consider some offsets within the robot 110. These offsets may need to be considered and accounted for due to inconsistencies in the unified robot description format (URDF) file. For example, in some instances, the position values ​​(e.g., the X-axis of the joint) of a joint (e.g., the actual joint of the robot 110) may not match the values ​​described in the URDF file. Such offset values ​​may be provided to the controller 108 in a table. The controller 108 may, in some instances, consider these offset values ​​while mathematically modeling the robot 110. In some examples, after the robot 110 is mathematically modeled, the controller 108 may determine whether the arm of the robot 110 can mechanically reach (or be at) a state defined by a node or node-waypoint pair.

[0060] As described above, the controller 108 can evaluate whether the robot 110, when placed in the position and orientation defined by the waypoint-node pair, will collide or interfere with anything in the workspace 101, including the part 114, the fixture 116, or the robot 110 itself. Once the controller 108 determines the conditions in which the robot arm can exist, the controller 108 may use a second calculation to make the aforementioned evaluation (e.g., as to whether the robot will collide with anything in its environment).

[0061] FIG. 9 is a flowchart of a method 900 for performing autonomous welding, according to various examples. More specifically, FIG. 9 is a flowchart of a method 900 for operating and controlling a welding robot (e.g., robot 110 of FIG. 1 ), according to some examples. In step 902, the method 900 includes acquiring image data of a workspace (e.g., workspace 101 of FIG. 1 ) using one or more sensors (e.g., sensor(s) 102 of FIG. 1 ). The image data may include 2D and / or 3D images of the workspace. As described above, one or more parts to be welded, fixtures and / or clamps capable of securely holding the parts, may be located in the workspace. In some examples, a point cloud may be generated from the image data. For example, the images may be overlaid on one another to reconstruct three-dimensional image data. The three-dimensional image data may be collated to generate the point cloud.

[0062] At step 904, the method 900 includes identifying a set of points on the parts to be welded based on the sensor data, which may be an image. The set of points may represent potential seams to be welded. In some examples, a neural network may perform pixel-by-pixel segmentation on the image data to identify the set of points. Fixtures and clamps in the image data may be classified by the neural network based on image classification. Portions of the image data associated with the fixtures and / or clamps may be segmented so that those portions of the image data are not used to identify the set of points, thereby reducing the computational resources required to identify the set of points to be welded by reducing the search space. In such examples, the set of points may be identified from other portions of the image data (e.g., non-segmented portions of the image data).

[0063] In step 906, the method 900 includes identifying candidate seams from the point set. For example, a subset of points in the point set may be identified as candidate seams. A neural network may perform image classification and / or depth classification to identify the candidate seams. In some examples, the candidate seams may be located relative to the part. For example, to locate the candidate seams, the position and orientation of the candidate seams relative to the part may be determined.

[0064] Additionally, method 900 further includes verifying whether the candidate seam is an actual seam. As described above, the sensor(s) can collect image data from multiple angles. For each image captured from a different angle, a confidence value can be determined that represents whether the candidate seam determined from that angle is an actual seam. If the confidence value exceeds a threshold based on the views taken from the multiple angles, the candidate seam can be verified as an actual seam. In some aspects, method 900 also includes classifying the candidate seam as a seam type. For example, a neural network can determine whether the candidate seam is a butt seam, a corner seam, an edge seam, a lap seam, a T-joint, etc.

[0065] In some examples, after the candidate seams are identified and verified, the subsets of points can be clustered together to form a continuous, seamless seam. At step 908, the method 900 includes generating welding instructions for a welding robot based on the candidate seams. For example, the welding instructions can be generated by tracing a path from one end of the subset of points to the other end of the subset of points. This can generate a path for the seam. In other words, welding can be performed by tracing this path with a welding head. Furthermore, path planning can be performed based on the identified and located candidate seams. For example, path planning can be performed based on the seam path, which can be generated by clustering the subset of points.

[0066] In some examples, the welding instructions may be based on a seam type (e.g., butt joint, corner joint, edge joint, lap joint, T-joint, etc.). In some examples, the welding instructions may be updated based on input from a user via a user interface (e.g., user interface 106 of FIG. 1). A user may select a candidate seam to be welded from all available candidate seams via the user interface. Path planning may be performed on the selected candidate seam, and welding instructions may be generated for the selected candidate seam. In some examples, a user may update welding parameters via the user interface. The welding instructions may be updated based on the updated welding parameters.

[0067] In this manner, a welding robot can be operated and controlled by performing method 900 without prior information (e.g., a CAD model) of the parts to be welded. As the parts are scanned to generate welding instructions, a representation of the scanned image of the part can be annotated (e.g., via a user interface) with one or more candidate seams. The annotated representation can be used to define a 3D model of the part. The 3D model of the part can be saved in a database for subsequent welding of additional instances of the part.

[0068] FIG. 10 is a flowchart of a method 1000 for performing autonomous welding, according to various examples. More specifically, FIG. 10 is a flowchart of a method 1000 for operating and controlling a welding robot (e.g., robot 110 of FIG. 1 ), according to some examples. In step 1002, method 1000 includes identifying an expected orientation and expected location of a candidate seam on a part to be welded based on a CAD model of the part. The expected orientation and expected location may be determined using annotations provided to the CAD model by a user / operator. Additionally or alternatively, a controller (e.g., controller 108 of FIG. 1 ) may also operate to identify the candidate seam based on model geometry. Object matching may be performed to match components on the part with components in the CAD model. In other words, the expected location and orientation of the candidate seam may be identified based on object matching.

[0069] In step 1004, method 1000 includes acquiring image data of a workspace (e.g., workspace 101 of FIG. 1 ) using one or more sensors (e.g., sensor(s) 102 of FIG. 1 ). The image data may include 2D and / or 3D images of the workspace. As described above, the workspace may include one or more parts to be welded and fixtures and / or clamps that can securely hold the parts. In some examples, a point cloud may be generated from the image data. For example, the images may be overlaid on one another to generate a reconstructed 3D image data. The 3D image data may be collated to generate a point cloud.

[0070] In some examples, to reduce the processing time for generating welding instructions, the sensor is configured to perform partial scans. In other words, instead of scanning the workspace from every angle, image data is collected from a small number of angles (e.g., angles where candidate seams are expected to be visible). In such examples, the point cloud generated from the image data is a partial point cloud. For example, generating a partial point cloud that does not include portions of the part that the model indicates do not include the seam to be welded can reduce scanning time and / or processing time.

[0071] In step 1006, the method 1000 includes identifying candidate seams based on the image data, the point cloud, and / or the subpoint cloud. For example, the controller 108 (FIG. 1) can identify candidate seams using the techniques described above.

[0072] At step 1008, method 1000 includes identifying an actual position and an actual orientation of the candidate seam. For example, at step 1002, a first subset of points may be identified as the modeled seam. At step 1006, a second subset of points may be identified as the candidate seam. In some examples, the first subset of points and the second subset of points may be compared (e.g., using the registration techniques described above with respect to FIG. 6). The first subset of points may be deformed to determine the actual position and orientation of the candidate seam. In some examples, the comparison of the first subset of points and the second subset of points may help determine a tolerance for the first subset of points (e.g., an expected position and expected orientation of the candidate seam). In such examples, the first subset of points may be deformed based on the tolerance to determine the actual position and orientation of the candidate seam. In other words, the expected position and expected orientation of the candidate seam may be refined (in some examples, based on the tolerance) to determine the actual position and actual orientation of the candidate seam. This deformation / refinement technique can take into account surface topography on the part that is not accurately represented in the CAD model (eg, in the CAD model in step 1002).

[0073] In step 1010, the method 1000 includes generating welding instructions for a welding robot based on the actual locations and actual orientations of the candidate seams. For example, path planning can be performed based on the actual locations and actual orientations of the candidate seams.

[0074] 9, once the actual location and actual orientation of the candidate seam are identified, method 1000 may include validating the candidate seam using the techniques described above. However, in contrast to method 900, user interaction in method 1000 may not be necessary because the seam or seams to be welded may already be annotated in the CAD model. Thus, in some cases, a welding robot may be operated and controlled by performing method 1000 without user interaction.

[0075] In addition to or instead of the steps described above with respect to method 1000, a welding robot (e.g., robot 110 of FIG. 1 ) may operate and control the welding robot to perform autonomous welding as follows: The controller 108 of the robot 110, and in particular the robot 110, may scan a workspace containing the part to determine the location of the part within the workspace (e.g., the location of the part on a positioning device) and generate a representation of the part (e.g., a point cloud representation). The controller 108 may be provided with an annotated CAD model of the part. The controller 108 may then determine expected locations and expected orientations of candidate seams on the part according to (or based on) the computer-aided design (CAD) model of the part and the representation of the part. For example, the controller 108 may operate to identify candidate seams based on model geometry—object matching may be performed to match components or features (e.g., topographical features) on the representation of the part with components or features in the CAD model—and the controller 108 may operate to use this object matching in determining the expected locations and expected orientations of candidate seams on the part. Once the expected position and orientation have been determined, the controller 108 may then determine the actual position and orientation of the candidate seam based at least in part on the representation of the part. The actual position and orientation may be determined using the deformation / refinement techniques described in step 1008.

[0076] FIG. 11 is a flow diagram of a method 1100 for performing autonomous welding, according to various examples. More specifically, FIG. 11 illustrates an exemplary method 1100 of operation of a manufacturing robot 110 (FIG. 1) in a robotic manufacturing environment. Assume that one or more parts 114 on which a manufacturing task (e.g., welding) is to be performed have been positioned and / or secured using a fixture 116 on another fixture 116 (e.g., a positioning device) within the manufacturing workspace 101. Following placement of the one or more parts 114, the method 1100 may begin. As an initial step, the method 1100 includes scanning the one or more parts (e.g., block 1110). The scan may be performed by one or more sensors 102 (e.g., scanners not coupled to the robot 110), and the controller 108 may be configured to determine the location of the part within the manufacturing workspace 101 and identify one or more seams on the part using image data acquired from the sensors and / or a point cloud derived from the image or sensor data (blocks 1112 and 1114). The part and seams may be located and identified based on one of the techniques described with respect to FIG. 1. Once the location of the part and seams is determined, the controller 108 plots a path for the manufacturing robot 110 along the identified seams (block 1116). The plotted path may include optimized motion parameters for the manufacturing robot 110 to complete the weld without colliding with itself or others within the manufacturing workspace 101. No human input is required in generating the optimized motion parameters for the manufacturing robot 110 to complete the weld. The path / trajectory may be planned based on one of the path planning techniques described above.

[0077] The terms "position" and "orientation" are described as separate entities in the above disclosure. However, the term "position," when used in the context of a part, means "a particular way in which a part is placed or arranged." The term "position," when used in the context of a seam, means "a particular way in which a seam on a part is positioned or oriented." Thus, the position of a part / seam may inherently consider the orientation of the part / seam. Thus, "position" can include "orientation." For example, position can include the relative physical location or orientation (e.g., angle) of a part or a candidate seam.

[0078] Unless otherwise specified, "about," "approximately," or "substantially" before a value means + / - 10 percent of the stated value. Unless otherwise specified, two objects described as "parallel" are side-by-side and have a distance between them that is constant or varies by 10 percent or less. Unless otherwise specified, two objects described as perpendicular intersect at an angle ranging from 80 degrees to 100 degrees. Modifications are possible in the examples described, and other examples are possible within the scope of the claims.

Claims

1. A method of manufacturing a machine tool comprising: a work space for holding a first part and a second part separate from the first part, the first part and the second part being positioned such that the first and second parts form an unwelded seam to be welded together; a sensor configured to capture a plurality of images within the workspace; a robot configured to weld the first and second parts together along the unwelded seam; identifying the unwelded seam based on the plurality of images; Autonomously planning a collision-free path for the robot to follow when welding the unwelded seam, the path including a plurality of different configurations of the robot; and Commanding the robot to weld the unwelded seam according to the planned path, the robot welding the first and second parts together without human input during welding. The controller and An autonomous robotic welding system, including:

2. The system of claim 1 , wherein each of the plurality of different configurations of the robot comprises a different angle at which an arm of the robot is held.

3. The system of claim 1 , wherein the controller is configured to determine if the robot collides with a structure in the workspace.

4. The system of claim 3 , wherein the controller is configured to plan the path based on the determination.

5. The system of claim 1 , wherein the sensor is configured to capture the images from multiple viewpoints within the workspace.

6. The system of claim 5 , wherein the controller is configured to determine a confidence value that the identified unwelded seam is in fact a seam based on the multiple images from the multiple viewpoints.

7. The system of claim 1 , wherein the controller is configured to constrain fixtures in the workspace to a particular configuration when welding the unwelded seam.

8. The system of claim 7 , wherein the controller is configured to constrain the fixture to a 1F or 2F welding position.

9. a manufacturing tool positioned in the manufacturing workspace for performing a manufacturing process on an unwelded seam formed between a first part and a second part separate from the first part; a robotic arm having multiple degrees of freedom and configured to move the manufacturing tool; one or more sensors configured to capture a plurality of images relating to the first and second parts; determining a location of the first and second parts within the manufacturing workspace using the plurality of images; identifying the unwelded seam where the manufacturing process is to take place; and generating motion parameters for the robot arm to perform the manufacturing process at the unwelded seam; a robot controller configured to perform one or more of the following: A manufacturing robot system comprising:

10. 10. The manufacturing robotic system of claim 9, wherein the one or more sensors are configured to capture image data related to other elements in the manufacturing workspace, the other elements including positioning devices and fixtures.

11. a positioning device within the manufacturing workspace; the first and second parts are positioned on the positioning device, and the robot controller is configured to generate additional motion parameters for the positioning device, the additional motion parameters facilitating coordinated motion between the robot arm and the positioning device.

10. The manufacturing robot system of claim 9.

12. 10. The manufacturing robot system of claim 9, wherein the robot controller is configured to use the images captured by the one or more sensors to determine whether the robot arm will collide with one or more objects in the manufacturing workspace.

13. 10. The manufacturing robotic system of claim 9, wherein the manufacturing workspace includes the manufacturing tool, the robotic arm, a positioning device, and the first and second parts positioned on the positioning device.

Images