Autonomous Welding Robot
Patent Information
- Application Number
- JP2023575689
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-24
- Filing Date
- 2022-02-24
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-02-24
AI Technical Summary
Traditional welding techniques are labor-intensive, inefficient, and inflexible, often requiring skilled operators to handle deviations and collisions, leading to misplaced welds and inefficiencies, especially in low- or medium-volume production scenarios.
A computer-implemented method using sensors to map manufacturing workspaces and dynamically generate welding instructions for a robot, enabling precise seam identification and path planning without prior information, such as CAD models, by employing neural networks and graph search techniques to avoid collisions.
Enables accurate and efficient welding operations in dynamic environments, reducing the need for skilled programmers and minimizing downtime by autonomously identifying seams and planning collision-free paths.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 63 / 153,109, filed February 24, 2021, entitled "SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT," and U.S. Provisional Patent Application No. 63 / 282,827, filed November 24, 2021, entitled "SYSTEMS AND METHODS FOR OPERATING AND CONTROLLING A WELDING ROBOT," the entire contents of each of which are incorporated herein by reference for all purposes. [Background technology]
[0002] background Robotic manufacturing involves the use of robots to perform one or more aspects of a manufacturing process. Robotic welding is one application in the field of robotic manufacturing. In robotic welding, a robot welds two or more components together along one or more seams. Such robots offer significant advantages in production time, reliability, efficiency, and cost because they automate processes that would normally be performed by humans or by machines directly controlled by humans. Summary of the Invention
[0003] overview In various examples, a computer-implemented method for generating instructions for a welding robot includes identifying expected locations of candidate seams on a part to be welded based on a computer-aided design (CAD) model of the part, scanning a workspace including the part to generate a representation of the part, identifying candidate seams on the part based on the representation of the part and the expected locations of the candidate seams, determining actual locations of the candidate seams, and generating welding instructions for the welding robot based at least in part on the actual locations of the candidate seams.
[0004] In an example, a computer-implemented method for generating welding instructions for a welding robot includes acquiring, via a sensor, image data of a workspace including parts to be welded, identifying a plurality of points on the parts to be welded based on the image data, identifying candidate seams on the parts to be welded from the plurality of points, and generating welding instructions for the welding robot based at least in part on the identification of the candidate seams. [Brief description of the drawings]
[0005] [Figure 1] FIG. 1 is a block diagram of an autonomous robotic welding system, in accordance with various examples. [Diagram 2] FIG. 1 is a schematic diagram of an autonomous robotic welding system, in accordance with various examples. [Diagram 3] FIG. 1 is a schematic diagram of an autonomous robotic welding system, in accordance with various examples. [Figure 4] 1 is an exemplary point cloud of a part having a weldable seam, in accordance with various examples. [Diagram 5] 1 is an exemplary point cloud of a part having a weldable seam, in accordance with various examples. [Figure 6] 1 is a block diagram illustrating an alignment process flow, according to various examples. [Figure 7] FIG. 1 is a schematic diagram of a graph search technique that may determine a path plan for a robot, in accordance with various examples. [Figure 8] 1 is a mathematical model of a robot arm, with various examples. [Figure 9] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. [Figure 10] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. [Figure 11] 1 is a flow diagram of a method for performing autonomous welding, in accordance with various examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0006] Detailed Description Traditional welding techniques are tedious, labor intensive, and inefficient. Traditional welding techniques are also not flexible enough to accommodate irregularities commonly encountered during the manufacturing process, leading to undesirable downtime and inefficiencies. For example, traditional welding techniques require skilled programmers to generate instructions for a welding robot to perform a welding job. These instructions instruct the welding robot on the movements, paths, trajectories, and welding parameters that must be used to perform a particular welding job. The instructions are written assuming that a large volume of jobs will be performed in which the same welding job is repeated many times. Thus, any deviations encountered during the welding process (e.g., different parts) can result in misplaced welds. Misplaced welds increase inefficiencies, costs, and other negative aspects of mass production.
[0007] In some cases, a computer-aided design (CAD) model of the part may be useful to the welding robot to facilitate the welding operation. For example, a CAD model of the part to be welded may be provided to the welding robot, and the welding robot may use the CAD model to guide its movement, such as the location of the seam to be welded. The seam(s) to be welded are annotated in the CAD model and the welding robot uses sensors to locate the seam and then applies the weld according to the annotations. While such approaches may reduce or eliminate the need for skilled programmers or manufacturing engineers, they have limitations. For example, welding operations are very precise operations. Generally, to create an acceptable weld, it is desirable for the welding tip to be located within 1 mm of the target location associated with the seam. When guiding the welding robot based on a CAD model, even if the part fits the CAD model exactly, the actual seam may be more than 1 mm from the modeled location, which may make it difficult or impossible for the welding tip to be positioned precisely to create an acceptable weld. If the CAD model is a simplification of the actual part, which is common in low volume production, the seam may be moved a centimeter or more from the modeled location. Thus, it may be difficult to precisely locate the seam based on the CAD model using known techniques. Thus, the welding robot may create an unacceptable weld, thereby resulting in a defective part.
[0008] Furthermore, prior solutions for controlling welding robots require a skilled operator to provide specific instructions to the welding robot (more specifically, the robot arm) to avoid collisions with other components (e.g., parts, sensors, clamps, etc.) as the welding robot (more specifically, the robot arm) moves through the manufacturing workspace along a path from a first point to a second point, such as a seam. Identifying a path (e.g., a path the robot may follow to weld a seam) that is free of obstacles and collisions is referred to herein as path planning. Requiring a skilled operator to do path planning for dozens or even hundreds of potential paths for the robot arm is inefficient, cumbersome, and costly. Furthermore, conventional welding robots are often programmed to repeatedly follow the same paths, the same movements, and the same trajectories. While this repetitive process may be acceptable in high-volume manufacturing situations where the manufacturing process is highly mature, in low- or medium-volume situations, components may be positioned in unexpected locations relative to the welding robot, which may lead to collisions, misaligned parts, poor tolerances, and other issues. Thus, in certain situations, a skilled operator may be required to facilitate welding.
[0009] The welding technique described herein is superior to prior welding robots and welding techniques because it can automatically and dynamically generate instructions useful for the welding robot to accurately and precisely identify and weld seams. Unlike prior systems and techniques, the welding technique described herein does not necessarily require a CAD model of the parts to be welded (although in some instances, as described below, a CAD model may be useful), nor does it necessarily require other prior information about the parts or manufacturing workspace. Rather, the welding technique described herein uses a movable sensor to map the manufacturing workspace (particularly the parts and seams) in three-dimensional (3D) space, and uses such a map to locate and weld seams with a high degree of accuracy and precision. The welding technique described herein includes various additional features that further distinguish it from prior inferior solutions, such as, among others, the ability to identify multiple candidate seams for welding, the ability to interact with a user to select candidate seams for welding, and the ability to dynamically change welding parameters and provide feedback on the welding operation. Additionally, the welding techniques described herein are configured to automatically and dynamically perform path planning using sensor acquired data, i.e., automatically and dynamically, without prior information, identify one or more paths in the manufacturing workspace along which the robot arm may travel without colliding with other components. The welding techniques described herein are also configured to dynamically perform path planning and welding using a combination of sensor acquired data and prior information (e.g., annotated CAD models). These and other examples are now described below with reference to the drawings.
[0010] FIG. 1 is a block diagram of an autonomous robotic welding system 100, according to various examples. The system 100 includes a manufacturing workspace 101, a user interface 106, a controller 108, and a storage device 109 storing a database 112. The system 100 may include other components or subsystems not explicitly described herein. The manufacturing workspace 101 is an area or enclosure within which a robotic arm(s) operates on one or more parts positioned on, coupled to, or otherwise supported by a pedestal or positioning device while assisted by information received by one or more sensors. In an example, the workspace 101 can be any suitable welding area designed with appropriate safety measures for welding. For example, the workspace 101 can be a welding area located in a workshop, job shop, manufacturing plant, assembly shop, etc. In examples, the manufacturing workspace 101 (or, more generally, the workspace 101) may include a sensor 102, a robot 110 configured to perform a welding-type process, such as welding, brazing, joining, etc., a part 114 to be welded (e.g., a part having a seam), and a fixture 116. The fixture 116 may hold, position, and / or manipulate the part 114 and may be, for example, a clamp, a table, a positioning device, or other type of fixture. The fixture 116 may be configured to securely hold the part 114. In examples, the fixture 116 is adjustable manually by a user or automatically by a motor. For example, the fixture 116 may dynamically adjust its position, orientation, or other physical configuration before or during the welding process. In some examples, the robot 110 may include one or more sensors 102. For example, the one or more sensors 102 may be positioned on an arm of the robot 110 (e.g., on a welding head attached to the arm). In another example, one or more sensors 102 may be positioned on a movable, non-welding robotic arm (which may be different from the robot 110).In yet another example, one of the one or more sensors 102 may be positioned on an arm of the robot 110 and another one of the one or more sensors 102 may be positioned on a movable piece of equipment in the workspace. In yet another example, one of the one or more sensors 102 may be positioned on an arm of the robot 110 and another one of the one or more sensors 102 may be positioned on a movable, non-welding robot arm.
[0011] The sensor 102 is configured to capture information about the workspace 101. In an example, the sensor 102 is an image sensor configured to capture visual information (e.g., two-dimensional (2D) images) about the workspace 101. For example, the sensor 102 may include a camera (e.g., a camera with a laser), a scanner (e.g., a laser scanner), and the like. The sensor 102 may include a sensor, such as a Light Detection and Ranging (LiDAR) sensor. Alternatively or additionally, the sensor 102 may be an acoustic sensor configured to emit and / or capture acoustics, such as an Acoustic Navigation and Ranging (SONAR) device. Alternatively or additionally, the sensor 102 may be an electromagnetic sensor configured to emit and / or capture electromagnetic (EM) waves, such as a Radio Detection and Ranging (RADAR) device. Through visual, acoustic, electromagnetic, and / or other sensing techniques, the sensor 102 may collect information about physical structures within the workspace 101. In examples, the sensors 102 collect static information (e.g., fixed structures within the workspace 101), in other examples, the sensors 102 collect dynamic information (e.g., moving structures within the workspace 101), and in still other examples, the sensors 102 collect a combination of static and dynamic information. The sensors 102 may collect any suitable combination of any and all such information regarding the physical structures within the workspace 101 and may provide such information to other components (e.g., the controller 108) to generate a 3D representation of the physical structures within the workspace 101. As discussed above, the sensors 102 may capture and communicate any of a variety of information types, but this description assumes that the sensors 102 primarily capture visual information (e.g., 2D images) of the workspace 101, which are then used together to generate a 3D representation of the workspace 101 as described below.
[0012] To generate a 3D representation of the workspace 101, the sensor 102 captures 2D images of physical structures within the workspace 101 from various angles. For example, a single 2D image of a fixture 116 or part 114 may be inadequate to generate a 3D representation of that component, and similarly, a set of multiple 2D images of a fixture 116 or part 114 from a single angle, view, or plane may also be inadequate to generate a 3D representation of that component, but multiple 2D images captured from multiple angles at various positions within the workspace 101 may be sufficient to generate a 3D representation of a component, such as a fixture 116 or part 114. This is because capturing 2D images at multiple orientations provides spatial information about the component in three dimensions, conceptually similar to the way that a plan drawing of a component that includes a front, side, and top view of the component provides all the information necessary to generate a 3D representation of the component. Thus, in examples, the sensors 102 are configured to move around the workspace 101 to capture sufficient information to generate a 3D representation of the structures within the workspace 101. In examples, the sensors are stationary, but are present in a sufficient number and in a sufficient variety of locations around the workspace 101 such that sufficient information is captured by the sensors 102 to generate said 3D representation. In examples where the sensors 102 are mobile, any suitable structure may be useful to facilitate such movement around the workspace 101. For example, one or more sensors 102 may be positioned on a motorized track system. While the sensors 102 are configured to move around the workspace 101 on the track system, the track system itself may be stationary. However, in some examples, the sensors 102 are mobile on the track system, and the track system itself is mobile around the workspace 101. In yet another example, one or more mirrors are arranged in the workspace 101 with sensors 102 that can pivot, swivel, rotate, or translate about a point and / or along an axis such that when the sensors 102 are in a first configuration, they capture a 2D image from an initial viewpoint, and when in a second configuration, the mirrors are used to capture a 2D image from another viewpoint.In yet other examples, the sensors 102 may be mounted on arms that may be configured to pivot, swivel, rotate, or translate about a point and / or along an axis, and the sensors 102 may be configured to capture 2D images from various viewpoints as these arms extend through their full range of motion.
[0013] Additionally or alternatively, one or more sensors 102 may be positioned on the robot 110 (e.g., on a welding head of the robot 110) and configured to collect image data as the robot 110 moves around the workspace 101. Because the robot 110 is movable in multiple degrees of freedom and therefore in multiple dimensions, the sensors 102 positioned on the robot 110 may capture 2D images from various perspectives. In yet other examples, the one or more sensors 102 may be stationary while a physical structure to be imaged moves around or within the workspace 101. For example, the part 114 to be imaged may be positioned on a fixture 116, such as a positioning device, and the positioning device and / or the part 114 may rotate, translate (e.g., in the x, y, and / or z directions), or otherwise move within the workspace 101 while a fixed sensor 102 (e.g., either a sensor coupled to the robot 110 or a sensor decoupled from the robot 110) captures multiple 2D images of various facets of the part 114.
[0014] In some examples, some or all of the above-mentioned sensor 102 configurations are implemented. Other sensor 102 configurations are contemplated and are within the scope of this disclosure.
[0015] With further reference to FIG. 1 , the robot 110 (e.g., a welding head of the robot 110) is configured to move within the workspace 101 according to a path plan received from the controller 108, as described below. The robot 110 is further configured to perform one or more suitable manufacturing processes (e.g., welding operations) on the part 114 according to instructions received from the controller 108. In some examples, the robot 110 may be a six-axis robot having a welding arm. The robot 110 may be any suitable robotic welding equipment, such as a YASKAWA® robot arm, an ABB® IRB robot, a KUKA® robot, or the like. The robot 110 may be configured to perform arc welding, resistance welding, spot welding, tungsten inert gas (TIG) welding, metal active gas (MAG) welding, metal inert gas (MIG) welding, laser welding, plasma welding, combinations thereof, and the like.
[0016] With further reference to FIG. 1 , the workspace 101, and specifically the sensor(s) 102 and robot 110 within the workspace 101, are coupled to a controller 108. The controller 108 is any suitable machine that is specifically and specifically configured (e.g., programmed) to perform the operations attributed herein to the controller 108, or more generally to the system 100. In some examples, the controller 108 is not a general-purpose computer, but instead is specifically programmed and / or hardware configured to perform the operations attributed herein to the controller 108, or more generally to the system 100. In some examples, the controller 108 is or includes an application specific integrated circuit (ASIC) configured to perform the operations attributed herein to the controller 108, or more generally to the system 100. In some examples, the controller 108 includes or is a processor, such as a central processing unit (CPU). In some examples, the controller 108 is a field programmable gate array (FPGA). In an example, the controller 108 includes a memory storing executable code that, when executed by the controller 108, causes the controller 108 to perform one or more of the operations attributed here to the controller 108, or more generally to the system 100. The controller 108 is not limited to the specific examples described herein.
[0017] The controller 108 controls the sensor(s) 102 and the robot 110 in the workspace 101. In some examples, the controller 108 controls the fixture(s) 116 in the workspace 101. For example, the controller 108 may control the sensor(s) 102 to move within the workspace 101 as described above and / or to capture 2D images, acoustic data, and / or EM data as described above. For example, the controller 108 may control the robot 110 as described herein to perform welding operations and move within the workspace 101 according to path planning techniques as described below. For example, the controller 108 may operate the fixture(s) 116, such as a positioning device (e.g., a table, a clamp, etc.), to rotate, translate, or otherwise move one or more parts within the workspace 101. The controller 108 may also control other aspects of the system 100. For example, the controller 108 may further interact with a user interface (UI) 106 by providing a graphical interface thereon through which a user may interact with and provide input to the system 100, and through which the controller 108 may interact with a user by providing and / or receiving various types of information from the user (e.g., identified seams that are candidates for welding, possible paths during path planning, welding parameter options or selections, etc.). The UI 106 may be any type of interface, including a touch screen interface, a voice-activated interface, a keypad interface, combinations thereof, etc.
[0018] Further, the controller 108 may interact with the database 112, for example, by storing data in the database 112 and / or retrieving data from the database 112. The database 112 may more generally be stored in any suitable type of storage device 109 configured to store any and all types of information. In some examples, the database 112 may be stored in the storage device 109, such as a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a flash memory, and the like. In some examples, the database 112 may be stored in a cloud-based platform. The database 112 may store any information useful to the system 100 in performing a welding operation. For example, the database 112 may store a CAD model of the part 114. As another example, the database 112 may store an annotated version of the CAD model of the part 114. The database 112 may also store a point cloud (also referred to herein as a CAD model point cloud) of the part 114 generated using the CAD model. Similarly, welding instructions for part 114, generated based on a 3D representation of part 114 and / or user input provided regarding part 114 (e.g., regarding which seams of part 114 to weld, welding parameters, etc.), may be stored in database 112. In examples, storage device 109 stores executable code 111 that, when executed, causes controller 108 to perform one or more operations attributed here to controller 108, or more generally to system 100. In examples, executable code 111 is a single self-contained program, and in other examples, executable code is a program having one or more function calls to other executable code that may be stored in storage device 109 or elsewhere. In some examples, one or more functions resulting from execution of executable code 111 may be implemented by hardware.For example, multiple processors may be useful for performing one or more separate tasks of the executable code 111 .
[0019] FIG. 2 is a schematic diagram of an exemplary autonomous robotic welding system 200, according to various examples. The system 200 is an example of the system 100 of FIG. 1, with like numerals referring to like components. For example, the system 200 includes a workspace 201. The workspace 201 further includes an optional movable sensor 202, a robot 210 (which may have one or more sensors 202 (other than the movable sensor 202) attached thereto), and a fixture 216. The robot 210 includes a number of joints and members (e.g., shoulders, arms, elbows, etc.) that allow the robot 210 to move with any suitable number of degrees of freedom. The robot 210 includes a welding head 210A that performs a welding operation on a part, such as a part that may be supported by a fixture 216 (e.g., a clamp). The system 200 further includes a UI 206 coupled to the workspace 201. During operation, the sensor 202 collects a 2D image of the workspace 201 and provides the 2D image to a controller (not explicitly shown in FIG. 2). The controller generates a 3D representation (e.g., a point cloud) of the workspace 201, including the fixture 216, the parts supported by the fixture 216, and / or other structures within the workspace 201. The controller uses the 3D representation to identify seams (e.g., parts supported by the fixture 216), plan a path for the robot 210 to weld the seams without colliding with structures within the workspace 201, and control the robot 210 to weld the seams, as described herein.
[0020] FIG. 3 is a schematic diagram of an autonomous robotic welding system 300, according to various examples. The system 300 is an example of the system 100 of FIG. 1 and the system 200 of FIG. 2, with like numerals referring to like components. For example, the system 300 includes a workspace 301. The workspace 301 further includes an optional movable sensor 302, a robot 310 (to which one or more sensors 302 may be attached), and a fixture 316 (e.g., a table or a positioning device). The robot 310 includes a number of joints and members (e.g., shoulders, arms, elbows, etc.) that allow the robot 310 to move with any suitable number of degrees of freedom. The robot 310 includes a welding head 310A that performs a welding operation on a part, for example, a part that may be supported by the fixture 316. The system 300 further includes a UI 306 coupled to the workspace 301. During operation, the sensor 302 collects a 2D image of the workspace 301 and provides the 2D image to a controller (not explicitly shown in FIG. 3 ). The controller generates a 3D representation (e.g., a point cloud) of the workspace 301, including the fixture 316, the parts supported by the fixture 316, and / or other structures within the workspace 301. The controller uses the 3D representation to identify seams (e.g., parts supported by the fixture 316), plan a path for the robot 310 to weld the seams without colliding with structures in the workspace 301, and control the robot 310 to weld the seams, as described herein.
[0021] Referring again to FIG. 1, as described above, the controller 108 is configured to receive the 2D images (and possibly other data, such as acoustic or EM data) from the sensor 102 and generate a 3D representation of the structure depicted in the 2D images. The 3D representation may also be referred to as a point cloud. A point cloud may be a collection of points, each of which represents a location in 3D space of a point on the surface of the part 114 and / or fixture 116. In some examples, one or more 2D images (e.g., image data captured by the sensor(s) 102 at a particular orientation relative to the part 114) may be overlaid and / or stitched together by the controller 108 to reconstruct and generate 3D image data of the workspace 101. The 3D image data may be collated to generate a point cloud having associated image data for at least some of the points of the point cloud.
[0022] In an example, the 3D image data can be collated by the controller 108 such that a point cloud generated from the data can have six degrees of freedom. For example, each point in the point cloud can represent an infinitesimally small position in 3D space. As described above, the sensor(s) 102 can capture multiple 2D images of a point from various angles. These multiple 2D images can be collated by the controller 108 to determine an average image pixel for each point. The average image pixel can be attached to the point. For example, if the sensor(s) 102 is a color camera with red, green, and blue channels, the six degrees of freedom can be {x position, y position, z position, red intensity, green intensity, and blue intensity}. For example, if the sensor(s) 102 is a black and white camera with black and white channels, four degrees of freedom can be generated.
[0023] Figure 4 is an example point cloud 400 of a part having a weldable seam, in accordance with various examples. More specifically, point cloud 400 represents part 402 and part 404 to be welded together along seam 406. Figure 5 is an example point cloud 500 of a part having a weldable seam, in accordance with various examples. More specifically, point cloud 500 represents part 502 and part 504 to be welded together along seam 506. Controller 108 (Figure 1) is configured to generate 3D point clouds 400, 500 based on 2D images captured by sensor 102, as described above. The controller 108 may then use the point clouds 400, 500 (or, in some examples, image data useful for generating the point clouds 400, 500) to identify and locate seams, such as seam 406, 506, plan a welding path along the seam 406, 506, and apply a weld along the seam 406, 506 using the robot 110 (FIG. 1) according to the path plan. The manner in which the controller 108 executes the executable code 111 (FIG. 1) to perform such operations, including seam identification and path planning, will now be described in detail.
[0024] Upon executing the executable code 111, the controller 108 performs pixel-by-pixel classification (e.g., using images captured by the sensor 102 or based on images captured by the sensor 102) and / or point-by-point classification (e.g., using one or more point clouds) using a neural network to identify and classify structures in the workspace 101. For example, the controller 108 may perform pixel-by-pixel and / or point-by-point classification to identify each imaged structure in the workspace 101, for example, as a part 114, as an interface seam (referred to herein as a candidate seam) on or between the parts 114, as a fixture 116, as a robot 110, etc. The controller 108 may, in examples, identify and classify pixels and / or points based on a neural network (e.g., a U-net model) trained using appropriate training data. The neural network may be trained on image data, point cloud data, spatial information data, or a combination thereof. Since the point cloud and / or image data includes information captured from various viewpoints within the workspace 101, the neural network can operate to classify the fixture 116 or candidate seams on the part(s) 114 from multiple angles and / or viewpoints. In some examples, the neural network can be trained to operate directly on the point set, such as, for example, a dynamic graph convolutional neural network, or the neural network may be implemented to analyze unorganized points on the point cloud. In some examples, a first neural network can be trained on the point cloud data to perform point-by-point classification, and a second neural network can be trained on the image data to perform pixel-by-pixel classification. The first neural network and the second neural network can individually identify the candidate seams and locate the candidate seams. The outputs from the first neural network and the second neural network can be combined as a final output to determine the position and orientation of one or more candidate seams on the part 114.
[0025] In some examples, where pixel-by-pixel classification is performed, the results can be projected onto the 3D point cloud data and / or a meshed version of the point cloud data, thereby providing information regarding the location of the fixture 116 within the workspace 101. If the input data is image data (e.g., a color image), spatial information such as depth information may be included along with the color data to perform pixel-by-pixel segmentation. In some examples, as described further below, pixel-by-pixel classification can be performed to identify candidate seams and locate the candidate seams relative to the part 114.
[0026] As described above, the controller 108 may identify and classify pixels and / or points as particular structures within the workspace 101, such as the fixture 116, the part 114, candidate seams on the part 114, etc. Portions of the image and / or point cloud data classified as non-part and non-candidate seam structures, such as the fixture 116, may be segmented and separated (e.g., deleted or otherwise removed) from the data, thereby isolating data identified and classified as corresponding to the part 114 and / or candidate seam(s) on the part 114. In some examples, after identifying the candidate seams and segmenting the non-part 114 and non-candidate seam data as described above (or optionally prior to such segmentation), a neural network may be configured to analyze each candidate seam to determine a seam type. For example, the neural network may be configured to determine whether the candidate seam is a butt joint, corner joint, edge joint, lap joint, T-joint, etc. A model (e.g., a U-net model) can classify seam types based on data captured from multiple viewpoints within the workspace 101.
[0027] If pixel-by-pixel classification is performed using image data, the controller 108 may project the pixels of interest (e.g., pixels representing the part 114 and candidate seams on the part 114) onto the 3D space to generate a 3D point set representing the part 114 and candidate seams on the part 114. Alternatively, if point-by-point classification is performed using point cloud data, the points of interest may already exist in the 3D space of the point cloud. In either case, to the controller 108, the 3D points are an unordered set of points, and at least some of the 3D points may be clumped together. To eliminate such noise and generate a subset of contiguous, unbroken points to represent the candidate seams, a Manifold Blurring and Mean Shift (MBMS) technique or a similar technique may be applied. Such a technique may cluster the points and eliminate noise. The controller 108 may then apply a clustering method to decompose the candidate seams into individual candidate seams. In other words, instead of having several subsets of points that represent multiple seams, clustering can decompose each subset of points into individual seams. Following clustering, the controller 108 can fit a spline to each individual subset of points. Thus, each individual subset of points can be an individual candidate seam.
[0028] In summary, using the techniques described above, but without limitation, the controller 108 receives image data captured by the sensor 102 from various positions and viewpoints within the workspace 101. The controller 108 performs pixel-by-pixel and / or point-by-point classification techniques using neural networks to classify and identify each pixel and / or point as a part 114, a candidate seam on the part 114 or at an interface between multiple parts 114, a fixture 116, etc. The structures identified as non-part 114 structures and non-candidate seam structures are segmented and separated, and the controller 108 may perform additional processing on the remaining points (e.g., to mitigate noise). By performing these operations, the controller 108 may generate a set of candidate seams on the part 114 that indicate the location and orientation of these seams. As will be described next, the controller 108 may then determine whether the candidate seams are in fact seams, and may optionally perform additional processing using prior information, such as CAD models of the parts and seams. The resulting data is suitable for use by the controller 108 to plan a path for applying a weld along the identified seam, as also described below.
[0029] In some cases, the identified candidate seam may not be a seam (i.e., the identified candidate seam may be a false positive). To determine whether the identified candidate seam is actually a seam, the controller 108 determines a confidence value using images captured by the sensor 102 from various viewpoints in the workspace 101. The confidence value represents the likelihood that the candidate seam determined from the corresponding viewpoint is an actual seam. The controller 108 may then compare the confidence values for the different viewpoints and eliminate candidate seams that are unlikely to be actual seams. For example, the controller 108 may determine the mean, median, maximum, or any other suitable summary statistic of the candidate values associated with a particular candidate seam. In general, a candidate seam that corresponds to an actual seam will have a consistently high (e.g., above a threshold) confidence value across the various viewpoints used to capture the candidate seam. If the summary statistic of the confidence values for the candidate seam is above a threshold, the controller 108 may designate the candidate seam as an actual seam. Conversely, if the summary statistics of the confidence values for a candidate seam are below a threshold, the candidate seam may be designated as a false positive that is not eligible for welding.
[0030] As described above, after identifying a candidate seam that is an actual seam, the controller 108 may use prior information, such as a CAD model (or a point cloud version of the CAD model), to perform additional processing, referred to herein as alignment. More specifically, in some cases, differences may exist between seam dimensions on the part and seam dimensions in the CAD model, and the CAD model needs to be transformed (e.g., updated) to account for any such differences, since the CAD model may later be used to perform path planning as described herein. Thus, the controller 108 compares a first seam (e.g., a candidate seam on the part 114 that is verified as an actual seam) to a second seam (e.g., a seam annotated (e.g., by an operator / user) on the CAD model that corresponds to the first seam) to determine the difference between the first seam and the second seam. The seam on the CAD model may be annotated as described above. The first seam and the second seam may be in approximately the same location if the CAD model and / or the controller 108 accurately predicts the location of the candidate seam. Alternatively, the first seam and the second seam may partially overlap if the CAD model and / or the controller 108 are partially accurate. The controller 108 may perform a comparison between the first seam and the second seam. This comparison between the first seam and the second seam may be based in part on the shape of both seams and their relative positions in space. If the shapes of the first seam and the second seam are relatively similar and in close proximity to each other, the second seam may be identified as the same as the first seam. In this manner, the controller 108 may take into account the topography of surfaces on the part that are not accurately represented in the CAD model. In this manner, the controller 108 may identify candidate seams and may sub-select or refine or update the candidate seams to the part using the CAD model of the part. Each candidate seam may be an updated set of points representing the position and orientation of the candidate seam relative to the part.
[0031] FIG. 6 is a block diagram illustrating an alignment process flow 600, according to various examples. Some or all of the steps of the alignment process flow 600 are performed by the controller 108 (FIG. 1). The controller 108 may first perform a coarse alignment 602 using a CAD model point cloud 604 and a scan point cloud 606 formed using images captured by the sensor 102. The CAD model point cloud 604 and the scan point cloud 606 may be sampled such that their points have a uniform or near uniform distribution and such that their points both have equal or near equal point density. In an example, the controller 108 downsamples the point clouds 604, 606 to randomly select and keep points in the clouds uniformly and discard the remaining unselected points. For example, in some examples, a Poisson disk sampling (PDS) downsampling algorithm may be implemented. The controller 108 may provide as input to the PDS algorithm the boundaries of the point clouds 604, 606, the minimum distance between samples, and the limit of samples to be selected before they are rejected. In some examples, a delta network may be useful to transform one model into another during the coarse registration 602. The delta network may be a Siamese network that takes the source and target models and encodes them into latent vectors. The controller 108 may use the latent vectors to predict point-by-point transformations that transform or update one model into another. The delta network may not require a training dataset. Given a CAD model and a scan point cloud 604, 606, the controller 108 in the context of a delta network spends one or more epochs learning the dissimilarity or similarity between the two point clouds. During these epochs, the delta network learns meaningful features that are useful for later registration. The delta network may use skip connections to learn the transformations, but in some examples, skip connections may not be used. In some cases, the CAD model includes surfaces that are not present in the 3D point cloud generated using the captured image (e.g., scan).In such a case, the delta network moves all points corresponding to the missing surface from the CAD model point cloud 604 to some points in the scan point cloud 606 (and updates the scan point cloud 606). Thus, during alignment, the controller 108 (e.g., the delta network) may transform (or update) the original CAD model using the learned features, or may transform the deformed CAD model using the learned features. In an example, the delta network may include an encoder network, such as a dynamic graph convolutional neural network (DGCNN). After the point cloud is encoded into features, a concatenated vector consisting of both the CAD and scan embeddings may be formed. After performing a pooling operation (e.g., max pooling), a decoder may be applied to the resulting vector. In some examples, the decoder may include five convolutional layers with specific filters (e.g., 256, 256, 512, 1024, Nx3 filters). The resulting output may be concatenated with the CAD model and scan embeddings, max pooled, and then provided to the decoder once again. The final result may include a point-wise transformation.
[0032] Extraneous data and noise in the data (e.g., the output of the coarse alignment 602) can affect the alignment of the parts 114. For at least this reason, it is desirable to remove as much of the extraneous data and noise as possible. The bounding box 608 is useful for removing this extraneous data and noise (e.g., fixtures 116) to limit the area in which the alignment takes place. Stated differently, data within the bounding box is retained, but all data from outside the bounding box, 3D or otherwise, is discarded. The aforementioned bounding box may be any shape that can surround or encapsulate (e.g., partially or completely) the CAD model itself. For example, the bounding box may be a dilated or enlarged version of the CAD model. Data outside the bounding box may be removed from the final alignment, or may still be included but weighted to reduce its impact.
[0033] With further reference to FIG. 6, during fine alignment 610, the controller 108 passes the output of the bounding box 608 as a patch through a set of convolutional layers in a neural network trained as an autoencoder. More specifically, the data may be passed through the encoder section of the autoencoder, and the decoder section of the autoencoder may not be used. The input data may be the XYZ positions of the points of the patch within the shape, e.g., (128, 3). The output may be a vector of length 1024, for example, which is useful for point-wise features.
[0034] The set of correspondence points that best supports the rigid transformation between the CAD and scan point cloud models should be determined during the registration. The correspondence candidates are stored as a matrix where each element stores the confidence or probability of a match between two points: It may be stored (e.g., in database 112) as TIFF2024508563000002.tif18128.
[0035] Various methods are available for finding corresponding points from this matrix, e.g., hard correspondences, soft correspondences, product manifold filters, graph cliques, covariance, etc. After the fine registration 610 is completed, the registration process is then completed (612).
[0036] As mentioned above, in some cases, the actual location of the seam on the part 114 may differ from the seam location determined by the controller 108 using sensor imaging (e.g., using a scan point cloud) and / or determined by a CAD model (e.g., using a CAD model point cloud). In such cases, a scanning procedure (sometimes referred to herein as a pre-scan) is useful to correct the determined seam location to more closely match the actual seam location on the part 114. In a scanning procedure, a sensor 102 (sometimes referred to herein as an on-board sensor) positioned on the robot 110 performs a scan of the seam. In some cases, this scanning may be performed using an initial movement and / or path plan generated by the controller 108 using a CAD model, a scan, or a combination thereof. For example, the sensor 102 may scan any or all areas of the workspace 101. During the execution of this initial movement and / or path plan, the sensor 102 may capture observed images and / or data. The observed images and / or data may be processed by the controller 108 to generate seam point cloud data. The controller 108 may use the seam point cloud data to correct seam locations when processing the point cloud(s) 604 and / or 606. The controller 108 may also use the seam point cloud data in correcting path and motion planning.
[0037] In some examples, the alignment techniques described above may be useful for comparing and matching seams determined using sensors 102 other than the on-board sensor 102 to seams identified by the on-board sensor 102. By matching seams in this manner, the robot 110 (more specifically, the head of the robot 110) may be positioned relative to the actual seams as needed.
[0038] In some examples, the pre-scan trajectory of the robot 110 is identical to the trajectory planned for welding along the seam. In some such examples, the movements taken for the robot 110 during the pre-scan may be generated separately to limit the probability of or reduce instances of collision, to better visualize the seam or critical geometry with the on-board sensor 102, or to scan the geometry around the seam in question.
[0039] In some examples, pre-scanning techniques may include scanning more than a particular seam or seams, but rather may also include scanning other geometries of the part(s) 114. The scan data may be useful for more precise application of any or all of the techniques described herein (e.g., alignment techniques) to find, locate, and detect the seams, and to ensure that the head of the robot 110 is positioned and moved along the seams as necessary.
[0040] In some examples, scanning techniques (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / weld head) may be useful to identify gap variability information about the seam, rather than position and orientation information about the seam. For example, scanned images captured by the sensor(s) 102 on the robot 110 during a scanning procedure may be useful to identify gap variability and adjust the weld trajectory or path plan to account for such gaps. For example, 3D points, 2D image pixels, or a combination thereof may be useful to locate variable gaps between the parts 114 to be welded. In some examples, variable gap finding is useful, where 3D points, 2D image pixels, or a combination thereof are useful to locate, identify, and measure the variable sizes of multiple gaps between the parts 114 to be welded together. In tack weld finding or general weld finding, previous welds or material deposits in the gaps between the parts 114 to be welded may be identified using 3D points and / or 2D image pixels. Any or all of such techniques may be useful for optimizing welding, including path planning. In some cases, gap variability may be identified in a 3D point cloud generated using images captured by the sensor 102. In yet other cases, gap variability may be identified by scanning techniques (e.g., scanning the actual seam using a sensor / camera attached to the welding arm / welding head) performed during welding operations for a task. In any one case, the controller 108 may be configured to dynamically adapt the welding instructions (e.g., welding voltage) based on the determined location and size of the gap. For example, by dynamically adjusting the welding instructions for the welding robot, precise welding of seams at variable gaps can be obtained. Adjusting the welding instructions may include adjusting one or more of the welder voltage, welder current, duration of the electric pulse, shape of the electric pulse, and material feed rate.
[0041] In an example, the user interface 106 may provide the user with an option to display candidate seams. For example, the user interface 106 may provide a graphical representation of the part 114 and / or the candidate seams on the part 114. Additionally or alternatively, the user interface 106 may group the candidate seams based on seam type. As described above, the controller 108 may identify the seam type. For example, candidate seams identified as lap seams may be grouped under the label "lap seam" and presented to the user via the user interface 106 under the label "lap seam." Similarly, candidate seams identified as edge seams may be grouped under the label "edge seam" and presented to the user via the user interface 106 under the label "edge seam."
[0042] The user interface 106 may further provide the user with the option to select a candidate seam to be welded by the robot 110. For example, each candidate seam on the part 114 may be presented as a push button on the user interface 106. When the user presses a particular candidate seam, the selection may be transmitted to the controller 108. The controller 108 may generate instructions for the robot 110 to perform a welding operation on that particular candidate seam.
[0043] In some examples, the user may be provided with the option to update the welding parameters. For example, the user interface 106 may provide the user with a list of different welding parameters. The user may select the particular parameter to be updated. Changes to the selected parameter may be made, for example, using a drop-down menu or by text entry. This update may be sent to the controller 108 so that the controller 108 can update the instructions for the robot 110.
[0044] In examples where the system 100 is not provided with prior information (e.g., a CAD model) of the part 114, the sensor(s) 102 can scan the part 114. A representation of the part 114 can be presented to a user via the user interface 106. This representation of the part 114 can be a point cloud and / or a mesh of a point cloud that includes projected 3D data of the scanned image of the part 114 obtained from the sensor(s) 102. The user can annotate the seams to be welded in the representation via the user interface 106. Alternatively, the controller 108 can identify candidate seams in the representation of the part 114. The candidate seams can be presented to the user via the user interface 106. The user can select the seam to be welded from among the candidate seams. The user interface 106 can annotate the representation based on the user's selection. In some examples, the annotated representation can be stored in the database 112.
[0045] After one or more seams on the part(s) 114 have been identified using the techniques described above (or using other suitable techniques) and corrected to the extent possible, the controller 108 plans a path for the robot 110 during the subsequent welding process. In some examples, graph matching and / or graph searching techniques may be useful in planning the path of the robot 110. A particular seam identified as described above may include multiple points, and the path planning techniques involve determining different states of the robot 110 for each such point along a given seam. The states of the robot 110 may include, for example, the position of the robot 110 in the workspace 101, and specific configurations of the arms of the robot 110 in any number of degrees of freedom that may be applied. For example, in the case of a robot 110 having an arm with six degrees of freedom, the state of the robot 110 would not only include the position of the robot 110 in the workspace 101 (e.g., the position of the welding head of the robot 110 in the three-dimensional xyz space), but also specific sub-states for each of the six degrees of freedom of the robot arm. Additionally, as the robot 110 transitions from a first state to a second state, the robot 110 may change its position within the workspace 101, which will entail the robot 110 traversing a particular path within the workspace 101 (e.g., along the seam being welded). Thus, specifying a set of states for the robot 110 entails specifying a path for the robot 110 to travel within the workspace 101. The controller 108 may perform a pre-scanning technique or a variant thereof after path planning is completed, and the controller 108 may use information captured during the pre-scanning technique to make any of a variety of appropriate adjustments (e.g., adjustments to the XYZ axes or coordinate system used to perform the actual weld along the seam).
[0046] FIG. 7 is a schematic diagram 700 of a graph search technique for determining a path plan for the robot 110 (e.g., by the controller 108). Each circle in the diagram 700 represents a different state of the robot 110 (e.g., a particular position of the robot 110 in the workspace 101 (e.g., the position of the welding head of the robot 110 in 3D space) and a different configuration of the arms of the robot 110, as well as the location and / or configuration of fixtures supporting the parts, such as positioners, clamps, etc.). Each row 702, 706, and 710 represents a different point along the seam to be welded. Thus, for a seam point corresponding to row 702, the robot 110 may be in any one of states 704A-704D. Similarly, for a seam point corresponding to row 706, the robot 110 may be in any one of states 708A-708D. Similarly, for a seam point corresponding to row 710, the robot 110 may be in any one of states 712A-712D. For example, if the robot 110 is in state 704A when at a seam point corresponding to row 702, the robot 110 may then transition to any of states 708A-708D for the next seam point corresponding to row 706. Similarly, upon entering states 708A-708D, the robot 110 may then transition to any of states 712A-712D for the next seam point corresponding to row 710, and so on. In some examples, entering a particular state may preclude entering other states. For example, entering state 704A may allow the possibility of subsequently entering states 708A-708C but not 708D, while entering state 704B may allow the possibility of subsequently entering states 708C and 708D but not states 708A-708B. The scope of the present disclosure is not limited to any particular number of seam points or any particular number of robot 110 states.
[0047] In some examples, to determine a path plan for the robot 110 using a graph search technique (e.g., according to the technique shown in diagram 700), the controller 108 may determine the shortest path from states 704A-704D to a state corresponding to the seam point N (e.g., states 712A-712D). An objective function may be designed by the controller 108 by assigning a cost to each state and each transition between states. The controller 108 finds the path that results in the minimum cost value of the objective function. For the freedom of having multiple start and end points to choose from, graph search methods such as Dijkstra's algorithm or A* may be implemented. In some examples, a brute force technique may be useful to determine an appropriate path plan. The brute force technique involves the controller 108 calculating all possible paths (e.g., through diagram 700) and selecting the shortest path.
[0048] The controller 108 may determine whether the state at each seam point is feasible, which means, at least in part, that the controller 108 may determine whether implementing a chain of states along a series of seam points of the seam would cause a collision between the robot 110 and a structure in the workspace 101, or even with parts of the robot 110 itself. To this end, the concept of realizing different states at different points of the seam may alternatively be expressed in the context of a seam having multiple waypoints. First, the controller 108 may discretize the identified seam into a series of waypoints. The waypoints may constrain the orientation of a welding head connected to the robot 120 with three (spatial / translational) degrees of freedom. Typically, constraints on the orientation of the welding head of the robot 120 are provided in one or two rotational degrees of freedom around each waypoint with the purpose of producing some desired weld of some quality, and the constraints are typically on the surface normal vectors emanating from the waypoints and the path of the weld seam. For example, the position of the welding head may be constrained in the x-axis, y-axis, and z-axis, as well as about one or two rotational axes perpendicular to the axis of the welding wire or tip of the welding machine, all relative to the waypoint and some nominal coordinate system attached to the waypoint. These constraints in some examples may be angle limits or tolerances. Those skilled in the art will recognize that the ideal or desired welding angle may vary based on the geometry of the part or seam, the direction of gravity relative to the seam, and other factors. In some examples, the controller 108 may constrain to a 1F or 2F welding position to ensure that the seam is perpendicular to gravity for one or more reasons (e.g., to find a balance between welding and path planning for optimization purposes). Thus, the position of the welding head may be held (constrained) by each waypoint at any suitable orientation relative to the seam. Typically, the welding head is not constrained about a rotational axis (θ) that is coaxial with the axis of the welding head.For example, each waypoint may define a position of a welding head of the welding robot 120 such that at each waypoint the welding head is at a fixed position and orientation relative to the weld seam. In some implementations, the waypoints are discretized fine enough to make the movement of the welding head substantially continuous.
[0049] The controller 108 may divide each waypoint into multiple nodes. Each node may represent a possible orientation of the welding head at that waypoint. As a non-limiting example, the welding head may be unconstrained about a rotation axis coaxial with the axis of the welding head such that the welding head may rotate (e.g., 360 degrees) along the rotation axis θ at each waypoint. Each waypoint may be divided into 20 nodes such that each node of each waypoint represents the welding head in 18 degree rotation increments. For example, a first waypoint node pair may represent 0 degrees of welding head rotation, a second waypoint node pair may represent welding head rotation from 18 degrees, a third waypoint node pair may represent welding head rotation from 36 degrees, and so on. Each waypoint may be divided into 2, 10, 20, 60, 120, 360, or any suitable number of nodes. Subdivision of the nodes may represent division of orientation beyond one degree of freedom. For example, the orientation of the welder tip around the waypoint may be defined by three angles. A welding path can be defined by connecting each waypoint-node pair, and the distance between the waypoints and the offset between adjacent waypoint-nodes can then represent the amount of translation and rotation of the welding head as it moves between the node-waypoint pairs.
[0050] The controller 108 can evaluate each waypoint node pair for feasibility of welding. Consider a non-limiting example of dividing the waypoints into 20 nodes. The controller 108 can evaluate whether the first waypoint node pair, representing the welding head held at 0 degrees, is feasible. In other words, the controller 108 can evaluate whether the robot 110 will collide or interfere with a part, a fixture, or the welding robot itself if placed in the position and orientation defined by that waypoint node pair. Similarly, the controller 108 can evaluate whether the second waypoint node pair, the third waypoint node pair, and so on, are feasible. The controller 108 can similarly evaluate each waypoint. In this manner, all feasible nodes for all waypoints can be determined.
[0051] In some examples, collision analysis as described herein may be performed by comparing a 3D model of the workspace 101 to a 3D model of the robot 110 to determine whether the two models overlap, and optionally, whether some or all of the triangles overlap. If the two models overlap, the controller 108 may determine that a collision is likely. If the two models do not overlap, the controller 108 may determine that a collision is unlikely. More specifically, in some examples, the controller 108 may compare the models for each of a set of waypoint node pairs (such as the waypoint node pairs described above) and determine that the two models overlap for a subset or possibly all of the waypoint node pairs. For the subset of waypoint node pairs for which model intersections are identified, the controller 108 may omit the waypoint node pairs in that subset from the planned path and may identify alternative pairs for those waypoint node pairs. The controller 108 may repeat this process as necessary until a collision-free path is planned. The controller 108 may use a flexible collision library (FCL), which includes various techniques for efficient collision detection and proximity calculation as a tool for collision avoidance analysis. The FCL is useful for making multiple proximity queries against different model representations and may be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used in conjunction with or in place of the FCL.
[0052] The controller 108 may generate one or more feasible simulated (or evaluated, both terms are used interchangeably herein) welding paths, if physically feasible. The welding path may be a path that the welding robot takes to weld the candidate seam. In some examples, the welding path may include all waypoints of the seam. In some examples, the welding path may include some, but not all, waypoints of the candidate seam. The welding path may include the movement of the robot and the welding head as the welding head moves between each waypoint-node pair. Once a feasible path between the node-waypoint pairs is identified, a feasible node-waypoint pair of the next successive waypoint may be identified, if one exists. Those skilled in the art will recognize that many search trees or other strategies may be employed to evaluate the space of feasible node-waypoint pairs. As discussed in more detail herein, a cost parameter may be assigned or calculated for movement from each node-waypoint pair to the subsequent node-waypoint pair. Cost parameters may be associated with the travel time, the amount of movement between node-waypoint pairs (e.g., including rotations), and / or the simulated / expected weld quality produced by the welding head during the movement.
[0053] If a node is not feasible for welding for one or more waypoints and / or no feasible path exists for traveling between a previous waypoint node pair and any of the waypoint node pairs for a particular waypoint, the controller 108 can determine alternative welding parameters such that at least some additional waypoint node pairs are feasible for welding. For example, if the controller 108 determines that none of the waypoint node pairs for a first waypoint are feasible, thereby causing the first waypoint to be unweldable, the controller 108 can determine alternative welding parameters, such as alternative welding angles, such that at least some waypoint node pairs for the first waypoint are weldable. For example, the controller 108 can remove or relax constraints on rotation about the x-axis and / or y-axis. Similarly, the controller 108 can allow the weld angle to vary in one or two additional rotational (angular) dimensions. For example, the controller 108 can split the unweldable waypoint into two-dimensional or three-dimensional nodes. Each node can then be evaluated for welding feasibility with the welding robot and weld held at various welding angles and rotations. Additional rotation or other degrees of freedom about the x-axis and / or y-axis may make the waypoint accessible to the welding head so that the welding head does not encounter any collisions. In some implementations, if a node is not feasible for welding one or more waypoints and / or no feasible path exists for moving between the previous waypoint node pair and any of the waypoint node pairs of the particular waypoint, the controller 108 can use the degrees of freedom provided by the positioner system in determining a feasible path between the previous waypoint node pair and any of the waypoint node pairs of the particular waypoint.
[0054] Based on the generated welding path, the controller 108 can optimize the welding path for welding. (Optimal and optimizing, as used herein, do not refer to determining the absolute best welding path, but generally refer to techniques that can reduce welding time and / or improve weld quality relative to less efficient welding paths.) For example, the controller 108 can determine a cost function that seeks local minima and / or minimums of the robot 110 movements. Typically, an optimal welding path minimizes the rotation of the welding head, since rotation of the welding head can increase the time to weld a seam and / or reduce weld quality. Thus, optimizing the welding path can include determining a welding path that passes through a maximum number of waypoints with a minimum amount of rotation.
[0055] In assessing the feasibility of welding at each of the partitioned nodes or node-waypoint pairs, the controller 108 may perform multiple calculations. In some examples, each of the multiple calculations may be mutually exclusive of one another. In some examples, a first calculation may include a kinematic feasibility calculation, which calculates whether the arm of the robot 110 of the welding robot being employed can mechanically reach (or be) at a state defined by the node or node-waypoint pair. In some examples, in addition to the first calculation, a second calculation, which may be mutually exclusive with the first calculation, may also be performed by the controller 108. The second calculation may include determining whether the arm of the robot 110 will encounter a collision (e.g., collide with the workspace 101 or a structure within the workspace 101) when accessing a portion of the seam (e.g., the node or node-waypoint pair in question).
[0056] The controller 108 may perform the first calculation before performing the second calculation. In some examples, the second calculation may only be performed if the result of the first calculation is positive (e.g., if it is determined that the arm of the robot 110 can mechanically reach (or be) in the state defined by the node or node-waypoint pair). In some examples, the second calculation may not be performed if the result of the first calculation is negative (e.g., if it is determined that the arm of the robot 110 cannot mechanically reach (or be) in the state defined by the node or node-waypoint pair).
[0057] Kinematic feasibility may correlate with the type of robot arm employed. For purposes of this description, it is assumed that the welding robot 110 includes a six-axis robotic welding arm with a spherical wrist. A six-axis robotic arm may have six degrees of freedom, i.e., three in the X, Y, Z Cartesian coordinates, and an additional three degrees of freedom due to the wrist-like nature of the robot 110. For example, the wrist-like nature of the robot 110 provides a fourth degree of freedom in the up-down direction of the wrist (e.g., the wrist moves in the +y and -y directions), a fifth degree of freedom in the lateral direction of the wrist (e.g., the wrist moves in the -x and +x directions), and a sixth degree of freedom in rotation. In some examples, a welding torch is attached to the wrist portion of the robot 110.
[0058] To determine whether the employed robot 110 arm can mechanically reach (or reside) in a state defined by a node or node-waypoint pair, i.e., to perform the first calculation, the robot 110 may be mathematically modeled as shown in the exemplary model 800 of FIG. 8. In some examples, the controller 108 may solve for the first three joint variables based on the wrist position and the other three joint variables based on the wrist orientation. Note that the torch is rigidly attached to the wrist; therefore, the transformation between the torch tip and the wrist is assumed to be fixed. Geometric techniques (e.g., the law of cosines) may be employed to find the first three joint variables (e.g., variables S, L, U in 802, 804, 806, respectively).
[0059] After the first three joint variables (i.e., S, L, U) are successfully calculated, the controller 108 may then solve for the last three joint variables (i.e., R, B, T in 808, 810, 812, respectively), for example, by considering the wrist orientation as ZYZ Euler angles. The controller 108 may take into account some offsets in the robot 110. These offsets may need to be considered and accounted for due to inconsistencies in the unified robot description format (URDF) file. For example, in some examples, the position values (e.g., X-axis of the joint) of the joints (e.g., the actual joints of the robot 110) may not match the values described in that URDF file. Such offset values may be provided to the controller 108 in a table. The controller 108 may take into account these offset values while mathematically modeling the robot 110, in some examples. In some examples, after the robot 110 is mathematically modeled, the controller 108 may determine whether an arm of the robot 110 can mechanically reach (or be) at a state defined by a node or node-waypoint pair.
[0060] As described above, the controller 108 can evaluate whether the robot 110, when placed in a position and orientation defined by a waypoint-node pair, will collide or interfere with other things in the workspace 101, including the part 114, fixture 116, or the robot 110 itself. Once the controller 108 has determined the conditions the robot arm can be in, the controller 108 may use a second calculation to make the aforementioned evaluation (e.g., as to whether the robot will collide with something in its environment).
[0061] FIG. 9 is a flow diagram of a method 900 for performing autonomous welding, according to various examples. More specifically, FIG. 9 is a flow diagram of a method 900 for operating and controlling a welding robot (e.g., robot 110 of FIG. 1), according to some examples. In step 902, the method 900 includes acquiring image data of a workspace (e.g., workspace 101 of FIG. 1) using one or more sensors (e.g., sensor(s) 102 of FIG. 1). The image data may include 2D and / or 3D images of the workspace. As described above, one or more parts to be welded, fixtures and / or clamps that can securely hold the parts may be located in the workspace. In some examples, a point cloud may be generated from the image data. For example, the images may be overlaid on one another to reconstruct and generate three-dimensional image data. The three-dimensional image data may be matched to one another to generate a point cloud.
[0062] At step 904, the method 900 includes identifying a set of points on the parts to be welded based on the sensor data, which may be an image. The set of points may represent possible seams to be welded. In some examples, the neural network may perform pixel-by-pixel segmentation on the image data to identify the set of points. Fixtures and clamps in the image data may be classified by the neural network based on image classification. Portions of the image data associated with fixtures and / or clamps may be segmented and isolated such that those portions of the image data are not used to identify the set of points, which may reduce computational resources required to identify the set of points to be welded by reducing the search space. In such examples, the set of points may be identified from other portions of the image data (e.g., non-segmented portions of the image data).
[0063] At step 906, the method 900 includes identifying a candidate seam from the point set. For example, a subset of points in the point set may be identified as the candidate seam. A neural network may perform image classification and / or depth classification to identify the candidate seam. In some examples, the candidate seam may be located relative to the part. For example, to locate the candidate seam, a position and orientation of the candidate seam relative to the part may be determined.
[0064] Moreover, the method 900 further includes verifying whether the candidate seam is an actual seam. As described above, the sensor(s) can collect image data from multiple angles. For each image captured from a different angle, a confidence value can be determined that represents whether the candidate seam determined from that angle is an actual seam. If the confidence value is above a threshold based on the views taken from the multiple angles, the candidate seam can be verified as an actual seam. In some aspects, the method 900 also includes classifying the candidate seam as a seam type. For example, a neural network can determine whether the candidate seam is a butt seam, a corner seam, a edge seam, a lap seam, a T-joint, etc.
[0065] In some examples, after the candidate seams are identified and verified, the subsets of points may be clustered together to form a continuous, unbroken seam. At step 908, the method 900 includes generating welding instructions for a welding robot based on the candidate seams. For example, the welding instructions may be generated by tracing a path from one end of the subset of points to the other end of the subset of points. This may generate a path for the seam. In other words, welding may be performed by tracing this path with a welding head. Further, path planning may be performed based on the identified and located candidate seams. For example, path planning may be performed based on a path for the seam that may be generated by clustering the subset of points.
[0066] In some examples, the welding instructions may be based on the seam type (e.g., butt joint, corner joint, edge joint, lap joint, T-joint, etc.). In some examples, the welding instructions may be updated based on input from a user via a user interface (e.g., user interface 106 of FIG. 1). A user may select a candidate seam to be welded from all available candidate seams via the user interface. Path planning may be performed on the selected candidate seam, and welding instructions may be generated for the selected candidate seam. In some examples, a user may update welding parameters via the user interface. The welding instructions may be updated based on the updated welding parameters.
[0067] In this manner, a welding robot can be operated and controlled by performing method 900 without prior information (e.g., a CAD model) of the parts to be welded. As the parts are scanned to generate welding instructions, a representation of the scanned image of the part can be annotated (e.g., via a user interface) with one or more candidate seams. The annotated representation can be used to define a 3D model of the part. The 3D model of the part can be stored in a database for subsequent welding of additional instances of the part.
[0068] FIG. 10 is a flow diagram of a method 1000 for performing autonomous welding, according to various examples. More specifically, FIG. 10 is a flow diagram of a method 1000 for operating and controlling a welding robot (e.g., robot 110 of FIG. 1), according to some examples. In step 1002, the method 1000 includes identifying an expected orientation and an expected location of a candidate seam on the part to be welded based on a CAD model of the part. The expected orientation and expected location may be determined using annotations provided to the CAD model by a user / operator. Additionally or alternatively, the controller (e.g., controller 108 of FIG. 1) may also operate to identify the candidate seam based on model geometry. Object matching may be performed to match components on the part with components in the CAD model. In other words, the expected location and orientation of the candidate seam may be identified based on object matching.
[0069] In step 1004, the method 1000 includes acquiring image data of a workspace (e.g., workspace 101 of FIG. 1) using one or more sensors (e.g., sensor(s) 102 of FIG. 1). The image data may include 2D and / or 3D images of the workspace. As discussed above, the workspace may include one or more parts to be welded and fixtures and / or clamps that may securely hold the parts. In some examples, a point cloud may be generated from the image data. For example, the images may be overlaid on one another to reconstruct and generate 3D image data. The 3D image data may be matched against one another to generate the point cloud.
[0070] In some examples, to reduce the processing time for generating the welding instructions, the sensor is configured to perform partial scans. In other words, image data is collected from a small number of angles (e.g., angles where candidate seams are expected to be visible) instead of scanning the workspace from every angle. In such examples, the point cloud generated from the image data is a partial point cloud. For example, scanning time and / or processing time can be reduced by generating a partial point cloud that does not include portions of the part that the model indicates do not include a seam to be welded.
[0071] At step 1006, the method 1000 includes identifying candidate seams based on the image data, the point cloud, and / or the subpoint cloud. For example, the controller 108 (FIG. 1) can identify the candidate seams using techniques described above.
[0072] At step 1008, the method 1000 includes identifying an actual location and an actual orientation of the candidate seam. For example, at step 1002, a first subset of points may be identified as the modeled seam. At step 1006, a second subset of points may be identified as the candidate seam. In some examples, the first subset of points and the second subset of points may be compared (e.g., using the registration technique described above with respect to FIG. 6). The first subset of points may be deformed to determine an actual location and orientation of the candidate seam. In some examples, the comparison of the first subset of points and the second subset of points may help determine a tolerance for the first subset of points (e.g., an expected location and an expected orientation of the candidate seam). In such examples, the first subset of points may be deformed based on the tolerance to determine an actual location and orientation of the candidate seam. In other words, the expected location and expected orientation of the candidate seam may be refined (in some examples, based on the tolerance) to determine an actual location and actual orientation of the candidate seam. This deformation / refinement technique can take into account surface topography on the part that is not accurately represented in the CAD model (eg, in the CAD model in step 1002).
[0073] In step 1010, the method 1000 includes generating welding instructions for a welding robot based on the actual locations and actual orientations of the candidate seams. For example, path planning can be performed based on the actual locations and actual orientations of the candidate seams.
[0074] 9, once the actual location and actual orientation of the candidate seam are identified, the method 1000 may include validating the candidate seam using the techniques described above. However, in contrast to the method 900, user interaction in the method 1000 may not be necessary because the seam or seams to be welded may already be annotated in the CAD model. Thus, in some cases, the welding robot may be operated and controlled by performing the method 1000 without user interaction.
[0075] In addition to or in lieu of the steps described above with respect to method 1000, a welding robot (e.g., robot 110 of FIG. 1) may operate and control the welding robot to perform autonomous welding as follows: A controller 108 of the robot 110, and in particular the robot 11, may scan a workspace containing the part to determine a location of the part in the workspace (e.g., the location of the part on a positioning device) and generate a representation of the part (e.g., a point cloud representation). The controller 108 may be provided with an annotated CAD model of the part. The controller 108 may then determine expected locations and expected orientations of candidate seams on the part according to (or based on) the computer-aided design (CAD) model of the part and the representation of the part. For example, the controller 108 may operate to identify candidate seams based on model geometry--object matching may be performed to match components or features (e.g., topographical features) on the representation of the part with components or features in the CAD model) and the controller 108 may operate to use this object matching in determining expected locations and expected orientations of candidate seams on the part. Once the expected position and orientation have been determined, the controller 108 may then determine an actual position and actual orientation of the candidate seam based at least in part on the representation of the part. The actual position and orientation may be determined using the deformation / refinement techniques described in step 1008.
[0076] FIG. 11 is a flow diagram of a method 1100 for performing autonomous welding, according to various examples. More specifically, FIG. 11 illustrates an exemplary method 1100 of operation of a manufacturing robot 110 (FIG. 1) in a robotic manufacturing environment. Assume that one or more parts 114 on which a manufacturing task (e.g., welding) is to be performed are positioned and / or secured using a fixture 116 on another fixture 116 (e.g., a positioning device) in the manufacturing workspace 101. Following placement of the one or more parts 114, the method 1100 may begin. As an initial step, the method 1100 includes scanning the one or more parts (e.g., block 1110). The scan may be performed by one or more sensors 102 (e.g., a scanner not coupled to the robot 110), and the controller 108 may be configured to determine a location of the part within the manufacturing workspace 101 and identify one or more seams on the part using image data acquired from the sensor and / or a point cloud derived from the image or sensor data (blocks 1112 and 1114). The part and seams may be located and identified based on one of the techniques described with respect to FIG. 1. Once the location of the part and seams is determined, the controller 108 plots a path for the manufacturing robot 110 along the identified seams (block 1116). The plotted path may include optimized motion parameters of the manufacturing robot 110 to complete the weld without colliding with itself or others within the manufacturing workspace 101. No human input is required in generating the optimized motion parameters of the manufacturing robot 110 to complete the weld. The path / trajectory may be planned based on one of the path planning techniques described above.
[0077] The terms "location" and "orientation" are described as separate entities in the above disclosure. However, the term "location", when used in the context of a part, means "a particular way in which a part is placed or arranged". The term "location", when used in the context of a seam, means "a particular way in which a seam on a part is positioned or oriented". Thus, the location of a part / seam may inherently take into account the orientation of the part / seam. Thus, "location" can include "orientation". For example, location can include the relative physical location or orientation (e.g., angle) of a part or a candidate seam.
[0078] Unless otherwise stated, "about," "approximately," or "substantially" preceding a value means + / - 10 percent of the stated value. Unless otherwise stated, two objects described as "parallel" are side-by-side and have a distance between them that is constant or varies by 10 percent or less. Unless otherwise stated, two objects described as perpendicular intersect at an angle ranging from 80 degrees to 100 degrees. Modifications are possible in the described examples, and other examples are possible within the scope of the claims.
Claims
1. 1. A computer-implemented method for generating instructions for a welding robot, comprising: identifying expected locations of candidate seams on the parts to be welded based on a computer-aided design (CAD) model of the parts; scanning a workspace containing the part to generate a representation of the part; identifying the candidate seams on the part based on the representation of the part and the expected locations of the candidate seams; determining an actual location of the candidate seam; and generating welding instructions for the welding robot based at least in part on the actual locations of the candidate seams.
4. A computer-implemented method comprising:
2. Determining the actual location of the candidate seam comprises: updating the predicted locations of the candidate seams based at least in part on the representation of the part.
2. The computer-implemented method of claim 1, further comprising:
3. Determining the actual location of the candidate seam comprises: determining a tolerance for the expected location of the candidate seam based at least in part on the representation of the part; and refining the predicted location of the candidate seam based at least in part on the tolerance.
2. The computer-implemented method of claim 1, further comprising:
4. The computer-implemented method of claim 1 , wherein identifying the expected locations of the candidate seams further comprises matching representations of components in the CAD model with the components on the parts to be welded.
5. The computer-implemented method of claim 1 , wherein the CAD model includes annotations of the candidate seams.
6. The step of identifying candidate seams comprises: Identifying a plurality of points on the part to be welded based on the representation of the part, at least some points from the plurality of points forming a seam.
2. The computer-implemented method of claim 1, further comprising:
7. The step of identifying candidate seams comprises: identifying a plurality of points on the part to be welded based on the representation of the part, where at least some points from the plurality of points form a seam; verifying whether the candidate seam is a seam; and identifying the candidate seams as a seam type using a neural network; 2. The computer-implemented method of claim 1, further comprising:
8. verifying whether the candidate seam is a seam; analyzing at least a subset of image data, the subset of image data including a plurality of images of the part, each image from the plurality of images capturing a portion of the part from a different angle; For each image from the plurality of images, determining a confidence value that the candidate seam is an actual seam; and verifying that the candidate seam is the seam based on a confidence value for each image from the plurality of images.
8. The computer-implemented method of claim 7, further comprising:
9. 1. A computer-implemented method for generating welding instructions for a welding robot, comprising: acquiring image data via a sensor of a work space containing the parts to be welded; identifying a plurality of points on the parts to be welded based on the image data; identifying candidate seams on the parts to be welded from the plurality of points; and generating welding instructions for the welding robot based at least in part on the identification of the candidate seams.
4. A computer-implemented method comprising:
10. The step of identifying candidate seams comprises: Locating the candidate seam relative to the part based on the image data.
10. The computer-implemented method of claim 9, further comprising:
11. 10. The computer-implemented method of claim 9, wherein identifying the candidate seam further comprises identifying a subset of points of the plurality of points that form the candidate seam.
12. The computer-implemented method of claim 9 , wherein the welding instructions include a welding path for the welding robot to weld the part.
13. classifying at least one object contained in the workspace as at least one of a clamp or a fixture based on the image data using a neural network.
10. The computer-implemented method of claim 9, further comprising:
14. The step of classifying the at least one object further comprises: performing a pixel-by-pixel classification of said image data; 14. The computer-implemented method of claim 13, further comprising:
15. identifying a first portion within at least one image included in the image data that includes a representation of the at least one object based on the pixel-by-pixel classification; and Identifying another candidate seam from a second portion in the at least one image.
15. The computer-implemented method of claim 14, further comprising:
16. identifying the candidate seam as belonging to a group of seam types consisting of a butt seam, a corner seam, a edge seam, a lap seam, and a T-joint, and wherein the welding instructions are generated based at least in part on the seam type.
10. The computer-implemented method of claim 9, further comprising:
17. The step of identifying candidate seams comprises: verifying whether the candidate seam is a seam; and identifying the candidate seams as a seam type using a neural network; 10. The computer-implemented method of claim 9, further comprising:
18. verifying whether the candidate seam is a seam; analyzing at least a subset of the image data, the subset of the image data including a plurality of images of the part, each image from the plurality of images capturing a portion of the part from a different angle; For each image from the plurality of images, determining a confidence value that the candidate seam is an actual seam; and verifying that the candidate seam is the seam based on a confidence value for each image from the plurality of images.
20. The computer-implemented method of claim 17, further comprising:
19. The step of identifying candidate seams comprises: verifying whether the candidate seam is a seam; identifying the candidate seams as a seam type using a neural network; and in response to verifying that the candidate seam is the seam, defining a group of a subset of points of the plurality of points thereby forming the seam.
10. The computer-implemented method of claim 9, further comprising:
20. the candidate seam is a first candidate seam, and the computer-implemented method further comprises: displaying to a user and via a user interface a plurality of candidate seams available to be welded, the plurality of candidate seams including the first candidate seam; receiving an indication via the user interface that a second candidate seam from the plurality of candidate seams should be welded; and updating the welding instructions for the welding robot such that the welding robot is instructed to weld the second candidate seam.
10. The computer-implemented method of claim 9, further comprising:
21. receiving changes to the welding parameters from a user via a user interface; and updating the welding instructions for the welding robot based at least in part on the changes in the welding parameters.
10. The computer-implemented method of claim 9, further comprising:
22. scanning the parts to be welded; receiving user input from a user including instructions to weld at least one candidate seam on the parts to be welded; and Defining a 3D model of the parts to be welded based at least in part on the user input and the scan.
10. The computer-implemented method of claim 9, further comprising:
23. scanning the parts to be welded; receiving user input from a user including instructions to weld at least one candidate seam on the parts to be welded; and defining a 3D model of the parts to be welded based at least in part on the user input and the scan; and storing the 3D model of the parts to be welded in a database; 10. The computer-implemented method of claim 9, further comprising:
24. 1. A computer-implemented method for generating instructions for a welding robot, comprising: scanning the workspace containing the part to determine a location of the part within the workspace and to generate a representation of the part; determining expected locations of candidate seams on the part to be welded according to a computer-aided design (CAD) model of the part and the representation of the part; and determining an actual location of the candidate seam based at least in part on the representation of the part; 4. A computer-implemented method comprising:
25. 25. The computer-implemented method of claim 24, wherein the CAD model includes annotations of the candidate seams.
26. 26. The computer-implemented method of claim 25, wherein determining the actual location of the candidate seam further comprises updating the predicted location of the candidate seam based at least in part on the representation of the part.
