Autonomous Assembly Robot

Abstract

The present disclosure provides systems, methods, and apparatus, including computer programs encoded on a computer storage medium, for operation of an assembly robotic system. In one aspect of the disclosure, the assembly robotic system includes a tool coupled to a robotic device and configured to selectively couple to a first object. The assembly robotic system also includes a welding tool, one or more sensors configured to generate sensor data, and a controller. The controller is configured to control the tool to couple the tool to the first object based on the sensor data, control the robotic device to bring the first object into a spatial relationship with a second object, and generate welding instructions to cause the welding tool to weld a seam formed between the first object and the second object. Other aspects and features are also claimed and described.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of U.S. Provisional Patent Application No. 63 / 332,416, entitled "AUTONOMOUS ASSEMBLY ROBOT," filed April 19, 2022, the entire contents of which are expressly incorporated by reference herein.

[0002] The present disclosure relates generally to robotic systems, and more specifically to robotic fabrication, assembly, or manufacturing systems, such as welding robotic systems. [Background technology]

[0003] Traditional manufacturing and processing may include various tasks or processes, such as, by way of illustrative and non-limiting examples, material handling, machining, and assembly. Compared to other tasks or processes, such as machining and material handling, assembly tasks or processes are generally considered to be more complex. An assembly task or process typically includes the act or process of bringing two or more objects (e.g., parts, pieces, components, etc.) together in a coupled relationship. By way of example, an assembly task or process may include joining or connecting two objects to form a new component or a finished product. By way of illustrative and non-limiting examples, joining or connecting two objects may include or correspond to welding or riveting the two objects.

[0004] A traditional assembly task or process may include a robot performing one or more assembly tasks. For example, a single robot may be configured to perform a task repeatedly, such as an insertion task. To enable a single robot to perform a task repeatedly, fixtures such as welding frames and integrated motion axes may be used to account for positional uncertainties. For complex assemblies, a teach pendant or optimization of assembly parameters on the pendant may also be used to control and operate a single robot.

[0005] However, traditional single-manipulator robotic solutions, such as rigid fixtures and teach-pendant or on-pendant devices, for assembly tasks or processes can suffer from various limitations. For example, traditional single-manipulator robotic solutions can be limited with respect to the type of assemblies they can handle and the number of assemblies they can complete. As another example, when a single-manipulator robotic solution includes a rigid fixture, the rigid fixture can increase the cost of setup, installation, maintenance, and operation, and may be customized for a specific assembly task, making the system inherently inflexible or unable to adapt to different assembly tasks. Additionally or alternatively, when a single-manipulator robotic solution relies on or is used in conjunction with a teach pendant or on-pendant device, changes to the assembly task or process, or operation of the corresponding robot, may require human (e.g., programmer or machine operator) intervention to provide input to the teach pendant or on-pendant device. This requires human presence, increasing operational costs, and / or reducing productivity and increasing production downtime when humans are not available. Summary of the Invention

[0006] The following summarizes some aspects of the present disclosure to provide a basic understanding of the discussed technology. This summary is not an extensive overview of all envisioned features of the present disclosure, and is not intended to identify key or critical elements of all aspects of the present disclosure, nor to delineate the scope of any or all aspects of the present disclosure. Its sole purpose is to present some concepts of one or more aspects of the present disclosure in summary form as a prelude to the more detailed description that is presented later.

[0007] The present disclosure relates to apparatus, systems, and methods that provide robotic fabrication, assembly, or manufacturing. In some embodiments, a robotic system, such as an assembly robotic system, is configured to join multiple objects or components to form an assembled component, sub-component, or product. As illustrative and non-limiting examples, the multiple objects may be joined via welding, brazing, soldering, or by performing other manufacturing operations on the multiple objects, such as riveting, pinning, bolting, etc.

[0008] In some embodiments, the robotic system is configured to identify a position of one or more objects. For at least one object of the one or more objects, the robotic system may determine a pose of the at least one object, identify a grip location on the at least one object, and pick up (e.g., lift, move, etc.) the at least one object at the grip location. As used herein, the term "pose" includes both position information, such as the position and orientation of an object in a coordinate system (e.g., x, y, z coordinates) and orientation information (e.g., the relative angle between the object and the ground or a holder on which the object is placed). In some implementations, the pose includes the position of the object (typically represented by x, y, z coordinates) and the orientation of the object (typically represented by roll, pitch, yaw angles). As another example, for each object of the one or more objects, the robotic system may determine a pose of the object, identify a grip location on the object, and pick up the object at the grip location. The position of the object may refer to the placement of the object in a coordinate system or physical space. For example, the position may describe where the object is located, but typically does not provide or include information about the orientation of the object. An object's position may be similar to an object's location, since position may also refer to the placement of an object within a coordinate system. However, an object's position is often used in the context of a specific point or reference within the object, such as the center of gravity or a specific corner of the object. For example, the position of a first object relative to the position of a second object may refer to a first edge or corner of the first object relative to the surface of the second object.

[0009] In some implementations, a robotic system configured to perform one or more manufacturing operations may include multiple robots, such as two or more robots. The multiple robots may be configured to perform one or more operations autonomously, individually or in coordination with other robots. The one or more operations may include picking up an object, coupling the picked up object to another object (e.g., bringing multiple objects together, mating, or moving them closer together), performing a manufacturing operation on an object (e.g., welding, bonding, blotting, pinning, etc.), or combinations thereof. Additionally or alternatively, the operations may include identifying an object on which a manufacturing task (e.g., welding, bonding, blotting, pinning, etc.) may be performed, picking one or more objects, positioning the objects relative to one another, and performing the manufacturing task (e.g., welding, bonding, bolting, pinning, etc.). In some implementations, the operations performed by the one or more robots may be performed in conjunction with or based on a controller of the robotic system.

[0010] In one aspect of the disclosure, an assembly robotic system includes a first robotic device disposed in a workspace and a first tool coupled to the first robotic device. The first tool is configured to be selectively coupled to a first object. The assembly robotic system further includes one or more sensors each configured to generate sensor data related to the workspace. The assembly robotic system also includes a welding tool configured to join two or more objects. For example, the welding tool may be configured to fuse, join (e.g., bolts or pins), or weld the two or more objects. To further illustrate, in some implementations, the welding tool can weld two objects together by laying a weld metal and fusing / joining the two or more objects. The assembly robotic system includes a controller configured to generate welding instructions based on the sensor data to control the first tool to be coupled to the first object, control the first robotic device to bring the first object into a spatial relationship with the second object such that a seam is formed between the first object and the second object, and cause the welding tool to weld the seam to join the first object and the second object. For example, the welding tool can lay a weld metal along the seam to join the first object and the second object.

[0011] In an additional aspect of the present disclosure, the assembly robotic system includes a first robotic device disposed in the workspace and a first tool coupled to the first robotic device. The first tool is configured to be selectively coupled to a first object. The assembly robotic system includes a second robotic device disposed in the workspace and a second tool coupled to the second robotic device. The second tool is configured to be selectively coupled to a second object. The first object and the second object are configured to form a first seam between the first object and the second object. The assembly robotic system also includes a third robotic device disposed in the workspace and one or more sensors each configured to generate sensor data related to the workspace. At least a first sensor of the one or more sensors is coupled to the third robotic device. The assembly robotic system further includes a fourth robotic device disposed in the workspace and a welding tool coupled to the fourth robotic device. The welding tool is configured to weld the first seam formed between the first object and the second object. For example, the welding tool may be configured to lay a weld metal along the first seam formed between the first object and the second object. As another example, the welding tool may be configured to weld (e.g., heat seal) a first seam formed between a first object and a second object. The assembly robotic system includes a controller configured to control a first tool to be coupled to a first object at a first location on the first object based on the sensor data and to control a second tool to be coupled to a second object at a second location on the second object based on the sensor data. The controller is further configured to generate welding instructions to control the first robotic device, the second robotic device, or both, to bring the first object and the second object into a first spatial relationship such that a first seam is formed between the first object and the second object and to cause the welding tool to weld the first seam to bond the first object and the second object. For example, the welding tool lays weld metal along the first seam to bond the first object and the second object.

[0012] In an additional aspect of the present disclosure, a method of operating an assembly robotic system is disclosed that includes a first tool coupled to a first robotic device, a second tool coupled to a second robotic device, one or more sensors, a welding tool, and a controller. The method includes receiving, by a controller of the assembly robotic system, first sensor data from the one or more sensors. The method also includes controlling, by the controller, the first tool to couple to the first object at a first location on the first object based on the first sensor data. The method further includes controlling, by the controller and based on the first sensor data, the second tool to couple to the second object at a second location on the second object. The method includes controlling, by the controller, the first robotic device, the second robotic device, or both, to bring the first object and the second object into a spatial relationship such that a seam is formed between the first object and the second object. The method also includes generating, by the controller, welding instructions, and welding the seam to join the first object and the second object by a welding tool responsive to the welding instructions.

[0013] There has been outlined above, rather broadly, the features and technical advantages of embodiments of the present disclosure in order that the detailed description that follows may be better understood. Additional features and advantages are set forth below. The concept and specific examples disclosed may be readily utilized as a basis for modifying or designing other structures for carrying out the same purposes of the present disclosure. Such equivalent structures do not depart from the scope of the appended claims. The features of the concepts disclosed herein, both their organization and method of operation, together with associated advantages, will be better understood from the following description when considered in conjunction with the accompanying figures, each of which is provided for purposes of illustration and description, and not as a definition of the limits of the claims. [Brief description of the drawings]

[0014] Further understanding of the nature and advantages of the present disclosure may be realized by referring to the following drawings. In the attached figures, similar components or features may have the same reference numerals. Furthermore, various components of the same type may be distinguished by adding a dash after the reference numeral and a second numeral to distinguish between the similar components. When only the first reference numeral is used in this specification, this specification applies to any one of the similar components having the same first reference numeral, regardless of the second reference numeral.

[0015] [Figure 1] FIG. 2 is a block diagram illustrating details of an embodiment of a robotic system according to one or more aspects.

[0016] [Diagram 2] FIG. 2 is a block diagram illustrating details of another example of a robotic system in accordance with one or more aspects.

[0017] [Diagram 3] FIG. 1 illustrates details of another example of a robotic system according to one or more aspects.

[0018] [Figure 4] FIG. 4 is a perspective view illustrating details of the robotic system of FIG. 3 from one or more sides.

[0019] [Diagram 5] 1 is a flow diagram illustrating an example process for performing an assembly task using an autonomous assembly robot in accordance with one or more aspects.

[0020] [Figure 6] FIG. 1 illustrates an image of a 3D representation of a point cloud of multiple parts from one or more side views.

[0021] [Figure 7] 11A-11D show images of an example of a positioning process according to one or more aspects.

[0022] [Figure 8A]14A-14D show images of example different gripping positions of a finger-based gripper with a cylindrical object with one or more sides. [Figure 8B] 14A-14D show images of example different gripping positions of a finger-based gripper with a cylindrical object with one or more sides. [Figure 8C] 14A-14D show images of example different gripping positions of a finger-based gripper with a cylindrical object with one or more sides.

[0023] [Figure 9] 1A-1C illustrate example images of the tool center point (TCP) of the gripper and the desired coordinate frame of the object, according to one or more aspects.

[0024] [Figure 10] 1A-1C are images showing 3D representations of a picked up object before and after positioning, according to one or more aspects.

[0025] [Figure 11] FIG. 1 shows example images of two objects positioned together at a fixed pose in space.

[0026] [Figure 12] 11A-11D show images of an example of a positioning process according to one or more aspects.

[0027] [Figure 13] FIG. 13 shows images of another example of a positioning process according to one or more aspects.

[0028] [Figure 14] 1A-1C show examples of representations of a robotic arm from one or more sides.

[0029] [Figure 15] 1 is a flow diagram illustrating an example of a process for operating a robotic assembly system in accordance with one or more aspects.

[0030] [Figure 16] FIG. 1 is a block diagram illustrating details of an example robotic system according to one or more aspects.

[0031] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The detailed description set forth below in conjunction with the accompanying drawings is intended to describe various configurations and is not intended to limit the scope of the present disclosure. Rather, the detailed description includes specific details for the purpose of providing a thorough understanding of the subject matter of the present invention. It will be apparent to those skilled in the art that these specific details are not required in all cases, and in some cases, well-known structures and components are shown in block diagram form for clarity of presentation.

[0033] The present disclosure relates to devices, systems, and methods that provide robotic fabrication, assembly, or manufacturing. In some embodiments, a robotic system, such as an assembly robotic system, is configured to perform an assembly task, such as joining multiple objects or components in a combined state to form an assembled component, subcomponent, or product. An assembly task may include performing multiple processes to complete the assembly task. For example, an assembly task may include locating one or more objects, determining a pose of one or more objects, picking a set of objects for one or more objects, determining a pose of the set of objects relative to a robotic device after picking, positioning the set of objects in a desired relative pose, coordinating the movement of one or more robotic devices, performing an assembly or manufacturing task or operation (e.g., welding, joining, bolting, pinning, etc.), or completing a combination thereof.

[0034] Performing one or more of the multiple processes associated with an assembly task may be computationally and algorithmically challenging. For example, performing one or more of the processes may use a large amount of computing power, be computationally and algorithmically infeasible given the limitations of classical computers, or a combination thereof. Such computational and algorithmic challenges may be multiplied for systems such as assembly robotic systems that include robotic devices with multiple degrees of freedom (DOF), such as six DOF. Assembly tasks have historically been manual, given their computationally intensive nature, the variability or variety of objects (e.g., objects may exist in any shape), the variability or variety of manufacturing tasks (e.g., welding, riveting, bolting, pinning, painting, packaging), or a combination thereof.

[0035] The use of one or more robots enables agile assembly solutions that can easily adapt to changes in the manufacturing process. In some embodiments, multiple robots can be used in the assembly process. At least one of the one or more robots can be used in the assembly process as a dynamic fixture to be able to grasp, reposition, and / or manipulate parts or objects to complete more complex assembly steps. The use of one or more robots in an assembly process presents various challenges to enable assembly algorithms, such as difficulty and complexity of programming, integration and coordination, advanced sensing and control capabilities, and the like, or a combination thereof.

[0036] The present disclosure relates to apparatus, systems, and methods that provide robotic fabrication, assembly, or manufacturing. In some embodiments, a robotic system, such as an assembly robotic system, is configured to join multiple objects or components to form an assembled component, sub-component, or product. For example, the system may be configured to locate and identify one or more parts or objects to be welded, determine at least one pose of the one or more parts or objects, position the at least one part or object in space, and perform a manufacturing operation or task (e.g., welding, joining, bolting, pinning, etc.) on the at least one part or object. In some implementations, the robotic system includes multiple robots and is configured to perform one or more robotic manufacturing operations, such as welding, bolting, pinning, brazing, soldering, riveting, as illustrative and non-limiting examples. Additionally or alternatively, one or more aspects of the robotic systems described herein relate to sensing capabilities, control capabilities, or a combination thereof, to enable agile assembly solutions.

[0037] In some embodiments of the system of the present disclosure, the system (e.g., an assembly robot system) includes a tool coupled to a first robotic device and configured to selectively couple to a first object. In some embodiments, the tool includes a gripper or clamp, as illustrative and non-limiting examples. Additionally or alternatively, the first robotic device may include a robotic arm configured to have six degrees of freedom. The system may also include a manufacturing tool (e.g., a welding tool), one or more sensors configured to generate sensor data, and a controller. The controller may be configured to determine a position of the first object at which the first object is to be gripped. The controller may be configured to control the tool to couple to the first object based on the sensor data, and to generate manufacturing instructions (instructions) for controlling the first robotic device to bring the first object into a spatial relationship with a second object and to cause an operation to be performed by the manufacturing tool. For example, the manufacturing instructions may include welding instructions for causing a manufacturing tool (e.g., a welding tool) to weld a seam formed between the first object and the second object, such as along a seam formed by the first object and the second object. By way of example, the welding instructions may cause the manufacturing tool to tack weld and / or lay down weld metal along a seam formed between a first object and a second object.

[0038] In some implementations, the system may include one or more additional robotic devices, such as a second robotic device coupled to a second tool (e.g., a gripper), a third robotic device coupled to at least one of the one or more sensors, a fourth robotic device coupled to a manufacturing tool, or combinations thereof. The controller may be configured to control the one or more additional robotic devices to combine multiple objects or components to form an assembled component, subcomponent, or product. By way of example, the controller may receive model information, such as point cloud data associated with the first object, the second object, or the combined (e.g., combined) first object and second object. In some implementations, the model information may include a computer-aided design (CAD) model of a representation of a structure including the first object and the second object, object identifiers or parameter information of the first object or the second object, or a combination thereof.

[0039] The controller may also receive first sensor data from one or more sensors. In some embodiments, the controller is configured to control the third robotic device to cause at least one sensor to generate at least a portion of the first sensor data. The first sensor data may include, as illustrative and non-limiting examples, image data, sonar data, ultrasonic data, etc. Based on the first sensor data, the controller may identify the first object, the second object, or a combination thereof. For example, to identify the first object, the controller may determine an object ID (e.g., a serial number, a barcode, a QR code, etc.) of the first object based on the first sensor data, match point cloud information associated with the first sensor data to point cloud information included in the model information, or a combination thereof.

[0040] After identifying the first object or the second object, the controller may perform a registration process to determine a pose of the first object or the second object. For example, with reference to the first object, the controller may perform a registration process to determine a first pose of the first object. To perform the registration process, the controller may perform a comparison or match operation based on the model information (e.g., point cloud data) and at least a portion of the first sensor data associated with the first object. Based on the first pose of the first object, the controller may determine a first position on the first object for the first tool to be coupled (e.g., grasped and / or picked up) to the first object, and control the first robotic device and the first tool such that the first tool is coupled to the first object based on the first position. In some embodiments, the controller may identify one or more positions on the first object for the first tool to be coupled to the first object, and perform a reachability analysis, a collision analysis, or a combination thereof for each of the one or more positions. After the first tool is coupled to the first object, the controller may receive second sensor data from one or more sensors related to the first object coupled to the first tool. In some implementations, the controller may control the third robotic device, the at least one sensor, or a combination thereof to cause the at least one sensor to generate at least a portion of the second sensor data. The controller may perform a positioning process to determine a second pose of the first object based on the model information and the second sensor data. Determining the second pose of the first object allows the system to compensate for an actual coupling between the first tool and the first object that differs from the first position. In some implementations, the controller may perform a reachability analysis, a collision analysis, or a combination thereof based on the second pose of the first object. Also, although one or more operations are described with respect to the first object, the controller may perform one or more operations with respect to the second object.

[0041] After the first tool is coupled to the first object and the second tool is coupled to the second object, the controller may control the first robotic device, the first tool, the second robotic device, the second tool, or a combination thereof to bring the first object and the second object into a spatial relationship such that a seam is formed between the first object and the second object. After the first object and the second object are brought together (towards a spatial relationship), the controller may verify that the first object and the second object are in a spatial relationship, for example, verifying whether a distance (separation) between the first object and the second object is less than or equal to a threshold, such as a threshold indicated by or included in the model information. By verifying that the first object and the second object are in a spatial relationship, the system can verify that the first object and the second object are in a spatial relationship before performing a manufacturing or assembly operation, such as a welding operation, on the first object and the second object.

[0042] To confirm that the first object and the second object are in a spatial relationship, the controller may receive third sensor data related to the first object and the second object. In some embodiments, the controller may control the third robotic device, at least one sensor, or a combination thereof to cause the at least one sensor to generate at least a portion of the third sensor data. The controller may determine whether a separation between the first object and the second object is equal to or less than a threshold based on the model information and the third sensor data. As a non-illustrated example, the threshold indicates that the first object and the second object are in contact with each other or are close to each other, such as within a distance of 1 mm. If the separation is greater than the threshold, the controller may perform a positioning process on the first object or the second object, and may control the first robotic device, the first tool, the second robotic device, the second tool, or a combination thereof to guide the first object and the second object toward a spatial relationship, or a combination thereof. The controller may determine whether the first object and the second object are in a spatial relationship after the positioning process or after controlling the first robotic device, the first tool, the second robotic device, the second tool, or a combination thereof. If the separation is greater than a threshold, the controller may generate a manufacturing instruction.

[0043] The controller can transmit or execute manufacturing instructions. Execution of the manufacturing instructions can cause a manufacturing tool to perform an operation. For example, the manufacturing instructions can include welding instructions to cause a manufacturing tool (e.g., a welding tool) to weld a seam formed between a first object and a second object, such as along a seam formed by the first object and the second object. By way of example, the welding instructions can cause the manufacturing tool to lay weld metal along a seam formed between the first object and the second object. Thus, the system can include one or more robotic devices configured to autonomously pick the first object, the second object, or both, bring the first object together with the second object, and perform manufacturing tasks (e.g., weld, join, bolt, pin, etc.) on the first object and the second object.

[0044] Certain embodiments of the subject matter described in this disclosure may be implemented to realize one or more of the following potential advantages or benefits. In some aspects, the present disclosure provides techniques for performing assembly tasks. Additionally, the techniques described herein may provide improved sensing and control capabilities compared to conventional systems. The techniques for performing assembly tasks may enable the system to couple, verify, or update the pose of one or more objects during the performance of an assembly task to maintain coordination and precision within the system. Additionally or alternatively, the present technology may provide or enable agile assembly solutions that can easily accommodate changes in the manufacturing process, such as complex assembly of different components or products with the same system.

[0045] 1, a diagram illustrating details of an example robotic system 100 according to one or more embodiments. In some embodiments, the robotic system 100 may include or correspond to an assembly robotic system. The robotic system 100 may be configured to join a first object 140 and a second object 142. For example, the first object 140 and the second object 142 may be designed to form a seam between the first object 140 and the second object 142. Each of the first object 140 and the second object 142 may be, without limitation, any part, component, subcomponent, combination of parts or components, or the like.

[0046] In some embodiments, the robotic system 100 may include or be associated with a workspace 102, such as a three-dimensional space. The workspace 102 may also be referred to as a manufacturing workspace. In some embodiments, the workspace 102 may be any suitable welding area or space designed with appropriate safety measures for welding. For example, the workspace 102 may be a welding area or space located in a workshop, shop, manufacturing plant, fabrication shop, outdoor construction area, or the like. In some embodiments, at least a portion of the robotic system 100 is disposed in the workspace 102. For example, the workspace 102 may be an area or space in which one or more robotic devices (e.g., robotic arm(s)) are configured to operate on one or more objects (or parts). The one or more objects may be located, coupled, stored, or otherwise supported on one or more platforms, containers, bins, racks, holders, or positioners. The one or more robotic devices may be configured to operate on one or more objects based on information received from one or more sensors, as further described herein. In some embodiments, the workspace 102 may include a pre-treatment area, a post-treatment area, or a combination thereof.

[0047] Robotic system 100 includes a first robotic device 110, a first tool 120, a manufacturing tool 126, one or more sensors 130 (hereinafter referred to as “sensors 130”), and a controller 150. Controller 150 is coupled to first robotic device 110, first tool 120, manufacturing tool 126, sensors 130, or a combination thereof. For example, controller 150 may be communicatively coupled to first robotic device 110, first tool 120, manufacturing tool 126, sensors 130, or a combination thereof via a wired connection, a wireless connection, or a combination thereof.

[0048] First robotic device 110 includes a mechanical device such as a robotic arm. In some embodiments, first robotic device 110 may be configured to have six degrees of freedom or less than six degrees of freedom. First robotic device 110 may include one or more components such as motors, servos, hydraulics, or combinations thereof, as illustrative and non-limiting examples. In some embodiments, first robotic device 110 includes a robotic arm, such as a robotic arm manufactured by YASKAWA®, ABB® IRB, KUKA®, or Universal Robots®.

[0049] In some embodiments, first robotic device 110 may be coupled to or include one or more tools. For example, based on the functionality the robot performs, a robot arm may be coupled to a tool configured to enable the functionality (e.g., perform at least a portion of the function). By way of example, a tool such as first tool 120 may be coupled to an end of first robotic device 110. In some embodiments, first robotic device 110 may be coupled to or include multiple tools, such as a manufacturing tool (e.g., a welding tool), a sensor, a picker or holder tool, a cleaner, a cutter, a grinder, or a combination thereof. In some embodiments, first robotic device 110 may be configured to operate with another device, such as another robotic device, as further described herein.

[0050] First tool 120 may include one or more tools. For example, first tool 120 may include a manufacturing tool (e.g., a welding tool), a sensor (e.g., sensor 130), a picker tool or a holder tool, or a combination thereof. As shown, first tool 120 is a picker tool or a holder tool configured to be selectively coupled to a first set of one or more objects, such as the first set of one or more objects including first object 140. In some embodiments, the picking tool or holder tool may include or correspond to a gripper, a clamp, a socket, a magnet, or a vacuum, as illustrative and non-limiting examples. For example, first tool 120 may include a three-fingered gripper, such as one manufactured by OnRobot®. As another example, first tool 120 may include a two-fingered gripper.

[0051] In some embodiments, first robotic device 110, first tool 120, or a combination thereof may be configured to change (e.g., adjust or manipulate) the pose of first object 140 while first object 140 is coupled to first tool 120. For example, the configuration of first robotic device 110 may be changed to change the pose of first object 140. Additionally or alternatively, first tool 120 may be adjusted (e.g., rotated or tilted) relative to first robotic device 110 to change the pose of first object 140.

[0052] The manufacturing tool 126 is configured to perform one or more manufacturing tasks or operations. The one or more manufacturing tasks or operations may include, as illustrative and non-limiting examples, welding, brazing, soldering, riveting, cutting, drilling, grinding, filing, bolting, pinning, and the like. In some embodiments, the manufacturing tool 126 is a welding tool configured to join two or more objects. For example, the welding tool may be configured to weld two or more objects together, such as welding the first object 140 to the second object 142. By way of example, the welding tool may be configured to lay a weld metal along a seam formed between the first object 140 and the second object 142. Additionally or alternatively, the welding tool may be configured to melt the first object 140 and the second object 142 together, such as melting the seam formed between the first object 140 and the second object 142 to join the first object 140 and the second object 142. In some embodiments, manufacturing tool 126 may be configured to perform one or more manufacturing tasks or operations in response to a manufacturing order, such as a welding order.

[0053] Sensor 130 is configured to generate sensor data, such as sensor data 180. Sensor 130 may be located on or coupled to, as illustrative and non-limiting examples, a robotic device (e.g., a robotic arm), a structure or frame, or the like. Additionally or alternatively, sensor 130 may be stationary or dynamic (e.g., movable). In some embodiments, sensor 130 may be coupled to a movable device, configured to rotate, configured to move along a rail or cable, or combinations thereof, as illustrative and non-limiting examples.

[0054] The sensors 130 may be configured to generate sensor data 180 associated with the robotic system 100, the workspace 102, or a combination thereof. The sensors 130 may include, as illustrative and non-limiting examples, visual sensors, Light Detection and Ranging (LiDAR) sensors, audio sensors, electromagnetic sensors, ultrasonic sensors, or combinations thereof. For example, the sensors 130 may include a camera (e.g., a camera with a laser) or a scanner (e.g., a laser scanner) configured to capture visual information (e.g., a two-dimensional (2D) image or a three-dimensional (3D) scan). In some implementations, the sensors 130 may be a 3D sensor, such as a scanner sold by Phtoneo®. As another example, the sensors 130 may be audio sensors, such as a Sound Navigation and Ranging (SONAR) device configured to emit or capture sound. Additionally or alternatively, the sensors 130 may be electromagnetic sensors, such as a Radio Detection and Ranging (RADAR) device configured to emit and / or capture electromagnetic (EM) waves. Through visual, audio, electromagnetic, and / or other sensing technologies, the sensors 130 can gather information about physical structures and objects associated with or within the workspace 102. Thus, the first sensor data 180 may include, as illustrative and non-limiting examples, image data, sonar data, ultrasound data, etc.

[0055] The sensors 130 may be configured to collect static information associated with stationary structures or objects, dynamic information associated with moving structures or objects, or a combination thereof. The sensors 130 can generate sensor data 180 (e.g., 2D image data, 3D image data, audio data, electromagnetic data, or other data) that is provided to another device, such as the controller 150, to generate a 3D representation of the physical structure or object. As described above, the sensors 130 can capture multiple 2D images from different angles (e.g., multiple fields of view).

[0056] In some implementations, the sensor 130 may be a 3D sensor configured to reconstruct the shape of a 3D surface of an inspected object (e.g., the first object 140 or the second object 142) using structured light projection. For example, one or more structured patterns may be projected onto the scene (background) or object during scanning, allowing depth to be calculated from one or more images of the scene. Information regarding the surface shape of the scanned object or scene may be included in the sensor data 180 as a depth map or in the form of a point cloud, as illustrative and non-limiting examples.

[0057] Control device 150 includes controller 152, one or more I / O devices 157, and network interface 160. Controller 152 may be any suitable machine specifically and specially configured (e.g., programmed) to perform one or more operations described herein. In some implementations, controller 152 is not a general-purpose computer, but is specifically programmed or hardware configured to perform one or more operations described herein. Additionally or alternatively, controller 152 is or includes an application specific integrated circuit (ASIC), a central processing unit (CPU), a field programmable gate array (FPGA), or a combination thereof.

[0058] In some implementations, the controller 152 includes a processor 154 and a memory 156. The processor 154 may include various forms of processor-based systems in accordance with aspects described herein. For example, the processor 154 may include a general-purpose computer system (e.g., a personal computer (PC), a server, a tablet device, etc.) and / or a special-purpose processor platform (e.g., an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), etc.).

[0059] Processor 154 may include one or more processors. Processor 154 may be configured to process sensor data 180, perform object detection and / or recognition, perform coordination or control functions, or perform combinations thereof, as illustrative and non-limiting examples. Additionally or alternatively, processor 154 may be configured to generate control information, such as control information 182 for another device, such as first robotic device 110, first tool 120, manufacturing tool 126, or sensor 130. For example, processor 154 may be configured to perform one or more operations as described herein. By way of example, processor 154 may be configured to perform one or more operations described with reference to at least FIGS. 1-5, 15, or 16. It should be noted that one or more operations described herein with reference to controller 150, controller 152, or processor 154 may be performed while robotic system 100 is online or offline, may be learned based on previous execution or operation of robotic system 100, or may be a combination thereof.

[0060] The memory 156 may include a ROM device, a RAM device, one or more HDDs, flash memory devices, SSDs, other devices configured to store data in a persistent or non-persistent state, or a combination of different memory devices. The memory 156 includes or is configured to store instructions 162, model information 164, sensor data 165, attitude information 166, and system information 168. In one or more aspects, the memory 156 may store instructions 162, such as executable code, that when executed by the processor 154, causes the processor 154 to perform operations according to one or more aspects of the present disclosure, as described herein. In some implementations, the instructions 162 (e.g., executable code) are a single, self-contained, program. In other implementations, the instructions (e.g., executable code) are a program with one or more function calls to other executable code that may be stored in storage or elsewhere. One or more functions resulting from the execution of the executable code may be implemented by hardware. For example, multiple processors may be used to perform one or more separate tasks of the executable code.

[0061] The instructions 162 include control logic 170, model logic 171, perception logic 172, adaptation logic 173, and kinematic reachability and collision logic 174. The control logic 170 is configured to control the operation or send commands or requests to one or more devices or components of the robotic system 100, such as the first robotic device 110, the first tool 120, the manufacturing tool 126, or sensors. In some embodiments, the control logic 170 may include assembly control logic or welding control logic. The welding control logic may be configured to control the welding process performed via the welding tool. Although described as being included in the controller 152, in other embodiments, the welding control logic may be included in another device configured to communicate with the controller 152.

[0062] Model logic 171 may be configured to perform one or more operations related to model information 164. For example, model logic 171 may be configured to analyze or process a CAD file (e.g., a CAD model) to identify one or more objects, one or more sub-components, one or more intermediate assemblies, a final assembly, annotation information, a point cloud model of the object, intermediate assembly, or final assembly, welding instructions, a welding plan, or a combination thereof. Additionally or alternatively, model logic 171 may be configured to analyze or process a CAD file (e.g., a CAD model) to determine an assembly sequence, primitive components or shapes in the CAD model, modifications and deformations to the primitive shapes or entire assemblies to accommodate physical parts, material properties (reflectivity, friction, etc.), or a combination thereof. In some implementations, the CAD file (e.g., a CAD model) may be used to determine primitive assembly tasks required for a particular assembly / manufacturing task. For example, a CAD model helps in planning an assembly process including tasks such as inserting, aligning, and positioning parts. In other words, a CAD model helps in understanding the basic steps required to assemble a product or structure.

[0063] Perception logic 172 may be configured to perform one or more operations related to processing sensor data 180, generating a representation based on sensor data 180 (e.g., a 2D representation using one or more images and / or a 3D representation using a laser scanner), locating and identifying objects, generating pose information, performing object joining operations, or combinations thereof, as illustrative and non-limiting examples. In some implementations, perception logic 172 may be configured to perform one or more operations based on or in combination with fitting logic 173, kinematic reachability and collision logic 174, or combinations thereof. Although one or more operations of perception logic 172 are described herein with reference to 3D representations such as point clouds, similar operations may be performed by perception logic 172 with respect to 2D data and / or 2D representations such as images.

[0064] In some implementations, the perception logic 172 can use the sensor data 180 to generate a representation associated with one or more objects associated with the workspace 102 or the assembly task. For example, the sensor data 180 can include one or more images (e.g., 2D image data captured by the sensor 130 at a particular orientation relative to the first object 140 or the second object 142). The perception logic 172 can overlay or stitch together multiple images to generate a representation, such as 2D image data or 3D image data associated with the representation. The representation can be a 2D representation (e.g., one or more 2D images that may be used in combination with a trained neural network) or a 3D representation (e.g., a point cloud representation). In some implementations, one or more operations of the perception logic 172 can use a depth map, a CAD and a 2D image, a CAD and a partial mesh / point cloud, a CAD and a single image, or a single 2D image. For example, the perception logic 172 may be configured to detect or identify an instance of an object and place a mask on the object based on a depth map, a CAD and a 2D image, a CAD and a partial mesh / point cloud, a CAD and a single image, or a single 2D image. The depth map may include one or more 2D images that include information about the distance of the object from the camera. The depth map may allow the object to be located in 3D space. For a CAD and a 2D image (of the object), the CAD may be used to locate and segment the object in the image. For a CAD and a partial mesh / point cloud (e.g., a point cloud of a collection of data points in 3D space), the CAD may be used to locate and segment the object in the partial mesh / point cloud. For a CAD and a single image, the CAD may be used to locate and segment the object in a single image. For a CAD and a single image, the object may be located and segmented in a single image. After the object is found or identified, the object may be segmented and the pose of the object may be estimated.For example, the segmented object may be provided to a pose interpreter, such as a pose interpreter network, to estimate the pose of the object. The pose interpreter may be included in or part of the perception logic 172. The pose interpreter may then position the object, as described further below. For example, a positioning process may be performed that includes aligning the segmented object with a CAD model, as estimated by the object interpreter. Thus, the examples described herein with reference to 3D representations are also applicable to 2D representations.

[0065] In some implementations, if a representation (e.g., a 2D or 3D representation) includes multiple objects, the perception logic 172 executes a clustering algorithm (e.g., K-means, affinity propagation, DBSCAN, mean shift, etc.) to segment or divide the 3D representation into different classes or clusters. Additionally or alternatively, the clustering algorithm can locate and identify target objects, such as objects to be assembled, within the 2D or 3D representation.

[0066] In some implementations, to identify the target object, the perception logic 172 can utilize information present in the representation of the final assembled object (e.g., annotations or user input from a user, shapes of objects forming the final object, identity information of objects forming the final assembled object present in a CAD file, etc.) to identify and locate the target object within the 2D or 3D representation. For example, the perception logic 172 can execute a shape identification algorithm to determine the shape of one or more objects within the 2D or 3D representation. To illustrate, the perception logic 172 (or model logic 171) can determine the shape of objects within the final assembled object present in a CAD file (e.g., by implementing shape information in the file or a feature or shape recognition algorithm) and use the determined shape to identify the target object within the 2D or 3D representation. In some implementations, annotations from a user can include annotations of seams, annotations corresponding to locations where tack welds will be made, etc.

[0067] In some implementations, the perception logic 172 may not perform the clustering algorithm in situations where the target object has identification information (e.g., unique ID information) associated with it. This identification information may be present in the CAD model of the final assembled object. Additionally or alternatively, the identification information may be loaded (or provided) to the controller 152 from the sensor 130. In such implementations, after generating a representation (e.g., a 2D or 3D representation) and receiving a unique ID associated with the object that will form the final assembled object, the perception logic 172 may use the unique ID or other information provided to the controller 152 to identify the target object within the 2D or 3D representation. After the object is identified, the perception logic 172 may determine a location of the target object, such as determining the location of the target object within the workspace 102 or in relation to the devices of the robotic system 100. For example, after identifying the target object, the perception logic 172 may determine a location within the 2D or 3D representation, which may be translated into a real-world location of the target object.

[0068] In some implementations, the perception logic 172 may be configured to analyze the object to determine one or more characteristics of the object, such as the size, shape, etc. of the object. For example, the perception logic 172 may analyze the object based on a CAD file / model, 2D information, 3D information, or a combination thereof. The perception logic 172 may be configured to identify one or more tools (e.g., tool 120) that may be used to pick up or grasp the object. By way of example, the perception logic 172 may be configured to identify or select which of a plurality of available tools is best suited to pick up or grasp the object based on a CAD file / model, 2D information, 3D information, or a combination thereof. For example, if the object is small and irregularly shaped, a gripper with soft, flexible fingers may be selected. Alternatively, if the object is large and heavy, a gripper with a strong, hard grip may be selected. In some implementations, the perception logic 172 may be configured to generate one or more instructions that are implemented during the assembly process to cause the robot's tools to use or change to the particular tool selected to pick up or grasp the object. For example, the perception logic 172 can instruct the robot to change from a first tool to a second tool. By determining and selecting a particular tool, the perception logic 172 can enable the robot to use the particular tool without requiring manual intervention. Additionally or alternatively, the perception logic 172 can enable a tool changer to pick up the selected tool in a situation where another tool is coupled to the robot, and the tool changer can operate to replace the other tool with the selected tool.

[0069] In some implementations, the perception logic 172 may determine a pose of the target object. Determining the pose of the target object may assist in determining a location where the tool 120 may pick up or engage with the target object (e.g., the first object 140). Note that the location where the first tool 120 may hold the first object 140 may be referred to as a pick-up location. The location where the first tool 120 may pick up or hold the first object 140 may depend on the pose of the target object (e.g., the first object 140). In some implementations, at least some information regarding the pose (e.g., position and orientation) of the target object may be determined based on a positioning technique performed by the adaptation logic 173.

[0070] In some implementations, the perception logic 172 may be configured to perform or mimic object joining techniques to bring two or more objects, such as the first object 140 and the second object 142, into a spatial relationship (e.g., into a fixed relative pose). The object joining techniques may determine how to orient objects in space relative to one another to position the objects in a fixed pose to perform a manufacturing task, and manage mobility and reachability constraints associated with one or more robotic devices, one or more tools, or a combination thereof. The object joining techniques may include defining a virtual space, such as a bounding box, the workspace 102 in which the objects may be assembled. This virtual space may be a cube (or other shape) of a certain length, width, and height. The perception logic 172 may perform the object joining techniques based on or in conjunction with the kinematic reachability and collision analysis performed by the kinematic reachability and collision logic 174. For example, controller 152 may perform reachability and collision analysis on one or more robotic devices or objects that may need to enter the virtual space to determine whether the one or more robotic devices or objects may be present (alone or with other objects) in one or more states in the virtual space. Perception logic 172 or kinematic reachability and collision logic 174 may take a pose of one object as a reference pose and perform reachability and collision analysis of the other object with respect to the reference pose of the one object. Additionally or alternatively, reachability and collision analysis may be performed based on a CAD model of the assembled objects.

[0071] The fit logic 173 may be configured to analyze or determine a placement, alignment, or fit of one or more objects. For example, the fit logic 173 may determine a placement, alignment, or fit of a first object relative to a second object. In some implementations, the fit logic 173 may analyze or determine a placement, alignment, or fit that minimizes errors or misalignments caused by CAD-to-object differences (e.g., differences between an actual object and an object represented by a CAD model / file). When CAD-to-object differences exist, the fit logic 173 may enable the actual object to be placed, aligned, or fit such that the object has a desired fit or alignment with one or more other objects.

[0072] In some implementations, the fitting logic 173 is configured to perform a registration process. The registration process is configured to transform or align data from different sources, such as a point cloud and a 3D representation of a CAD model, in the same coordinate frame or system. By way of example, the fitting logic 173 may perform the registration process using the point cloud of the CAD model of the first object 140 and a representation (e.g., a 2D or 3D representation) of the first object 140 (generated using the sensor data 180) by sampling the point cloud and the representation of the CAD model. The sampling may be performed such that the points in the CAD model point cloud and the representation have a uniform or near uniform distribution, or have equal or near equal point density. Based on the sampling, the coordinate systems of the model and the representation may be coarsely (e.g., with a resolution of 1 cm) and finely (e.g., with a resolution of 1 mm).

[0073] The kinematic reachability and collision logic 174 is configured to perform kinematic reachability and collision analysis. In evaluating kinematic reachability and collision, the kinematic reachability and collision logic 174 may perform multiple different calculations. In some examples, these calculations may be mutually exclusive. In some examples, the kinematic reachability calculation calculates whether the arm of the picking robot is capable of (is capable of) the mechanical reach required to pick the target object from the pick-up location. In some examples, collision calculations may be performed in addition to kinematic feasibility calculations. The kinematic reachability and collision logic 174 may use a flexible collision library (FCL) in performing the collision calculations. The FCL includes various techniques for efficient collision detection and proximity calculation as a tool for collision avoidance analysis or calculation. The FCL is useful for performing multiple proximity queries on different model representations and can be used to perform probabilistic collision identification between 3D representations. Additional or alternative resources may be used in combination with or in place of the FCL.

[0074] Aspects of kinematic feasibility analysis for a particular type of robotic arm are described as an example to provide a basic understanding of the described technology. This example of aspects of kinematic feasibility analysis is not an extensive overview of all possible features of all different kinds and types of robotic arms. The example of a robotic arm having six degrees of freedom and six joints is not intended to define the scope of any or all aspects of the present disclosure. In other examples of robotic arms, the number of degrees of freedom and joints may differ from each other (e.g., the number of joints may be five and the number of degrees of freedom may be six).

[0075] As an example, in evaluating the feasibility of welding at each of the plurality of nodes or node-waypoint pairs, the kinematic reachability and collision logic 174 may perform a plurality of calculations. In some implementations, each of the plurality of calculations may be mutually exclusive. In some implementations, a first calculation may include a kinematic feasibility calculation that calculates whether a robot arm coupled to the welding tool employed can mechanically reach (or be) at a state defined by the node or node-waypoint pair. In some implementations, in addition to the first calculation, a second calculation (which may be mutually exclusive from the first calculation) may also be performed. The second calculation may include calculating whether the robot arm will encounter a collision (e.g., a collision with the workspace or something within the workspace) when accessing a portion of the seam (e.g., the node or node-waypoint pair in question).

[0076] The first calculation may be performed before performing the second calculation. In some embodiments, the second calculation may be performed only if the result of the first calculation is positive (e.g., if it is determined that the robot arm can mechanically reach (or be in) the state defined by the node or node-waypoint pair). In some embodiments, the second calculation may not be performed if the result of the first calculation is negative (e.g., if it is determined that the robot arm cannot mechanically reach (or be in) the state defined by the node or node-waypoint pair).

[0077] Kinematic feasibility may correlate with the type of robot arm employed. Herein, the robot arm is assumed to include a six-axis robot welding arm with a spherical wrist. The six-axis robot arm may have six degrees of freedom in total, three in the Cartesian coordinates of X, Y, and Z, plus three additional degrees of freedom for the wrist-like nature of the robot arm. For example, the wrist-like nature of the robot arm provides a fourth degree of freedom in the wrist up and down direction (e.g., wrist moves in +y and -y directions), a fifth degree of freedom on the wrist side (e.g., wrist moves in -x and +x directions), and a sixth degree of freedom in rotation. In some implementations, the welding tool is attached to the wrist portion of the robot arm.

[0078] Referring to FIG. 14, an example of a representation of a robot arm according to one or more aspects is shown. The representation of the robot arm includes a Base, a Wrist Center, and links B, L, R, S, T, and U, which may be considered as joint variables. To determine whether the employed robot arm can mechanically reach (or be) in a state defined by a node or a node-waypoint pair, i.e., to perform a first calculation, the robot arm (including the welding tool) may be mathematically modeled, for which the following mathematical modeling may be employed. In some implementations, the kinematic reachability and collision logic 174 may solve the first three joint variables based on the wrist position and solve the other three joint variables based on the wrist orientation. It is noted that the torch of the welding tool may be rigidly attached to the wrist. Therefore, it is assumed that the transformation between the torch tip and the wrist is fixed. A geometric approach (e.g., the law of cosines) may be employed to first determine the three joint variables (e.g., variables S, L, U).

[0079] Once the first three joint variables (S, L, U) are successfully calculated, the kinematic reachability and collision logic 174 can solve for the last three joint variables (R, B, T), for example, by considering the wrist orientation as ZYZ Euler angles. Note that the kinematic reachability and collision logic 174 may need to consider some offsets of the robot. These offsets may need to be considered and accounted for due to inconsistencies in the Unified Robot Description Format (URDF) file. For example, in some implementations, the position values ​​(e.g., the X-axis of the joint) of the joints (e.g., the actual joints of the welding robot) may not match the values ​​noted in the URDF file. Such offset values ​​may be provided to the controller in a table. In some implementations, the kinematic reachability and collision logic 174 may take these offset values ​​into account while mathematically modeling the robot arm. In some implementations, once the robot arm is mathematically modeled, the kinematic reachability and collision logic 174 may determine whether the robot arm can mechanically reach (or be) in a state defined by a node or a node-waypoint pair.

[0080] As described above, the kinematic reachability and collision logic 174 can evaluate whether the robot arm will collide or interfere with a part, fixture, or anything else in the robot arm's environment if the robot arm is placed in a position and orientation defined by that waypoint-node pair. Once the kinematic reachability and collision logic 174 has determined the states the robot arm can be in, it can use a second calculation to perform the aforementioned evaluation (e.g., as to whether the robot arm will collide with something in its environment).

[0081] The model information 164 may include manufacturing information 176 and thresholds 177. The manufacturing information 176 may include a computer-aided design (CAD) model of a representation of a structure (e.g., a final assembly) including the first object 140, the second object 142, or a combination thereof. For example, the representation may include one or more 2D representations, one or more 3D representations, such as a point cloud of the final assembled object, or a combination thereof. The CAD model may indicate a spatial relationship between the first object 140 and the second object 142, may include annotations related to the first object 140, the second object 142, or the seam 144 formed between the first object 140 and the second object 142, or may include a combination thereof. Additionally or alternatively, the manufacturing information 176 may include a point cloud of the object (also referred to herein as a CAD model point cloud) generated using the CAD model, welding instructions or plans for the first object 140 and the second object 142, tool center point (TCP) information related to a tool, or a combination thereof. In this disclosure, the terms welding plan and welding instructions are used interchangeably. Manufacturing information 176 may also include identification information associated with one or more objects. The identification information may include RFID tag information, markers (e.g., bar codes or QR codes), part numbers, the geometric configuration of the object (e.g., shape and dimensions), physical characteristics of the object (e.g., weight), other relevant information, or combinations thereof.

[0082] The thresholds 177 may include or correspond to one or more design constraints associated with the end product. The one or more design constraints may include one or more maximum values, one or more minimum values, one or more ranges, or a combination thereof.

[0083] The sensor data 165 may include or correspond to sensor data 180 received by the control device. The pose information 166 may include or correspond to the pose of the first object 140, the second object 142, or a combination thereof. The system information 168 may include information related to one or more devices of the robotic system 100 (e.g., the robotic device 110, the tool 120, the manufacturing tool 126, or the sensor 130). To illustrate, the system information 168 may include, as illustrative and non-limiting examples, ID information, a communication address, one or more parameters, or a combination thereof.

[0084] The one or more I / O devices 157 may include, by way of illustrative and non-limiting examples, a mouse, a keyboard, a display device, a camera, other I / O devices, or combinations thereof. The network interface 160 may be configured to communicatively couple the controller 150 to one or more other devices, such as the first robotic device 110, the first tool 120, the manufacturing tool 126, the sensor 130, or a combination thereof. For example, the controller 150 may be configured to receive sensor data 180 from the sensor 130, output control information 182 to one or more other devices, or a combination thereof. By way of example, the controller 150 may be configured to communicate via an Ethernet connection or a wireless communication protocol, such as Wi-Fi.

[0085] In some implementations, the controller 150 may be a single device configured to perform the operations described herein. One skilled in the art can recognize that although FIG. 1 illustrates the controller 150 as a single block, the implementation of the controller 150 is not limited to a single component, but may instead be distributed across multiple components. For example, the operations of the controller 150 may be distributed across multiple devices configured to perform all or a portion of the operations of the controller 150 in accordance with the present disclosure. Implementing the functionality of the controller 150 across multiple devices can improve the efficiency, processing time, and reliability of the robotic system 100.

[0086] During operation of the robotic system 100, the controller 152 (e.g., control device 150) may receive and store model information 164, such as 2D data, image data, 3D data, or point cloud data, associated with the first object 140, the second object 142, or the first object 140 and the second object 142 bonded together (e.g., bonded state). In some implementations, the model information 164 may include a CAD model of a representation of a structure including the first object 140 and the second object 142, object identifiers or parameter information of the first object 140 or the second object 142, or a combination thereof. In some implementations, the controller 152 may perform one or more pre-processing operations, such as one or more operations as further described herein with reference to at least FIG. 16 .

[0087] Controller 152 may also receive sensor data 180 from sensor 130. In some implementations, controller 152 is configured to control sensor 130 to generate at least a portion of sensor data 180. Based on sensor data 180, controller 152 may identify first object 140, second object 142, or a combination thereof. For example, to identify first object 140, controller 152 may determine an object ID (e.g., a serial number, a barcode, a QR code, etc.) of first object 140 based on first sensor data 180, match a representation (e.g., a 3D representation) associated with sensor data 180 to point cloud information included in model information 164, or a combination thereof.

[0088] After identifying the first object 140 or the second object 142, the controller 152 may perform a fitting process (e.g., a positioning process) to determine a pose of the first object 140 or the second object 142. For example, with reference to the first object 140, the controller 152 may perform the fitting process (using the fitting logic 173) to determine a first pose of the first object 140. To perform the fitting process (e.g., a positioning process), the controller 152 may perform a comparison or matching operation based on the model information (e.g., point cloud data) and at least a portion of the sensor data 180 associated with the first object 140.

[0089] Based on the first pose of the first object 140, the controller 152 may determine a first position on the first object 140 for the tool 120 to be coupled (e.g., grasped and / or picked up) to the first object 140 and control the robotic device 110 and the first tool 120 such that the first tool 120 is coupled to the first object 140 based on the first position. In some embodiments, the controller may identify one or more positions on the first object 140 for the first tool 120 to be coupled to the first object 140 and perform a reachability analysis, a collision analysis, or a combination thereof for each of the one or more positions.

[0090] The controller 152 may generate and transmit control information to the robotic device 110 or the tool 120 to enable or cause the tool 120 to be coupled to the first object 140. After the tool 120 is coupled to the first object 140, the controller 152 may receive additional sensor data (e.g., 180) from the sensor 130. The additional sensor data may be associated with the first object 140 coupled to the first tool 120. The controller 152 may perform a fitting process (e.g., a positioning process) based on the model information 164 and the additional sensor data to determine a second pose of the first object 140. In some implementations, the controller may perform a reachability analysis, a collision analysis, or a combination thereof based on the second pose of the first object 140.

[0091] The controller 152 may control the robotic device 110 to direct the first object 140 into a spatial relationship with the second object 142 such that a seam 144 is formed between the first object 140 and the second object 142. For example, the controller 152 may be configured to control the tool 120, the robotic device 110, or a combination thereof to adjust the second pose of the first object 140. In some embodiments, the controller 152 may change the configuration of the robotic device 110 to bring the first object 140 and the second object 142 together. The configuration of the robotic device 110 may include a pose in which the first object 140 is held.

[0092] To verify that the first object 140 and the second object 142 are in a spatial relationship, the controller 152 may receive sensor data (e.g., 180) related to the first object 140 and the second object 142. The controller 152 may determine whether a separation (e.g., gap) between the first object 140 and the second object 142 is less than or within a threshold (e.g., 177) based on the model information and the sensor data. If the separation is greater than the threshold, the controller 152 may perform a fitting process (e.g., a positioning process) on the first object 140 or the second object 142 and control the robotic device 110 or the tool 120 to bring the first object 140 and the second object 142 toward a spatial relationship. The controller 152 may perform the fitting process (e.g., a positioning process) again after adjusting the first object 140 to be in a spatial relationship with the second object 142. If the separation is less than the threshold, the controller 152 may generate a manufacturing instruction. While the above description specifies using a fitting process to ascertain / detect spatial relationships, in other implementations the spatial relationships may be performed using vision or force servoing techniques (as further described herein with reference to at least FIG. 16), or a combination of the two.

[0093] A manufacturing instruction (e.g., 182) may cause manufacturing tool 126 to perform an action. Controller 152 may transmit or execute the manufacturing instruction. Execution of the manufacturing instruction may cause an action to be performed by manufacturing tool 126. For example, a manufacturing instruction may include one or more bolt instructions or one or more pin instructions to cause manufacturing tool 126 to provide a bolt or pin that penetrates one or more objects. A bolt instruction may also include an instruction to cause manufacturing tool 126 to couple a nut to a bolt. As another example, a manufacturing instruction may include one or more welding instructions to cause manufacturing tool 126 (e.g., a welding tool) to weld a seam formed between first object 140 and second object 142, such as welding along the seam formed by first object 140 and second object 142. By way of example, manufacturing tool 126 may lay a weld metal along the seam, spot weld (or tack weld) the seam, fuse the seam, or a combination thereof.

[0094] In some implementations, after the first object 140 and the second object 142 are joined, the controller 152 is further configured to control the tool 120 to be uncoupled from the first object 140 and coupled to a third object at a third location on the third object. For example, after the weld metal is laid to join the first object 140 and the second object 142, the controller 152 is further configured to control the tool 120 to be uncoupled from the first object 140 and coupled to the third object at a third location on the third object. The controller 152 may also control the robotic device 110 to direct the third object into a second spatial relationship with the joined first object 140 and second object 142 such that a second seam is formed between the third object and the first object 140 or the second object 142. The controller 152 may generate further instructions to cause the manufacturing tool 126 to join the third object to the first object 140 and the second object 142, such as welding instructions to cause the manufacturing tool 126 to weld a second seam to join the third object, the first object 140, and the second object 142.

[0095] In some embodiments, the robotic system 100 may include one or more other components. For example, the robotic system 100 may include a positioner, a structure (e.g., a frame), or a combination thereof. The positioner may be configured to hold (e.g., host) the first object 140, the second object 142, or a combination thereof. For example, the positioner may host the first object 140 before the first object 140 is picked up by the tool 120. The structure may be configured to support the sensor 130. Additionally or alternatively, the robotic system 100 may include one or more additional robotic devices.

[0096] In some embodiments, the robotic system 100 includes a robotic device 110 disposed in the workspace 102 and a tool 120 coupled to the robotic device 110. The tool 120 is configured to be selectively coupled to a first object 140. The robotic system 100 further includes a sensor 130 configured to generate sensor data 180 related to the workspace 102. The robotic system 100 also includes a manufacturing tool 126 (e.g., a welding tool) configured to join two or more objects. For example, the manufacturing tool 126 (e.g., a welding tool) may be configured to lay down a weld metal to join two or more objects. The robotic system 100 includes a controller 152 configured to control the tool 120 to be coupled to the first object 140 based on the sensor data 180, control the robotic device 110 to place the first object 140 in a spatial relationship with the second object 142 such that a seam 144 is formed between the first object 140 and the second object 142, and generate welding instructions (e.g., 182) to the welding tool to lay weld metal along the weld at the seam 144 to join the first object 140 and the second object 142. For example, the welding instructions (e.g., 182) may cause the welding tool to lay weld metal along the weld seam 144 to join the first object 140 and the second object 142.

[0097] One advantage provided by the robotic system 100 of FIG. 1 is that the robotic system 100 provides techniques for performing assembly tasks. For example, the robotic system 100 provides a robotic device configured to autonomously pick up a first object 140, bring the first object 140 with a second object 142, and enable a manufacturing task (e.g., welding) to be performed on the first object 140 and the second object 142. Additionally, the robotic system 100 includes improved sensing and control capabilities compared to conventional systems such that the robotic system 100 can verify or update the pose of one or more objects during the performance of an assembly task to maintain coordination and precision within the system.

[0098] 2, which is a block diagram illustrating details of another example of a robotic system 200 according to one or more embodiments. The robotic system 200 may include or correspond to the robotic system 100 of FIG.

[0099] 1, the robotic system 200 includes a plurality of robotic devices. Illustratively, the plurality of robotic devices includes four robotic devices, including a first robotic device (e.g., 110), a second robotic device 212, a third robotic device 214, and a fourth robotic device 216. Furthermore, the sensor 130 includes a plurality of sensors, such as a first sensor 234 and a second sensor 236. Also, the robotic system 200 includes a positioning device 240 (e.g., a positioner device), a structure 242, and a second tool 222 in addition to the first tool 120.

[0100] The workspace 102 of the robotic system 200 may include one or more devices or components of the robotic system 200. As shown, the workspace 102 includes a first robotic device 110, a first tool 120, a second robotic device 212, a second tool 222, a first sensor 234, and a manufacturing tool 126. In other embodiments, the workspace 102 may include fewer or more components or devices than are shown in FIG. 2. For example, the workspace 102 may include a third robotic device 214, a fourth robotic device 216, a second sensor 236, a positioning device 240, a structure 242, a controller 150, or a combination thereof.

[0101] In some embodiments, the plurality of robotic devices can include or correspond to robotic device 110. For example, at least one of the plurality of robotic devices (e.g., 110, 212, 214, 216) can include a robotic arm providing six degrees of freedom, as non-limiting examples. In embodiments, the robotic arm can be manufactured by YASKAWA®, ABB® IRB, KUKA®, Universal Robots®. Additionally or alternatively, the robotic arm can be configured to couple to one or more tools.

[0102] The second robotic device 212 may include a second robotic arm. A second tool 222 may be coupled to an end of the second robotic arm. In some embodiments, the second tool 222 may include or correspond to the first tool 120. For example, the second tool 222 may be configured to selectively couple to a second set of one or more objects including the second object 142. The second set of objects may be the same or different from the first set of objects to which the first tool 120 is configured to be coupled.

[0103] The third robotic device 214 may include a third robotic arm. The first sensor 234 may be coupled to an end of the third robotic arm. In some embodiments, the first sensor 234 is configured to generate first sensor data (e.g., 180). For example, the first sensor 234 is configured to capture one or more images of the first object 140, the second object 142, or a combination thereof.

[0104] The fourth robotic device 216 includes a fourth robotic arm. A manufacturing tool 126 (e.g., a welding tool) is coupled to the end of the fourth robotic arm.

[0105] The second sensor 236 is configured to generate second sensor data (e.g., 180). For example, the second sensor 236 is configured to capture one or more images of the first object 140, the second object 142, or a combination thereof. In some implementations, the second sensor 236 is disposed on or coupled to the structure 242. The structure 242, such as a frame or weld, may be dynamic or static. In either the dynamic or static configuration of the structure 242, the second sensor 236 may be configured to be dynamic or static with respect to the structure 242, for example, if the second sensor 236 is dynamic, the second sensor may be configured to rotate (e.g., pan) or tilt.

[0106] The positioning device 240 may be configured to host (e.g., hold or securely hold) the first object 140, the second object 142, or a combination thereof. By way of example, the positioning device 240 may be dynamic or adjustable, either manually by a user or automatically by a motor. For example, the positioning device 240 may dynamically adjust its pose (e.g., position, orientation, or other physical configuration) before or during an assembly process or during a welding process based on instructions received from the controller 152. In some implementations, the positioning device 240 is configured to adjust the pose or position of an object hosted by the positioning device 240 while the object hosted by the positioning device 240 is hosted by the positioning device 240. For example, the positioning device 240 may be configured to hold, position, or manipulate the first object 140, the second object 142, or a combination thereof. In some implementations, the positioning device 240 may be configured to hold, position, or affix one or more objects, such as using a clamp or other type of fixture. 2 illustrates a single positioning device, in other implementations, the robotic system may include multiple positioning devices. In some implementations, the positioning device 240 may be essentially static in that it may be a bin-like storage unit that can house objects (e.g., the positioning device 240 may not be configured to adjust the pose or position of an object). In some such implementations, the objects may be randomly placed within the unit (e.g., in a random pose).

[0107] During operation of the robotic system 200, the controller 152 may perform one or more operations as described with reference to the robotic system 100. Additionally, the processor may perform additional operations as described herein.

[0108] The controller 152 may receive sensor data (e.g., 180) from the first sensor 234, the second sensor 236, or a combination thereof to identify and locate the first object 140 and the second object 142. In some embodiments, the controller 152 is configured to control the third robotic device 214 to cause at least one sensor to generate at least a portion of the sensor data 180. After identifying the first object 140 or the second object 142, the controller 152 may perform a fitting process (e.g., a positioning process) to determine a pose of the first object 140 or the second object 142. For example, the controller 152 may use fitting logic 173 to determine the pose of the first object 140 or the second object 142.

[0109] In some implementations, before instructing the first tool 120 to pick up the first object 140 or before instructing the second tool 222 to pick up the second object 142, the controller 152 may control the positioning device 240 to change the pose of the first object 140 or the pose of the second object 142. In some implementations, the controller 152 may perform a fitting process (e.g., a positioning process) after the positioning device 240 changes the pose of the first object 140 or the pose of the second object 142. The controller 152 may determine a first pick-up position of the first object 140 or a second pick-up position of the second object 142 after the positioning device 240 changes the pose of the first object 140 or the pose of the second object 142.

[0110] After the first tool 120 is coupled to the first object 140 and the second tool 222 is coupled to the second object 142, the controller 152 can control the first robotic device 110, the first tool 120, the second robotic device 212, the second tool 222, or a combination thereof to direct the first object 140 and the second object 142 toward a spatial relationship such that a seam 144 is formed between the first object 140 and the second object 142. After the first object 140 and the second object 142 are brought together (toward a spatial relationship), the controller 152 may verify whether the first object 140 and the second object 142 are in a spatial relationship, for example, verify whether a separation between the first object 140 and the second object 142 is below a threshold, such as a threshold indicated by or included in the model information. By verifying that the first object 140 and the second object 142 are in a spatial relationship, the robotic system 200 can verify that the first object 140 and the second object 142 are in a spatial relationship before performing a manufacturing or assembly operation, such as a welding operation, on the first object 140 and the second object 142.

[0111] To confirm that the first object 140 and the second object 142 are in a spatial relationship, the controller 152 may receive additional sensor data (e.g., 180) related to the first object 140 and the second object 142. In some embodiments, the controller 152 may control the third robotic device 214, the first sensor 234, or a combination thereof to cause the first sensor 234 to generate at least a portion of the additional sensor data. The controller 152 may determine whether a separation between the first object 140 and the second object 142 is less than or equal to a threshold based on the model information 164 and the additional sensor data. If the separation is greater than the threshold, the controller 152 may perform a fitting process on the first object 140 or the second object 142 and may control the first robotic device 110, the first tool 120, the second robotic device 212, the second tool 222, or a combination thereof to direct the first object 140 and the second object 142 toward being in a spatial relationship, or a combination thereof. The controller 152 may determine whether the first object 140 and the second object 142 are in a spatial relationship after the matching process or after controlling the first robotic device 110, the first tool 120, the second robotic device 212, the second tool 222, or a combination thereof. If the separation is less than a threshold, the controller 152 may generate manufacturing instructions. In addition to executing the manufacturing instructions, the controller 152 may control the fourth robotic device 216 to position the manufacturing tool 126 relative to the seam 144. In some embodiments, the controller 152 may control the first robotic device 110, the first tool 120, the second robotic device 212, the second tool 222, or a combination thereof to change the orientation of the first object 140 and the second object 142 while they are in a spatial relationship such that the first object 140 and the second object 142 are positioned in a position having a seam 144 adjacent to the manufacturing tool 126 while they are in a spatial relationship.

[0112] Controller 152 can transmit or execute manufacturing instructions. Execution of the manufacturing instructions causes manufacturing tool 126 to perform an action. For example, the manufacturing instructions can include one or more assembly instructions to cause manufacturing tool 126 to perform an action, such as inserting a bolter or pin through first object 140, second object 142, or both. As another example, the manufacturing instructions can include one or more welding instructions to cause manufacturing tool 126 (e.g., a welding tool) to weld a seam formed between first object 140 and second object 142, such as seam 144 formed by first object 140 and second object 142. By way of illustration, the one or more welding instructions can cause the manufacturing tool (e.g., a welding tool) to lay weld metal along a seam formed between first object 140 and second object 142, such as along seam 144 formed by first object 140 and second object 142. Thus, the system may include one or more robotic devices configured to autonomously pick a first object 140, a second object 142, or both, bring the first object 140 together with the second object 142, and perform a manufacturing task (e.g., welding) on ​​the first object 140 and the second object 142.

[0113] In some embodiments, the assembly robot system includes a first robotic device disposed in the workspace and a first tool coupled to the first robotic device. The first tool is configured to be selectively coupled to the first object. The assembly robotic system includes a second robotic device disposed in the workspace and a second tool coupled to the second robotic device. The second tool is configured to be selectively coupled to the second object. The first object and the second object are configured to form a first seam between the first object and the second object. The assembly robotic system also includes a third robotic device disposed in the workspace and one or more sensors configured to generate sensor data related to the workspace. At least a first sensor of the one or more sensors is coupled to the third robotic device. The assembly robotic system further includes a fourth robotic device disposed in the workspace and a welding tool coupled to the fourth robotic device. The welding tool is configured to weld the first seam formed between the first object and the second object. For example, the welding tool may be configured to lay a weld metal along the first seam formed between the first object and the second object. The assembly robotic system includes a controller configured to control a first tool to be coupled to a first object at a first location on the first object based on the sensor data and to control a second tool to be coupled to a second object at a second location on the second object based on the sensor data, The controller is further configured to generate welding instructions to control the first robotic device, the second robotic device, or both, to bring the first object and the second object into a first spatial relationship such that a first seam is formed between the first object and the second object, and to cause the welding tool to metallically weld the first seam to couple the first object to the second object.

[0114] The principles and techniques described herein (e.g., principles relating to positioning target objects, such as first object 140 or second object 142, in workspace 102, determining a pose of the target objects, identifying a pick-up location on each of the target objects, picking up the target objects, placing the target objects in a spatial relationship or fixed pose relative to one another, and performing a manufacturing operation (e.g., welding) on ​​one or more objects in a spatial relationship) are not limited to robotic system 100, robotic system 200, or other systems in which the principles are described (such as the systems of FIG. 3, FIG. 4, or FIG. 16). In other words, the principles are not limited to 1-robot or 4-robot systems; rather, the principles may be used to implement an n-robot system, where n is a positive integer.

[0115] With reference to Fig. 3, a diagram illustrating details of an example of a robotic system 300 according to one or more embodiments. In some embodiments, the robotic system 300 may include or correspond to the robotic system 100 or the robotic system 200. With further reference to Fig. 4, Fig. 4 is a perspective view illustrating details of the robotic system 300 of Fig. 3 from one or more sides. The robotic system 200, such as a robotic assembly system, includes or is associated with a workspace 302. The workspace 302 may include or correspond to the workspace 102.

[0116] 3, in some embodiments, a robotic system 300 may include multiple robots 301, 311, 321, 331, frames 350, 360, a welding controller 307, a main controller 308 (hereinafter referred to as "controller 308"), and positioning devices 355, 365 that host objects 322, 32, respectively. Still referring to FIG. 3, the workspace 302, and specifically, one or more sensors (hereinafter referred to as "sensors 130") within the workspace 302, the welding controller 307, the positioning devices 355, 365, and the robots 301, 311, 321, 331 are (at least communicatively) coupled to the controller 308.

[0117] Robots 301, 311, 321, and 331 may include or correspond to the robotic devices of Figures 1 and 2. For example, robot 311 may include or correspond to the first robotic device 110, robot 331 may include or correspond to the second robotic device 212, robot 321 may include or correspond to the third robotic device 214, and robot 301 may include or correspond to the fourth robotic device 216.

[0118] The positioning devices 355, 365 may include or correspond to the positioning device 240. Additionally, the objects 322, 332 may include or correspond to the first object 140 and the second object 142. The frames 350, 360 may include or correspond to the structure 242. The sensors 330 of the robotic system 300 may include or correspond to the sensors 130, such as the first sensor 234, the second sensor 236, or a combination thereof. The welding controller 307 may include or correspond to the controller 150, such as the controller 152 (e.g., the control logic 170). The controller 308 may include or correspond to the controller 150, such as the controller 152.

[0119] Each of the plurality of robots 301, 311, 321, 331 may include, as a non-limiting example, a robot arm providing six degrees of freedom. In some implementations, the robot arm 305, 315, 325, 335 of each of the robots 301, 311, 321, 331 may be any suitable robot arm, such as those manufactured by YASKAWA®, ABB® IRB, KUKA®, Universal Robots®. The robot arm 305, 315, 325, 335 may have one or more tools coupled to its tip. The plurality of robots 301, 311, 321, 331 may include a robot arm 305, 315, 325, 335, and each of the robot arms 305, 315, 325, 335 may be attached to a tool 310, 320, 330, 340, respectively. As an example, based on the functionality that the robot performs, the robot arm may be coupled to a tool configured to enable the functionality (e.g., perform at least a portion of the function). To illustrate, the robot arm 305 may be coupled to a welding tool, making the robot 301 a welding robot. Similarly, the robot arm 315 may be coupled to one or more sensors, making the robot 311 a sensing robot. The robot arms 325, 335 may be coupled to a picking tool, making them picking robots. In some implementations, a single robot may perform multiple functions. For example, both a welding tool and one or more sensors may be coupled to a robot arm, making a single robot perform both sensing and welding functions. In some implementations, it is contemplated that a single picking robot picks a single object. However, if a single picking robot cannot pick an object (e.g., the weight of the object exceeds the weight-carrying capacity of a single robot), in other implementations, two or more picking robots may be configured to pick the object.

[0120] The tool 330 may be disposed on a robot arm, such as the robot arm 325. In some embodiments, the sensor 330 may be disposed on a welding robot arm (e.g., on a robot arm to which a welding tool is attached). Additionally or alternatively, the sensor 330 may be disposed on a structure or frame 350, 360 disposed in the workspace 102, as described and illustrated herein with reference to at least FIG. 4. In some embodiments, the sensor 330 may be disposed in one or more of the configurations described above. For example, the workspace 302 may include a combination of one or more sensors (e.g., 330) on the robot arm and one or more sensors (e.g., 330) on the frame 350, 360. The functionality of the sensor 330 is described in more detail below in conjunction with the controller 308.

[0121] The sensors 330 may be one or more image sensors configured to capture visual information (e.g., two-dimensional (2D) images or three-dimensional (3D) scans) regarding the workspace 302. For example, the sensors 330 may include one or more cameras (e.g., cameras with built-in lasers), scanners (e.g., laser scanners), and the like. In some implementations, the sensors 330 may include sensors such as Light Detection and Ranging (LiDAR) sensors. Alternatively or additionally, the sensors 330 may be or include one or more audio sensors configured to emit and / or capture sounds, such as Sound Navigation Ranging (SONAR) devices. Alternatively or additionally, the sensors 330 may be or include one or more electromagnetic sensors configured to emit and / or capture electromagnetic (EM) waves, such as Radio Detection and Ranging (RADAR) devices. Through visual, audio, electromagnetic, and / or other sensing technologies, the sensors 330 may collect information regarding physical structures and objects within the workspace 302. In some examples, the sensors 330 may collect static information (e.g., stationary structures in the workspace), in other examples, the sensors 330 may collect dynamic information (e.g., moving structures in the workspace), and in still other examples, the sensors 330 may collect a combination of static and dynamic information. The sensors 330 may collect any suitable combination of such information regarding the physical structures in the workspace 302 and may provide such information to other components (e.g., the controller 308) to generate representations, such as 3D representations, of the physical structures in the workspace 302 (e.g., the positioners 355, 365, the objects 322, 332, and the robots 301, 311, 321, 331, or combinations thereof, etc.). As described above, the sensors 330 can capture and communicate any of a variety of information types, but this description assumes that the sensors 330 primarily capture visual information (e.g., 2D images or 3D scans) of the workspace 302, which are then used collectively to generate a 3D representation of the workspace 302 (or objects placed in the workspace 302), as described below.

[0122] The positioning devices 355, 365 are configured to hold, position, and / or manipulate the objects 322, 332, respectively. In some embodiments, the positioning devices 355, 365 may hold, position, or affix the objects, for example, using a clamp or other type of fastener. For example, the positioning devices 355, 365 may be configured to securely hold the objects 322, 332. In some embodiments, the positioning devices 355, 365 are dynamic or adjustable, either manually by a user or automatically by a motor. For example, the positioning devices 355, 365 may dynamically adjust their pose (e.g., position, orientation, or other physical configuration) before or during the welding process based on instructions received from the controller 308, as described below. Although FIG. 3 illustrates the objects 322, 332 positioned on different positioning devices (e.g., the positioning devices 355, 365), in some embodiments, the objects 322, 332 may be positioned on the same positioning device. Additionally or alternatively, the positioning devices 355, 365 may be bin-like storage units, shelves, racks, or combinations thereof that can house objects. In such implementations, the objects may be randomly positioned (e.g., in random poses) within the units.

[0123] Welding controller 307 may have electronic and computational hardware to appropriately control the welding process performed via a welding tool mounted on the end of a robotic arm. In some implementations, welding controller 307 may have the ability to communicate with other computerized devices (e.g., controller 308, described below), where communication may include control or action requested from any suitable device or may mean sending information regarding the status of the welding tool, the weld or weld being applied, or the welding controller itself. Welding controller 307 may function as or interface with a welding power source.

[0124] 3 , the workspace 302, and specifically the sensors 330, welding controller 307, positioning devices 355, 365, and the robots 301, 311, 321, 331 within the workspace 302, are (at least communicatively) coupled to a controller 308. The controller 308 is any suitable machine specifically and specially configured (e.g., programmed) to perform the operations attributed herein to the controller 308, or more generally to the robotic system 300. In some examples, the controller 308 is not a general-purpose computer, but instead is specifically programmed and / or hardware configured to perform the operations attributed herein to the controller 308, or more generally to the robotic system 300. In some examples, the controller 308 is or includes an application specific integrated circuit (ASIC) configured to perform the operations ascribed to the controller 308, or more generally, the robotic system 300, herein. In some examples, the controller 308 includes or is a processor, such as a central processing unit (CPU), a graphics processing unit (GPU), or the like. In some examples, the controller 308 is a field programmable gate array (FPGA). In other examples, the controller 308 includes a memory that stores executable code that, when executed by the controller 308, causes the controller 308, or more generally, the robotic system 300, to perform one or more of the operations ascribed to herein. The controller 308 is not limited to the specific examples described herein.

[0125] The controller 308 controls one or more sensors in the workspace 302, the weld controller 307, the positioning devices 355, 365, and the robots 301, 311, 321, 331 (including tools attached to the robot arms). For example, the controller 308 can control a robot having one or more sensors (e.g., 330) attached thereto to move within the workspace 302 and capture data (e.g., one or more images, audio data, and / or EM data, as described below). For example, the controller 308 can control the robot to locate one or more objects within the workspace 302, pick up one or more objects at specific pick-up locations, approach one or more objects, and perform welding operations according to a weld plan. In some implementations, the controller 308 can also manipulate the positioning devices to rotate, translate, or otherwise move one or more objects within the workspace 302. The positioning devices can be controlled by the controller 308 to change the pose of a part placed thereon. For example, the positioning device may be motorized in that it can be operated to move / rotate (or change pose) the object relative to the ground or the welding robot. The controller 308 may also control other aspects of the robotic system 300. For example, the controller 308 may further interact with a user interface (UI) (not explicitly shown in FIG. 3 ) by providing a graphical interface on the UI through which the user may interact with the robotic system 300, provide input to the robotic system 300, and through which the controller 308 may provide and / or receive various types of information from the user (e.g., identified seams that are candidates for welding, possible weld plans, welding parameter options or selections, etc.). The UI may be any type of interface including a touch screen interface, a voice-activated interface, a keypad interface, a combination thereof, etc.

[0126] Additionally, the controller 308 may interact with a database 312, for example, by storing data in the database 312 and / or retrieving data from the database 312. The database 312 may include or correspond to the memory 156. The database 312 may more generally be stored in any suitable type of storage device configured to store any type of information. In some examples, the database 312 may be stored in a storage such as a random access memory (RAM), a memory buffer, a hard drive, an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), a read-only memory (ROM), a flash memory, or the like. In some examples, the database 312 may be a cloud-based platform. The database 312 may store any information useful to the robotic system 300 in performing an assembly operation. For example, the database 312 may store information captured by one or more sensors. The database 312 may further store one or more CAD models, for example, of the final assembly of the objects 322, 332. As another example, the database 312 may store an annotated version of the CAD models of the objects 322, 332 and / or the final assembly of the objects 322, 332. The database 312 may also store a point cloud of the object (also referred to herein as a CAD model point cloud) generated using the CAD model. Similarly, the welding instructions or plans for the objects 322, 332 may be stored in the database 312. In some embodiments, the database 312 may further store Tool Center Point (TCP) information associated with tools coupled to the robotic arm. For example, the database 312 may include TCP information for welding tools, grippers, etc.

[0127] In examples, the storage stores executable code 313 that, when executed, causes the controller 308, or more generally the robotic system 300, to perform one or more of the operations ascribed herein. In some examples, the executable code is a single, self-contained, program, while in other examples, the executable code is a program having one or more function calls to other executable code that may be stored in the storage or elsewhere. Additionally or alternatively, one or more functions resulting from execution of the executable code may be implemented by hardware. For example, multiple processors are useful for performing one or more individual tasks of the executable code.

[0128] Returning to the sensors, the controller 308 is configured to receive data from the sensors (e.g., image data, such as 2D images, 3D images, and possibly other data, such as audio or electromagnetic data). In some implementations, the information captured by the sensors 330 may be used to generate one or more representations, such as a 3D representation, of the structures captured in the images. The 3D representations may include a point cloud. A point cloud may be a collection of points, each of which represents a location in 3D space of a point on the surface of an element (e.g., object 322, 332, positioning device 355, 365, etc.) in the workspace 302.

[0129] In some examples, one or more images (e.g., 2D image data captured by the sensor(s) 330 at a particular orientation relative to the object 322, 332) are superimposed or stitched together by the controller 308 to reconstruct and generate 2D or 3D image data of the workspace 302. The 3D image data can be collated to generate a point cloud having associated image data for at least some of the points in the point cloud. In examples, the 3D image data can be collated by the controller 308 in such a manner that the point cloud generated from the data can have six degrees of freedom. For example, each point in the point cloud can represent an infinitesimal position in 3D space. As described above, the sensor 330 can capture multiple 2D images of the points from various angles. These multiple 2D images can be collated by the controller 308 to determine an average image pixel for each point. The average image pixel can be attached to the point. For example, if the sensor 330 is a color camera with red, green and blue channels, the six degrees of freedom may be {x position, y position, z position, red intensity, green intensity and blue intensity}. For example, if the sensor 330 is a black and white camera with black and white channels, four degrees of freedom may be generated.

[0130] In some implementations, instead of using a sensor that captures 2D images, a 3D sensor (such as a scanner sold by Photoneo®) may be implemented. In some implementations, the 3D sensor may use structured light projection to reconstruct the shape of the 3D surface of the inspected object (e.g., object 322). For example, one or more structured patterns may be projected onto the background (scene) during scanning, which allows the depth to be calculated from one or more images of the background. After scanning, information about the surface shape of the scanned object and / or the background may be provided to the controller 308, for example as a depth map or in the form of a point cloud. In some implementations, the scanning process of the sensor consists of three stages: capture (or acquisition), processing (computation), and transfer. In the first stage, the sensor projects a series of light patterns onto the background. Once the data is acquired, the sensor (in cooperation with the controller 308) calculates the visible points on the background and transmits this data to the controller 308, for example via an Ethernet connection or other communication protocol such as Wi-Fi.

[0131] Referring to FIG. 4, FIG. 4 of the robotic system 300 illustrates example locations of additional sensors (e.g., 330), such as sensors 351, 352, 361, 362, within the workspace 302. For example, sensors 351, 352, 361, 362 may be disposed on frames 350, 360, respectively. Sensors 351, 352, 361, 362 are disposed on frames 350, 360 and affixed to the comers of frames 350, 360, although the placement of sensors 351, 352, 361, 362 may be different. As an example, sensors 351, 352, 361, 362 may be affixed to different locations on frames 350, 360. Additionally or alternatively, frames 350, 360 may be static in nature, while sensors 351, 352, 361, 362 disposed on such frames may be movable. For example, each of the sensors 351, 352, 361, 362 may be configured to bend and rotate to scan the workspace 302. Additionally or alternatively, both the frame and the sensors may be movable.

[0132] The shapes of the frames 350, 360 shown in Figures 3 and 4 are for illustrative purposes. In some implementations, frames of different shapes may be employed. For example, one or more sensors may be coupled to a tripod-style frame structure, and each sensor may be configured to bend or rotate to scan the workspace 302. In some implementations, the use of a frame may be avoided, and the sensors 351, 352, 361, 362 may be placed on the ground. In such implementations, the sensors may be movable, for example, the sensors may be configured to bend or rotate.

[0133] 16 is a block diagram illustrating details of an example of a robotic system 1600 according to one or more embodiments. The robotic system 1600 may include or correspond to robotic system 100, robotic system 200, robotic system 300, or a combination thereof. Thus, the robotic system 1600 may be configured to perform one or more operations as described herein with reference to at least robotic system 100, robotic system 200, robotic system 300, or a combination thereof. Additionally or alternatively, the robotic system 1600 may be configured to perform one or more operations as further described herein with reference to at least FIG. 5 or FIG. 15.

[0134] The robotic system 1600 may include one or more components, one or more stages, or a combination thereof. For example, the robotic system 1600 may include pre-processing 1610 (e.g., a pre-processing stage), assembly 1620 (e.g., an assembly stage), welding 1630 (e.g., a welding stage), post-processing 1640 (e.g., a post-processing stage), and a controller 1650. The pre-processing 1610, assembly 1620, welding 1630, post-processing 1640, or a combination thereof, may be included in or associated with a workspace, such as workspace 102 or workspace 302. Additionally or alternatively, the robotic system 1600 may include one or more robotic devices, such as robotic devices 110, 212, 214, 216, 301, 311, 321, 331, etc.

[0135] In some embodiments, the robotic system 1600 may include a different number of stages, a different number of components, or a combination thereof. Additionally or alternatively, a single stage may be divided into multiple stages, or multiple stages may be combined into a single stage. In some embodiments, the robotic system 1600 may include an assembly line or be part of an assembly line. The assembly line may be configured to receive pre-processed objects from a pre-processing center / stage (e.g., including laser cutting) of the assembly line.

[0136] Pre-processing 1610 may be configured to receive or provide a material, an object (e.g., a pre-processed object), or a combination thereof. The material, object, or combination thereof may be used by robotic system 1600 to form a new part or a finished product. The material, object (e.g., a part, a portion, a component, etc.), or combination thereof may include or correspond to object 140, 142, 322, 332, or another object. In some implementations, the material may be considered a raw material, such as a sheet of metal or a tube of metal.

[0137] Pre-processing 1610 may include subtractive processing 1612 (e.g., a subtractive processing stage, a fabrication stage, or a laser cutting stage), inspection 1613 (e.g., an inspection stage), a cleaner 1614, or a combination thereof. The subtractive processing 1612 may be configured to create one or more objects. For example, the subtractive processing 1612 may be configured to receive a material and create one or more objects based on the material. By way of example, the subtractive processing 1612 may receive a material, perform a laser cutting operation, and output a shaped object based on the laser cutting operation. Additionally or alternatively, the subtractive processing 1612 may be configured to cut one or more objects from a metal sheet, form one or more grooves (e.g., V-grooves or U-grooves) in an object, form slots or holes in an object, separate or debond one or more cut objects, or a combination thereof. Although described as performing a laser cutting operation, subtractive processing 1612 may be configured to perform additional or alternative fabrication operations such as scanning, measuring, registering, bending, drilling, grinding, etc., as illustrative and non-limiting examples.

[0138] In some implementations, the subtractive process 1612 may be configured to operate with path planning logic (e.g., adaptation logic 173 or kinematic reachability and collision logic 174) to plan a trajectory. The subtractive process 1612 may also be configured to adjust a force applied to a surface, such as a surface of an object or material. For example, the subtractive process 1612 may also be configured to adjust a force applied to a surface by a robotic device or tool. If the subtractive process 1612 removes layers of a surface, the force may need to be adjusted after removal of one or more layers or after each layer. Additionally or alternatively, the subtractive process 1612 may be configured to perform visual detection of the workspace and / or work piece. For example, the subtractive process 1612 may include or operate with one or more sensors (e.g., 130) to perform visual detection. In some implementations, the subtractive process 1612 may be configured to project a pattern onto the object or material using a projector (e.g., tool 120) based on a CAD model. The visual detection operations may be configured to recognize one or more known patterns that may explain the absence of features on an object or material. The subtractive processing 1612 may also be configured to select a particular tool to perform a desired operation, the selection being made manually via a GUI or automatically based on an algorithm. For example, given a bank of available tools (e.g., available tool positioners) and one or more operations to be performed on an object or material, an algorithm may query the appropriate tool to be used.

[0139] Inspection 1613 may be configured to perform monitoring, quality control, or a combination thereof. For example, inspection 1613 may be configured to monitor the number of objects / parts made (at pre-processing 1610), the number of objects / parts passing (through pre-processing 1610), or a combination thereof. To monitor the number of objects / parts made or passing through, inspection 1613 may be configured to perform object detection, segmentation, counting, or a combination thereof. In some implementations, object detection, segmentation, counting, or a combination thereof may be performed in conjunction with one or more sensors 130. Additionally or alternatively, if an object or material (e.g., a metal sheet) has dimensions that differ from the desired dimensions, the object or material may be laser cut (e.g., 1612) or fixed before being sent to a subsequent stage. In other words, inspection 1613 may determine whether one or more dimensions of the object or material meet a threshold (e.g., within a tolerance). Identification of a "difference" with respect to the desired dimensions may be object specific. For example, if the object is sheet metal, then the Inspect 1613 may use edge detection techniques to measure one or more dimensions of the sheet metal. Additionally or alternatively, the Inspect 1613 may use positioning techniques (such as those described above with reference to the matching logic 173) based on objects such as subcomponents to determine or identify differences.

[0140] In some implementations, the inspection 1613 may be configured to scan, measure, or register an object. By way of example, the inspection 1613 may be configured to perform one or more quality control operations including visual inspection (e.g., using one or more sensors 130) to inspect an object, material, or joint (e.g., weld quality). The visual inspection may include comparing the object, material, or joint (e.g., weld) to a CAD file / model. Additionally or alternatively, to inspect a joint (e.g., weld quality), the inspection 1613 may be configured to perform a visual inspection, monitor sound, determine a heat signature associated with a weld, determine a weld puddle shape, or perform a combination thereof, as illustrative and non-limiting examples. The inspection 1613 may determine whether one or more dimensions of the object are within a threshold (e.g., 177) or tolerance of the object. If the object is out of tolerance, one or more actions may be performed on the object (e.g., by subtractive processing 1612) to bring the object within tolerance or the object may be discarded. If the object is within tolerance, the object may be provided to cleaner 1614 or assembly 1620. In some embodiments, inspection 1613 may receive the object from cleaner 1614 and determine whether the object may be provided to assembly 1620.

[0141] In some embodiments, inspection 1613 may be configured to perform one or more quality control operations during pre-processing 1610, after welding 1630 (e.g., during a welding or bonding operation), or during post-processing 1640. For example, inspection 1613 may be configured to perform one or more quality control operations during pre-processing as part of part, object, or material inspection. Inspection of parts, objects, or materials may include verifying that individual parts, materials, or objects meet required specifications and are within acceptable tolerances. This ensures that each component, object, or material is suitable for use in the assembly of a final product. Any parts that are out of tolerance are identified and replaced or modified before proceeding to assembly 1620. As another example, inspection 1613 may be configured to perform one or more quality control operations after welding 1630 is performed to weld or bond two or more objects together (but before a final assembly is formed). Inspection between pre-processing 1610 and post-processing 1640 may be similar to those described with reference to the quality control operations performed during pre-processing 1610. For example, the one or more quality control operations may include verifying that two or more joined objects are within an acceptable tolerance. As another example, inspection 1613 may be configured to perform one or more quality control operations during post-processing 1640 to evaluate the final assembly. Illustrating that once all components have been inspected, the final assembly process can begin, and one or more quality control operations may be configured to ensure that the finished product meets predefined tolerances. The quality control operations during post-processing 1640 can maintain the overall quality and functionality of the finished product.

[0142] In some implementations, the inspection 1613 may be configured to apply periodic and / or non-periodic forces to inspect a part, object, or material. For example, the repetitive and / or non-repetitive forces may be applied in addition to or as an alternative to one or more visual inspection or non-destructive testing processes. The periodic forces may be applied using one or more robots. For example, one or more robots may be configured (e.g., programmed) to apply repetitive, cyclic forces to a test object (e.g., a component, object, or material) simulating real-world conditions such as vibration, impact, or repetitive stress. These tests help determine the fatigue life of a material and identify potential weaknesses in a design, allowing improvements to be made before final production. Non-periodic forces may include applying a single, non-repetitive force to a material under test. This helps evaluate the material's response to a sudden, one-time event, such as an impact or collision. This information is useful in understanding the overall strength and resilience of the material being tested. In some implementations, the cyclic and / or non-cyclic forces can include destructive testing techniques such as hammering, twisting, or a combination thereof, as illustrative and non-limiting examples. Destructive testing techniques provide insight into the structural integrity and durability of the component, object, or material being tested. In some situations, destructive testing methods such as hammering, twisting, or tearing may be useful to fully understand the performance limits and failure modes of a material, object, or component. By applying such extreme forces, the robot can accurately measure the breaking point and deformation characteristics of the material, providing valuable data that can be used to optimize designs and ensure long-term durability.

[0143] In some embodiments, the inspection 1613 can be configured to perform different levels of inspection, such as a micro-level or a macro-level. By way of example, a micro-level inspection can include or correspond to a weld inspection, such as porosity detection, undercut, weld size, comparison of weld geometry to a reference geometry, etc. A macro-level inspection can relate to out-of-tolerance, cylindricity, and distortion determination due to weld heat.

[0144] In some implementations, the inspection 1613 may be configured to generate or provide an output related to the results of one or more operations, such as one or more quality control operations. For example, the inspection 1613 may be configured to generate or provide the output via a GUI. The GUI, such as the I / O devices 157, may be configured to display one or more color-coded values, with different colors representing various levels of deviation from a tolerance, such as an ideal tolerance. The one or more color-coded values ​​enable a user to quickly identify components or assemblies that are out of tolerance and take appropriate corrective action. Additionally or alternatively, the GUI may be configured to generate or output a list of one or more components, objects, or materials that are out of tolerance along with the degree to which they deviate from a desired fit or tolerance. Such information is useful in pinpointing problems and determining one or more adjustments to improve product quality. Additionally or alternatively, the inspection 1613 may be configured to generate or output a text file indicating parts, objections, or materials and / or their deviations from an ideal fit at each waypoint in the manufacturing process. This text file can act as a record for quality control purposes, making it easier to track and address recurring issues in the manufacturing process.

[0145] The cleaner 1614 is configured to perform one or more operations to prepare the object for additional processes or operations, such as a tacking operation or a welding operation. For example, the cleaner 1614 may be configured to perform one or more operations, such as rust removal, debris removal, build-up removal, sandblasting, sanding, grinding, filing, sharp edge removal, chemical treatment, or combinations thereof, as illustrative and non-limiting examples. In some embodiments, the cleaner 1614 may include a robotic device, tool, or combination. For example, the robotic device may include a robotic arm including a tool coupled to the robotic arm. The tool may include or correspond to tools 120, 126, 222, 310, 320, 330, 340. After the object is cleaned, the object may be inspected by inspection 1613, placed, or held in assembly 1620.

[0146] In some implementations, the cleaner 1614 may be configured to perform visual detection of the workspace and / or work pieces. For example, the cleaner 1614 may include or operate with one or more sensors (e.g., 130) to perform visual detection, such as visual detection to detect rust, debris, deposits, sharp edges, or combinations thereof. In some implementations, the cleaner 1614 is configured to apply air pressure. For example, the cleaner 1614 may be configured to detect debris on an object or material and apply air pressure to remove the debris. Additionally or alternatively, the cleaner 1614 may be configured to detect rust or debris (e.g., visual detection based on the sensor 130) and perform force control of a tool (e.g., a file or sander) to remove the debris or rust, or the like. In some implementations, the cleaner 1614 may be configured to operate with path planning logic (e.g., adaptation logic 173 or kinematic reachability and collision logic 174) to plan a trajectory of an object, material, one or more tools, or combinations thereof.

[0147] In some implementations, the cleaner 1614 is configured to prepare one or more objects for further processes or operations, such as tack welding or other welding operations. For example, the cleaner 1614 can identify areas requiring cleaning, such as by detecting issues, such as rust, paint, debris, etc. In some implementations, by analyzing visual data collected (e.g., by the sensor 130), the cleaner 1614 can locate the exact location needing attention and then perform the necessary cleaning tasks. These tasks can include rust removal, sandblasting, polishing, filing, or chemical treatment, among other techniques.

[0148] Assembly 1620 may be configured to identify an object, select an object, position an object (e.g., relative to another object), verify the position of an object, adjust the position of an object, or combinations thereof. After one or more operations associated with assembly 1620, a welding operation may be initiated or performed, as described further herein.

[0149] Assembly 1620 may include part location 1622 (e.g., a part location phase), picking 1624 (e.g., a picking phase), part bonding 1626 (e.g., a part bonding phase), scanplanning 1628 (e.g., a scanplanning phase), part matching 1629 (e.g., a part matching phase), or a combination thereof. In some embodiments, assembly 1620 can include or correspond to perception logic 172, matching logic 173, or a combination thereof.

[0150] Part localization 1622 may include identifying or recognizing an object, selecting an object, or a combination thereof. To illustrate, an object may be received from preprocessing 1610. For example, the object may be located in a staging area, such as a positioning device. The object in the staging area may be positioned in a predefined pose, alone, with other similar or different objects, in an organized or unorganized manner, or a combination thereof. The staging area (e.g., positioning device) may include a table, a rack, a bin, a carousel (e.g., a carousel including multiple bins or racks), a holder, or a combination thereof. The staging area (e.g., positioning device) may include or correspond to positioning devices 240, 355, 365. In some embodiments, the object may be identified based on an identifier on the object, a CAD model, a part assembly sequence, or a combination thereof. In some embodiments, part localization 1622 may identify or select an object based on scan data associated with the object, such as described herein above with reference to at least FIG. 3 and FIG. 4.

[0151] Picking 1624 may include determining a point / location at which to pick up or grasp an object, grasping an object, or a combination thereof. For example, picking 1624 may determine a location to grasp an object after the object is identified and / or selected for grasping by part orientation 1622. Picking 1624 may perform one or more actions based on a future state or position of the object (e.g., a desired position or relationship).

[0152] To pick up or grasp an object, picking 1624 may include determining a mesh or pose of the object, determining one or more candidate gripping poses, checking one or more constraints, ranking one or more gripping poses, selecting a gripping pose, determining or validating a path of movement of the grasped object, or combinations thereof. Illustratively, to determine a mesh or pose of the object, picking 1624 may generate or determine mesh or pose information, such as 3D or point cloud information of the object. To determine the one or more candidate gripping poses, picking 1624 may generate one or more candidate gripping poses associated with the part. For example, the gripping poses may include or be associated with a gripping pose of a tool, such as a gripper device, clamp, vacuum, etc., coupled to the robotic device. The one or more candidate gripping poses may be generated randomly, at uniformly spaced positions across the object, or a combination thereof. Additionally or alternatively, the candidate gripping poses may be determined based on a weld plan or weld positions associated with the object. Thus, the search space (or number of grasp positions) of the object, the number of candidate grasp poses, or a combination thereof may be reduced based on available information related to the robotic system 1600.

[0153] In some implementations, picking 1624 can select grasp pose candidates using a model, such as a trained model or a neural network. The model may be trained using training data such that the model can output one or more grasp pose candidates or a best pose candidate. The model may be used by picking 1624 in conjunction with a scoring scheme, such as the exemplary grasp scoring scheme described further herein, or independently of the scoring scheme.

[0154] Picking 1624 may check one or more constraints, such as one or more geometric constraints, for at least one of the one or more candidate gripping poses. For example, picking 1624 may check one or more constraints to ensure that proper contact can be achieved between a tool (e.g., a gripper device) and an object. In some implementations, picking 1624 can check one or more constraints for each of the one or more candidate gripping poses.

[0155] The one or more constraints may include, as illustrative and non-limiting examples, a tool-object collision constraint, a gripping contact constraint, an assembly feasibility constraint, a welding accessibility constraint, or a combination thereof. A tool-object collision constraint may be related to a tool gripping / holding an object such that the tool does not collide with the part. To illustrate, when a tool includes two grippers (e.g., fingers) coupled to a base of the tool, a tool-object collision constraint verifies that the base of the tool does not or will not contact the object. A gripping contact constraint may be related to verifying that the grippers (e.g., fingers) of the tool properly contact the object. An assemblability constraint may be related to verifying that a gripping position or a position / pose of the tool does not or will not interfere with another object, such as another object to which the object is to be joined (e.g., welded). Welding feasibility constraints may relate to verifying that the gripping position or tool position / position is not too close to the weld area (e.g., within a threshold distance) and / or does not impede (e.g., limit or restrict) the mobility of the welding torch.

[0156] Picking 1624 may rank the one or more gripping pose candidates. In some implementations, picking 1624 may rank the one or more gripping pose candidates that meet or pass the constraint check. Picking 1624 may determine a grip score for one of the one or more gripping pose candidates. In some implementations, picking 1624 may determine a grip score and / or rank the one or more gripping pose candidates based on a grip location, a center of gravity of the object, an amount of contact of the gripper (or fingers) with the object, or a combination thereof. As an illustrative and non-limiting example, a grip score for a gripping pose candidate may be determined as follows: where α is a first constant value and β is a second constant value.

number

[0157] The picking 1624 may rank one or more candidate gripping poses (or a portion thereof) based on the determined gripping score(s). Additionally or alternatively, the picking 1624 may select the gripping score with the highest value and determine or plan a path for the tool (e.g., gripper) to the gripping position associated with the selected gripping score. In some implementations, the picking 1624 may determine the path using kinematic reachability analysis and collision analysis. If the picking 1624 determines that the path of the gripper to grip the object based on the candidate gripping pose is possible or valid, the candidate gripping pose is selected. In some implementations, the path may include a path of the gripper relative to the object, a path of the gripper to grip the object from a position / pose where the gripper grips the object to a position / pose where the weld operation is performed on the object, or a combination thereof. If the picking 1624 determines that the path is not possible or invalid, the picking 1624 selects another candidate gripping pose based on the determined gripping score. For example, picking 1624 may select the next highest grasp score and determine whether the path associated with the grasp location of the next highest grasp score is possible or valid.

[0158] In some implementations, a picking position may not provide an optimal pose for a downstream manufacturing task. To address this issue, picking 1624 is designed to generate a set of alternative picking poses and one or more corresponding paths, allowing a smooth transition from an undesired pick to a more appropriate, desired pick. In some implementations, the one or more corresponding paths may be determined using perception logic 172, adaptation logic 173, kinematic reachability and collision logic 174, or a combination thereof. In addition to determining or selecting a picking position, picking 1624 may also analyze the stability of the pose when placing or positioning the object, such as ensuring that the object remains stable and correctly oriented during placement. Additionally or alternatively, picking 1624, perception logic 172, adaptation logic 173, kinematic reachability and collision logic 174, or a combination thereof may consider the speed and acceleration at which the object is picked up and moved. By considering these factors, the planner can minimize the risk of dropping the object or causing any damage during the transfer process.

[0159] In some implementations, picking 1624 may be configured to determine that the object cannot be grasped at an ideal position / location on the object. For example, picking 1624 may identify an ideal position / location on the object based on a CAD model, perception logic 172, fit logic 173, kinematic reachability and collision logic 174, or a combination thereof. To illustrate, the object may be placed (e.g., on a table, in a bin, or on a positioning device) such that picking 1624 cannot grasp the object at the ideal position / location on the object. In some such implementations, the object may be grasped at a first position on the object and placed on the positioning device (or other structure) so that the object can be re-grasped at the desired position / location on the object. In some implementations, picking 1624 may be configured to determine how to position the object on the positioning device (or other structure) so that the object is stable and can be re-grasped at the desired position / location on the object. In some implementations, picking 1624 may be configured to determine that the grasp is inappropriate (e.g., "off") after positioning or fitting the object. Additionally or alternatively, the picker may determine that a re-grasp needs to be performed based on potential collisions, robot movement, calibration issues (e.g., determining that the object cannot be brought into correct spatial relationship with other objects), etc. In such a situation, the picker 1624 may place the object in an intermediate state to allow the object to be re-grasped.

[0160] Based on the selection of the candidate gripping poses (and / or paths), the object is gripped. In some implementations, part coupling 1626 is configured to grip the object and move the object from a location / position where the gripper grips the object to a location / position associated with a welding operation to be performed on the object subject. Once the object is in a location / position associated with the welding operation, assembly 1620 can ensure that the object (and one or more other objects / components) are positioned such that when assembled, the objects will fit seamlessly and precisely to result in a structurally sound and effectively functioning product. Such operations performed by assembly 1620 may be referred to as "part fit" and may include scanplan 1628, part fit 1629, or a combination thereof. Establishing part fit may include considering factors such as tolerances, material properties, physical dimensions of one or more objects, or a combination thereof, as illustrative and non-limiting examples. As illustrated, the above factors may be considered such that one or more objects fit precisely without gaps or misalignment (e.g., gaps or misalignment outside of acceptable tolerances).

[0161] In some implementations, part fitting can be performed or achieved using high resolution micro 3D sensors and can be used for very low tolerance parts. Additionally or instead, part fitting can be performed or achieved using visual servoing and force control. Visual servoing and force control can be performed sequentially, simultaneously (e.g., simultaneously), or a combination thereof. In some implementations, continuous positioning (which can be considered a type of visual servoing) may be used to perform or achieve part fitting.

[0162] Visual servoing involves using a camera or other visual sensor to guide a robotic device based on visual feedback. For example, the robotic device is guided in real-time based on visual feedback. In some implementations, a camera may be configured to capture images of one or more objects, and the images may be processed to determine position and / or orientation information of the one or more objects (individually or as a group). The position and / or orientation information may be used to adjust or control the movement of the robotic device for part fit. Additionally or alternatively, assembly 1620 may use force control. Force control involves measuring a force applied by an end effector (e.g., a tool holding an object) of the robotic device and generating force information based on the measured force. The force information may be used to adjust or control the movement of the robotic device. For example, if the robotic device encounters resistance (e.g., as indicated based on the measured force or force information) when attempting to insert an object into a proper / desired position relative to another object, the movement of the robotic device may be adjusted to apply more / less force as necessary to achieve proper / desired part fit. In some implementations, visual servoing may be used in conjunction with force control to achieve accurate part fit. Cameras or other visual sensors can provide real-time feedback on the object's position and orientation, while force control can ensure that the robotic device is applying the correct amount of force to achieve the proper / desired part fit.

[0163] The scanplan 1628 is configured to determine the position and / or orientation of an object, such as the position and / or orientation relative to one or more other objects. For example, the scanplan 1628 may be configured to determine one or more 2D measurements, one or more 3D measurements, other information, or a combination thereof. By way of example, the scanplan 1628 may perform one or more operations to determine a 3D point cloud, a 2D image, a camera matrix, initial seam information, or a combination thereof. In some implementations, the scanplan 1628 may be configured to guide a scanner or other sensor (e.g., 130) such that the scanner or other sensor provides information for one or more other operations or phases using vision techniques, force techniques, or a combination thereof, or the like. For example, the scanplan 1628 may be configured to optimize the pose of the scanner / robot according to the needs (e.g., vision or force) of one or more operations or phases. Additionally or alternatively, the scanplan 1628 may determine whether the gap contour and / or one or more measurements are within a tolerance range (e.g., whether they meet a threshold or are within a threshold range), or a combination thereof.

[0164] In some implementations, the scanplan 1628 is configured to scan an object to generate scan data (e.g., composite data). In some implementations, the scan may generate one or more 2D images (e.g., 2D image data). The scanplan 1628 may also perform 2D gap detection. The 2D gap detection may include generating or processing the 2D image data, performing gap detection, determining gap contours, or a combination thereof. In some implementations, the gap detection may be performed using a neural network.

[0165] The scanplan 1628 may also be configured to generate 3D data. For example, the scanplan 1628 may be configured to identify primitive components or shapes from a CAD model, scan areas around seams in 3D, project the 3D onto a 2D image, interpolate in the 2D area, uplift to 3D, or combinations thereof. Additionally or alternatively, the scanplan 1628 may be configured to perform edge reordering.

[0166] Part Fit 1629 may be configured to adjust the object. For example, part fit 1629 may be configured to adjust the position or orientation of the object based on data or information acquired or determined by scanplan 1628. For example, part fit 1629 may be configured to perform an edge reordering operation. Edge reordering can adjust the object to position an edge of the object to form a desired seam, such as a seam that meets one or more tolerances. Illustratively, part fit 1629 may cause a robotic device having a gripper to grasp an object to perform one or more operations, such as one or more movements. In some embodiments, part fit 1629 may have event-based force control. For example, part fit 1629 may measure a force or resistance to a movement that indicates that the object is in contact with another object. After a movement of the object to adjust the position or orientation of the object, scanplan 1628 may check whether the seam or contour is within the respective tolerance. In some embodiments, part fit 1629 may be configured to perform a fitting operation based on vision data, force data, or a combination thereof. Additionally or alternatively, Part Adaptation 1629 may be configured to perform adaptation actions based on other data such as sound, conductivity, etc. In some embodiments, Part Adaptation 1629 may be configured to execute a adaptation algorithm, such as a learning-based method / algorithm that takes as input a sequence of images or point clouds, or a mesh and joint states of the robot, and outputs / predicts a sequence of actions that will guide the robot to perform an assembly task.

[0167] After assembly 1620, welding 1630 may be configured to perform one or more welding operations on the object or a seam formed at least in part by the object. Welding 1630 may include welding plan 1632 (e.g., a welding plan stage), modification 1634 (e.g., a modification stage), tack / weld 1636 (e.g., a tack / weld stage), or a combination thereof. Welding plan 1632 may be configured to execute the welding plan. Modification 1634 may be configured to verify / confirm the welding plan. If the welding plan is not verified / confirmed, another welding plan may be generated by welding plan 1632 and verified / confirmed by modification 1634. Based on the welding plan verified / confirmed by modification 1634, tack / weld 1636 may perform a tack welding operation, a lap welding operation, or a combination thereof. In some implementations, tack / weld 1636 may perform one or more tack welding operations and then ungrip and move the gripper (e.g., a robotic device) from the object or region of the object. After the gripper is moved, the tack / weld 1636 can perform one or more welding operations, for example, one or more lap welding operations.

[0168] After one or more operations by tack / weld 1636, post-processing 1640 may be configured to perform one or more post-processing operations. Post-processing 1640 may include a cleaner 1642, a compliance 1644 (e.g., a compliance stage), a palletizing 1646 (e.g., a palletizing stage), or a combination thereof. The cleaner 1642 may be configured to perform one or more cleaning operations on the object after one or more tack / weld operations are performed on the object. The cleaner 1642 may include or correspond to the cleaner 1614. In some embodiments, the cleaner 1642 may perform one or more operations before or after one or more operations performed by compliance 1644, before one or more operations performed by palletizing 1646, or a combination thereof.

[0169] Compliance 1644 may be configured to perform one or more compliance checks on the welded object. In some implementations, the one or more compliance checks may include verifying that the product meets regulatory or industry standards, such as those related to strength and durability. The one or more compliance checks may include one or more scanning operations, destructive testing, non-destructive testing, or a combination thereof. Non-destructive testing techniques may include visual inspection, ultrasonic testing, x-ray testing, or a combination thereof. Non-destructive testing techniques allow for identifying defects or flaws in an assembled / finished weld without damaging the weld. Destructive testing techniques include damaging a product to evaluate product performance under stress. Destructive testing techniques include exposing a product to extreme temperatures, pressures, or other forces to see how the product reacts.

[0170] Destructive testing can be performed on small sample sizes, such as randomly selected products during multiple production runs of the same product. Destructive testing can provide valuable information about the strength and durability of a product, but is not suitable for all situations because it involves the destruction of the product. If a product fails any of the tests, an analysis may be performed (e.g., by the controller 1650) to generate hypotheses based on the results of the tests and adapt one or more aspects of the assembly line / stage. For example, one or more aspects may be to reduce tolerances on part fit, recheck after pre-processing, or make other adaptations. Palletizing 1646 may be configured to package the product (e.g., the object after one or more tacking / welding operations). In some implementations, palletizing 1646 may be configured to maximize the number of parts that can be placed on a pallet or in a box.

[0171] The controller 1650 may be configured to control or execute one or more operations of the robotic system 1600. For example, the controller 1650 may be configured to control or execute one or more operations of pre-processing 1610, assembly 1620, welding 1630, post-processing 1640, or combinations thereof. In some embodiments, the controller 1650 may be configured to control at least a portion of the robotic system 1600 to perform coordinated operations to reach difficult or welding areas under different part orientations (e.g., 1F, 2F, 1G, 2G, etc.) while assembling a complex part.

[0172] The controller 1650 is configured to determine which parts need to be welded by the assembly robot and which parts need to be welded by another welder down the assembly line. For example, the controller 1650 may be configured to learn a part assembly sequence 1652. The part assembly sequence 1652 may include or indicate an assembly sequence of one or more objects to manufacture a product, e.g., the order in which the objects / parts need to be picked up and brought together. To illustrate, the controller 1650 may determine a first sequence / process of assembling the product. For example, the controller 1650 may determine the first sequence / process (e.g., part assembly sequence 1652) based on a CAD model of the final product. Based on the execution of the first sequence / process, the controller 1650 may obtain data and feedback. Based on the data and feedback, the controller 1650 may determine a second sequence / process of assembling the product.

[0173] The robotic system 1600 of Figure 16 may enable handling of raw materials, creation of sub-components, and manufacturing of end products, such as complex end products. In some implementations, the robotic system 1600, such as pre-processing 1610, assembly 1620, welding 1630, post-processing 1640, controller 1650, or combinations thereof, is configured to perform one or more operations as further described herein with reference to at least Figure 5.

[0174] 5 is a flow diagram illustrating a process (e.g., method 500) for performing assembly tasks using an autonomous assembly robot, according to various embodiments. The method 500 disclosed in FIG. 5 will be described in conjunction with the robotic system 300 shown in FIGS. 3 and 4 illustrate a four-robot system in which two of the robotic arms (e.g., robotic arms 315 and 335) are configured to be utilized to pick an object (e.g., robotic arm 315 may pick object 322 and robotic arm 335 may pick object 332), robotic arm 325 may be configured to be used to scan (e.g., to perceive) the workspace (and objects 322 and 332 within the workspace), and the last one of the four robotic arms (e.g., robotic arm 305) may be used to weld the object (e.g., a part picked by the robotic arm configured to pick the part). Note that one or more aspects of method 500 are also applicable to robotic system 100 of FIG. 1, robotic system 200 of FIG. 2, or robotic system 1600 of FIG. 16.

[0175] Method 500 will now be described in conjunction with the four-robot system shown in Figures 3 and 4. Method 500 includes, at step 510, robotic system 300 scanning and locating one or more objects for assembly. For example, robotic system 300 can locate different parts to be picked up and assembled. As another example, assembly 1620 of Figure 16 may be configured to locate different parts to be picked up and assembled.

[0176] In step 520, the robotic system 300 may obtain a first object and scan the first object. For example, the robotic system 300 may pick up object 322 from positioning device 355 or object 332 from positioning device 365 and rescan objects 322, 332 to improve accuracy of the object position on the tool of the robotic arm. As another example, the assembly 1620 of FIG. 16 may be configured to pick up and scan an object.

[0177] In step 530, the robotic system 300 may determine a path to move the first object. For example, the robotic system 300 may execute an algorithm to move the picked up object based on the reachability and mobility of one or more robots included in the robotic system 300. As another example, the assembly 1620 of FIG. 16 may be configured to move the picked up object based on the reachability and mobility of one or more robots.

[0178] In step 540, the robotic system 300 may position the first object in a spatial relationship (e.g., a fixed pose) with a second object. For example, the robotic system 300 may implement a visual servoing approach that uses perceptors to find the position or pose of the object with high precision and may move the object relatively to bring it into the desired position or relationship. As another example, the assembly 1620 of FIG. 16 may be configured to position the first object in a spatial relationship with another object.

[0179] In step 550, the robotic system 300 may perform a manufacturing operation. For example, the robotic system 300 may implement a visual servoing approach to locate a welding tool, move the welding tool to a desired position, and operate the welding tool. As another example, the welding 1630 may be configured to perform a manufacturing operation.

[0180] In step 560, the robot may determine a next action. For example, the robotic system 300 may release the first object, the second object, or both the first object and the second object. As another example, the controller 1650 of FIG. 16 may determine the next action.

[0181] After releasing one or both of the first and second objects, method 500 may return to step 510 to identify one or more objects for the next assembly task. Additionally or alternatively, after releasing one of the first or second objects, method 500 may return to step 520 to obtain scans of the first and second objects that are joined together.

[0182] Further details of one or more operations that may be performed as part of method 500 are described further herein. Referring to step 510 of method 500, the robotic system 300 scans the workspace 302 with one or more sensors, such as sensor 330, mounted on the robot. Additionally or alternatively, the workspace 302 may be scanned with sensors located elsewhere in the workspace, such as on the frame. For example, the workspace 302 may be scanned with sensors 351, 352, 361, 362 located on frames 350, 360.

[0183] The data captured by the sensors can be used to generate a 2D or 3D representation of the workspace 302 or elements (e.g., objects) within the workspace 302. An example of such a 3D representation is described further herein with reference to at least FIG. 6. FIG. 6 is an image of a 3D representation of a point cloud of multiple parts. The multiple parts (e.g., objects) are positioned on a positioning device, such as positioning device 365.

[0184] 5, in an embodiment in which multiple objects are included in the background or workspace 302 (e.g., multiple robots and positioning devices, each positioning device having multiple objects disposed thereon), the robotic system 300 may perform a clustering algorithm (e.g., K-means, affinity propagation, DBSCAN, mean shift, etc.) to segment or divide the 3D representation into different classes or clusters. Additionally or alternatively, the clustering algorithm may find and identify a target object within the 2D or 3D representation. The target object may include the object to be assembled or another object.

[0185] In some implementations, the target object may be located and identified within the 3D representation (or 2D representation) based on a representation of the final assembled object. In some examples, the representation of the final assembled object may include a CAD model (or a point cloud of the CAD model) of the final assembled object, which may be provided to the controller 308 (and stored in the database 312) before the start of the assembly process or during the assembly process. In some implementations, the controller 308 may use some information present within the representation of the final assembled object (e.g., annotations from the user, shapes of the objects forming the final object, identification information of the objects forming the final assembled object present in the CAD file, etc.) to identify and locate the target object within the 3D representation of the background. In some implementations, the controller 308 may implement a shape identification algorithm to determine the shape of the object within the 3D representation. For example, the controller 308 may determine the shape of the object within the final assembled object present in the CAD file (e.g., by implementing shape information in the file or a feature or shape recognition algorithm) and use the determined shape to identify the target object within the 3D representation. In some implementations, annotations from the user may include seam annotations, annotations corresponding to where tack welds will be made, and the like.

[0186] During operation, the sensor captures one or more images of the workspace 302, which may be converted into one or more 3D representations. A clustering algorithm (informed by information captured from a CAD model of the final assembled object) may then be performed on the one or more 3D representations. As mentioned above, the information captured from the CAD model may include the shapes of the objects that will form the final object, which may be used to classify and identify the target object from the 3D representation of the background.

[0187] In some implementations, the use of a clustering algorithm to identify target objects within the 3D representation may be avoided (e.g., not used) in implementations in which the target objects have identification information (e.g., unique ID information) associated with them. This ID information may be present in a CAD model of the final assembled object. Additionally or alternatively, the ID information may be loaded (or provided) to the controller 308 when the object is placed on the positioning device 365, 355 (e.g., by one or more sensors that scan the ID of the target object). In such implementations, after generating a 3D representation of the background and receiving the unique IDs associated with the objects that will form the final assembled object, the controller 308 may use the unique IDs or other information provided to the controller 308 to identify the target objects within the 3D representation (e.g., the 3D representation may project or annotate the unique IDs of the objects within the 3D representation).

[0188] In some implementations, the ID information may be carried by an RFID tag on the target object, which may include unique identification information associated with the object. Additionally or alternatively, the object positioned on the positioning device may include a barcode, an ArUco marker (e.g., QR code), or a serial number that includes a unique ID associated with the object. During operation, one or more sensors may scan the ID marker (e.g., ArUco, RFID, etc.) and associate the scanned information with object ID information in a CAD model of the final assembled object to identify the object. In some implementations, the unique ID may also include additional information about the object, such as the geometric configuration of the object (e.g., shape and dimensions), the physical characteristics of the object (e.g., weight), and other relevant information.

[0189] Once the target objects are identified, they may be positioned within the workspace 302. For example, if the controller 308 identifies the target objects, the controller 308 may determine their positions within the 3D representation of the workspace 302, which may be translated to the real-world positions of the objects. In operation, after identifying the target objects within the 3D representation, the controller 308 may identify the positions of the target objects within the workspace 302. In some implementations, the target objects may be positioned on the same positioning device or on different positioning devices. As shown in FIGS. 3 and 4, the target objects (e.g., objects 322 and 332) are positioned on separate positioning devices (e.g., positioning devices 355 and 365).

[0190] Referring to step 520 of method 500, the robotic system 300 picks up a target object. Although the controller 308 has determined the location of the target object, in some implementations it may be beneficial to refine that information and determine the pose of the target object. Determining the pose of the target object may help determine where the pick-up robot (e.g., robot 311 including robot arm 315) picks up the target object. Note that the position where the robot may hold the object may be referred to as the pick-up position. The position where the pick-up robot picks up / holds the object may depend at least in part on the pose in which the object is positioned on the positioning device. At least some information regarding the pose (e.g., position and orientation) of the object may be determined using a positioning technique. The positioning technique includes or is associated with the adaptation logic 173.

[0191] The positioning process is configured to transform or align data from different sources (e.g., the point cloud and the 3D representation of the CAD model) in the same coordinate frame or system. To illustrate, the controller 308 can perform the positioning process using the point cloud of the CAD model of the target object and the 3D representation of the target object (generated using data captured from one or more sensors) by sampling the point cloud and the 3D representation of the CAD model. The sampling is performed such that the points in the point cloud and the 3D representation of the CAD model have a uniform or nearly uniform distribution or have equal or nearly equal point densities. After sampling, the coordinate system of the model and the coordinate system of the scanned information may be coarsely (e.g., with a resolution of 1 cm) and finely (e.g., with a resolution of 1 mm). FIG. 7 includes images illustrating an example of the positioning process. For example, FIG. 7 includes a first image of the 3D representation of the collection of objects before positioning and a second image of the 3D representation of the collection of objects after positioning.

[0192] 5, after the objects have been scanned, located, positioned, and their relative object poses determined, they may be picked up by a corresponding picking robot. For example, object 322 may be picked up by robot 311 including robotic arm 315, and object 332 may be picked up by robot 331 including robotic arm 335.

[0193] As described herein, the object has, at least in part, the shape of a hollow cylinder, and the picker or gripper is a three-finger gripper, such as that manufactured by OnRobot®. Examples of pickers or grippers coupled to the object are further described herein with reference to FIGS. 8A-8C. In other implementations, the object may have a different shape, and the picker or gripper may be a picker or gripper other than a three-finger gripper, or a combination thereof. It should be noted that the location at which the pick-up robot may grasp or hold the object may depend at least in part on the geometric configuration (e.g., shape) of the object. Additionally or alternatively, the location at which the pick-up robot may hold the object may depend at least in part on the type of picker or gripper used to hold or pick the object.

[0194] 8A-8C include images of examples of different gripping positions for a finger-based gripper gripping a cylindrical object. As shown, FIG. 8A shows an end outside grip, where the picker's three fingers grip the cylindrical object from the outside. FIG. 8B shows an end inside grip, where the picker's fingers are inside the recessed portion of the cylinder. FIG. 8C shows a side grip, where the picker grips the object from the side.

[0195] Referring again to FIG. 5, in some implementations, the controller 308 may perform kinematic reachability and collision analysis on the possible grip positions to determine the final grip pose. The kinematic reachability and collision analysis may include or correspond to the kinematic reachability and collision logic 174. In evaluating the kinematic reachability and collision, the controller 308 may perform several different calculations. In some examples, these calculations may be mutually exclusive. In some examples, the kinematic feasibility calculation calculates whether the arm of the picking robot is capable (or present) of the mechanical reach required to pick the target object from the pick-up location. In some examples, a collision calculation may be performed in addition to the kinematic feasibility calculation. The controller 308 may use a flexible collision library (FCL) in performing the collision calculation. The FCL includes various techniques for efficient collision detection and proximity calculation as a tool for collision avoidance analysis or calculation. The FCL is useful for performing multiple proximity queries on different model representations and can be used to perform probabilistic collision identification between 3D representations. Additional or alternative resources may be used in combination with or in place of the FCL.

[0196] The collision calculation is configured to determine whether the robot's arm will encounter a collision (e.g., a collision with something in the workspace 302) during access when going to pick up the object. The controller 308 may perform the first calculation before performing the second calculation. In some examples, the second calculation may only be performed if the result of the first calculation is positive (e.g., it is determined that the robot's 301 arm can mechanically reach (or be present) in the state). In some examples, the second calculation may not be performed if the result of the first calculation is negative (e.g., it is determined that the robot's 301 arm cannot mechanically reach (or be present) in the state). The kinematic and collision analysis may be used to select or finalize one of the three pick-up locations.

[0197] After the final grip position is determined, the controller 308 may assign a desired coordinate frame value to the target object. The controller 308 may then use the desired coordinate frame of the target object to grip the target object. For example, the controller 308 may move the TCP of the gripper to the desired coordinate frame to pick up the object. Figure 9 includes an example image of the TCP of the gripper and the desired coordinate frame of the object (referred to as the potential TCP frame).

[0198] Referring again to FIG. 5, although optional, in some implementations, after picking up the target object, the controller 308 can instruct a sensor (either a static sensor on the frame or a sensor coupled to the robot arm) to rescan the target object. The rescanning process can include capturing one or more images of the picked up object from multiple different angles. The captured data can be combined to generate a synthetic scan in which the above-mentioned positioning techniques can be employed. Reapplication of the positioning process can use the synthetic scan in place of the 3D representation of the object generated after the first scan. Reapplication of the positioning techniques can allow the controller 308 to provide a more accurate view of reality. FIG. 10 includes images showing a 3D representation of the picked up object before and after positioning.

[0199] Referring again to step 530 of method 500 in FIG. 5, after the pick-up location is determined and the object is picked up, the robotic system 300 can place the picked-up object in a fixed pose relative to another object. This other object can be another target object to be picked up by another picking robot using the techniques described above. By way of example, the robot 331 can pick the object 332 and the robot 311 can pick the object 322. In step 530, the robots 311 and 331 can perform coordinated actions to place the objects 322 and 332 in a fixed pose relative to each other. In some implementations, the objects 322 and 332 can be positioned in a fixed pose in space as shown with reference to FIG. 11. FIG. 11 is an image of an example of two objects positioned together in a fixed pose in space. For example, FIG. 11 shows the object 322 coupled to the robot 311 and the object 332 coupled to the robot 331. As shown, the objects 322 and 332 are positioned together in a fixed pose in space, e.g., in a spatial relationship. FIG. 11 also shows the robot 301 positioned to perform a welding operation on the objects 322 and 332 .

[0200] In some implementations, the objects 322, 332 in a fixed relative pose may physically touch each other and form a seam. In other implementations, the objects 322, 332 may not touch each other and have a gap between them. In some implementations, the gap may be in the range of 0.1 mm to 1 cm. In some implementations, this gap may be reduced and the objects 322, 332 may be brought together in contact using techniques described with reference to step 540 of method 500. In other implementations, this gap may not be reduced and the objects 322, 332 may remain in a fixed pose and manufacturing tasks are performed on the objects 322, 332 while they are in a fixed pose where they are not physically touching each other.

[0201] 5, in some implementations, the following techniques (referred to herein as "object bonding techniques") can be used to bring parts into a fixed relative pose: Object bonding techniques can position objects in a fixed pose, determine how objects are oriented in space relative to one another to perform manufacturing tasks such as welding, and manage mobility and reachability constraints associated with picking and welding robots.

[0202] In some implementations, the object joining technique may include defining a virtual space in the workspace 302 where objects may be assembled. This virtual space may be or be defined by a boundary or cube of a certain length, width, height. In some implementations, the controller 308 may perform kinematic reachability and collision analysis of the robot that may be required to enter the virtual space. For example, the controller 308 may perform reachability and collision analysis on a picking robot to determine whether the robot can exist in one or more states (alone or with other objects) in the virtual space. In some implementations, the controller 308 may consider one pose of the object as a reference pose and then perform reachability and collision analysis of the second object with respect to the reference pose of the first object. The reachability and collision analysis may be performed considering a CAD model of the assembled object. The controller 308 may also perform reachability and collision analysis on the welding robot so that the welding tool can exist in a state according to the welding plan. For example, the controller 308 may perform reachability and collision analysis based on the location of a tack weld to be performed on the object. The reachability and conflict analysis may be similar to the reachability and conflict analysis described above with reference to step 520 of method 500 .

[0203] Referring to step 540 of method 500, in some implementations where the objects 322, 332 are physically in contact with one another in a fixed relative pose, method 500 may proceed to step 550. Note that in some implementations, if the parts are already in contact in a fixed relative pose, a fitting approach (e.g., 1629) may be implemented to ensure proper alignment and positioning of the objects. By utilizing a visual servoing method and / or positioning process, the robotic system can make minor adjustments to the position and / or orientation of the objects to achieve a desired fit and ensure a seamless connection. This adaptability allows the fitting approach to be used regardless of whether the parts are initially in contact or not. In some implementations where the objects 322, 332 are not physically in contact with one another in a fixed relative pose, the robotic system 300 can use a part fitting approach (e.g., part fitting 1629) to bring the objects 322, 332 together in contact with one another in step 540. In some implementations, the part fitting approach includes using a visual servoing approach. A visual servoing approach may use feedback information extracted from visual sensors (such as one or more sensors in workspace 302) to control at least one movement of a robot (e.g., robot 331 including robot arm 335) carrying an object (e.g., object 332). A visual servoing approach may include performing a positioning process (as described above) on the object while it is in space. In operation, after selecting target objects and positioning them close to each other (while they are not touching), controller 308 may direct sensors (either static sensors on the frame or sensors coupled to the robot arm) to rescan the object. The rescanning process may include capturing one or more images of the picked up object from multiple different angles. The captured data may be combined to generate a synthetic scan in which the positioning techniques described above may be employed.Reapplication of the positioning process for part fit can use a synthetic scan instead of the 3D representation of the object generated after the initial scan. Reapplication of the positioning technique can provide a more accurate view of reality by the controller 308. FIG. 12 includes images of an example of a positioning process in accordance with one or more aspects. For example, FIG. 12 includes a first image of the 3D representation before the positioning process and a second image of the 3D representation after the positioning process.

[0204] Referring again to FIG. 5, after reapplication of the positioning process, the controller 308 can instruct the picking robot to move the objects closer together. In some implementations, the positioning process may be applied iteratively to improve the accuracy of the part fit. FIG. 13 includes images of another example of a positioning process according to one or more embodiments. For example, FIG. 13 includes a first image of the 3D representation before the positioning process and a second image of the 3D representation after the positioning process. For example, FIG. 13 includes a first image of the 3D representation after the positioning process (left image) and a second image (right image) after another positioning process is performed after generating the first image. Thus, the robot system 300 may perform the positioning process multiple times before the objects come into contact with each other.

[0205] 5, in some implementations, in addition to visual servoing, force servoing may be implemented by the robotic system 300. For example, force servoing may be implemented based on a determination by the controller 308 that objects are in contact with one another (e.g., using images captured by sensors in the workspace 302). The iterations for visual servoing may be fixed. Additionally or alternatively, other data modalities such as sound or electricity (e.g., using arc techniques) may be implemented by the robotic system 300 in addition to visual servoing and / or force servoing.

[0206] During reapplication of the positioning, or even during the initial application of the positioning process, the controller 308 can determine appropriate positions where the sensor in the workspace 302 should capture images of the object based on a scan optimization technique. In other words, the goal of the scan optimization technique is to find a set of camera positions (or end effector positions) to provide optimal visual information. For example, explicitly, the controller 308 finds the optimal positions that maximize a visual score function given a discrete set of camera positions. Alternatively, the controller 308 may be configured to generate a minimal set of camera positions to maximize the desired information. As the controller 308 performs iterations of the scan optimization, the controller 308 can track what was actually seen. This allows the controller 308 to inform future actions regarding where to go next, subject to a priori estimation. In some implementations, the first step is to determine a finite number of valid (based on kinematic and collision analysis) camera or end effector positions. The second step may include performing a simulation of the visual information from the positions, which can be accomplished via hidden point removal, ray casting, realistic rendering, etc. The second step can generate estimates of the visual data from the proposed camera or end effector positions. The visual data from each proposed position is then scored.

[0207] Given visual data, a scalar value is generated that represents the importance of that data relative to what has been seen before. The controller 308 can then calculate a difference score a: difference(a,p) between the proposed coverage p and what has actually been seen so far. The controller 308 can then perform a scoring operation on the point cloud difference to generate an information gain score. This information gain score reflects how much new information the controller 308 expects to gain from visiting this new location. The gain score is used to finalize the sensor position or location.

[0208] Referring to step 550 of method 500, the objects may be in physical contact with one another and the robotic system 300 may perform the weld based on the weld plan. Before the weld is performed, the welding robot may be used to move the welding tool to the position where the weld is to be performed. Because the TCP of the welding tool is known and the annotation information of the CAD model may be used to determine the weld plan (e.g., tack location and its coordinates), the TCP of the welding tool may be moved to the weld coordinates to perform the weld.

[0209] In some implementations, the CAD model may include annotations of the seams. For example, the CAD model may annotate the seams. In other implementations, the seams may be determined using one or more images captured by a sensor in the workspace 302. For example, the controller 308 may identify the seams using a 3D representation. In some examples, the matching techniques described above may be useful to compare and match seams determined using the sensor with seams annotated on the CAD model.

[0210] In some implementations, after identifying the seam, the controller 308 may automatically generate a welding path for the seam. In determining the welding path, the controller 308 may generate multiple points indicative of the seam. In some implementations, the path planning involves determining different states of the welding robot for each point along a given seam. The states of the robot may include, for example, the position of the robot in the workspace 302 and a specific configuration of the robot's arm in any number of degrees of freedom that may be applied. For example, for a robot having an arm with six degrees of freedom, the state of the robot includes not only the position of the robot in the workspace 302 (e.g., the position of the robot's welding head in a three-dimensional xyz space) but also specific sub-states for each of the six degrees of freedom of the robot arm. Furthermore, when the robot transitions from a first state to a second state, the position in the workspace 302 may change, and in such a case, the robot will necessarily traverse a specific path in the workspace 302 (e.g., along the seam being welded). Thus, identifying a set of states of the robot entails identifying the path the robot will traverse in the workspace 302.

[0211] In some implementations, at each point of the seam, the controller 308 can determine whether the robot can be in a feasible state at the given point, meaning, at least in part, that the controller 308 can determine whether a collision occurs between the robot and a structure in the workspace 302 or between a part of the robot itself by implementing a set of states along the sequence of seam points of the seam. To this end, the concept of realizing different states at different points of the seam may alternatively be expressed in the context of a seam having multiple waypoints. First, the controller 308 may discretize the identified seam into a sequence of waypoints. The waypoints may constrain the orientation of a welding head connected to the robot in three (spatial or translational) degrees of freedom. Typically, for the purpose of performing a certain quality weld, constraints on the orientation of the welding head of the robot are provided with one or two rotational degrees of freedom for each waypoint. The constraints are typically relative to the surface normal vector emanating from the waypoint and the path of the weld seam. For example, the position of the weld head may be constrained about one or two rotational axes perpendicular to the axis of the welding wire or the tip of the welder, and about the x-axis, y-axis, and z-axis. All of these are relative to the waypoints and some nominal coordinate system associated with them. These constraints are, in some examples, angle bounds or tolerances. One skilled in the art will recognize that the ideal or desired weld angle may vary based on the part or seam geometry, the direction of gravity relative to the seam, and other factors. In some examples, the controller 308 may limit the 1F or 2F weld position so that the seam is perpendicular to gravity for one or more reasons (e.g., to balance welding and path planning for optimization purposes). Thus, the position of the weld head may be held (constrained) by each waypoint in any suitable orientation relative to the seam. Typically, the weld head is not constrained for the rotational axis (9) that is coaxial with the axis of the weld head.For example, each waypoint may define a position of a welding head of a tool 310 of a robot 301 (e.g., a welding robot), where at each waypoint the welding head is at a fixed position and orientation relative to the weld seam. In some implementations, the waypoints are discretized fine enough such that the movement of the welding head is substantially continuous.

[0212] The controller 308 may divide each waypoint into multiple nodes. Each node may represent a possible orientation of the weld head at that waypoint. As a non-limiting example, the weld head may be unconstrained with respect to an axis of rotation coaxial with the axis of the weld head, such that the weld head may rotate (e.g., 360 degrees) along the axis of rotation (9) at each waypoint. Each waypoint may be divided into 20 nodes, with each node of each waypoint representing the weld head at an 18 degree rotation increment. For example, the first waypoint node pair represents the rotation of the weld head at 0 degrees, the second waypoint node pair represents the rotation of the weld head from 18 degrees, the third waypoint node pair represents the rotation of the weld head at 36 degrees, and so on. Each waypoint may be divided into 2, 19, 29, 69, 129, 369, or any suitable number of nodes. Subdivision of the nodes may represent division of orientation in one or more degrees of freedom. For example, the orientation of the welder tip relative to a waypoint can be defined by three angles. Each waypoint-node pair can be linked to define a welding path. Thus, the distance between waypoints and the offset between adjacent waypoint nodes can represent the amount of translation and rotation of the welding head as it moves between the node-waypoint pairs.

[0213] The controller 308 can evaluate the feasibility of welding for each waypoint-node pair. Consider a non-limiting example of dividing the waypoints into 20 nodes. The controller 308 can evaluate whether the first waypoint-node pair, representing the welding head held at 0 degrees, is feasible. In other words, the controller 308 can evaluate whether the robot 301 will collide or interfere with the part, the fixed tool, or the welding robot itself if placed in the position and orientation defined by that waypoint-node pair. Similarly, the controller 308 can evaluate whether the second waypoint-node pair, the third waypoint-node pair, and so on, are feasible. The controller 308 can similarly evaluate each waypoint. In this manner, all feasible nodes for all waypoints can be determined.

[0214] In some implementations, a collision analysis as described herein may be performed to compare the 3D model of the workspace 302 with the 3D model of the robot 301 to determine whether the two models overlap, and optionally, whether some or all of the triangles overlap. If the two models overlap, the controller 308 may determine that a collision is likely. If the two models do not overlap, the controller 308 may determine that a collision is unlikely. More specifically, in some examples, the controller 308 may compare the models of each of a set of waypoint node pairs (such as the waypoint node pairs described above) and determine that the two models overlap for a subset of the waypoint node pairs, or possibly all of the waypoint node pairs. For the subset of waypoint nodes for which the models are identified to intersect, the controller 308 may omit the waypoint node pairs in that subset from the planned path and identify alternatives for those waypoint node pairs. The controller 308 may repeat this process as necessary until a collision-free path is planned. The controller 308 may use a flexible collision library (FCL) as a tool for collision avoidance analysis, which includes various techniques for efficient collision detection and proximity computation. The FCL is useful for performing multiple proximity queries on different model representations and may be used to perform probabilistic collision identification between point clouds. Additional or alternative resources may be used in conjunction with or in place of the FCL.

[0215] The controller 308 may generate one or more feasible welding paths, if physically feasible. A welding path is a path that the welding robot follows to weld a candidate seam. In some examples, the welding path may include all waypoints of the seam. In some examples, the welding path may include some, but not all, waypoints of the candidate seam. The welding path may include the movement of the robot and the welding head as the welding head moves between each pair of waypoint nodes. Once a feasible path is identified between a pair of node waypoints, a feasible pair of node waypoints, if any, is identified for the next successive waypoint. Those skilled in the art will recognize that many search trees or other strategies may be employed to evaluate the space of feasible node waypoint pairs. As described in more detail herein, a cost parameter may be assigned or calculated for the movement from each pair of node waypoints to the next pair of node waypoints. The cost parameter may be related to the time of movement, the amount of movement (e.g., including rotation) between the pair of node waypoints, or a simulation or prediction of the quality of the weld produced by the welding head during the movement.

[0216] Based on the generated weld path, the controller 308 can optimize the weld path for welding. As used herein, the terms optimal and optimization do not mean determining the absolute best weld path, but generally mean techniques that reduce weld time or improve weld quality compared to less efficient weld paths. For example, the controller 308 can determine a cost function for the robot 301 motion that seeks a local or global minimum. Typically, an optimal weld path is one that minimizes weld head rotations, which may increase the weld time of a seam or reduce the quality of the weld. Thus, optimizing the weld path can include determining a weld path that passes through a maximum number of waypoints with a minimum of rotations.

[0217] Although shown and described as using a single welding torch, the number of welding robots may vary. For example, in some implementations, multiple robots having welding tools (attached thereto) may be used to perform a welding task. In such a situation, the multiple welding tools may be brought close to the object when they are in fixed relative positions. The controller 308 may use the techniques described with reference to step 550 of method 500 to position each of the multiple welding tools in a desired position. In such a situation, the controller 308 may employ a different welding plan for each of the multiple welding tools. For example, one welding robot may perform tack welding while the other welding robot performs a different type of welding (e.g., full seam welding). In some implementations, the multiple welding robots may perform the same type of welding but with an emphasis on welding different locations on the object. It should be noted that the robotic systems described herein may include coordination and / or cooperation between different robots, such as a first robot performing a first action / function (e.g., picking up a first object) while a second robot performs a second action / function (e.g., a welding action) and a third robot holds the second object.

[0218] Once the welding process is complete, either of the picking robots can transport and place the assembled object onto positioning device 355, 365, or another positioning device, etc.

[0219] As mentioned above, although FIGS. 3 and 4 show four robot systems, the number of robots may be different since the assembly tasks may be performed by a single robot system. For example, the robot systems 100, 200, and 300 may include one robot coupled with a picking tool. In such an implementation, a first target object may be positioned on a positioning device and a second object may be positioned on the same positioning device or on a different positioning device. The robot may be configured to locate, identify, and pick up the first object using the techniques described above. The robot may then manipulate the pose of the first object to move it closer to the second object using the techniques described above. In such an implementation, the second object may be secured to the positioning device (e.g., using a clamp) and the positioning device may be configured to change the pose of the second object. In some implementations, the positioning device may change the pose of the second object utilizing some of the principles described above. For example, when a picking robot moves a picked up object closer to a second object, the positioning device (which may be actuated to move or rotate) may change the pose of the second object in coordination with the picked up object to bring the first and second objects closer to a fixed pose relative to each other. In such an implementation, the welding tool is stationary (e.g., not moving) and the robot and positioning device may coordinate to bring the object closer to the welding tool while keeping it in a fixed pose. As the robot and positioning device change the pose of the object while keeping it in a fixed pose relative to each other, the welding tool may perform welding according to the welding plan.

[0220] In some implementations, an assembly task may be performed by two robotic systems. For example, each of the robotic systems 100, 200, 300 may include two robots coupled with picking tools. In such implementations, a first object may be placed on a positioning device and a second object may be placed on the same or a different positioning device. Both robots may be configured to find, identify, and pick up their respective objects (e.g., one robot picks up the first object and the second robot picks up the second object). In picking up the objects, both robots may use the techniques described above. The robots may then cooperatively manipulate the pose of the picked-up object to a fixed relative pose. In such implementations, the welding tool may remain stationary (e.g., not moving) and the robot may bring the object closer to the welding tool to facilitate welding of the object. For example, the robot changes the pose of the object while keeping the relative pose of the object fixed, and the welding tool performs the welding according to the welding plan.

[0221] In some implementations, the assembly task may be performed by three robotic systems. For example, the robotic system 300 may include two robots, each coupled with a picking tool. In such an implementation, a first object may be placed on a positioning device, and a second object may be placed on the same or a different positioning device. Both robots may be configured to locate, identify, and pick up their respective objects (e.g., the first robot picks up the first object, and the second robot picks up the second object). In picking up the objects, both robots may use the techniques described above. The robots may then cooperatively manipulate the pose of the picked-up object to a fixed relative pose. In a three-robot system, a welding tool may be coupled to the third robot (similar to the embodiment described in Figs. 3 and 4). Similar to the description of Figs. 3 and 4, the welding tool may be brought close to the picked-up object, and the welding tool may perform a weld according to a welding plan.

[0222] A picking robot picks an object and manipulates its pose. A welding robot manipulates the pose of a welding tool. A sensing robot manipulates the pose of one or more sensors. These robots essentially manipulate the pose of a body, such as an object or a sensor. Thus, in some implementations, these robots may be referred to as pose-manipulated robots or pose-manipulated tools. For example, robot 301 may be referred to as a pose-manipulated tool coupled to a welding tool. Similarly, robot 311 may be referred to as a pose-manipulated tool coupled to a picking tool, whereby the pose-manipulated tool is configured to pick an object and change its pose. The same pose-manipulation concepts apply to robots 321 and 331.

[0223] FIG. 15 is a flow diagram illustrating an example process (e.g., method 1500) of operating a robotic assembly system. The method 1500 disclosed in FIG. 15 may be executed by a robotic system such as robotic systems 100, 200 (e.g., control device 150), robotic system 300 (e.g., controller 308), or robotic system 1600 (e.g., control device 1650). In some implementations, the robotic system includes a first tool coupled to a first robotic device, a second tool coupled to a second robotic device, one or more sensors, a welding tool, and a controller. For example, the sensor device (e.g., 130) may be coupled to or installed on a welding robot, a picking robot, or a perception robot. Additionally or alternatively, the sensor device may be moved or changed on the robot using a tool changer or other robot. The first tool may include or correspond to a first tool 120 coupled to a first robotic device 110, or a tool 320 coupled to a robot 311. The second tool may include or correspond to the second tool 222 coupled to the second robotic device 212, or the tool 340 coupled to the robot 331. The one or more sensors may include or correspond to one or more of the sensor 130, the first sensor 234, the second sensor 236, the sensor 330, or the sensors 351, 352, 361, 362. The welding tool may include or correspond to the manufacturing tool 126 or the tool 310. The controller may include or correspond to the control device 150, the controller 152, the welding controller 307, the controller 308, or the database 312.

[0224] Method 1500 may include a controller receiving first sensor data from one or more sensors at step 1502. For example, the first sensor data may include or correspond to sensor data 180. In some implementations, the first sensor data includes image data.

[0225] The method 1500 includes a controller controlling the first tool to couple to the first object at a first location on the first object based on the first sensor data at 1504. The first object may include or correspond to the first object 140 or object 322. To control the first tool, the controller may generate and transmit control information 182.

[0226] The method 1500 includes a controller controlling, at step 1506, based on the first sensor data, to couple a second tool to a second object at a second location on the second object. The second object may include or correspond to the second object 142 or the object 332. To control the second tool, the controller may generate and transmit control information 182.

[0227] Method 1500 includes a controller controlling the first robotic device, the second robotic device, or both, to direct the first object and the second object into a spatial relationship such that a seam is formed between the first object and the second object, at step 1508. The seam may include or correspond to seam 144.

[0228] Method 1500 includes a controller generating manufacturing instructions at step 1510. For example, the manufacturing instructions may include welding instructions. In some implementations, the controller may transmit or execute the manufacturing instructions. For example, if the manufacturing instructions include welding instructions, the welding instructions may be executed to cause a welding tool to perform a weld to join the first object and the second object. To weld the seam, the welding tool may lay down a weld metal along the seam to join the first object and the second object. In some implementations, the controller may transmit the manufacturing instructions to a welding controller, such as welding controller 307, to be executed.

[0229] In some implementations, the method 1500 may include a controller receiving model data associated with a computer-aided design (CAD) model of a representation of a structure including a first object and a second object. The model data may include or correspond to the model information 164, the manufacturing information 176, or the threshold value 177. The controller may identify the first object based on the first sensor data, the model data, or a combination thereof. Additionally or alternatively, the controller may perform a positioning process to determine a first pose of the first object based on the model data, the first sensor data, or a combination thereof. In some implementations, the controller may generate a first 3D representation of the first object based on the first sensor data, and perform a positioning process based on the first 3D representation to generate a second 3D representation of the first object. To perform the positioning process, the controller may execute the fitting logic 173.

[0230] In some implementations, the controller may determine a first position on the first object based on a first pose of the first object. Additionally or alternatively, the controller may determine the first position based on the first 3D representation, the second 3D representation, the model data, or a combination thereof. In some implementations, the controller may perform a reachability analysis, a collision analysis, or a combination thereof based on the first position. To perform the reachability analysis or the collision analysis, the controller may execute kinematic reachability and collision logic 174.

[0231] In some implementations, after the first tool is coupled to the first object, the controller receives second sensor data from one or more sensors related to the first object coupled to the first tool. The controller may perform a positioning process to determine a second pose of the first object based on the model data and the second sensor data. To perform the positioning process, the controller may execute adaptation logic 173.

[0232] In some implementations, after the first object and the second object are in a spatial relationship, the controller receives third sensor data related to the first object and the second object from one or more sensors. The controller can determine whether a separation between the first object and the second object is less than or equal to a threshold value based on the model data and the third sensor data. For example, the threshold value includes or corresponds to threshold value 177.

[0233] The above-described embodiments generally relate to robots configured to autonomously perform assembly tasks. Semi-autonomous or autonomous welding robots typically include, but are not limited to, sensors for scanning specific objects, software-based algorithms for automatically identifying and locating objects to be welded, machine-readable code-based algorithms for controlling the robot's movements to pick up the objects, and machine-readable code-based algorithms for controlling the robot's movements to perform the weld. The machine-readable code-based algorithms may also be configured to control other devices, such as motorized positioners, to reposition the object to perform the weld. Additionally or alternatively, semi-autonomous or autonomous welding robots may include, but are not limited to, sensors for scanning specific objects, software-based algorithms for automatically recognizing, identifying, and registering a predefined model of the object if the object has already been designated (annotated) in some way (e.g., annotated with the location of a tack weld), and software-based algorithms for automatically programming the robot's movements, the picking robot, the welding robot, and other devices (such as motorized positioners) to weld the seam accurately or as desired.

[0234] A semi-autonomous or autonomous robot may only have some of the above capabilities, and may require user-specified or selected parameters, or may require the user to be involved in other ways, such as placing objects on a positioning device or adding annotations to a model (e.g., a computer-aided design (CAD) model).

[0235] While various implementations have been described above, it should be understood that they are presented by way of example only and not limitation. While various implementations have been described as having certain combinations of features and / or components, other implementations are possible having combinations of any of the features and / or components from any of the examples and / or additional features and / or components, where appropriate. For example, although not described in detail above, in some implementations, the method of determining the shape of a portion of a part may include a calibration step that determines distortions in the detector, the lens on the detector, the combination of the detector and lens, and / or the relative position of the camera with respect to the test surface or fixture onto which the pattern is projected.

[0236] Certain features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Where the methods described above indicate the occurrence of certain events in a particular order, the order of certain events may be changed. Additionally, certain events may be performed simultaneously in parallel processes where possible, and may also be performed in the order described above.

[0237] Similarly, although the figures depict operations in a particular order, this should not be construed as requiring the operations to be performed in the order depicted or in a sequential order, or as requiring all of the depicted operations to be performed, in order to achieve desirable results. Additionally, the figures may depict one or more exemplary processes in the form of a flow diagram. However, other operations not depicted may be incorporated into the exemplary process depicted in the schematic. For example, one or more additional operations may be performed before, after, simultaneously with, or between any of the depicted operations. In certain circumstances, multitasking and parallel processing may be advantageous. Additionally, the separation of various system components in the above-described embodiments should not be construed as requiring a similar separation in all embodiments, and the program components and systems described above may generally be integrated into a single software product or packaged into multiple software products. Additionally, certain other embodiments are within the scope of the following claims. In some cases, the operations recited in the claims may be performed in a different order to achieve desirable results.

[0238] Those skilled in the art will appreciate that information, messages, and signals may be represented using any of a variety of different technologies and techniques. For example, the data, instructions, commands, information, and signals that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0239] The components, functional blocks, and modules illustrated and described herein may include processors, electronic devices, hardware devices, electronic components, logic circuits, memories, software code, firmware code, and the like, or any combination thereof. Software may be broadly construed to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, or functions, and may also be referred to as software, firmware, middleware, microcode, hardware description languages, and other names. Additionally, the functionality described herein may be implemented by dedicated processor circuitry, executable instructions, or combinations thereof.

[0240] Those skilled in the art will further appreciate that the various exemplary logic blocks, modules, circuits, and algorithmic steps described in connection with the present disclosure can be implemented as a combination of electronic hardware, computer software, or both. To clearly illustrate this interchangeability of hardware and software, the above description outlines various exemplary components, blocks, modules, circuits, and steps based on their functionality. Various exemplary logic, logic blocks, modules, circuits, and algorithmic processes described in connection with the implementations disclosed herein can be implemented as a combination of electronic hardware, computer software, or both. In one or more aspects, the described functionality may be implemented in hardware, digital electronic circuitry, computer software, firmware, or any combination thereof, including the structures disclosed herein and structural equivalents thereof. Implementations of the subject matter described herein may also be implemented as one or more computer programs, i.e., modules of one or more computer program instructions encoded in a computer storage medium for controlling or performing the operation of a data processing apparatus.

[0241] The hardware and data processing devices used to implement the various example logic, logic blocks, modules, and circuits described in connection with the aspects disclosed herein may be implemented or performed using general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, or any conventional processor, controller, microcontroller, or state machine. In some implementations, a processor may be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or other similar configurations. In some implementations, particular processes and methods may be performed by circuitry that is specific to a given function.

[0242] If implemented in software, the functions may be stored or transmitted as one or more instructions or codes on a computer-readable medium. The processes of a method or algorithm disclosed herein may be implemented in a processor-executable software module residing on a computer-readable medium. Computer-readable media includes both computer storage media and communication media, including any medium that can enable a computer program to be transferred from one place to another. A storage medium may be any medium accessible by a computer. By way of example, and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection may be properly referred to as a computer-readable medium. Additionally, the operations of a method or algorithm may be present in one or any combination or set of code and instructions on a machine-readable medium and a computer-readable medium, which may be incorporated into a computer program product.

[0243] Some of the implementations described herein relate to methods or processing events. It should be understood that such methods or processing events can be computer-implemented. That is, when a method or other event is described herein, it should be understood that it can be performed by a computing device having a processor and memory. The methods described herein can be performed locally, for example, on a computing device physically co-located with the robot or on a local computer / controller associated with the robot. They can also be performed remotely, such as on a server or on the "cloud."

[0244] The memory of a computing device may also be referred to as a non-transitory computer readable medium and may contain instructions or computer code for performing various computer-implemented operations. A computer readable medium (or processor readable medium) is non-transitory in the sense that it does not itself contain a transitory propagating signal (e.g., electromagnetic waves that propagate information on a transmission medium such as space or cable). The medium and computer code (also referred to as code) may be designed and constructed for a specific purpose. Examples of non-transitory computer readable media include, but are not limited to, magnetic storage media such as hard disks, floppy disks, magnetic tapes, compact disks / digital video disks (CDs / DVDs), compact disk read-only memories (CD-ROMS), and holographic devices; magneto-optical storage media (such as optical disks); carrier wave signal processing modules, read-only memories (ROMs), random access memories (RAMs), and the like. One or more processors are communicatively coupled to the memory and capable of executing the code stored on the non-transitory processor readable medium. Examples of processors include general purpose processors (e.g., CPUs), graphic processing units, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), etc. Examples of computer code include, but are not limited to, microcode or microinstructions, machine instructions such as those produced by a compiler, code used to generate web services, and files containing higher level instructions that are executed by a computer using an interpreter. For example, it may be implemented using an imperative programming language (e.g., C, Fortran), a functional programming language (e.g., Haskell, Erlang), a logic programming language (e.g., Prolog), an object-oriented programming language (e.g., Java, C++), or other suitable programming language and / or development tools.Examples of computer code include, but are not limited to, control signals, encryption code, compression code, and the like.

[0245] As used herein, various terms are intended to describe particular implementations only and are not intended to limit the implementation. For example, as used herein, ordinal numbers (e.g., "first," "second," "third," etc.) used to modify elements such as structures, components, operations, etc., do not in themselves indicate a priority or order of the element in relation to other elements, but merely distinguish it from other elements having the same name (except for the use of the ordinal number). The term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. Two items that are "coupled" may be integral with one another. The terms "a" and "an" are defined as one or more, unless this disclosure expressly requires otherwise.

[0246] The term "about" as used herein allows for a degree of variability in a value or range, including the exact stated value or range, such as within 10%, within 5%, or within 1% of a stated value or limit of a stated range. The term "substantially" is defined as approximately what is specified (including what is specified, e.g., substantially 90 degrees includes 90 degrees and substantially parallel includes parallel), but not necessarily exactly what is specified, as understood by those skilled in the art. In any disclosed implementation, the term "substantially" may be replaced with the phrase "within [a percent] of what is specified," where percent includes 0.1, 1, or 5 percent. Also, the term "approximately" may be replaced with the phrase "within 10 percent of what is specified." The phrase "substantially X to Y" has the same meaning as "substantially X to substantially Y" unless otherwise specified. Similarly, the phrase "substantially X, Y, or substantially Z" has the same meaning as "substantially X, substantially Y, or substantially Z" unless otherwise specified. Unless otherwise stated, the word "or" as used herein is an inclusive "or" and is synonymous with "and / or," and when "or" is used with a list of two or more items, it means that any one of the listed items can be used alone, or any combination of two or more of the listed items can be used. For example, "A, B, or C" includes A only, B only, C only, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C. Similarly, "A, B, C, or any combination thereof" or "A, B, C, or any combination thereof" includes A only, B only, C only, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B, and C.

[0247] Throughout this document, numerical values ​​expressed in range format should be interpreted flexibly as including not only the numerical values ​​expressly recited as the limits of that range, but also all individual numerical values ​​or subranges subsumed within that range. For example, a range such as "about 0.1% to about 5%" or "about 0.1% to 5%" should be interpreted to include not only "about 0.1% to about 5%" but also the individual numerical values ​​(e.g., 1%, 2%, 3%, 4%) and subranges within that range (e.g., 0.1% to 0.5%, 1.1% to 2.2%, 3.3% to 4.4%).

[0248] The terms "comprise" (and all forms thereof, such as "comprise" and "comprising"), "have" (and all forms thereof, such as "has" and "having"), "include" (and all forms thereof, such as "includes" and "including"), and "contain" (and all forms thereof, such as "contain" and "containing"). As a result, an apparatus before which "comprises", "has", "includes", or "contains" is used may possess one or more elements, but not necessarily exclusively those elements. Similarly, a method before which "comprises", "has", "includes", or "contains" is used may possess one or more steps, but not necessarily exclusively those steps.

[0249] Any system, method, or article of manufacture implementation may consist of, or consist essentially of, any of the recited steps, elements, or functions. Thus, in any claim, the term "consisting of" or "consisting essentially of" may be replaced with any of the open-ended linking verbs listed above, thereby modifying the scope of the claim differently than would be the case if an open-ended linking verb were used. Additionally, the term "wherein" may be used interchangeably with the term "where."

[0250] Furthermore, a device or system configured in a certain manner may be configured in at least that manner, but in other manners than specifically described. Unless expressly prohibited by the nature of this disclosure or the implementation, features or characteristics of one implementation may be applied to other implementations, even if not described or illustrated.

[0251] The claims are not intended to, and should not be construed to, include any "means plus" or "step plus function" limitations unless such limitations are expressly recited in the applicable claim using the phrase "means for" or "step for."

[0252] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications of the disclosure will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure and the following claims are not intended to be limited to the embodiments and designs described herein, but should be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. 1. An assembly robot system, comprising: A first robot device disposed in a working space; a first tool coupled to the first robotic device and configured to selectively couple to a first object; one or more sensors configured to generate sensor data associated with the workspace; a welding tool configured to join two or more objects; A controller; Equipped with The controller: Controlling the first tool to be coupled to the first object based on the sensor data; controlling the first robotic device to spatially relate the first object to the second object such that a seam is formed between the first object and the second object; generating welding instructions to cause the welding tool to weld the seam to join the first object and the second object; Assembly robot system.

2. the first robotic device includes a first robotic arm; the sensor data generated by at least one of the one or more sensors includes image data; The controller: further configured to identify a first location on the first object based on the image data; Controlling the first tool to couple to the first object at the first location on the first object; controlling the first tool, the first robotic arm, or a combination thereof to adjust a pose of the first object; The assembly robot system according to claim 1 .

3. A second robot device disposed within the working space; a second tool coupled to the second robotic device and configured to selectively couple to the second object; Further equipped with The controller: Controlling the second tool to be coupled to the second object based on the sensor data; controlling the second robotic device to orient the second object in a spatial relationship with the first object such that the seam is formed between the first object and the second object; The assembly robot system according to claim 1 .

4. the controller controls the first robotic device, the first tool, the second robotic device, the second tool, or a combination thereof to change a pose of the first object and the second object while the first object and the second object are in the spatial relationship. The assembly robot system according to claim 3 .

5. a third robotic device disposed within the workspace; a first sensor of the one or more sensors is coupled to the third robotic device; the sensor data generated by the first sensor includes first image data; the controller controls the first tool to be coupled to the first object based on the first image data; The assembly robot system according to claim 1 .

6. The controller: controlling the third robotic device to move the first sensor closer to the first object after the first tool is coupled to the first object; receiving additional sensor data from the first sensor; the additional sensor data includes additional image data associated with the first object coupled to the first tool. The assembly robot system according to claim 5 .

7. a fourth robotic device disposed in the workspace; the welding tool is coupled to the fourth robotic device; the controller controls the fourth robotic device to position the welding tool adjacent to the first object and the second object; the welding tool places a weld metal along the seam to join the first object and the second object based on the welding instruction. The assembly robot system according to claim 1 .

8. the controller receiving data associated with a computer-aided design (CAD) model of a representation of a structure including the first object and the second object; the CAD model includes a spatial relationship between the first object and the second object, an annotation associated with the seam, or a combination thereof; The assembly robot system according to claim 7 .

9. The controller: determining a first location of the seam based on user input, annotations included in the CAD model, or a combination thereof; controlling the fourth robotic device to position the welding tool relative to a first location of the seam. The assembly robot system according to claim 8 .

10. the one or more sensors include a second sensor; the sensor data generated by the second sensor includes second image data; the controller controls the first robotic device based on the second image data after the first tool is coupled to the first object. The assembly robot system according to claim 1 .

11. a positioning device configured to host the first object; the positioning device adjusts an attitude or position of the first object while the first object is hosted by the positioning device. The assembly robot system according to claim 1 .

12. the controller controls the first tool, the first robotic device, or a combination thereof to adjust the attitude of the first object. The assembly robot system according to claim 1 .

13. 1. An assembly robot system, comprising: A first robot device disposed in a working space; a first tool coupled to the first robotic device and configured to selectively couple to a first object; A second robot device disposed in the working space; a second tool coupled to the second robotic device and configured to selectively couple to a second object, the first object and the second object configured to form a first seam between the first object and the second object; and A third robot device disposed in the working space; one or more sensors configured to generate sensor data related to the workspace, at least a first sensor of the one or more sensors being coupled to the third robotic device; A fourth robot device disposed in the working space; a welding tool coupled to the fourth robotic device and configured to weld the first seam formed between the first object and the second object; A controller; Equipped with The controller: controlling the first tool to be coupled to the first object at a first position of the first object based on the sensor data; controlling the second tool to be coupled to the second object at a second position of the second object based on the sensor data; controlling the first robotic device, the second robotic device, or both, to bring the first object and the second object into a first spatial relationship such that the first seam is formed between the first object and the second object; generating welding instructions to cause the welding tool to weld the first seam to join the first object and the second object; Assembly robot system.

14. the controller is coupled to the first robotic device, the second robotic device, the third robotic device, and the fourth robotic device; The controller includes a memory for storing instructions; and a processor coupled to the memory and configured to execute the instructions to perform one or more operations. The assembly robot system according to claim 13.

15. the workspace includes a three-dimensional space; the first robotic device includes a first robotic arm and the first tool coupled to an end of the first robotic arm; the second robotic device includes a second robotic arm and the second tool coupled to an end of the second robotic arm; the third robotic device includes a third robotic arm and the first sensor coupled to an end of the third robotic arm and configured to capture one or more images of the first object, the second object, or a combination thereof; the fourth robotic device including a fourth robot arm and the welding tool coupled to an end of the fourth robot arm and configured to weld the first seam according to a welding plan in response to the welding command; The assembly robot system according to claim 13.

16. the one or more sensors include a second sensor configured to capture a set of one or more images of the first object, the second object, or both after the first object is coupled to the first tool, after the second object is coupled to the second tool, or a combination thereof; the sensor data generated by the second sensor includes second image data; the controller controls the first robotic device based on the second image data after the first tool is coupled to the first object. The assembly robot system according to claim 15.

17. the first tool is configured to be selectively coupled to a third object; After the welding tool has laid down a weld metal to join the first object and the second object, the controller: controlling the first tool to be detached from the first object and attached to the third object at a third location on the third object; controlling the first robotic device, the second robotic device, or both, to bring the third object into a second spatial relationship with the joined first and second objects such that a second seam is formed between the third object and the first or second object; generating welding instructions to cause the welding tool to weld the second seam to join the third object to the first object and the second object; The assembly robot system according to claim 15.

18. 1. A method of operating an assembly robotic system including a first tool coupled to a first robotic device, a second tool coupled to a second robotic device, one or more sensors, a welding tool, and a controller, comprising: receiving, by the controller of the assembly robotic system, first sensor data from the one or more sensors; controlling, by the controller, the first tool to be coupled to the first object at a first position on the first object based on the first sensor data; controlling, by the controller, the second tool to be coupled to the second object at a second position on the second object based on the first sensor data; controlling, by the controller, the first robotic device, the second robotic device, or both, to orient the first object and the second object in a spatial relationship such that a seam is formed between the first object and the second object; generating welding instructions by the controller; welding the seam with the welding tool in response to the welding instruction to join the first object and the second object; The method according to claim 1,

19. receiving, by the controller, model data associated with a computer-aided design (CAD) model of a representation of a structure including the first object and the second object; identifying, by the controller, the first object based on the first sensor data, the first sensor data including image data; executing, by the controller, a positioning process to determine a first pose of the first object based on the model data and the first sensor data; determining, by the controller, the first position on the first object based on the first orientation of the first object; performing, by the controller, a reachability analysis, a collision analysis, or a combination thereof based on the first location; 20. The method of claim 18, comprising:

20. receiving, by the controller after the first tool is coupled to the first object, second sensor data associated with the first object coupled to the first tool from the one or more sensors; executing, by the controller, the positioning process based on the model data and the second sensor data to determine a second pose of the first object; receiving, by the controller, third sensor data associated with the first object and the second object from the one or more sensors after the first object and the second object are brought into the spatial relationship; determining, by the controller, whether a distance between the first object and the second object is less than or equal to a threshold based on the model data and the third sensor data; 20. The method of claim 19, comprising:

Images