Multi-modal data collection system for artificial intelligence training and enhancement and control of robotic welding systems

Abstract

A system for system for controlling a robotic welding system having a robotic welding torch manipulator and a welding power source, comprising a motion control interface configured for connection to one of a plurality of brands of robot controller of the robotic welding torch manipulator, a weld control interface configured for connection to one of a plurality of brands of welding power source, an image acquisition interface configured to receive image data from a camera positioned to capture real-time images of a welding operation, a user interface comprising a display for displaying images of the welding operation and at least one input device for receiving user commands specifying adjustments to the welding operation, and control circuitry comprising one or more processors configured to convert the user commands into configuration specifications for the robotic welding system, provide motion control adjustment instructions to the motion control interface utilizing a protocol for a selected one of the plurality of brands of robot controller, and provide weld control adjustment instructions to the weld control interface utilizing a protocol for a selected one of the plurality of brands of welding power source.

Description

MULTI-MODAL DATA COLLECTION SYSTEM FOR ARTIFICIAL INTELLIGENCE TRAINING AND ENHANCEMENT AND CONTROL OF ROBOTIC WELDING SYSTEMSCross-Reference to Related Application

[0001] This patent application claims priority to United States provisional patent application No. 63 / 590,561 filed October 16, 2023, the entire contents of which is incorporated by reference herein.Technical Field

[0002] The present disclosure relates to controlling autonomous welding robots and cobots. Particular embodiments provide systems and methods for versatile data collection and artificial intelligence (Al) development for enabling autonomous welding supervision and control across multiple modalities and sensors including vision, motion, and process variables.Background

[0003] Existing industrial robots, such as welding robots and other types of robots, often allow users to specify, for example, a series of teach points for the robotic manipulator to move through. However, current systems often require brand specific programming of the robot for more sophisticated motion control, and may not provide any means to modify the path or operation parameters of robotic welding system between teach points.

[0004] Autonomous welding robots generally require Al-driven cognition and perception. This autonomy includes capturing the welder's responses to welding scenes, assessing their dexterity in motion, and adapting to various welding parameters based on different welding conditions and scenarios. Existing systems often require pretraining and offline development within a mock-up production environment. Therefore, the development of Al solutions faces significant delays before reaching full production readiness, along with the requirement for substantial technical refinement before achieving complete autonomy.

[0005] The inventors have determined a need for improved systems, methods and related apparatus for multi-modal data collection, training and supervision of autonomous welding robots in production environments.Summary

[0006] One aspect of the present disclosure provides a system for controlling a robotic welding system having a robotic welding torch manipulator and a welding power source. The system comprises a motion control interface configured for connection to one of a plurality of brands of robot controller of the robotic welding torch manipulator, a weld control interface configured for connection to one of a plurality of brands of welding power source, an image acquisition interface configured to receive image data from a camera positioned to capture real-time images of a welding operation, a user interface comprising a display for displaying images of the welding operation and at least one input device for receiving user commands specifying adjustments to the welding operation, and control circuitry comprising one or more processors configured to convert the user commands into configuration specifications for the robotic welding system, provide motion control adjustment instructions to the motion control interface utilizing a protocol for a selected one of the plurality of brands of robot controller, and provide weld control adjustment instructions to the weld control interface utilizing a protocol for a selected one of the plurality of brands of welding power source. In some embodiments the motion control interface is directly connected to the robot controller and the weld control interface is directly connected to the welding power source. In some embodiments both the motion control interface and the weld control interface are connected to the robot controller, and the weld control adjustment instructions are relayed to the welding power source by the robot controller. In some embodiments, the control circuitry is configured to record operator inputs and capture related images, robot movement parameters, and other welding parameters and data related to the operator inputs for use in training of an Al model.

[0007] Another aspect of the present disclosure provides s method for controlling an autonomous robotic welding system having a robotic welding torch manipulator and a welding power source. The method comprises: determining a brand of the roboticwelding torch manipulator and a brand of the welding power source; receiving an initial robotic welding program specifying a welding path and welding parameters to complete a welding operation; capturing real-time images of the welding operation; processing the real-time images determine of the welding operation to determine one or more welding key performance indicators (KPIs); overlaying the real-time images with the welding KPIs to generate augmented images and displaying the augmented images on a display of a user interface; receiving one or more operator inputs from an operator using the user interface specifying adjustments the welding path and / or welding parameters; converting the one or more operator inputs into motion control adjustment instructions utilizing a robot control protocol for the determined brand of the robotic welding torch manipulator, and into weld control adjustment instructions utilizing a weld control protocol for the determined brand of the welding power source. In some embodiments, the method further comprises recording timestamped operator inputs and corresponding related images, robot movement parameters, and welding parameters related to the welding operation at the time of each operator input for use in training of an artificial intelligence (Al) welding control model.

[0008] Further aspects of the present disclosure and details of example embodiments are set forth below.Drawings

[0009] The following figures set forth embodiments in which like reference numerals denote like parts. Embodiments are illustrated by way of example and not by way of limitation in the accompanying figures.

[0010] Figure 1 shows an example system for controlling a robotic welding system according to one embodiment of the present disclosure.

[0011] Figure 2 shows an example software architecture for a system for controlling a robotic welding system according to one embodiment of the present disclosure.Detailed Description

[0012] The following describes example control systems, methods and related apparatus that allow operators of robotic welding systems to make real time adjustments to robotic path and welding parameters. As described further below, systems and methods according to certain embodiments of the present disclosure enable concurrent multi-modal data collection and Al development alongside robotic welding production, ensuring a seamless maturation process with minimal disruption to existing production workflows. In some embodiments, the systems, methods and apparatus are also configured to record operator inputs and capture related images, robot movement parameters, and other welding parameters and data related to the operator inputs for use in training of machine learning or other Al systems.

[0013] Systems according to certain embodiments of the present disclosure are compatible with both traditional industrial robots and collaborative robots, or “cobots”, and increase the flexibility and ease of use of robots and cobots by enabling modifications to welding operations beyond just setting teach points. Systems and methods according to the present disclosure are capable of modifying welding operations between teach points by both controlling the robot movement and also the process parameters for a quality weld.

[0014] In some embodiments, all the interactions of the system with the robot are recorded with timestamps so an Al model can be trained mimicking what the operator is doing to make the changes. Systems according to certain embodiments of the present disclosure enable end to end Al control based on inputs from one or more sensory systems, including without limitation cameras / image sensors, temperature sensors, sound sensors, information from encoders associated with the robot’s actuators which indicate the locations of joints, and welding process parameters such as current, voltage, etc.

[0015] In some embodiments, an end-to-end Al model is configured for continuous improvement over time as more data is collected and fed into the system, such that a welding robot can complete a welding operation at a level that is as accurate as, or more accurate than, a skilled human operator. When the accuracy is deemedacceptable, the end-to-end Al model can take over the control such that the welding robot operates completely autonomously, thereby completely freeing up the operator.

[0016] In some embodiments, the end-to-end Al model may be configured for new operator skill training and monitoring purposes without controlling the robot by operating the system in an operator training mode, such that the Al model may be used for evaluating the new operator’s actions based on the trained model. In training mode, an operator being trained is shown previously recorded images of a welding operation and the operator is instructed to make adjustments where they think it would be appropriate, which may be scored utilizing the Al model. In some embodiments, systems according to the present disclosure may be used in the operator training mode for training a new operator and providing a score based on how closely the new operator’s adjustments match those that would have been made ty the Al model.

[0017] In some embodiments, the system is configured to stop training the end- to-end Al model once it reaches acceptable performance thresholds. In other embodiments, the system is configured to continue training the Al model even after acceptable performance thresholds are reached, utilizing online Al learning methods and deploying periodic updates to the model.

[0018] Some embodiments of the present disclosure provide a system comprising one or more cameras or other image sensors, an industrial computer and control module, and an operator control pendant (wired or wireless), with all the protocols of a plurality of leading OEM robotic and welding power source brands to allow for integration with a wide variety of robotic welding systems. Systems according to the present disclosure allow operators to fine tune operation of their existing robotic welding systems for their particular needs, without the need for the manual reprogramming that would otherwise be required in many existing systems.

[0019] Systems and methods according to some embodiments of the present disclosure make working environments in welding shops safer because the operator does not need to get closer to the robot or other elements of the welding system to make fine tune reprogramming as they can now make modifications on the fly if needed from behind a human-machine interface (HMI) screen through a pendant.

[0020] For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements. Numerous details are set forth to provide an understanding of the examples described herein. The examples may be practiced without these details. In other instances, well- known methods, procedures, and components are not described in detail to avoid obscuring the examples described. The description is not to be considered as limited to the scope of the examples described herein.

[0021] Figure 1 shows an example system 100 for controlling an industrial welding robot according to one embodiment of the present disclosure. The system 100 comprises a control unit 110 having a motion control interface configured for connection a robot controller 10 which controls operation of a robotic welding torch manipulator of the industrial welding robot, and a weld control interface configured for connection to a welding power source 20. In the illustrated example, the system 100 is configured for the motion control interface to be directly connected to the robot controller 10, and for the weld control interface to be directly connected to the welding power source 20 by an independent connection. In other embodiments, for example for use with a robotic welding system wherein the robot controller 10 is configured to control the welding power source 20, the system may be configured for both the motion control interface and the weld control interface to be connected to the robot controller 10, and weld control instructions from the weld control interface are relayed to the welding power source 20 via the robot controller 10. As described further below, the system 100 is configured with protocols for a plurality of brands of robot controllers and a plurality of brands of welding power sources, such that system 100 can readily be integrated with a variety of types of robotic welding systems. For example, in some embodiments, the control unit 110 is programmed with protocols for robot arms and welding machines from substantially all major manufacturers thereof, such that the system 100 can be used to provide for autonomous operation, or improve the autonomous operation, of many if not most existing welding shop setups.

[0022] In the illustrated example, the system 100 comprises a user interface in the form of an operator control pendant 102 and a touch-sensitive display 104, and a camera 106 configured to capture real-time images of a welding operation, which areconnected (wired or wirelessly) to the control unit 110. The operator control pendant 102 may, for example, comprise one or more buttons, touch screens, or haptic interfaces that an operator interacts with to provide user commands specifying adjustments to the welding operation. The system 100 converts the user commands into instructions for controlling operations of the robot controller 10 and welding power source 20 of any a variety of types of robotic welding systems.

[0023] The example system 100 shown in Figure 1 comprises a user interface that includes an operator control pendant 102 and a touch-sensitive display 104, but it is to be understood that the user interface could be differently configured in other embodiments. For example, in some embodiments the user interface could comprise only a touch-sensitive display configured to both display images of the welding operation and receive user commands. In some embodiments, the user interface could comprise a non-touch-sensitive display and a control pendant with dedicated welding control buttons thereon.

[0024] In the illustrated example, the control unit 110 comprises control circuitry that includes an embedded computer 112 and a programmable logic controller (PLC) system 114. The control unit 110 is also configured for connection to a wide area network (WAN) 108 and a typical 110VAC / 15A electrical power supply 109. In some embodiments, the control circuitry of the control unit 110 is configured to record operator inputs and capture related images, robot movement parameters, and other welding parameters and data related to the operator inputs for use in training of an Al model for machine learning or other Al systems.

[0025] The example system 100 shown in Figure 1 comprises an integrated camera / light assembly 106 and a control unit 110 configured with a liquid cooling system 116 for providing cooling fluid thereto, but in other embodiments different types of cameras or other image sensors may be used for providing images to the system 100. There are many possibilities for the camera(s) used to provide images to the control unit. In some implementations, a near infrared (NIR) camera is used to capture real-time images of the welding operation. In certain specific implementations the camera is an NIR camera with a resolution of 2048x2048, an 8-bit depth, and the control unit 110 is provided with images wherein each pixel corresponds to an area ofabout 0.02 mm by 0.02 mm. The camera can be of different types, and may have a different resolution, bit depth, lens, or other parameters in other implementations. In some implementations, the camera is a stereo camera. In some implementations, the camera / light assembly 106 includes multiple cameras operably coupled to the control unit 110. In other embodiments, other types of cameras may be used, including without limitation such as infrared camera, ultraviolet cameras, or X-ray cameras.

[0026] Figure 2 shows an example control circuitry and software architecture 200 for a system for controlling an industrial welding robot according to one embodiment of the present disclosure, such as for example system 100 of Figure 1 . An image acquisition block 210 synchronizes operation of a camera (and optionally an illumination device) for capturing real-time images of a welding operation (e.g. images showing the welding arc, weld pool, and edges of the workpieces), and provides time-stamped frames to a pre-processing block 220. The pre-processing block 220 enhances the images, performs rectification for any lens distortions or other corrections, and provides the pre-processed frames to a media management block 230, comprising a frame handling module that buffers, queues and routes the frames to both a user interface 240 and an image analysis block 260. The media management block 230 also comprises a video recorder module for encoding and compressing the frames, a data logger for associating the frames with other time-stamped data regarding the welding operation and the robotic welding system, and local storage for storing the image data and related data. Data in the local storage may also be provided to a cloud integration block 290 for upload to cloud storage.

[0027] The user interface 240 comprises a visualization module that displays images of the welding operation (with only a slight delay from real-time) to a user, and a data overlay module that receives key performance indicators (KPIs) of the weld as determined by the image analysis block 260 for display to the operator. The operator can specify parameters of the robotic welding system via the user interface 240, and also provide real-time user commands, which are converted to system configuration specifications 250 that specify adjustments to the welding operation. The system configuration specifications 250 are provided to a real-time data transmission module 270, along with any adjustments that may be determined by the image analysis block260, including data fusion and latency compensation. In some embodiments, the system is connected to receive sensor data relating to the welding operation from one or more additional sensors (e.g., temperature sensors, sound sensors, encoder sensors, current sensors, and / or voltage sensors), and the image analysis block 260 is configured to perform data fusion utilizing the real-time images and the additional sensor data to determine the KPIs and provide the KPIs to the user interface 240.

[0028] The real-time data transmission module 270 provides the configuration specifications to a control block 280 comprising a motion control module programmed with protocols for communicating with a plurality of types of robotic control systems, and a weld control module programmed with protocols for communicating with a plurality of types of welding power sources, such that the user commands are converted into the desired modifications in the operation of the robotic welding system. All communications with the robotic welding system are also provided to the local storage and / or cloud storage for future analysis. In the illustrated embodiment, all data and communications are depicted as first being stored in local storage of the media management block 230 before being uploaded to cloud storage via the cloud integration block 290, but in other embodiments data and / or communications could additional or alternately be uploaded to cloud storage directly.

[0029] The embodiments of the systems and methods described herein may be implemented in a combination of both hardware and software. These embodiments may be implemented on programmable computers, each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof), and at least one communication interface. For example, the programmable computers may be a server, network appliance, connected or autonomous vehicle, set-top box, embedded device, computer expansion module, personal computer, laptop, personal data assistant, cloud computing system or mobile device. A cloud computing system is operable to deliver computing service through shared resources, software and data over a network.

[0030] Program code is applied to input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices. In some embodiments, the communication interface may be anetwork communication interface. In embodiments in which elements may be combined, the communication interface may be a software communication interface, such as those for inter-process communication. In still other embodiments, there may be a combination of communication interfaces implemented as hardware, software, and combination thereof.

[0031] Each program may be implemented in a high level procedural or object oriented programming or scripting language, or both, to communicate with a computer system. However, alternatively the programs may be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program may be stored on a storage media or a device (e.g. ROM or magnetic diskette), readable by a general or special purpose programmable computer, for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein. Embodiments of the system may also be considered to be implemented as a non- transitory computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0032] Furthermore, the system, processes and methods of the described embodiments are capable of being distributed in a computer program product including a physical non-transitory computer readable medium that bears computer usable instructions for one or more processors. The medium may be provided in various forms, including one or more diskettes, compact disks, tapes, chips, magnetic and electronic storage media, and the like. The computer useable instructions may also be in various forms, including compiled and non-compiled code.

[0033] Throughout the foregoing discussion, numerous references may be made regarding servers, services, interfaces, portals, platforms, or other systems formed from computing devices. It should be appreciated that the use of such terms is deemed to represent one or more computing devices having at least one processor configured to execute software instructions stored on a computer readable tangible, non-transitory medium. For example, a server can include one or more computers operating as a webserver, database server, or other type of computer server in a manner to fulfill described roles, responsibilities, or functions.

[0034] The technical solution of embodiments of the present disclosure may be in the form of a software product. The software product may be stored in a non-volatile or non- transitory storage medium, which can be a compact disk read-only memory (CD- ROM), a USB flash disk, or a removable hard disk. The software product includes a number of instructions that enable a computer device (personal computer, server, or network device) to execute the methods provided by the embodiments.

[0035] The embodiments described herein are implemented by physical computer hardware, including computing devices, servers, receivers, transmitters, processors, memory, displays, and networks. The embodiments described herein provide useful physical machines and particularly configured computer hardware arrangements.

[0036] It will be appreciated that numerous specific details are set forth in order to provide a thorough understanding of the exemplary embodiments described herein.However, it will be understood by those of ordinary skill in the art that the embodiments described herein may be practiced without these specific details. In other instances, well-known methods, procedures and components have not been described in detail so as not to obscure the embodiments described herein. Furthermore, this description is not to be considered as limiting the scope of the embodiments described herein in any way, but rather as merely describing implementation of the various example embodiments described herein.

[0037] The description provides many example embodiments of the inventive subject matter. Although each embodiment represents a single combination of inventive elements, the inventive subject matter is considered to include all possible combinations of the disclosed elements. Thus, if one embodiment comprises elements A, B, and C, and a second embodiment comprises elements B and D, then the inventive subject matter is also considered to include other remaining combinations of A, B, C, or D, even if not explicitly disclosed.

[0038] As will be apparent to those skilled in the art in light of the foregoing disclosure, many alterations and modifications are possible to the methods and systemsdescribed herein. While a number of exemplary aspects and embodiments have been discussed above, those of skill in the art will recognize certain modifications, permutations, additions and sub-combinations thereof. It is therefore intended that the following appended claims and claims hereafter introduced are interpreted to include all such modifications, permutations, additions and sub-combinations as may reasonably be inferred by one skilled in the art. The scope of the claims should not be limited by the embodiments set forth in the examples, but should be given the broadest interpretation consistent with the foregoing disclosure.

[0039] The present disclosure may be embodied in other specific forms without departing from its spirit or essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive.

Claims

Claims1 . A system for controlling a robotic welding system having a robotic welding torch manipulator and a welding power source, the system comprising: a motion control interface configured for connection to one of a plurality of brands of robot controller of the robotic welding torch manipulator; a weld control interface configured for connection to one of a plurality of brands of welding power source; an image acquisition interface configured to receive image data from one or more cameras positioned to capture real-time images of a welding operation; a user interface comprising a display for displaying images of the welding operation and at least one input device for receiving operator inputs specifying adjustments to the welding operation; and control circuitry comprising one or more processors configured to convert the operator inputs into configuration specifications for the robotic welding system, provide motion control adjustment instructions to the motion control interface utilizing a protocol for a selected one of the plurality of brands of robot controller, and provide weld control adjustment instructions to the weld control interface utilizing a protocol for a selected one of the plurality of brands of welding power source.

2. The system of claim 1 wherein the motion control interface is directly connected to the robot controller and the weld control interface is directly connected to the welding power source.

3. The system of claim 1 wherein both the motion control interface and the weld control interface are connected to the robot controller, and the weld control adjustment instructions are relayed to the welding power source by the robot controller.

4. The system of any one of claims 1 to 3 wherein the control circuitry is configured to record timestamped operator inputs and corresponding related images, robot movement parameters, and welding parameters related to the welding operation at thetime of each operator input for use in training of an artificial intelligence (Al) welding control model.

5. The system of claim 4, wherein the control circuitry is configured to be operable in an autonomous mode wherein the control circuitry controls the robotic welding utilizing the Al welding control model to automatically make real-time adjustments to a welding program being carried out by the robotic welding system.

6. The system of claim 4 or 5, wherein the control circuitry is configured to be operable in an operator training mode wherein the control circuitry monitors adjustments made by a new operator to a welding program being carried out by the robotic welding system and utilizes the Al welding control model to evaluate the adjustments made by the new operator.

7. The system of any one of claims 4 to 6, wherein the control circuitry comprises a local storage storing the Al welding control model.

8. The system of any one of claims 1 to 7, wherein the control circuitry comprises a cloud integration block for uploading the timestamped operator inputs and corresponding related images, robot movement parameters, and welding parameters related to the welding operation to cloud storage.

9. The system of any one of claims 1 to 8, wherein the control circuitry is connected to receive sensor data relating to the welding operation from one or more additional sensors, determine one or more welding key performance indicators (KPIs) based on the real-time images and the sensor data from the one or more additional sensors, generate augmented images by overlaying the welding KPIs on the real-time images, and provide the augmented images to the user interface for displaying on the display.

10. The system of claim 9 wherein the one or more additional sensors comprise one or more temperature sensors, sound sensors, encoder sensors, current sensors, or voltage sensors.

11. A method for controlling an autonomous robotic welding system having a robotic welding torch manipulator and a welding power source, the method comprising: determining a brand of the robotic welding torch manipulator and a brand of the welding power source; receiving an initial robotic welding program specifying a welding path and welding parameters to complete a welding operation; capturing real-time images of the welding operation; processing the real-time images determine of the welding operation to determine one or more welding key performance indicators (KPIs); overlaying the real-time images with the welding KPIs to generate augmented images and displaying the augmented images on a display of a user interface; receiving one or more operator inputs from an operator using the user interface specifying adjustments the welding path and / or welding parameters; converting the one or more operator inputs into motion control adjustment instructions utilizing a robot control protocol for the determined brand of the robotic welding torch manipulator, and into weld control adjustment instructions utilizing a weld control protocol for the determined brand of the welding power source.

12. The method of claim 11 , further comprising recording timestamped operator inputs and corresponding related images, robot movement parameters, and welding parameters related to the welding operation at the time of each operator input for use in training of an artificial intelligence (Al) welding control model.

13. The method of claim 11 or claim 12, further comprising receiving sensor data relating to the welding operation from one or more additional sensors comprising or more temperature sensors, sound sensors, encoder sensors, current sensors, or voltage sensors, and wherein processing the real-time images to determine the one ormore welding KPIs comprises combining the sensor data from the one or more additional sensors with the real-time images.

Images