Hybrid sensing and processing device for a robot

Abstract

A hybrid sensing and processing device for a robot with an articulated wrist and a processing tool has an interposable frame, a laser camera affixed to a reference surface portion of the frame, video cameras integrated into the frame, an electronic circuit for control of the laser camera and other purposes, a communication interface integrated into the frame, and an onboard controller mounted into the frame and connected to the electronic circuit and the communication interface. The device provides calibrated machine vision data for on-site and remote robot or cobot operation and monitoring purposes.

Description

[0001] HYBRID SENSING AND PROCESSING DEVICE FOR A ROBOT

[0002] FIELD OF THE INVENTION

[0003] The invention relates to robotics and more particularly to a hybrid sensing and processing device for a robot which may be a cobot.

[0004] BACKGROUND

[0005] In the case of a welding robot whose task is to perform precise joining of a workpiece made for example of two parts, many concerns need to be addressed. For example, the parts must be well located with respect to a programmed trajectory of the welding tool. Any deviation of the welding tool with respect to the parts, such as offsets, misalignments, gaps, will cause a bad location of the welding bead, leading to a defective joint. Correcting the trajectory of the welding robot and its welding tool using a laser vision system for seam tracking or seam finding is known, e.g. as taught in US patent No. 11396057 (Boillot et al.).

[0006] Vision systems must be perfectly mechanically referenced on the robot wrist and calibrated within the robot environment to provide data which can be used to position the welding tool. Such referencing typically requires human trained resources to be achieved. For example, calibration between the vision systems and the welding tool involves mechanical design knowledge of the vision systems with respect to the welding tool for providing appropriate control data to the robot. The calibration requires a calibration routine to be executed in order to establish the relationship between vision data and a Tool Center Point (TCP) of the robot. This routine requires the operator to teach an accurate TCP and to define a coordinate system in the robot environment to support calibration. These steps are known to be complex and often lead to a non-optimal calibration which compromises the use of the vision system.

[0007] Furthermore, robotic technology is no longer just used for manufacturing process automation but also for gathering and recording complex process data which may be produced by various models of devices proposed by different manufacturers.

[0008] SUMMARY

[0009] An object of the invention is to provide a hybrid sensing and processing device for a robot, which may be perfectly mechanically referenced on the robot wrist, compact, and easily calibrated within the robot environment.

[0010] Another object of the invention is to provide a hybrid sensing and processing device for a robot, which may integrate data processing components requiring no external processing devices to provide full operative vision data as may be needed by the robot to perform its task, and which may further provide process data to other devices such as a tablet for remote monitoring, setting and other possible purposes.

[0011] According to an aspect of the invention, there is provided a hybrid sensing and processing device for a robot with an articulated wrist and a processing tool, the hybrid sensing and processing device comprising: a frame having opposite top and bottom mounting assemblies respectively compatible with a flange of the articulated wrist and the processing tool for interposing the frame between the articulated wrist and the processing tool; a laser camera affixed to a reference surface portion of the frame, the laser camera having a laser emitter positioned with respect to the frame to project a laser beam at a look-ahead distance from a tool center point of the processing tool, and an image sensor with a field of view over a target area where the laser beam is projected; first and second video cameras integrated into the frame, the first video camera having a field of view directed over a process monitoring area of the processing tool, the second video camera having a field of view directed over the tool center point of the processing tool, the fields of view of the first and second video cameras having an internal calibration in relation with the field of view of the image sensor of the laser camera; an electronic circuit comprising a laser controller connected to the laser emitter of the laser camera and a 3D profiler connected to the image sensor of the laser camera; a communication interface integrated into the frame; and an onboard controller mounted into the frame and connected to the electronic circuit and the communication interface, the onboard controller configured to perform operations comprising: controlling operation of the laser emitter via the laser controller; producing process data expressed into a common reference based on robot positions and process time, the process data including image data derived from the image sensor of the laser camera and the video cameras; transmitting the process data via the communication interface; and receiving setting and control data affecting the operations performed by the onboard controller via the communication interface.

[0012] According to another aspect of the invention, there is provided a hybrid sensing method for a robot with an articulated wrist and a processing tool, the hybrid sensing method comprising the steps of: providing a frame interposable between a flange of the articulated wrist and the processing tool; affixing a laser camera to a reference surface portion of the frame so that the laser camera has a field of view over a target area where a laser beam is projected by the laser camera; integrating first and second video cameras into the frame so that the first video camera has a field of view directed over a process monitoring area of the processing tool and the second video camera has a field of view directed over a tool center point of the processing tool; internally calibrating the field of views of the video cameras in relation with the field of view of the laser camera; controlling operation of the laser camera to produce the laser beam; producing process data expressed into a common reference based on robot positions and process time, the process data including image data derived from the image sensor of the laser camera and the video cameras; transmitting the process data to the robot; and receiving setting and control data affecting operations performed by the laser camera, a processing of the image data and a producing of the process data.

[0013] The following provides an outline of possible aspects, features and advantages of the invention which are to be considered in a non-restrictive and non-limiting manner and which will be more fully disclosed hereinafter.

[0014] The design of the hybrid sensing and processing device of the invention allows direct interposition of the device between to the robot and the processing tool without requiring mounting adapters as it replicates the robot wrist pattern for attachment of the processing tool which would otherwise be directly attached to the robot wrist. Thus, the device allows direct and factory calibration with respect to a robot articulated arm wrist so that its setup and applications are simplified. The device integrates and combines vision and processing components to provide 2D HDR (High Dynamic Range) video channels I streams, 3D range measurement I channel, audio acquisition and global process sensing. The integrated vision components thus eliminate the need to perform calibration between usually independent vision components of various models and positions and the robot as the vision components have indexed and fixed positions relative to a robot flange. The device can be factory-calibrated to provide all process data expressed directly in the robot coordinate system. The device also integrates bleeding edge control technology to further simplify its integration into a robotic cell with no external control box, and a direct Wi-Fi link to any device including portable tablets to get access to its implementable web-based user interface. This seamless integration makes adaptive process control more accessible for the collaborative robot (cobot) industry. The two 2D image sensors integrated into the device allows direct and real time monitoring of the TCP and the surrounding process environment, and real time correction of its position and trajectory. The data processing power of the device allows implementing video overlay functions for quick referencing of the TCP position and process tool alignment for advanced process monitoring even in presence of a possible welding arc. Internal calibration of the 2D image sensors in relation with the 3D image sensor providing the 3D range measurement allows generating colored 3D surfaces maps of the workpiece which can be used in advanced vision processing algorithms combining 3D laser range data and 2D video image data. Integrated sensors such as a microphone, an inertial motion unit (IMU), 2D channels, 3D channel, provide full complex process monitoring and data gathering capabilities. For example, in a welding environment, such gathered data may include lateral and / or vertical part deviations, unwelded joint characteristics such as gap and mismatch, bevel preparation, etc. All those valuable information may be used for preventive actions to avoid welding quality issues. Also, combining the vision data with post-process weld inspection data may contribute to improve the performance of weld inspection and satisfying Industrial Internet of Things (HOT) requirements. Different vision packages may be supported by the device. Integrated peripherals such as a LED ring, an OLED display, control or command buttons, provide optimized control to an operator directly from the device attached to the robot arm wrist. Connectivity with the vision system of the device and the robot allows a diagnosis of the complete robotic cell and provides access to all components. The processing power of the device simplifies the overall cell layout. The ruggedized assembly of the device provides protection against electromagnetic interferences, high temperatures, fine metallic dust, molten metal spatter and high-frequency interferences commonly found in welding or industrial production environments.

[0015] As the present invention may integrate various devices inside one factory- calibrated frame, all data produced by the various devices may be processed and expressed into a common referential and time-based system. The processed data may advantageously include robot or cobot positions and a process timeline or process timestamps so that all the information may be synchronized, combined and accumulated. Such information may be used for real time control and development of control rules for a robotic task. The common referential and timebased system may be expressed in a coordinate system of the robot (or cobot) as acquired from position data derived from the robot using a communication protocol such as an advantageous one developed by the company SERVO-ROBOT. All information may be synchronized using timestamps included in each message exchanged between the device and the robot controller. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] A detailed description of preferred embodiments will be given herein below with reference to the following drawings.

[0017] FIG. 1 is a perspective exploded schematic view of a hybrid sensing and processing device mountable between a robot wrist and a processing tool according to an embodiment of the invention.

[0018] FIG. 2 is a perspective exploded schematic view of a hybrid sensing and processing device according to an embodiment of the invention.

[0019] FIG. 3 is a perspective schematic view of a hybrid sensing and processing device according to an embodiment of the invention.

[0020] FIG. 4 is a schematic block diagram of a hybrid sensing and processing device according to an embodiment of the invention.

[0021] FIG. 5 is a perspective schematic view illustrating a wide angle field of view of one video camera and a laser beam projection plane of a laser emitter of a hybrid sensing and processing device according to an embodiment of the invention.

[0022] FIG. 6 is a perspective schematic view illustrating wide and narrow angle fields of view of two video cameras of a hybrid sensing and processing device according to an embodiment of the invention.

[0023] FIG. 7 is a schematic view illustrating an example of overlay added to image data produced by a hybrid sensing and processing device according to an embodiment of the invention.

[0024] FIG. 8 is a perspective schematic view illustrating a wide angle field of view of one video camera and a laser beam projection plane of a laser emitter mounted on a modular laser arm of a hybrid sensing and processing device according to an embodiment of the invention.

[0025] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] Referring to FIG. 1 , there is shown a hybrid sensing and processing device 2 for a robot with an articulated wrist 4 and a processing tool 6 according to the invention. The processing tool 6 may be, for example, a GMAW (gas metal arc welding) torch, another type of welding torch or device, a material dispenser as a sealant dispenser, a material handling device as a robot claw, or other devices requiring machine vision and also signal and data processing for robot or cobot applications.

[0027] The hybrid sensing and processing device comprises a frame 8 having opposite top and bottom mounting assemblies (or robot flange adapters) 10, 12 respectively compatible with a flange 18 of the articulated wrist 4 and the processing tool 6 for interposing the frame 8 between the articulated wrist 4 and the processing tool 6. In an embodiment, the top and bottom mounting assemblies have electric insulations with respect to the flange 18 and the processing tool 6. In an embodiment, the top and bottom mounting assemblies are mountable using dowel pins (not shown).

[0028] Referring to FIG. 3, a laser camera 24 (as shown in FIG. 2) is affixed to a reference surface portion of the frame 8. The reference surface portion may advantageously have a shape matching a shape of the laser camera 24 so that the laser camera 24 integrates into or form a complementary part of the frame 8. The reference surface portion may also be an outer surface portion of the frame 8, in which case the laser camera 24 protrudes from the frame 8. The reference surface portion may also be a mix of the above embodiments so that the laser camera 24 partially integrates into the frame 8 and partially protrudes from the frame 8. The laser camera 24 has a laser emitter 26 (as shown in FIG. 4) positioned with respect to the frame 8 to project a laser beam (or stripe) 28 at a look-ahead distance from a TCP 30 (or tip) of the processing tool 6 (as shown e.g. in FIG. 5). In an embodiment, the laser emitter 26 is formed of a laser diode. In an embodiment, the laser emitter 26 is a class 2 red laser emitter. In a further embodiment, the laser emitter 26 may be operated according to different classes of operating modes. The laser camera 24 also has an image sensor 32 (as shown in FIG. 4) with a field of view over a target area where the laser beam 28 is projected in order to produce laser range data representative of a profile of a workpiece 46 (as shown e.g. in FIG. 5). The image sensor 32 may advantageously be a CMOS imager. The laser camera 24 may be considered, with appropriate electronics e.g. for 3D triangulation purposes, as a 3D range finder or scanner.

[0029] First and second video cameras 36, 38 (as shown in FIG. 4) are integrated into the frame 8. The first video camera 36 has a wide angle field of view 40 (as shown e.g. in FIG. 6) directed over a process monitoring area or an important process zone, which allows a global view for real time process monitoring purposes. The first video camera 36 thus provides a global video channel, which may be used for example to transmit color or monochrome images or video of the environment. The second video camera 38 has a narrow angle field of view 42 (as shown in FIG. 6) directed over or aligned with the TCP 30 of the processing tool 6 (as shown in FIG. 6), which allows a view of the TCP 30 and its immediate surrounding for real time process monitoring purposes. The second video camera 38 thus provides a zoomed video channel, which may be used for example to transmit geometric data of a molten weld pool. Both fields of view 40, 42 have an internal calibration in relation with the field of view 28 of the image sensor 32 of the laser camera 24, which may be calibrated with the frame 14 as they have all fixed positions relative to the frame 14. In an embodiment, one or both video cameras 36, 38 use high dynamic range (HDR) sensors 104, 106 (as shown in FIG. 4) to produce the image data. The video cameras 36, 38 may have respective connectors 120, 122 mounted onto the frame 8 for direct cable connections if desired.

[0030] Referring to FIG. 4, the hybrid sensing and processing device 2 has an electronic circuit 44 which may advantageously be implemented using a FPGA (Field Programmable Gate Array). Other electronics may be used if desired, for example discrete electronic components. The electronic circuit 44 comprises a laser controller 48 connected to the laser emitter 26 of the laser camera 24 (as shown in FIG. 2) through a laser driver and protection circuit 114, and a 3D profiler 50 connected to the image sensor 32 of the laser camera 24. The laser controller 48 and the 3D profiler 50 may be formed by the FPGA. In an embodiment, the image sensor 32 is connected to the electronic circuit 44 using a MIPI (Mobile Industry Processor Interface) interface. A memory such as an EEPROM 116 may be provided to store laser configuration settings usable by the laser controller 48.

[0031] A communication interface 52 is integrated into the frame 8 (as shown e.g. in FIG. 1 ).

[0032] An onboard controller 54 is mounted into the frame 8 and is connected to the electronic circuit 44 and the communication interface 52. The onboard controller 54 is configured to perform operations comprising controlling operation of the laser emitter 26 via the laser controller 48, producing process data expressed into a common reference based on robot positions and process time and including image data derived from the image sensor 32 of the laser camera 24 and the video cameras 36, 38, transmitting the process data via the communication interface 52, and receiving setting and control data affecting the operations performed by the onboard controller 54 via the communication interface 52.

[0033] In an embodiment, the video cameras 36, 38 are connected to the onboard controller 54 using a MIPI interface. The video cameras 36, 38 may be provided with LED lights 108, 110 for illuminating the processing area. One or both video cameras 36, 38 may be used to read a QR code for part identification purposes.

[0034] Referring back to FIG. 2, in an embodiment, the device 2 may be equipped with a modular laser arm 132 affixed to and projecting from a reference surface portion of the frame 8. The modular laser arm 132 has a distal end provided with a laser emitter 134 which may be controlled in a same way as the laser emitter 26 (as shown in FIG. 4) of the laser camera 24. The modular laser arm 132 has a shape and a size adapted to the processing tool 6 (shown e.g. in FIG. 1) so that, as shown in FIG. 8, a laser beam produced by the laser emitter 134 is projected in a predefined projection plane 28' intersecting the workpiece 46 in the field of view of the image sensor 32 (as shown in FIG. 4) of the laser camera 24 (as shown in FIG. 2). Different modular laser arms 132 may be provided and used depending on the model of the processing tool 6 so that a profile of the workpiece 46 illuminated by the laser beam produced by the laser emitter 134 is in the field of view of the image sensor 32 (as shown in FIG. 4) of the laser camera 24. Depending on a Z-axis rotation angle of the device 2 through action of the wrist 4 (as shown e.g. in FIG. 1), the resulting image data produced by the image sensor 32 may be used for example for seam tracking or bead checking purposes in a welding environment. FIG. 8 illustrates a Z-axis rotation angle of the device 2 with the modular arm 132 and the laser emitter 134 in position for seam tracking. For bead checking, the wrist 18 would rotate the device 2 by 180° in the Z-axis so that the laser emitter 134 is positioned behind a moving direction of the device 2. In an embodiment, the laser emitter 134 may replace the laser emitter 26 (as shown in FIG. 4). In an embodiment, the modular laser arm 132 is an add-on.

[0035] In an embodiment, the top mounting assembly 10 comprises a LED ring 56 upwardly projecting from a top surface portion 58 of the frame 8 and having a central recess 60 into which the flange 18 of the articulated wrist 4 is mountably engageable. The top surface portion 58 may be formed by a top cover of the frame 8. With reference to FIG. 4, the electronic circuit 44 then comprises a LED controller 62 connected to the LED ring 56 and responsive to a control signal produced by the onboard controller 54, the control signal defining a color and a blinking mode of the LED ring 56 according to the setting and control data. The LED ring 56 is visible by a user or operator over 360 degrees and may be used to indicate an operating state of the device and to provide warnings if desired. For example, the color and blinking mode of the LED ring 56 may be used to report a system booting state, a system critical event, a warning event, a system ready state, etc. In an embodiment, the LED ring 56 may be illuminated by a LED illumination ring (not shown) integrated into the frame 8 and arranged to project light having a desired color through openings 136 in the top cover of the frame 8.

[0036] Referring to FIG. 2, in an embodiment, one or more push buttons 64 (two in the illustrated case) are externally positioned onto the frame 8. With reference to FIG. 4, the electronic circuit 44 then comprises a user input controller 66 connected to the push button(s) 64 for producing interaction signals in response to the push button(s) 64 being pressed by an operator. The operations performed by the onboard controller 54 then also comprise responding to the interaction signals according to the setting and control data.

[0037] In an embodiment, a display 66 responsive to display data produced by the onboard controller 54 (as shown in FIG. 4) according to the setting and control data is also externally positioned onto the frame 8. The display 66 may be an OLED display (for real-time) showing a 3D profile of the workpiece 46 (as shown e.g. in FIG. 5), providing system messages to the user or operator, displaying a user interactive menu (e.g. involving use of the push buttons 64) and providing system information such as an IP address and a Wi-Fi SSID (Service Set Identifier) of the device. The display 66 may thus provide direct feedback to the operator and may be used for diagnostic purposes during process execution and for guiding purposes during manual operations such as installation, configuration and teaching of the device. The push buttons 64 and the display 66 preferably work together and are programmed in the onboard controller 54 to provide various options and features relating to use and operation of the device. For example, possible supported features may be to take a snapshot, to switch between the video cameras 36, 38 as video source, to control the LCD shutters 100, 102 and the LED lights 108, 110, to control the laser diode 26, to trigger audio command inputs, etc. The push buttons 64 may also be used to acknowledge (and possibly supersede) alarm conditions detected by the onboard controller 54.

[0038] Referring to FIG. 3, in an embodiment, the laser camera 24 and the video cameras 36, 38 (as shown in FIG. 2) have respective protective heads (or nozzles) 34, 68, 70 extending on an underside surface 16 of the frame 8 and responsive to control signals produced by the onboard controller 54 (as shown in FIG. 4) according to the setting and control data. The protective heads 34, 68, 70 may advantageously be as disclosed in US patent No. 9541755 (Boillot et al.), with protective shutters as disclosed in US patent No. 11396057 (Boillot et al.), in particular electronic LCD shutters 100, 102 (as shown in FIG. 4 for the video cameras 36, 38), provided with (replaceable) optical lenses (not shown) to cope with possible welding fume and spatters or other process inconveniences. Lens-detect sensor(s) 130 (as shown in FIG. 4) may be provided to detect presence of the lenses. The protective heads 34, 68, 70 may also have optical filters (not shown) in front of the lens and focus lens (not shown) if desired. In the case where protective heads 34, 68, 70 are used, the frame 8 may have an air inlet 124 outwardly projecting from the frame 8, connectable to a compressed air hose (not shown), and in communication with the protective heads 34, 68, 70.

[0039] Referring again to FIG. 4, in an embodiment, a microphone 72 is integrated into the frame 8 (as shown e.g. in FIG. 3) and connected to the onboard controller 54. The microphone 72 may have a sound inlet 74 extending through the underside surface 16 of the frame 8 (as shown in FIG. 3) so that, for example, sound of a welding process may be monitored. The operations performed by the onboard controller 54 may then comprise adding sound data derived from a sound output of the microphone 72 to the process data.

[0040] In an embodiment, an inertial measurement unit 76 (IMU) is integrated into the frame 8 and connected to the onboard controller 54. The operations performed by the onboard controller 54 may then comprise computing rotation angles derived from measurement data produced by the inertial measurement unit 76 and adding orientation data of the processing tool 6 (as shown in FIG. 1) derived from the rotation angles to the process data. The IMU 76 may be used to detect a collision of the device 2 with respect to its environment.

[0041] In an embodiment, the communication interface 52 comprises a Wi-Fi communication circuit 78 and an Ethernet communication circuit 80, or only one of them if desired. An antenna 118 (as shown in FIG. 2) of the Wi-Fi communication circuit 78 may be mounted onto the frame 8 so as to upwardly project from its top surface portion 58. The Ethernet communication circuit 80 may be intended in particular to communicate with the robot, or with an external camera or PC (not shown). An interconnection kit (not shown) may be used to provide power supply (e.g. 24 V) to the device and other possible accessories, an Ethernet link with the Ethernet communication circuit 80, and an enable laser input which may be required by the onboard controller 54 for safely enabling activation of the laser emitter 26.

[0042] In an embodiment, the onboard controller 54 comprises a processor 82 (CPU), a memory 84 coupled to the processor 82, and a graphics processing unit 86 (GPU) coupled to the processor 82 and the memory 84. The memory 84 stores instructions executable by the processor 82 and the graphics processing unit 86 to define the operations performed by the onboard controller 54 and store the setting and control data in the memory 84. In an embodiment, the onboard controller 54 also comprises a neural processing unit 88 (NPU) coupled to the processor 82, the graphics processing unit 86 and the memory 84. The memory 84 then stores instructions executable by the neural processing unit 88 enabling pretrained and scalable neural network processing to predict and perform a process task using image data processed by the graphics processing unit 86 for automated machine vision. The onboard controller 54 may be implemented using a system-on-chip (SOC). In an embodiment, the onboard controller 54 also has a video processing unit 112 (VPU) for running machine vision algorithms stored in the memory 84.

[0043] In an embodiment, the graphics processing unit 86 has video overlay functions under control of the processor 82 to add a video overlay to the image data. With reference to FIG. 7, an example of such a video overlay is illustrated, where the video overlay comprises a crosshair reticle 90 centered onto the TCP 30 by the graphics processing unit 86, and lines 126 aligned with a nozzle portion 128 of the processing tool 6 (as shown e.g. in FIG. 1). Other video overlays may be user settable and stored in the memory 84.

[0044] In an embodiment, the 3D profiler 50 comprises a configuration interface 92 connected to the image sensor 32 and responsive to configuration data received from the onboard controller 54, a pixel deserializer 94 connected to the image sensor 32, a 3D profile generator 96 under control of the onboard controller 54 and connected to the pixel deserializer 94 for producing 3D range profile data derived from the image data (e.g. raw pixels) processed by the pixel deserializer 94, and a 3D profile memory 98 connected to the 3D profile generator 96 for storing the 3D laser range data forming the image data produced by the image sensor 32 and transmitted to the onboard controller 54 for processing by the CPU 82.

[0045] In an embodiment, the setting and control data comprise laser class definitions, and the operations performed by the onboard controller 54 comprise switching and controlling a class of the laser emitter 26 (or 26') depending on features of the workpiece 46 to be processed by the processing tool 6 (as shown e.g. in FIG. 5). For example, the laser class may be switched from class 2 to class 3b if the workpiece 46 is very shiny. In an embodiment, a welding process may be detected by image video channels provided by the video cameras 36, 38 and switching from class 2 to class 3b may be performed accordingly. Such switching between class 2 and class 3b may result from detecting an intensity saturation in a known torch wire tip or TCP area in the images provided by any one of the video cameras 36, 38. Switching from class 2 to class 3b may be triggered upon detecting such intensity saturation, and switching from class 3b to class 2 may be triggered when no such intensity saturation is detected.

[0046] In an embodiment, the memory 84 may store software ran by the CPU 82 for e.g.:

[0047] - vision analysis to detect features on the workpiece 46 (shown e.g. in FIG. 5);

[0048] - seam finding and part location computing;

[0049] - calibration to convert all 2D and 3D vision data into a coordinate frame of the device or the robot or a user-defined coordinate frame;

[0050] - communication with the robot for applying seam finding corrections;

[0051] - communication with a (web) Human Machine Interface (HMI) for system monitoring and configuration purposes;

[0052] - encapsulating a format of the image data into MP4 format or another format;

[0053] - transferring the image data encapsulated into a given format to an external machine possibly using a web browser which may include TCP positioning display data.

[0054] In an embodiment, the memory 84 stores software ran by the GPU 86 for HDR display of the TCP 30 (as shown in FIG. 7) which may consist in e.g.: - acquiring the image data produced by the video cameras 36, 38 at different integration times which may be controlled by the CPU 82;

[0055] - applying specific masks on each image derived from the image data;

[0056] - combining images derived from the image data;

[0057] - compressing the image data into h.264 format or another format using the VPU 112;

[0058] - detecting presence of a welding arc or another processing tool effect.

[0059] As a result, the device according to the invention may provide full scene display data for display e.g. on a tablet or similar device as a PC (not shown) via the Wi-Fi communication interface 78 or the Ethernet interface 80 using standard web browsing applications. Also, operator pool monitoring viewing in real time as possible with the device may allow an operator to shift the robot program in real time.

[0060] In an embodiment, the process-related sound signal picked by the microphone 72 may be memorized in the memory 84 and synchronized with a robot position. Data from other sources (not shown) may be processed by the device 2, for example peripheral torch sensor data such as arc voltage, gas flow, nozzle temperature, etc. Orientation of the robot wrist 4 and flange 18 and orientation of the device 2 and the processing tool 6 relative to gravitational forces, as provided by the IMU 76, may also be taken into account and all the data sources may be combined into common reference. Since all the data sources are factory calibrated to a common reference (the device 2 itself), all information is synchronized to a current robot position within a timeline of the process. The data may first be stored in a volatile memory of the memory 84 and retrieved at an end of a process sequence in the form of a set of files using a predefined format. The device 2 may be connected to an external database (not shown) so that all information may be automatically uploaded for later analysis or retrieval. The stored and / or extracted data may be used in various ways, for example for visual analysis of the data in correspondence with a robot position, combination of intensity, color and 3D range in hybrid analysis algorithms to compute weld defects, creation of labelled data blocks to train Al models for detection of abnormal conditions (e.g. matching an arc sound with possible 3D defect sounds).

[0061] Based on the foregoing, the invention may be also expressed as a method residing in:

[0062] - providing a frame 8 interposable between the flange 18 of the articulated wrist 4 and the processing tool 6;

[0063] - affixing the laser camera 24 to the reference surface portion of the frame 8 so that the laser camera 24 has a field of view over a target area where the laser beam 28 is projected by the laser camera 24;

[0064] - integrating the video cameras 36, 38 into the frame 8 so that the first video camera 36 has a field of view directed over a process monitoring area of the processing tool 6 and the second video camera 38 has a field of view directed over the TCP 30 of the processing tool 6;

[0065] - internally calibrating the field of views of the video cameras 36, 38 in relation with the field of view of the laser camera 24;

[0066] - controlling operation of the laser camera 24 to produce the laser beam 28;

[0067] - producing process data expressed into a common reference based on robot positions and process time and including image data derived from the laser camera 24 and the video cameras 36, 38;

[0068] - transmitting the process data to the robot; and

[0069] - receiving setting and control data affecting operations performed by the laser camera 24, a processing of the image data and a producing of the process data.

[0070] While embodiments of the invention have been illustrated in the accompanying drawings and described above, it will be evident to those skilled in the art that modifications may be made therein without departing from the invention.

Claims

CLAIMS:1 . A hybrid sensing and processing device for a robot with an articulated wrist and a processing tool, the hybrid sensing and processing device comprising: a frame having opposite top and bottom mounting assemblies respectively compatible with a flange of the articulated wrist and the processing tool for interposing the frame between the articulated wrist and the processing tool; a laser camera affixed to a reference surface portion of the frame, the laser camera having a laser emitter positioned with respect to the frame to project a laser beam at a look-ahead distance from a tool center point of the processing tool, and an image sensor with a field of view over a target area where the laser beam is projected; first and second video cameras integrated into the frame, the first video camera having a field of view directed over a process monitoring area of the processing tool, the second video camera having a field of view directed over the tool center point of the processing tool, the fields of view of the first and second video cameras having an internal calibration in relation with the field of view of the image sensor of the laser camera; an electronic circuit comprising a laser controller connected to the laser emitter of the laser camera and a 3D profiler connected to the image sensor of the laser camera; a communication interface integrated into the frame; and an onboard controller mounted into the frame and connected to the electronic circuit and the communication interface, the onboard controller configured to perform operations comprising: controlling operation of the laser emitter via the laser controller; producing process data expressed into a common reference based on robot positions and process time, the process data including image data derived from the image sensor of the laser camera and the video cameras; transmitting the process data via the communication interface; and receiving setting and control data affecting the operations performed by the onboard controller via the communication interface.

2. The hybrid sensing and processing device according to claim 1 , wherein the top mounting assembly comprises a LED ring upwardly projecting from a top surface portion of the frame and having a central recess into which the flange of the articulated wrist is mountably engageable, and the electronic circuit comprises a LED controller connected to the LED ring and responsive to a control signal produced by the onboard controller, the control signal defining a color and a blinking mode of the LED ring according to the setting and control data.

3. The hybrid sensing and processing device according to claim 1 , further comprising one or more push buttons externally positioned onto the frame, and wherein the electronic circuit comprises a user input controller connected to said one or more push buttons for producing interaction signals in response to said one or more push buttons being pressed by an operator, the operations performed by the onboard controller comprising responding to the interaction signals according to the setting and control data.

4. The hybrid sensing and processing device according to claim 1 , further comprising a display externally positioned onto the frame and responsive to display data produced by the onboard controller according to the setting and control data.

5. The hybrid sensing and processing device according to claim 1 , wherein the laser camera and the video cameras have respective protective heads extending on an underside surface of the frame and responsive to control signals produced by the onboard controller according to the setting and control data.

6. The hybrid sensing and processing device according to claim 1 , further comprising at least one of: a microphone integrated into the frame and connected to the onboard controller, the microphone having a sound inlet extending through an underside surface of the frame, the operations performed by the onboard controller comprising adding sound data derived from a sound output of the microphone to the process data; andan inertial measurement unit integrated into the frame and connected to the onboard controller, the operations performed by the onboard controller comprising computing rotation angles derived from measurement data produced by the inertial measurement unit and adding orientation data of the processing tool derived from the rotation angles to the process data.

7. The hybrid sensing and processing device according to claim 1 , wherein the communication interface comprises at least one of a Wi-Fi communication circuit and an Ethernet communication circuit.

8. The hybrid sensing and processing device according to claim 1 , wherein the onboard controller comprises a processor, a memory coupled to the processor, and a graphics processing unit coupled to the processor and the memory, the memory storing instructions executable by the processor and the graphics processing unit to define the operations performed by the onboard controller and store the setting and control data in the memory.

9. The hybrid sensing and processing device according to claim 8, wherein the onboard controller further comprises a neural processing unit coupled to the processor, the graphics processing unit and the memory, the memory storing instructions executable by the neural processing unit enabling pretrained and scalable neural network processing to predict and perform a process task using image data processed by the graphics processing unit for automated machine vision.

10. The hybrid sensing and processing device according to claim 8, wherein the graphics processing unit has video overlay functions under control of the processor to add a video overlay to the image data.

11. The hybrid sensing and processing device according to claim 10, wherein the video overlay comprises a crosshair centered onto the tool center point by the graphics processing unit.

12. The hybrid sensing and processing device according to claim 1 , wherein the electronic circuit comprises a FPGA, the laser controller and the 3D profiler being formed by the FPGA.

13. The hybrid sensing and processing device according to claim 1 , wherein the 3D profiler comprises a configuration interface connected to the image sensor and responsive to configuration data received from the onboard controller, a pixel deserializer connected to the image sensor, a 3D profile generator under control of the onboard controller and connected to the pixel deserializer for producing 3D range profile data derived from the image data processed by the pixel deserializer, and a 3D profile memory connected to the 3D profile generator for storing the 3D laser range data forming the image data produced by the image sensor and transmitted to the onboard controller.

14. The hybrid sensing and processing device according to claim 1 , wherein the top and bottom mounting assemblies have electric insulations with respect to the flange and the processing tool.

15. The hybrid sensing and processing device according to claim 1 , wherein the video cameras use high dynamic range sensors to produce the image data.

16. The hybrid sensing and processing device according to claim 1 , wherein the setting and control data comprise laser class definitions, and the operations performed by the onboard controller comprise switching and controlling a class of the laser emitter depending on features of a workpiece to be processed by the processing tool or an intensity saturation detected by the controller with respect to the tool center point in the image data derived from any one of the video cameras.

17. The hybrid sensing and processing device according to claim 1 , further comprising a modular arm affixed to and projecting from a reference surface portion of the frame, the modular arm having a distal end provided with a laser emitter controlled by the onboard controller, and a shape and a size so that a laser beam produced by the laser emitter on the modular arm is projected in apredefined projection plane intersecting a workpiece in the field of view of the image sensor of the laser camera.

18. A hybrid sensing method for a robot with an articulated wrist and a processing tool, the hybrid sensing method comprising the steps of: providing a frame interposable between a flange of the articulated wrist and the processing tool; affixing a laser camera to a reference surface portion of the frame so that the laser camera has a field of view over a target area where a laser beam is projected by the laser camera; integrating first and second video cameras into the frame so that the first video camera has a field of view directed over a process monitoring area of the processing tool and the second video camera has a field of view directed over a tool center point of the processing tool; internally calibrating the field of views of the video cameras in relation with the field of view of the laser camera; controlling operation of the laser camera to produce the laser beam; producing process data expressed into a common reference based on robot positions and process time, the process data including image data derived from the image sensor of the laser camera and the video cameras; transmitting the process data to the robot; and receiving setting and control data affecting operations performed by the laser camera, a processing of the image data and a producing of the process data.

Images