Spray tip calibration

Abstract

Misalignment, tip degradation, and manufacturing inconsistencies in a spraying end effector at a distal end of a robotic arm can cause unintended deviations to a width and / or thickness of sprayed band of material and / or cause an untended offset of the sprayed band of material relative to the intended center line. Calibration techniques can be implemented to determine offsets in pitch, yaw, and / or roll, fan width, and / or thickness. Offsets in the pitch, yaw, and roll axes, fan width, and thickness can be determined using algorithms (e.g., computer vision algorithms, digital signal processing algorithms, and machine learning algorithms) and / or from user input. Compensation can be applied to the tool path parameters so that errors in spraying and / or deviations from intended spraying can be corrected in subsequent sprays.

Description

SPRAY TIP CALIBRATION

[0001] Priority Application

[0002] This application claims priority to and / or receives benefit from US Provisional Application No. 63 / 649,843, filed on 20 May 2024 and titled “SPRAY TIP CALIBRATION”. The US Provisional Application is hereby incorporated by reference in its entirety.

[0003] Technical Field

[0004] The present disclosure generally relates to robotic systems, and, more specifically, to calibration for a spraying end effector attached to a robotic arm.

[0005] Introduction

[0006] In construction, surface finishing, such as drywall finishing, is a labor intensive task that is often completed by skilled and experienced professionals. One part of drywall finishing is to apply a joint compound or material that can fill in some areas of the surface so that a smooth, flat finish can later be achieved by sanding. A professional may use a handheld spray gun to spray and apply a layer of joint compound onto the surface.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The various advantages and features of the present technology will become apparent by reference to specific implementations illustrated in the appended drawings. A person of ordinary skill in the art will understand that these drawings show only some examples of the present technology and would not limit the scope of the present technology to these examples. Furthermore, the skilled artisan will appreciate the principles of the present technology as described and explained with additional specificity and detail through the use of the accompanying drawings in which:

[0008] FIGS. 1 and 2 are exemplary perspective drawings of an exemplary robotic system, according to some aspects of the disclosed technology.

[0009] FIG. 3 is a block diagram illustrating components of an exemplary robotic system, according to some aspects of the disclosed technology.

[0010] FIG. 4 illustrates an exemplary spraying end effector spraying material in a fan, according to some aspects of the disclosed technology.

[0011] FIG. 5 illustrates an exemplary diagram of a seam sprayed with a band of material, according to some aspects of the disclosed technology.

[0012] FIG. 6 illustrates tip alignment issues with an exemplary spraying end effector, according to some aspects of the disclosed technology.

[0013] FIG. 7 illustrates a good spray burst or a calibrated spray burst, according to some aspects of the disclosed technology.

[0014] FIG. 8 illustrates a spray burst having a pitch offset in a first direction, according to some aspects of the disclosed technology.

[0015] FIG. 9 illustrates a spray burst having a pitch offset in a second direction, according to some aspects of the disclosed technology.

[0016] FIG. 10 illustrates a spray burst having a yaw offset in a first direction, according to some aspects of the disclosed technology.

[0017] FIG. 11 illustrates a spray burst having a yaw offset in a second direction, according to some aspects of the disclosed technology.

[0018] FIG. 12 illustrates a spray burst having a roll offset in a first direction, according to some aspects of the disclosed technology.

[0019] FIG. 13 illustrates a spray burst having a roll offset in a second direction, according to some aspects of the disclosed technology.

[0020] FIG. 14 illustrates a spray burst having a first deviation from expected fan width, according to some aspects of the disclosed technology.

[0021] FIG. 15 illustrates a spray burst having a second deviation from expected fan width, according to some aspects of the disclosed technology.

[0022] FIG. 16 is a block diagram illustrating components of an exemplary robotic system, according to some aspects of the disclosed technology.

[0023] FIG. 17 is a flow diagram illustrating a calibration method that utilizes user input, according to some aspects of the disclosed technology.

[0024] FIG. 18 is a flow diagram illustrating a calibration method that utilizes perception, according to some aspects of the disclosed technology.

[0025] FIG. 19 is a flow diagram illustrating method 1900 executing calibration procedure or a verification procedure, according to some aspects of the disclosed technology.

[0026] FIG. 20 depicts an exemplary first pattern type, according to some aspects of the disclosed technology.

[0027] FIG. 21 depicts an exemplary second pattern type, according to some aspects of the disclosed technology.

[0028] FIG. 22 depicts an exemplary first vertical measurement associated with pitch offset, according to some aspects of the disclosed technology.

[0029] FIG. 23 depicts an exemplary second vertical measurement associated with pitch offset, according to some aspects of the disclosed technology.

[0030] FIG. 24 depicts determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology.

[0031] FIG. 25 depicts an exemplary first horizontal measurement associated with yaw offset, according to some aspects of the disclosed technology.

[0032] FIG. 26 depicts an exemplary second horizontal measurement associated with yaw offset, according to some aspects of the disclosed technology.

[0033] FIG. 27 depicts determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology.

[0034] FIG. 28 depicts exemplary measurements associated with roll offset, and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology.

[0035] FIG. 29 depicts an exemplary measurement associated with deviation from expected fan width, according to some aspects of the disclosed technology.

[0036] FIG. 30 depicts an end effector and a depth sensor at a distal end of a robotic arm, according to some aspects of the disclosed technology.

[0037] FIG. 31 A depicts a measurement associated with pitch offset and determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology.

[0038] FIG. 31 B depicts a measurement associated with pitch offset and determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology.

[0039] FIG. 32A depicts a measurement associated with yaw offset and determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology.

[0040] FIG. 32B depicts a measurement associated with yaw offset and determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology.

[0041] FIG. 33A depicts a measurement associated with roll offset and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology.

[0042] FIG. 33B depicts a measurement associated with roll offset and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology.

[0043] FIG. 34A depicts a measurement associated with fan width, according to some aspects of the disclosed technology.

[0044] FIG. 34B depicts a measurement associated with fan width, according to some aspects of the disclosed technology.

[0045] FIGS. 34C-D depict determining a fan width and a thickness from sensor data, according to some aspects of the disclosed technology.

[0046] FIG. 35 is a flow diagram illustrating method 3500 executing calibration procedure or a verification procedure, according to some aspects of the disclosed technology.

[0047] FIG. 36 is a flow diagram illustrating method 3600 executing calibration procedure or a verification procedure, according to some aspects of the disclosed technology.

[0048] FIG. 37 is a block diagram of an exemplary computing device, according to some embodiments of the disclosure.DETAILED DESCRIPTION

[0049] The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configurations in which the subject technology may be practiced. The appended drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for the purpose of providing a more thorough understanding of the subject technology. However, it will be clear and apparent that the subject technology is not limited to the specific details set forth herein and may be practiced without these details. In some instances, structures and components are shown in block diagram form to avoid obscuring the concepts of the subject technology.

[0050] Overview

[0051] Misalignment, tip degradation, and manufacturing inconsistencies in a spraying end effector at a distal end of a robotic arm can cause unintended deviations to a width and / or thickness of sprayed band of material and / or cause an untended offset of the sprayed band of material relative to the intended center line.

[0052] Tool path represents the path through space and time that the end effector follows to cause an effect or generate an output, e.g., spraying a band of material having certain parameters such as a position of a center line, width of the band, length of the band, thicknesses of the band at various positions, etc. When the end effector is misaligned, it becomes more difficult for a robotic arm to consistently produce outputs that have accurate and / or precise characteristics. Repeatability can beimpacted when an effector becomes misaligned. Inaccurate spraying due to spray tip misalignment can cause inconsistencies or deviations between an expected / intended output of material from the spraying end effector (e.g., based on a model of tool path parameters and dynamics of the spraying end effector, robotic arm, and / or tool paths therefore) and what output is actually produced. Thus, misalignment of the tip can prevent such robotic systems from producing precise application of material. For a spraying end effector, unintended changes to the width and unintended offset can be undesirable and can cause inconsistent and inaccurate spraying of materials on surfaces. Inaccurate spraying may lead to wasteful use of materials. If parameters of the band of spray materials are incorrect, the robotic system may take longer to sand down the material to reach a smooth flat surface. In some cases, misalignment can result in unacceptable finish quality or coverage since the material is not deposited as planned. Misalignment may lead to additional rework, such as additional (manual) sanding and / or touch-up.

[0053] An end effector may be assumed to have a certain pose, e.g., having certain degrees alignment about the pitch, roll, and yaw axes or {0P°, 0y°, 0r°} and a certain position, e.g., {x,y,z}. Characteristics of results created by using the end effector can be expected based on the assumed pose and / or assumed position. In many scenarios, the assumed pose and / or assumed position may not align with the actual pose and / or position of the end effector. The results created by the end effector may have deviations from expected or intended characteristics. An end effector may degrade over time or during use. An end effector may have manufacturing inconsistencies across presumably identical end effectors. The consistency of material (e.g., viscosity, temperature, moisture content, curing speed, etc.) being applied by a spraying end effector may change over time. The material (e.g., viscosity, temperature, moisture content, curing speed, etc.) being applied by a spraying end effector can be inconsistent across presumably identical materials. Such issues may cause results to have deviations from expected or intended characteristics.

[0054] Calibration techniques can be implemented, e.g., at certain times, to determine the extent of deviations from expected or intended characteristics such that the deviations can be compensated for or reduced in subsequent uses of the end effector using appropriate measures.

[0055] A spraying end effector can be actuated to make one or more spray patterns. The spray pattern can be sensed, and the sensor data can be analyzed to extract characteristics of the spray pattern.

[0056] Calibration techniques can be implemented to determine one or more characteristics, such as pitch offset, yaw offset, roll offset, fan width, deviation from expected fan width, thickness, anddeviation from expected thickness from sensor data. The characteristics can be determined using algorithms (e.g., computer vision algorithms, digital signal processing algorithms, and machine learning algorithms). The characteristics of the produced spray pattern can be detected by a perception system.

[0057] In some cases, one or more of offsets in the pitch, yaw, and roll axes, fan width, deviation from expected fan width, thickness, and deviation from expected thickness, can be determined from user input. The characteristics of the produced spray pattern can be measured by a user and input into the robotic system.

[0058] The characteristics can be compared against the expected characteristics to determine the inconsistencies and deviations. Compensation to tool path parameters can be determined and applied based on the determined inconsistencies and deviations, errors in spraying can be corrected in subsequent sprays.

[0059] In some implementations, a calibration system can integrate sensors, such as cameras and depth sensors, to accurately measure spray characteristics, such as pitch, yaw, roll offsets, fan width deviations, and thickness variations. These measurements are processed using, e.g., computer vision techniques, digital signal processing algorithms, and machine learning models, to extract relevant features and determine calibration parameters. The system can apply compensation values to tool path parameters, ensuring that subsequent sprays align with intended specifications.

[0060] In some implementations, the calibration system can implement on-the-fly sensing during spraying. By monitoring spray characteristics (e.g., edge of spray) and comparing them to expected values, the calibration system can trigger a calibration. On-the-fly sensing minimizes disruptions to the spraying process, enabling seamless integration into workflows.

[0061] In some implementations, the calibration system can implement on-the-fly adjustments during spraying or during a spray job. By monitoring spray characteristics (e.g., edge of spray) and comparing them to expected values, the calibration system can update tool path parameters on-the-fly to maintain consistent application quality. On-the-fly adjustments minimize disruptions to the spraying process, enabling seamless integration into workflows.

[0062] In some implementations, the calibration system may extract other characteristics of the spraying end effector, such as tip wear and signs of clogging, from the sensor data. Detecting such characteristics can trigger one or more compensation actions to be performed, such as replacement of spraying end effector, a de-clogging procedure, etc.

[0063] Exemplary robotic systems, such as a robotic system suitable for surface finishing

[0064] FIGS. 1 and 2 are exemplary perspective drawings of an exemplary robotic system 100, according to some aspects of the disclosed technology. The robotic system 100 may be placed at a work site to perform surface finishing tasks. The robotic system 100 comprises one or more of: a base unit 101 , a robotic arm 108, and an end effector 116. The base unit 101 can include one or more of, among other things, a positioning system 102, a support 104 coupled to the positioning system 102, and a lift system 106 that can control the height of the support 104. Robotic arm 108 can include a base end 184 and a distal end 144. The end effector 116 may be coupled to the distal end 144 of the robotic arm 108. The base end of robotic arm 108 can be coupled to support 104.

[0065] The positioning system 102, the lift system 106, and the robotic arm 108 illustrate possible positioning mechanisms of a robotic system. Each positioning mechanism may have different degrees of freedom and / or limitations. The positioning mechanisms may cooperate to allow the end effector 116 to achieve a certain three-dimensional position within a work site. Other robotic positioning mechanisms are envisioned by the disclosure.

[0066] The positioning system 102 may change the (ground) position of the base unit 101 and can move the base unit 101. The positioning system 102 can be a coarse positioning system to mobilize the robotic system within a work site (enabling the end effector 116 to reach a region in space within the work site). The lift system 106 may change the height of support 104, such that the robotic arm 108 coupled to the support 104 may be able to reach higher regions in space within the work site. The robotic arm 108 may change a three-dimensional position of an end effector 116 within a three- dimensional space around the robotic system. The robotic arm can be a fine positioning system to mobilize the end effector 116 to a specific point in space within the work site.

[0067] The positioning system 102 may include a drive train system. As shown, the drive train system includes wheels. In some cases, the drive train system includes tracks. The drive train system is controllable to relocate the robotic system 100, on the ground, to and from different locations within an area. The drive train system may be controlled by a user. The positioning system 102 may navigate within the area autonomously (e.g., based on instructions or control signals generated by a control system).

[0068] In FIG. 1 , lift system 106 is in an unextended position. In FIG. 2, the lift system 106 is in an extended position, lifting the support 104 upwards. Lift system 106 may lift the support 104 up and down to change the height of the support 104. Lift system 106 can aid the robotic arm 108 to reach a larger range of positions.

[0069] The robotic arm 108 can comprise any suitable robotic arm or positioning stage system, which can include pneumatic actuators, electric actuators, and the like. Examples of robotic arm 108 includes articulated arm, cartesian robot arm, cylindrical robot arm, delta robot arm, spherical robot arm, Selective Compliance Articulated Robot Arm (SCARA), etc. Robotic arm 108 may include links joined together by arm joints. Robotic arm 108 can change the position of end effector 116 on the distal end of robotic arm 108 within a three-dimensional workspace of robotic arm 108. The robotic arm 108 can have any suitable number of degrees of freedom. In some embodiments, the distal end of robotic arm 108, e.g., a wrist of robotic arm 108, may be able to rotate or revolve the end effector 116. In some embodiments, the distal end of robotic arm 108, e.g., a wrist of robotic arm 108, may be able to change the angle or direction of the end effector 116. Robotic arm 108 may be controlled by a user. Robotic arm 108 may change position within the workspace autonomously (e.g., based on instructions or control signals generated by a control system). Other types of fine positioning mechanisms that can change one or more of the position, rotational position, and angular direction of the end effector 116 are envisioned by the disclosure.

[0070] In some embodiments, robotic system 100 can comprise one or more modular and / or multi-use end effector 116, which can be configured for various drywalling, construction, manufacturing, fabrication, or other tasks. For example, as discussed herein, end effectors such as end effector 116 can be configured for substrate planning, substrate hanging, applying coating or joint compound to hung substrate, spraying, sanding the coating, painting, scraping, smoothing, applying tape, drilling, vibrating, measuring, applying pressure, sculpting, and the like. Such end effectors may be selectively coupled to or decoupled from the robotic system 100 to configure it with an end effector corresponding to a particular task. In some cases, end effector 116 may include a plurality of selectively triggerable / controllable end effectors (e.g., end effectors may have electronic triggers to turn on or off, and / or electronic controls to modulate settings of a given end effector).

[0071] The robotic system 100 may include sensors 110, 112a, 112b, 114a, 114b, 186a, and 186b. Sensors can generate sensor data for a perception system. Sensors can generate sensor data for a localization system.

[0072] Sensor(s) 110 may include a distance / range sensor. Sensors 114a and 114b may include distance / range sensors. Examples of distance / range sensors may include, e.g., capacitive sensor, ultrasonics sensor, time-of-flight sensor, structured light sensor, light detection and ranging sensor (LIDAR), radio detection and ranging sensor (RADAR), etc. Sensors 110, 114a, and 114b maygenerate data that can measure the robotic system 100’s distance from a wall. Sensors 110, 114a, and 114b may generate data that can assist a localization system to determine the robotic system 100’s location within the worksite. Sensors 110, 114a, and 114b may generate data that can detect obstacles and / or other objects in the surroundings of the robotic system 100.

[0073] Sensors 112a and 112b may include a camera or imaging system (e.g. , infrared camera, thermal camera, stereo cameras, structured light camera, etc.). In one example, sensors 112a and 112b are 180-degree field of view cameras. Sensors 112a and 112b can capture images and video of the surroundings (almost 360-degree field of view) of the robotic system 100. The images and video may offer situation awareness of the robotic system 100.

[0074] Sensors 186a and 186b may include a camera or imaging system (e.g., infrared camera, thermal camera, stereo cameras, structured light camera, etc.). Sensor 186b is shown in dashed lines since sensor 186b is located on a different side of support 104 not seen in the perspective view. Sensors 186a and 186b may be positioned and configured to capture images or video of a surface in front of sensors 186a and 186b (e.g., a wall in front of robotic system 100 or a wall next to a side of robotic system 100). Images captured by sensors 186a and 186b may be provided to a perception system and / or a localization system.

[0075] In some cases, robotic system 100 may include cameras or imaging systems having a field of view pointing in any suitable direction away from robotic system 100. For example, robotic system 100 may include a camera or imaging system pointing upwards towards a ceiling. Robotic system 100 may include a camera or imaging system pointing downwards towards a floor. In some cases, robotic system 100 may include one or more cameras or imaging systems that can change its field of view (e.g., panning towards a different direction, zooming in or out, etc.).

[0076] Robotic system 100 may include one or more processors 172 and one or more non-transitory computer-readable media 174 to store instructions and / or data. The instructions may be executed by the one or more processors 172 to implement one or more functionalities relating to sensor data processing, localization, perception, planning, and controls. The data may include data generated by the sensors. The data may include data generated by the one or more processors 172.

[0077] Robotic system 100 may include an output device 170. The output device 170 may include a display, such as touch-sensitive screen. The output device 170 may include an audio speaker. The output device 170 may output (e.g., display) status information about the surface finishing information. The output device 170 may output audible information (e.g., speech, sound, etc.) to a user operatingthe robotic system 100. The audible information may include status information about the surface finishing information. The audible information may include audio instructions from a remote operator at a remote user input system 194. In some cases, the output device 170 may receive user input and operate as an input device as well.

[0078] Robotic system 100 may include a network adapter 180. Network adapter 180 may offer wireless and / or wired connectivity to the one or more processors 172 for computing devices which are near the robotic system 100 or computing devices remote from the robotic system 100. Network adapter 180 may be communicably coupled to a local area network (not shown explicitly in the FIG.). Network adapter 180 may be communicably coupled to a public communications network (e.g., cellular network 196).

[0079] In some embodiments, a local operator may operate and interact with robotic system 100 using a user input system 192 that is near robotic system 100 (e.g, at the same work site). The user input system 192 may be wirelessly communicably coupled with the one or more processors 172 via network adapter 180. The user input system 192 may be communicably coupled with the one or more processors 172 via a wired connection via network adapter 180. User input system 192 may be a mobile device, such as a smartphone or a tablet. User input system 192 may include user input interfaces and / or user output interfaces. User input system 192 may include a computing system. User input system 192 may have a graphical user interface. The graphical user interface may display information from systems such as perception system, localization system, planner, and controls. A local operator may provide user input using user input system 192. A local operator may send commands to robotic system 100 (e.g, to start execution of a task, to control positioning system 102, etc.) using user input system 192.

[0080] In some embodiments, a remote operator may remotely operate and interact with robotic system 100 using a remote user input system 194 that is remote from robotic system 100 (e.g, not at the work site). The remote user input system 194 may be wirelessly communicably coupled with the one or more processors 172 via network adapter 180, over a cellular network 196 (e.g, 5G cellular network). The remote user input system 194 may include a computing system. The remote user input system 194 may receive sensor data captured by sensors of the robotic system 100. The remote user input system 194 may implement similar functionalities as the user input system 192. Remote user input system 194 may include user input interfaces and / or user output interfaces. Remote user input system 194 may have a graphical user interface. The graphical user interface may display informationfrom systems such as perception system, localization system, planner, and controls. Graphical user interface may display video feeds from sensors 112a and 112b to monitor the surroundings of the robotic system. A remote operator may provide user input using remote user input system 194. A remote operator may send commands to robotic system 100 (e.g., to start execution of a task, to control positioning system 102, etc.) using remote user input system 194. In some cases, the remote user input system 194 may implement expert functionalities such as debugging of the robotic system. In some cases, the remote user input system 194 may implement expert functionalities such as controls of the robotic arm 108 and / or lift system 106.

[0081] FIG. 3 is a block diagram illustrating components of an exemplary robotic system, according to some aspects of the disclosed technology. Robotic system 300 can be a surface finishing system.

[0082] Robotic system 300 may include a base unit 370, a robotic arm 380, and one or more end effectors 350. For simplicity, the passages may refer to an end effector 350. The base unit 370 may include platform 302 and a cart 306, with a lift 304 disposed between platform 302 and cart 306. A base end 384 (end that is attached to platform 302) of robotic arm 380 may be physically coupled to the base unit 370, e.g, at platform 302, or at cart 306. Cart 306 can be disposed at or near the ground and may be movable by wheels 308 or other mechanisms such as a belt with treads. The lift 304 can raise platform 302 up and down, if used. In some examples, lift 304 may include a scissor lift or other suitable height adjustment mechanisms that can raise and lower platform 302 relative to cart 306.

[0083] In various embodiments, robotic arm 380 can comprise one or more suitable robotic arms or positioning systems, which can include pneumatic actuators, electric actuators, and the like. The robotic arm 380 can have any suitable number of degrees of freedom. Robotic arm 380 may include sensors to detect positioning of various parts of the robotic arm 380, and optionally objects or people in the vicinity of the robotic arm 380. In some cases, the robotic arm 380 may include a lifting mechanism or height adjustment mechanism, in combination with other positioning system(s) operable to adjust position of a distal end 382 of the robotic arm 380 in space.

[0084] In some cases, robotic arm 380 may be replaced by or supplemented with other types of positioning system such as a gantry (XY) positioning system (as illustrated in FIG. 4). A gantry positioning system may include a first end that is affixed to the base unit 370, and an attachment end. The gantry positioning system may have a first track for moving the attachment end in a first direction and a second track for moving the attachment end in a second direction perpendicular to the first direction. In some cases, the one or more end effectors 350 may be attached to the attachment end ofthe gantry positioning system, and the position of the one or more end effectors 350 may be adjusted using the gantry positioning system. In addition to the gantry positioning system, the robotic system 300 may include a fine movement component that connects the one or more end effectors 350 to the attachment end of the gantry positioning system. The fine movement component may adjust a position and / or pose the one or more end effectors 350. The fine movement component may rotate the orientation of one or more end effectors 350 (e.g. , to adjust a fan bias angle or roll angle of a spraying end effector). The fine movement component can perform a flicking movement in some cases.

[0085] An end effector 350 can be coupled or affixed at the distal end 382 of the robotic arm 380. In some examples, the robotic system 300 can comprise modular and / or multi-use end effectors 350, which can be configured for various material application tasks, wall, ceiling and / or floor preparation, wall, ceiling and / or floor finishing, drywalling tasks, construction tasks, or other tasks. For example, as discussed herein, end effectors 350 can be configured for substrate planning, substrate hanging, applying coating or joint compound to hung substrate, sanding the coating, painting, and the like. Although various examples herein relate to surface finishing, further embodiments of the robotic system 300 can be configured for any suitable tasks, including other construction tasks, manufacturing tasks, gardening tasks, farming tasks, inspection tasks, compliance tasks, planning tasks, domestic tasks, and the like. Accordingly, the discussions herein related to drywalling and construction should not be construed to be limiting on the wide variety of tasks that robotic system 300 can be configured for.

[0086] End effectors 350 can comprise various suitable devices, including a cutting device, hanging device, coating device, sanding device, painting device, vacuum device, a sprayer (or spraying device), fastening device, screwing device, fastener driver device, impact device, scraping device, vibrational device, tape application device, measuring device, material removal device, testing device, fan device, suction device, grinding device, heat application device, welding device, burning device, etching device, mixing device, sawing device, router or shaping device, tying device, bending device, riveting device, and the like. Other suitable devices can be part of an end effector 350 and can be selected based on any desired task that the end effector 350 may be used for.

[0087] Robotic system 300 may include sensors 344. Sensors may include one or more of: visible spectrum camera, radio detection and ranging (RADAR), LIDAR system, sound navigation and ranging (SONAR) system, camera (e.g., infrared camera, thermal camera, stereo cameras, structured light camera, and the like), laser scanners, time-of-flight sensors, inertial measurement unit (IMU), a vision system, an imaging system, capacitive sensors, temperature sensors, impedance sensors, pressuresensors, audio sensors, humidity sensor, air flow sensors, proximity sensors, laser curtains, force and torque sensors, limit switches, rotameter, spring and piston flow meter, ultrasonic flow meter, turbine meter, paddlewheel meter, variable area meter, positive displacement, vortex meter, pitot tube or differential pressure meters, magnetic meters, conductivity sensor, and depth or thickness sensors, and the like. In some cases, sensors 344 may be provided with robotic arm 380. In some cases, sensors 344 may be provided with the end effector 350.

[0088] In some embodiments, sensors 344 include a depth sensor, such as a time-of-flight sensor, or a three-dimensional (3D) laser point profile sensor. The laser point profile sensor is a high speed, noncontact measurement device designed for precise dimensional inspection. The laser point profile sensor utilizes laser triangulation technology to generate accurate 3D profiles of a target surface. The laser point profile sensor emits a laser beam that reflects off the target surface, and the reflected light is captured by an onboard camera. By analyzing the displacement of the laser line, the laser point profile sensor constructs a detailed 3D profile of the object's surface. The 3D profile can be in the form of depth measurements (representing a z-dimension) across the x and y dimensions. The data obtained from the laser point profile sensor can include dimensional measurements such as thickness, height, and surface roughness.

[0089] Robotic system 300 may include one or more input interfaces (not explicitly depicted in FIG. 3) that allow for various systems to couple with the robotic system 300. The one or more input interfaces may allow for resources provided by such systems to be provided to the robotic arm 380 and / or the end effector 350 coupled at a distal end 382 (end that has the end effector 350 affixed thereto) of the robotic arm 380. For example, a pneumatic source, a power source, a vacuum source, a paint source, a coating or joint compound source, or the like can be coupled to robotic system 300, at any suitable location of the robotic system 300.

[0090] For various end effectors 350, proper alignment relative to the distal end 382 of robotic arm 380 may impact accuracy and / or precision of the end effector 350 in performing the task. Components of end effectors 350 may become misaligned in pitch, roll, and / or yaw axes. Misalignment may be caused by use of one or more end effectors 350, change of one or more end effectors 350, and / or unintended movement of one or more end effectors 350. Distance of one or more end effectors 350 relative to the target surface may not be accurate. One or more end effectors 350 may undergo wear and tear, or other degradation from use or overtime, which may lead to offsets and / or deviation from expectedresults. One or more end effectors 350 may have manufacturing inconsistencies, which may lead to offsets and / or deviation from expected results.

[0091] In some embodiments, base unit 370 may include a coarse positioning system that can change the coarse position of the robotic system 300. In some embodiments, robotic arm 380 may serve as a fine positioning system that can change the fine position and pose of one or more end effectors 350. The position of one or more end effectors 350 may be defined in a three-dimensional coordinate system, having three coordinate values along three perpendicular axes, e.g., {x, y, z}. The pose of the one or more end effectors 350 may be defined according to pitch, roll, and yaw axes, e.g., {pitch degrees, roll degrees, and yaw degrees}. Pose may be measured relative to distal end 382 of robotic arm 380.

[0092] The robotic system 300 can further include a control system 334, which may include one or more of perception system 340, mapping and localization 392, planning 394, movement system 342, and actuation system 396. Control system 334 can perform one or more operations such as perception, mapping / localization, planning, actuation / movement control. The control system may perform operations that enable the robotic system 300 to operate autonomously or with little human inp ut / i ntervention . The control system 334 may cause robotic system 300 to carry out tasks, such as executing a calibration procedure, sensing characteristics of spray bursts / passes, determining calibration parameters, applying tool path parameters compensation, and executing a verification procedure.

[0093] Perception system 340 may receive input from the sensors 344 and perceive the environment surrounding the robotic system 300. Perception system 340 may determine location of seams of a wall assembly (or part of a building structure). Perception system 340 may determine the locations of structural components of a wall assembly (or part of a building structure). Perception system 340 may determine one or more characteristics of a target surface. Perception system 340 may determine one or more characteristics of one or more spray bursts applied to the target surface. Perception system 340 may determine one or more characteristics of one or more spray passes applied to the target surface.

[0094] Mapping and localization 392 may receive map data and / or input from sensors 344 and use the information to determine the robotic system 300 within a space. Mapping and localization 392 may construct a map representation of the environment of robotic system 300. The map may include a three-dimensional map of the environment. The map may include a floor plan or building plan. The mapmay capture spatial layout, including walls, floors, ceilings, obstacles, location of other robotic system(s), building features (windows, outlets, doorways), etc. Mapping and localization 392 may determine the position or location of robotic system 300 within the map, e.g., based on the map data and / or input from sensors 344. Mapping and localization 392 help robotic system 300 perform tasks at appropriate locations within the environment. Mapping and localization 392 may determine the position or location of robotic system 300 relative to a target surface, e.g., based on the map data and / or input from sensors 344.

[0095] Planning 394 may determine how robotic system 300 should perform a task or to execute an action. Planning 394 may receive information from perception system 340 and / or mapping and localization 392 to create a plan for executing a task. Planning 394 may receive feedback information from movement system 342 and / or actuation system 396. The plan may include tool path parameters. The plan may include requirements on the tool path parameters. Planning 394 may make decisions for how robotic system 300 would carry out the task or execute the action. Planning 394 may provide a plan to movement system 342 and / or actuation system 396. Planning 394 may update the plan based on new information from perception system 340 and / or mapping and localization 392. In some embodiments, planning 394 may determine calibration parameters based on user input and / or information determined in perception system 340. Planning 394 may apply tool path parameters compensation based on the calibration parameters.

[0096] Movement system 342 may generate commands based on the plan provided by planning 394. Movement system 342 may determine commands to actuate elements of the robotic system 300 to move one or more of: the robotic arm 380, lift 304, and wheels 308. Such commands may be generated for an electric motor, pneumatic actuators, piezoelectric actuator, and the like. In some cases, movement system 342 may command lift 304 to control the height of lift 304. In some cases, the movement system 342 may command wheels 308 to control the position and heading of the base unit 370. In some cases, the movement system 342 may command the robotic arm 380 to move, change the orientation of, rotate, or otherwise configure the end effector 350 (e.g., relative to a wall, ceiling, or floor). The commands may be transmitted as one or more signals to actuators of robotic system 300. Movement system 342 may cooperate with actuation system 396 to generate commands based on tool path parameters determined in planning 394 to execute a calibration procedure and / or a verification procedure. Movement system 342 may cooperate with actuation system 396 to generate commands based on tool path parameters determined in planning 394 to change the position and / or pose of oneor more end effectors 350. Movement system 342 may cooperate with actuation system 396 to generate commands based on tool path parameters determined in planning 394 to change the movement speed of one or more end effectors 350.

[0097] Actuation system 396 may generate commands based on the plan provided by planning 394. The commands may electronically trigger operations of the one or more end effectors 350. The commands may electronically change settings or operation modes of the one or more end effectors 350. The commands may be transmitted as one or more signals to one or more end effectors 350. Actuation system 396 may cooperate with movement system 342 to generate commands to execute a calibration procedure and / or a verification procedure. Actuation system 396 may cooperate with movement system 342 to generate commands based on tool path parameters determined in planning 394 to turn one or more end effectors 350 on or off at certain times.

[0098] Control system 334 may include a user interface 332 (e.g., communicably coupled to control system 334 via wired or wireless means). User interface 332 may generate output to one or more users of robotic system 300, via a graphical user interface provided by one or more user input / output devices. User interface 332 may receive user input from the one or more users of robotic system 300 via a graphical user interface provided by one or more user input / output devices.

[0099] Exemplary issues relating to accuracy and / or precision of an end effector

[0100] The robotic system 100 as illustrated in FIG. 1 and / or robotic system 300 (referred to herein as a robotic system or surface finishing system) illustrated in FIG. 3 can be used in a variety of situations, such as when to deposit material onto a surface with accuracy and / or precision. One example of such a situation is to accurately and selectively apply or deposit material in a region while avoiding depositing material in another region. Another situation is to selectively apply or deposit material with a certain thickness with accuracy. For instance, robotic system 100 may be used when selectively applying touch-up paint to a gap on a surface, when selectively covering seams with joint compound, and / or when spraying material to a certain thickness while avoiding areas that are already at a thickness level. To better understand one of such situations, the following passages describe joints and how bands of material can be sprayed to cover joints.

[0101] Joints are formed by abutting edges of adjacent components. For example, abutting edges of adjacent boards of substrate form a joint. The terms “joint” and “seam” are used interchangeably in the present disclosure. A tapered joint (also known as a “factory” joint) is formed by abutting tapered edges of adjacent components. A tapered joint creates a valley in which coating material can be applied tocreate a level surface relative to a face of the substrate. A butt joint is formed by abutting flat edges of adjacent components. In contrast to tapered joints, butt joints lack a formal valley in which a coating material can lie to create a level surface. Creating the appearance of flatness can be easier for tapered joints than it is for butt joints because the valley in tapered joints can hide much of the coating material. In contrast, for butt joints, much of the material extends beyond the plane of the surface of the substrate. A mixed joint (also known as a bastard joint) is formed by abutting a tapered edge and a flat edge of adjacent components. Mixed joints have only a portion of the valley for the coating material.

[0102] During coating work, a robotic system can apply a layer of coating material to joints that may have a thickness that is greater than is manually applied by human workers to allow for a sanding system (e.g., a sanding end effector) to sand down the compound to a desired plane. For example, in some examples, manual joint compound application mud can be profiled to taper from high points. The robotic system 100 can apply a thicker layer than normal, enabling a sanding system to sand down high points to be level to the adjacent surfaces.

[0103] FIG. 4 illustrates an exemplary diagram 400 of a spraying end effector 402 spraying material in a fan, according to some aspects of the disclosed technology. A spraying end effector 402 is located a distance, dsurface, from a surface 408 while spraying material in a fan 404 that has a spread angle, 02, and produces a spray pattern 406. The spray pattern 406 can be elliptical in shape. The spray pattern 406 has a major axis and a minor axis. The major axis has a corresponding major width, WMAJOR, measured along the major axis. The minor axis has a corresponding minor width, WMINOR, measured along the minor axis. The spraying end effector 402 can be rotated about an axis that passes through a center of the orifice of a spray tip of the spraying end effector 402 and the center of the fan 404; such rotation about the axis biases the spray pattern 406 at a fan bias angle, 9i. When the spraying end effector 402 is rotated at an angle of 0 or 360 degrees, the width of the material sprayed from the spraying end effector 402 is equal to the major width, WMAJOR. When the sprayed at an angle between 0 and 360 degrees, the width of the material sprayed from the spraying end effector 402 is less than the major width, WMAJOR.

[0104] The major width, WMAJOR and the minor width WMINOR are directly related to the distance dsurface of the spraying end effector 402. The major width, WMAJOR and the minor width WMINOR can be made bigger by increasing the distance dsurface of the spraying end effector 402 from the surface 408. The major width, WMAJOR and the minor width WMINOR can be made smaller by decreasing the distance dsurface of the spraying end effector 402 from the surface 408.

[0105] FIG. 5 illustrates an exemplary diagram 500 of a seam 502 sprayed with a band of material 504, according to some aspects of the disclosed technology. The band of material 504 can be formed by a spraying end effector (e.g., spraying end effector 402) when rotated at the fan bias angle, 61. The spraying end effector can moves in the direction indicated by arrow 510 while spraying the band of material 504. The fan bias angle 01, may be adjusted by changing the angle of the spraying end effector about the roll axis.

[0106] Because the spraying end effector 402 is rotated, the spray pattern 406 is biased at the fan bias angle, 01 and the effective width, WEFFECTIVE, of the spray pattern 406 is less than the major width, WMAJOR. Increasing the angle of the spraying end effector about the roll axis can increase the fan bias angle, 01 and reduce the effective width, WEFFECTIVE, of the spray pattern 406.

[0107] The thickness of the material deposited by the spraying end effector 402 varies based on, among other things, the speed at which the spraying end effector 402 moves in the direction of arrow 510 while spraying the material. For example, increasing the speed of the spraying end effector 402 reduces the thickness of the sprayed material and vice versa. A flow rate of the spraying end effector 402 can depend on the amount of pressure driving the material through the spraying end effector 402, material viscosity, a size of the orifice on the spray tip of spraying end effector 402, and an amount of wear around the orifice (e.g., more wear can increase the size of the orifice). Parameters of the sprayed material (e.g., effective width and thickness) are based on parameters of the spraying end effector (e.g., distance to the sprayed surface, spread angle of the fan, rotation of the end effector). Thus, surface finishing systems can vary the tool path parameters of the spraying end effector 402 to achieve desired or intended specifications of the sprayed material.

[0108] FIG. 6 illustrates tip alignment issues with an exemplary spraying end effector, according to some aspects of the disclosed technology. Diagram 600 shows a spraying end effector to spray a spray pattern 630 onto surface 620.

[0109] The spraying end effector can include a spray gun 602, and a manifold 680 (which is affixed to robotic arm 108 of FIGS. 1-2 or robotic arm 380 of FIG. 3). Manifold 680 may have (fluid) passages and ports to allow for various materials to be provided to and away from spray gun 602. A fluid supply hose having material to be sprayed (e.g., mud) may be connected to a port of manifold 680. Manifold 680 may have other passages or boreholes. Spray gun 602 may be mounted or mated with ports of the manifold 680. The manifold 680 may be mounted to a distal end 144 of robotic arm 108 of FIGS. 1-2 or distal end 382 of robotic arm 380 of FIG. 3).

[0110] The spraying end effector can include a spray guard 606 that can be fastened to the spray gun 602 with fastener (nut) 604. The spray guard 606 can be fully (and easily) rotatable in the roll axis while the fastener 604 is firmly fastened to the spray gun 602. When the spray guard 606 is rotated, the fan bias angle changes. When fan bias angle changes, the effective width WEFFECTIVE of the spray pattern 630 changes. Unintentional rotation of the spray guard 606 can cause unintentional changes to a width of a band of sprayed material.

[0111] The spraying end effector also includes spray tip 608 which can be inserted into a fitted opening of the spray gun. The spray tip 608 can include an orifice 614, e.g., in a cylindrical portion of the spray tip 608, that allows the spray gun to provide material through the orifice 614 and outward towards the surface 620. The spray tip 608 has a cylindrical portion, which allows the spray tip 608 to be rotatable in the pitch axis within the spray guard. The spray tip 608 can have an orifice 614 running along a line that is perpendicular to a long axis of a cylindrical portion of the spray tip 608, the orifice is to release material (from spray gun 602) in a fan (e.g., according to FIG. 4-6). Spray tip 608 can be inserted in spray guard 606 of the spraying end effector. The spray tip 608 may include a non-cylindrical portion that allows a user to hold, handle, or rotate the spray tip 608. The non-cylindrical portion may have one or more flat surfaces that a user may hold to rotate the spray tip 608.

[0112] When spray tip 608 is rotated, the direction of the orifice 614 can change (rotating upwards or rotating downwards) in the pitch axis. When the direction of orifice 614 changes, an offset occurs with the band of sprayed material formed with spray pattern 630. Unintentional rotation or deviation from the initial or proper alignment or position of the spray tip orifice 614 can cause a pitch offset from the intended center point of the band of sprayed material. While there may be mechanical features on the spray tip that may help to ensure a fixed alignment or direction of the orifice 614, those mechanical features may wear and tear over time (e.g, become loose), and their effectiveness to ensure alignment can degrade overtime.

[0113] After spray tip 608 is inserted into the fitted opening, the location of the orifice 614 can change (in the left or right directions) in the yaw axis or deviate from a center point. When the location of orifice 614 changes, an offset occurs with the band of sprayed material formed with spray pattern 630. Unintentional deviation of the position of the spray tip orifice 614 can cause a yaw offset from the intended center line of the band of sprayed material. While there may be mechanical features on the spray tip that may help to ensure a fixed position of the orifice 614, those mechanical features maywear and tear over time (e.g., become loose), and their effectiveness to ensure alignment can degrade overtime.

[0114] FIG. 7 illustrates a good spray burst or a calibrated spray burst, according to some aspects of the disclosed technology. The spray burst has a center point located at a target center point of the target surface, and has no offsets about the pitch, yaw, and roll axes. The spray burst has an expected spray width.

[0115] FIG. 8 illustrates a spray burst having a pitch offset in a first direction, according to some aspects of the disclosed technology. The spray burst has shifted upwards.

[0116] FIG. 9 illustrates a spray burst having a pitch offset in a second direction, according to some aspects of the disclosed technology. The spray burst has shifted downwards.

[0117] FIG. 10 illustrates a spray burst having a yaw offset in a first direction, according to some aspects of the disclosed technology. The spray burst has shifted to the right.

[0118] FIG. 11 illustrates a spray burst having a yaw offset in a second direction, according to some aspects of the disclosed technology. The spray burst has shifted to the left.

[0119] FIG. 12 illustrates a spray burst having a roll offset in a first direction, according to some aspects of the disclosed technology. The spray burst is rotated in a clockwise direction.

[0120] FIG. 13 illustrates a spray burst having a roll offset in a second direction, according to some aspects of the disclosed technology. The spray burst is rotated in a counterclockwise direction.

[0121] FIG. 14 illustrates a spray burst having a first deviation from expected fan width, according to some aspects of the disclosed technology. The spray burst has a wider (longer) spray width (shaded as gray) than the expected / intended spray width (with a dotted line as outline). It is not always necessary for the calibration system to establish an expected / intended / nominal spray width to implement the spray tip calibration procedure. The actual fan width can be measured, and the tool path parameters can be adjusted based on the measured actual fan width to achieve desired results of spraying.

[0122] FIG. 15 illustrates a spray burst having a second deviation from expected fan width, according to some aspects of the disclosed technology. The spray burst has a narrower (shorter) spray width (shaded as gray) than the expected / intended spray width (with a dotted line as outline). It is not always necessary for the calibration system to establish an expected / intended / nominal spray width to implement the spray tip calibration procedure. The actual fan width can be measured, and the tool path parameters can be adjusted based on the measured actual fan width to achieve desired results of spraying.

[0123] Exemplary robotic system having spray tip calibration

[0124] FIG. 16 is a block diagram illustrating components of an exemplary robotic system 1600, according to some aspects of the disclosed technology. Robotic system 1600 may include control system 334, one or more sensors 344, one or more user input / output devices 1664 providing user interface 332, one or more coarse positioning systems 1630, one or more fine positioning systems 1632, and one or more end effectors 350.

[0125] Control system 334 may include mapping and localization 392, movement system 342, user interface 332, perception system 340, planning 394, and actuation system 396. Control system 334 may include calibration manager 1696.

[0126] Calibration manager 1696 may be implemented as a manager to orchestrate a suitable sequence of calibration operations. One example of a sequence of calibration operations is illustrated as method 1700 of FIG. 17. Another example of a sequence of calibration operations is illustrated as method 1800 of FIG. 18. Another example of a sequence of calibration operations is illustrated as method 3500 of FIG. 35. Another example of a sequence of calibration operations is illustrated as method 3600 of FIG. 36.

[0127] In some cases, calibration manager 1696 may orchestrate calibration operations involving (autonomously) detecting one or more conditions for triggering tip calibration, executing a calibration procedure (e.g., illustrated as method 1900 of FIG. 19), determining characteristics of the result of the calibration procedure, determining calibration parameters, applying tool path parameters compensation, and executing a verification procedure (e.g., illustrated as method 1900 of FIG. 19). Calibration manager 1696 may trigger one or more components in control system 334 to complete one or more parts of the calibration operations.

[0128] In some cases, calibration manager 1696 may orchestrate calibration operations involving receiving user input requesting tip calibration, executing a calibration procedure (e.g., illustrated as method 1900 of FIG. 19), determining characteristics of the result of the calibration procedure based on user input, determining calibration parameters, applying tool path parameters compensation, and executing a verification procedure (e.g., illustrated as method 1900 of FIG. 19). Calibration manager 1696 may cause user interface 332 to display / output guidance, instructions, and / or prompts to guide a user to complete one or more parts of the calibration operations.

[0129] Calibration manager 1696 may be implemented to sense one or more conditions for triggering tip calibration. Calibration manager 1696 may track a state of robotic system 1600 to determine whether one or more conditions are met. Exemplary conditions may include:• First day in which robotic system 1600 is used at a particular worksite• Before first initial spray• After a spray tip change• After using the tip to spray a certain amount of material (e.g. , 2 drums of material)• After clearing a clog or executing of a de-clogging procedure• After sensing unintentional movement / jerking of the spraying end effector• After rotation or movement of the spray tip• After rotation of the spray tip in a closed position and back into the open position• After sensing by perception system 340 that resulting spray has deviations from expected characteristics (e.g., detected over spraying at intersections, detected non-overlapping bevel passes)• After detecting that verification procedure resulted in sprays that still had deviations from expected characteristics

[0130] Calibration manager 1696 may cooperate with movement system 342 to move robotic system 1600 and / or one or more end effectors 350 to cause one or more end effectors 350 to be at a distance that is to be used during normal operation for spraying material onto the target surface. Calibration manager 1696 may cooperate with actuation system 396 and / or movement system 342 to perform a calibration procedure (e.g., illustrated by method 1900 of FIG. 19). The calibration procedure is preferably performed at the distance that is to be used during normal operation for spraying material onto the target surface. Calibration manager 1696 may cooperate with actuation system 396 and / or movement system 342 to perform a verification procedure (e.g., illustrated by method 1900 of FIG. 19). The verification procedure is preferably performed at the distance that is to be used during normal operation for spraying material onto the target surface.

[0131] Mapping and localization 392 may receive information such as map data 1606 and / or sensor data from one or more sensors 344. Mapping and localization 392 may determine a position of the robotic system 1600 in three-dimensional space, e.g., such as position of robotic system 1600 within a worksite. Mapping and localization 392 may determine a location of target surfaces in three- dimensional space, such that the distance of an end effector relative to the target surface can bedetermined. Mapping and localization 392 may determine location of features of target surfaces (e.g., location and length of seams) in three-dimensional space and create a map of the features. Mapping and localization 392 may in some cases leverage information determined by perception system 340 to determine the location of the features.

[0132] Movement system 342 may transmit commands to one or more coarse positioning systems 1630 and / or one or more fine positioning systems 1632. In some embodiments, movement system 342 may transmit commands to one or more coarse positioning systems 1630 to move robotic system 1600 to a particular location of a worksite. A user of robotic system 1600 may provide user input 1604 via one or more user input / output devices 1664 to move robotic system 1600 using movement system 342. In some embodiments, movement system 342 may transmit commands to one or more fine positioning systems 1632 to adjust a position and / or pose of one or more end effectors 350 within the worksite and / or relative to a target surface.

[0133] Perception system 340 may include spray measurements determination 1682. Spray measurements determination 1682 may determine one or more characteristics of one or more sprays resulting from a calibration procedure. Spray measurements determination 1682 may determine one or more characteristics of one or more sprays resulting from a verification procedure.

[0134] Spray measurements determination 1682 may receive one or more camera images captured by one or more sensors 344 of a target surface at one or more points of the calibration procedure or verification procedure. One or more camera images may be processed or analyzed using one or more suitable computer vision techniques to identify or extract one or more key features of the one or more sprays. For example, the one or more camera images may undergo a semantic segmentation process to locate the sprays. The one or more camera images may undergo edge detection processing to locate the boundaries / contours of the sprays. Location of the sprays and / or boundaries of the sprays may be used to determine locations or points of the key features. One or more lengths / distances may be measured between key features. The lengths / distances, measured in pixels, may be translated into lengths / distances, e.g., through matrix transformation, according to a reference frame of robotic system 1600.

[0135] Spray measurements determination 1682 may receive 3D profile data captured by one or more sensors 344 of a target surface at one or more points of the calibration procedure or verification procedure. The 3D profile data may include depth measurements (representing a z-dimension) across the x and y dimensions. 3D profile data may be processed or analyzed using one or more suitabledigital signal processing techniques to identify or extract one or more key features of the one or more sprays (e.g., starting location, ending location, a center location, an edge point, a corner, a boundary, a centerline, etc.). For example, the 3D profile data may undergo a semantic segmentation process to locate the one or more sprays. The 3D profile data may undergo a template fitting process to locate the one or more sprays. In some implementations, an edge detection algorithm can be applied to the 3D profile data to identify the boundaries of the one or more sprays. In some implementations, a surface fitting algorithm (e.g., least square fitting or polynomial fitting) can be applied to the 3D profile data to identify the boundaries of the one or more sprays. In some implementations, a point cloud processing algorithm (filtering, segmentation, and registration) can be applied to the 3D profile data to identify the boundaries of the one or more sprays. In some implementations, a machine learning model (e.g., neural networks, random sample consensus model, support vector machines, K-means clustering, etc.) can be applied to the 3D profile data to identify the boundaries of the one or more sprays by classifying whether a particular depth measurement belongs to the one or more sprays or a surface on which the one or more sprays were applied. Locating the one or more sprays may include determining the boundaries / contours of the one or more sprays. Location of the sprays and / or boundaries of the sprays may be used to determine locations or points of the key features. One or more lengths / distances may be measured between key features. In some implementations, the 3D profile data before spraying may be subtracted from the 3D profile data after spraying to assess thickness of the one or more sprays. In some implementations, a two-dimensional (2D) characteristic thickness profile may be determined by averaging the depth data or thickness data over a certain distance along the x dimension or the y dimension. The lengths / distances, measured in the coordinate system of the 3D profile data, may be translated into physical offsets / lengths / distances, e.g., through matrix transformation, according to a reference frame of robotic system 1600.

[0136] Perception system 340 may include spray order determination 1684. Spray order determination 1684 may determine the order in which the sprays occurred in the calibration procedure. Spray order determination 1684 may determine the order in which the sprays occurred in the verification procedure. Spray order determination 1684 may receive one or more camera images captured by one or more sensors 344 of a target surface at one or more points of the calibration procedure or verification procedure, e.g., before and / or after each spray. Spray order determination 1684 may determine differences between the camera images by comparing pixel values. Spray order determination 1684 may determine whether the differences exceed a certain threshold. Spray order determination 1684may identify a region in the camera image associated with a new spray (thus deriving that spray order) if the differences exceed the threshold. Spray order determination 1684 may receive a camera image captured by one or more sensors 344 of a target surface at after the calibration procedure or verification procedure and determine whether the sprays captured in the camera image matches which one of two spray pattern types (exemplary spray pattern types are illustrated in FIGS. 20-21). Spray order determination 1684 may apply an image classifier to classify the spray pattern type. Spray order determination 1684 may derive the spray order based on the pattern type. The order in which the sprays occurred, or the spray order, may be used in determining a direction or a sign of an offset.

[0137] In some embodiments, user interface 332 may receive user input 1604 provided via user input / output devices 1664 selecting a pattern type, which may indicate the order in which the sprays occurred in the calibration procedure and / or verification procedure.

[0138] In some embodiments, user interface 332 may receive user input 1604 provided via user input / output devices 1664 indicating one or more lengths / distances associated with one or more sprays occurring in the calibration procedure and / or verification procedure.

[0139] Planning may include calibration parameters determination 1686. Calibration parameters determination 1686 may receive the one or more lengths / distances determined by spray measurements determination 1682. In some cases, calibration parameters determination 1686 may receive the one or more lengths / distances received by user interface 332. Calibration parameters determination 1686 may receive the spray order determined by spray order determination 1684. In some cases, calibration parameters determination 1686 may receive the spray order inferred from the pattern type selected in user input 1604. Calibration parameters determination 1686 may perform one or more calculations to determine one or more of: pitch offset, yaw offset, roll offset, actual fan width, and actual thickness.

[0140] Planning may include tool path parameters compensation 1688. Tool path parameters compensation 1688 may calculate compensation values to be applied to tool path parameters to compensate for or reduce one or more of: pitch offset, yaw offset, roll offset, actual fan width, and actual thickness.

[0141] Exemplary calculations performed by calibration parameters determination 1686 and / or tool path parameters compensation 1688 are described and illustrated with FIGS. 20-29.

[0142] Exemplary spray tip calibration techniques

[0143] FIG. 17 is a flow diagram illustrating calibration method 1700 that utilizes user input, according to some aspects of the disclosed technology. Method 1700 may be performed by calibration manager 1696 of FIG. 16. Calibration manager 1696 can orchestrate one or more components in robotic system 1600 to perform one or more operations illustrated in FIG. 17.

[0144] In 1702, a robotic system may be positioned at a suitable location for calibration to occur. A calibration manager may use the user interface to prompt the user to relocate or position the robotic system and / or the spraying end effector to a designated location. A user may provide user input via the user interface to relocate or position the robotic system and / or the spraying end effector to the designated location. The control system of the robotic system may transmit commands to a coarse positioning system and / or a fine positioning system to relocate or position the robotic system and / or the spraying end effector. Preferably, the suitable location or designated location is the location that the robotic system and / or the spraying end effector is expected to be positioned for normal operation.

[0145] In 1704, the calibration manager may configure the user interface to prompt the user to confirm whether to perform calibration or to skip calibration. The calibration manager may receive user input via the user interface requesting to perform calibration.

[0146] In 1706, the calibration manager may configure the user interface to prompt the user to initiate calibration, e.g., displaying a “START” button. The calibration manager may receive user input via the user interface initiating calibration.

[0147] In 1708, the calibration manager may cause a control system of the robotic system to execute a calibration procedure (e.g., method 1900 of FIG. 19).

[0148] In 1710, the calibration manager may configure the user interface to prompt the user to select one of two pattern types. A pattern type may indicate a particular spray order. The calibration manager may receive user input via the user interface identifying a particular pattern type. In some embodiments, the calibration manager may receive user input via the user interface specifying a particular spray order in lieu of a pattern type selection. Exemplary pattern types are illustrated in FIGS. 20-21. In some embodiments, 1710 may be omitted when calibration manager may use a perception system to determine the spray order.

[0149] In 1712, the calibration manager may configure the user interface to prompt the user to measure and input a vertical distance. The vertical distance may indicate a pitch offset. The calibration manager may receive user input having the vertical distance via the user interface. Exemplary vertical distances are illustrated in FIGS. 22-23.

[0150] In 1714, the calibration manager may configure the user interface to prompt the user to measure and input a horizontal distance. The horizontal distance may indicate a yaw offset. The calibration manager may receive user input having the horizontal distance via the user interface. Exemplary horizontal distances are illustrated in FIGS. 23-26.

[0151] In 1716, the calibration manager may configure the user interface to prompt the user to measure and input a fan width. The calibration manager may receive user input having the fan width via the user interface.

[0152] In some cases, the calibration manager may configure the user interface to prompt the user to measure and input spray thickness. The calibration manager may receive user input having the spray thickness via the user interface.

[0153] In some cases, the calibration manager may configure the user interface to prompt the user to measure and input roll measurements. The roll measurements may indicate a roll offset. The calibration manager may receive user input having the roll measurements via the user interface. Exemplary roll measurements are illustrated in FIGS. 28.

[0154] In 1718, the calibration manager may trigger a planning system to determine calibration parameters based on the user input. The planning system may determine one or more calibration parameters, such as pitch offset, yaw offset, roll offset, actual fan width, and actual spray thickness, based on the user input. Exemplary calculations are described and illustrated in FIGS. 20-29.

[0155] In 1720, the calibration manager may trigger a planning system to apply tool path parameters compensation based on the calibration parameters. The planning system may determine one or more calibration values based on the calibration parameters.

[0156] In some cases, the planning system may perform sanity checks to determine whether one or more of the calibration parameters (e.g., pitch offset, yaw offset, roll offset, actual fan width, and actual spray thickness) are within a correctable range. If a calibration parameter is outside of a correctable range, a maximum compensation value may be applied. Optionally, the calibration manager may configure the user interface to display / output a warning that the calibration parameter is out of range and that a maximum compensation value is applied. The calibration manager may configure the user interface to request confirmation from the user to proceed with the maximum compensation value. In some cases, if a calibration parameter is outside of a correctable range, the calibration manager may configure the user interface to display / output a recommendation to replace the spray tip and request the user to return to 1704 after the spray tip has been manually replaced.

[0157] Rather than using uncalibrated tool path parameters, the calibration values determined from the calibration parameters are applied to produce calibrated tool path parameters. Calibrated tool path parameters, when used, can cause resulting sprays to have expected characteristics. The calibrated tool path parameters can lead to sprays with reduced or minimized pitch offset, yaw offset, roll offset, and have a desired fan width. Specifically, for performing targeted spraying on factory seams and butt seams, the calibrated tool path parameters can lead to coatings on the factory seams and butt seams to have desirable characteristics, such as a precise width about a centerline of the seam, located at a precise distance from the centerline of the seam, have a precise overlap between overlapping (bevel) passes, have a precise fan width, starts and end precisely at a seam intersection, etc. Exemplary calculations are described and illustrated in FIGS. 20-29.

[0158] In 1722, the calibration manager may trigger configure the user interface to prompt the user to initiate verification / validation, e.g., displaying a “START” button. The calibration manager may receive user input via the user interface initiating verification / validation. The calibration manager may cause a control system of the robotic system to execute a verification procedure (e.g., method 1900 of FIG. 19) using calibrated tool path parameters. The result of the verification procedure, if calibrated tool path parameters are appropriately determined to minimize one or more of pitch offset, yaw offset, roll offset, and optionally achieve a desired fan width, would have spray bursts that are aligned and on top of each other, and optionally have the desired fan width. A user may confirm alignment of spray bursts and desired fan width of the result.

[0159] In some cases, 1722 may be optional.

[0160] 1706-1722 may be repeated if the result of the verification procedure did not result in spray bursts that are aligned and on top of each other or did not have the desired fan width.

[0161] FIG. 18 is a flow diagram illustrating calibration method 1800 that utilizes perception, according to some aspects of the disclosed technology. Method 1800 may be performed by calibration manager 1696 of FIG. 16. Calibration manager 1696 can orchestrate one or more components in robotic system 1600 to perform one or more operations illustrated in FIG. 18.

[0162] In 1802, the calibration manager may determine that one or more conditions for triggering calibration is met.

[0163] In 1804, the calibration manager may position the robotic system and / or the end effector to a designated location. The control system of the robotic system may transmit commands to a coarse positioning system and / or a fine positioning system to relocate or position the robotic system and / or thespraying end effector. Preferably, the designated location is the location that the robotic system and / or the spraying end effector is expected to be positioned for normal operation.

[0164] In 1806, the calibration manager may cause a control system of the robotic system to execute a calibration procedure (e.g., method 1900 of FIG. 19). In some embodiments, the calibration manager may initiate the calibration procedure in response to determining that the environment is safe to perform a calibration procedure. In some embodiments, the calibration manager may initiate the calibration procedure in response to receiving user input indicating that the calibration procedure may begin.

[0165] In 1808, the calibration procedure may trigger a perception system to determine characteristics of the spray pattern. Characteristics may include a spray order, lengths / distances of key features of one or more sprays, etc.

[0166] In 1810, the calibration manager may trigger a planning system to determine calibration parameters based on the determined characteristics. The planning system may determine one or more calibration parameters, such as pitch offset, yaw offset, roll offset, and actual fan width, based on the determined characteristics. Exemplary calculations are described and illustrated in FIGS. 20-29.

[0167] In 1812, the calibration manager may trigger a planning system to apply tool path parameters compensation based on the calibration parameters in a similar or same manner as 1720.

[0168] In 1814, the calibration manager may cause a control system of the robotic system to execute a verification procedure (e.g., method 1900 of FIG. 19) using calibrated tool path parameters. In some embodiments, the calibration manager may initiate the verification procedure in response to determining that the environment is safe to perform a verification procedure. In some embodiments, the calibration manager may initiate the verification procedure in response to receiving user input indicating that the verification procedure may begin. The result of the verification procedure, if calibrated tool path parameters are appropriately determined to minimize one or more of pitch offset, yaw offset, roll offset, and optionally achieve a desired fan width, would have spray bursts that are aligned and on top of each other, and optionally have the desired fan width. A perception system may determine from one or more camera images whether the spray bursts are aligned and whether a desired fan width is achieved.

[0169] In some cases, 1814 may be optional.

[0170] 1806-1814 may be repeated if the result of the verification procedure did not result in spray bursts that are aligned and on top of each other or did not have the desired fan width.

[0171] FIG. 19 is a flow diagram illustrating method 1900 executing a calibration procedure or a verification procedure, according to some aspects of the disclosed technology. The calibration procedure or verification procedure preferably involves two or more spray bursts that hold one or more tool path parameters constant while varying one or more tool path parameters to expose pitch offset, yaw offset, roll offset. In some cases, a third spray burst is included to assess easily (by a user or a perception system) the spray order. The calibration procedure or verification procedure involves at least one spray burst to expose actual fan width.

[0172] In 1902, a spraying end effector is actuated to perform a first spray burst (“SPRAY 1 ”). The tool path parameters of the first spray burst can include a fixed spraying duration (e.g. 0.25ms), 0° pitch degrees, 0° yaw degrees, 0° roll degrees, x1-coordinate position, y1 -coordinate position, z1 -coordinate position. {x1 , y1 , and z1 } may be set such that the spraying end effector is at a specific distance from the target surface (e.g, 0.8m), and at a specific height from the ground (e.g, 0.6m).

[0173] In 1904, a spraying end effector is rotated and actuated to perform a second spray burst (“SPRAY 2”). The tool path parameters of the second spray burst can include a fixed spraying duration (e.g. 0.25ms), 0° pitch degrees, 0° yaw degrees, 180° roll degrees, x1 -coordinate position, y1- coordinate position, z1-coordinate position. {x1, y1 , and z1 } may be set such that the spraying end effector is at a specific distance from the target surface (e.g, 0.8m), and at a specific height from the ground (e.g, 0.6m).

[0174] Measuring lengths / distances at key points between SPRAY 1 and SPRAY 2 can expose pitch offset and yaw offset. Measuring lengths / distances at key points of SPRAY 1 or key points of SPRAY 2 can reveal roll offset and actual fan width. The spray order may indicate the direction / sign of the pitch offset and yaw offset. The tilting of a spray may indicate the direction / sign of roll offset.

[0175] In 1906, a spraying end effector is moved and actuated to perform a second spray burst (“SPRAY 3”). The tool path parameters of the second spray burst can include a fixed spraying duration (e.g. 0.25ms), 0° pitch degrees, 0° yaw degrees, 180° roll degrees, x1 -coordinate position, y2- coordinate position, z1-coordinate position. {x1 , y2=y1 +OSPRAY_OFFSET, and z1} may be set such that the spraying end effector is at a specific distance from the target surface (e.g, 0.8m), at a specific height from the ground (e.g, 0.6m), and shifted to the right by a horizontal distance DSPRAY_OFFSET.

[0176] SPRAY 3 can indicate the spray order of the resulting spray patterns without requiring tracking of the spray order during the calibration / verification procedure. Depending on the direction / sign of the pitch offset, SPRAY 1 may be on top of SPRAY 2, or SPRAY 2 may be on top of SPRAY 1 . Byapplying SPRAY 3, SPRAY 2 will be accompanied by SPRAY 3 (offset by the horizontal distance DSPRAYJJFFSET), and it would be possible for a user or a perception system to ascertain which spray was SPRAY 1 and which spray was SPRAY 2 (without needing to keep track of the spray order during the calibration / verification procedure). The spray order may be used to ascertain the direction / sign of the pitch offset. The spray order may be used to ascertain the direction / sign of the yaw offset.

[0177] In some embodiments, 1906 is optional.

[0178] When performing method 1900 as a calibration procedure, 1906 may be skipped, and it is determined by a perception system and / or by a user whether SPRAY 1 and SPRAY 2 are aligned and on top of each other, and optionally whether the actual fan width of SPRAY 1 and SPRAY 2 is achieved.

[0179] Exemplary pattern types identifying spray ordering

[0180] FIG. 20 depicts an exemplary first pattern type, according to some aspects of the disclosed technology. FIG. 21 depicts an exemplary second pattern type, according to some aspects of the disclosed technology. The first pattern type, “double on the bottom” may indicate a first spray order. The first pattern type, “double on the bottom” may indicate that the pitch offset direction / sign is positive. The second pattern type, “double on top” may indicate a second spray order. The second pattern type, “double on top” may indicate that the pitch offset direction / sign is negative. A variable pitch_sign may be set to +1 if the second pattern type is selected, and the variable pitch_sign may be set to -1 if the first pattern type is selected.

[0181] Exemplary vertical measurements and compensation of pitch offset

[0182] FIG. 22 depicts an exemplary first vertical measurement associated with pitch offset, according to some aspects of the disclosed technology. FIG. 23 depicts an exemplary second vertical measurement associated with pitch offset, according to some aspects of the disclosed technology. First vertical measurement or second vertical measurement may measure a length / distance mpitch. A calibration parameter for pitch offset, dpitch, can be determined according to the following relationship:

[0183] The pitch offset calibration parameter dpitcflmay be half of the measured length / distance dpitch-

[0184] FIG. 24 depicts determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology. The spraying end effector pose (e.g., the pitch angle of thedistal end of the robotic arm) may be adjusted using the pitch compensation value. The pitch compensation value 9pitchmay be determined according to the following relationship:

[0185] The pitch compensation value 9p itchmay be the arctan of the pitch offset calibration parameter dpitch divided by the distance from the target surface dsurface. When applying pitch compensation value 9p itch, pitch_sign may be taken into account to ensure the pitch compensation is applied in the appropriate direction.

[0186] When applying the pitch compensation value 9p itch, the pitch compensation value 9pitch(e.g., pitch compensation value 9pitch* pitch_sign) may be added to the intended pitch angle tool path parameter to produce a calibrated pitch angle tool path parameter.

[0187] Exemplary horizontal measurements and compensation of yaw offset

[0188] FIG. 25 depicts an exemplary first horizontal measurement associated with yaw offset, according to some aspects of the disclosed technology. FIG. 26 depicts an exemplary second horizontal measurement associated with yaw offset, according to some aspects of the disclosed technology.

[0189] First horizontal measurement or second horizontal measurement may measure a length / distance myaw. A calibration parameter for yaw offset, dyaw, can be determined according to the following relationship: myaw — 2 ■ dyaw+ DSPRAY_OFFSET

[0190] The yaw offset calibration parameter dyawmay be half of the measured length / distance dyawsubtracted by the DSPRAYOFFSET- The yaw offset calibration parameter dyawhas a sign, which is equivalent to the direction / sign of the yaw offset.

[0191] FIG. 27 depicts determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology. The spraying end effector pose (e.g., the yaw angle of the distal end of the robotic arm) may be adjusted using the yaw compensation value. The yaw compensation value 9yawmay be determined according to the following relationship:

[0192] The yaw compensation value 9yawmay be the arctan of the yaw offset calibration parameter dyawdivided by the distance from the target surface dsurface. When applying pitch compensation value 9yaw, the sign of yaw compensation value 9yawmay be taken into account to ensure the yaw compensation is applied in the appropriate direction.

[0193] When applying the yaw compensation value 9yaw, the yaw compensation value 9yawmay be added to the intended yaw angle tool path parameter to produce a calibrated yaw angle tool path parameter.

[0194] Exemplary roll measurements and compensation of roll offset

[0195] FIG. 28 depicts exemplary measurements associated with roll offset, and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology. A first roll measurement may be mrouhof a spray burst (e.g. , SPRAY 1). A second roll measurement may be mrM hof the spray burst (e.g., SPRAY 1).

[0196] Manner of tilt of SPRAY 1 (e.g., left end higher than the right end, or right end higher than the left end) may determine the roll sign, e.g., rolLsign. RolLsign may be set to +1 if the left end is higher than the right end of SPRAY 1 . RolLsign may be set to -1 if the right end is higher than the left end of SPRAY 1.

[0197] Manner of tilt of SPRAY 2 (e.g., left end higher than the right end, or right end higher than the left end) may determine the roll sign, e.g., rolLsign. RolLsign may be set to -1 if the left end is higher than the right end of SPRAY 2. RolLsign may be set to -1 if the right end is higher than the left end of SPRAY 2.

[0198] Manner of tilt of SPRAY 3 (e.g., left end higher than the right end, or right end higher than the left end) may determine the roll sign, e.g., rolLsign. RolLsign may be set to -1 if the left end is higher than the right end of SPRAY 3. RolLsign may be set to -1 if the right end is higher than the left end of SPRAY 3.

[0199] The spraying end effector pose (e.g., the roll angle of the distal end of the robotic arm) may be adjusted using the roll compensation value. The roll compensation value 9roamay be determined according to the following relationship:

[0200] The roll compensation value 9roamay be the arctan of the first roll measurement mrou _hdivided by the second roll measurement mroU w. When applying roll compensation value 9roll,roll_sign may be taken into account to ensure the roll compensation is applied in the appropriate direction.

[0201] When applying the roll compensation value eron, the roll compensation value eroa(e.g., roll compensation value erM* roll_sign) may be added to the intended roll angle tool path parameter to produce a calibrated roll angle tool path parameter.

[0202] In some cases, more than one set of the first roll measurement and the second roll measurement of different sprays may be made to obtain a higher strength signal of the roll compensation value. The roll compensation value determined from individual sets can be averaged.

[0203] Exemplary fan width measurements and compensation of fan width or fan width error / deviation

[0204] FIG. 29 depicts an exemplary measurement associated with fan width, or an exemplary measurement associated with a deviation from expected fan width, according to some aspects of the disclosed technology. It is not always necessary for the calibration system to establish an expected / intended / nominal spray width to implement the spray tip calibration procedure. The actual fan width can be measured, and the tool path parameters can be adjusted based on the measured actual fan width to achieve desired results of spraying. A fan width measurement may be mspray widthof a spray burst (e.g., SPRAY 1). A fan width measurement may be mspray widthof SPRAY 2. A fan width measurement may be mspray widthof SPRAY 3. In some cases, more than one spray width measurement may be made to obtain a higher strength signal of the actual spray width. The actual spray width determined from individual sets can be averaged.

[0205] The fan width measurement mspray widthmay be used to adjust a distance of a spraying end effector from the target surface. If the fan width measurement mspray widtflis longer than the expected fan width, the distance of the spraying end effector from the target surface may be reduced, such that a desired effective width of a spray is achieved. If the fan width measurement mspray widthis shorter than the expected fan width, the distance of the spraying end effector from the target surface may be increased, such that a desired effective width of a spray is achieved.

[0206] The fan width measurement mspray widthmay be used to adjust an offset of a spray from a centerline of a seam. If the fan width measurement mspray widthis longer than the expected fan width, the offset of the spray may be reduced, such that a desired effective width of two parallel overlapping sprays is achieved. If the fan width measurement mspray widtflis shorter than theexpected fan width, the offset of the spray may be increased, such that a desired effective width of two parallel overlapping sprays is achieved.

[0207] The fan width measurement mspray widthmay be used to adjust a roll angle of a bevel (tilted) spray. If the fan width measurement mspray widthis longer than the expected fan width, the roll angle of the bevel spray may be increased, such that a desired width of a spray is achieved. If the fan width measurement mspray widthis shorter than the expected fan width, the roll angle of the spray may be increased, such that a desired effective width of two parallel overlapping sprays is achieved. The calibrated roll angle tool path parameter may be based on an arccos of a desired width bevel_width, I bevel width \ divided by the an width measurement mspray width, or arccos I - - - I.\mspray width /

[0208] Spray tip calibration system using a depth sensor and implementing on-the-fly sensing and / or adjustments

[0209] FIG. 30 depicts system 3000 having one or more end effector 350 and depth sensor 3002 at distal end 382 of robotic arm 380, according to some aspects of the disclosed technology. Depth sensor 3002 may include a 3D laser point profile sensor as described herein. Distal end 382 may be operated within the 3D space according to a reference coordinate system. Depth sensor 3002 may be affixed next to or adjacent to one or more end effectors 350 at distal end 382.

[0210] In some implementations, depth sensor 3002 may be calibrated relative to distal end 382 such that the relationship or transformation (e.g., the rotation and translation) that relates the coordinate system of depth sensor 3002 to the coordinate system of distal end 382 (the reference coordinate system) is known. The relationship or transformation may be defined as a transformation matrix. A calibrated depth sensor 3002 may obviate the need to perform an initial scan of the target surface before spraying. A calibrated depth sensor 3002 may obviate the need to make multiple sprays at differently, z} positions to establish expected characteristics of the spray pattern.

[0211] In some implementations, depth sensor 3002 may be operated to obtain sensor data when end effector 350 is not use. In some implementations, depth sensor 3002 may be operated to obtain sensor data while one or more end effectors 350 is in use (e.g., spraying material onto a surface).

[0212] The configuration depicted in FIG. 30 of having one or more end effectors 350 and depth sensor 3002 next to each other at distal end 382 can have one or more technical advantages.

[0213] In some implementations, depth sensor 3002 can be actuated to implement on-the-fly sensing during spraying. Depth sensor 3002, in cooperation with spray measurements determination 1682 andcalibration parameters determination 1686 of FIG. 16, can sense / monitor / measure one or more spray characteristics (e.g., edge of spray, fan width, thickness distribution) and compare them to expected values. In particular, depth sensor 3002 can measure characteristics while the spraying operation is ongoing. Calibration parameters determination 1686 and calibration manager 1696 may detect a significant deviation from an expected value, such as a significant offset in the spray's alignment, an unexpected change in fan width, irregularities in thickness, or a deviation exceeds a threshold, a spray tip calibration procedure can be triggered or initiated by calibration manager 1696 of FIG. 16. On-the-fly sensing minimizes disruptions to the spraying process, enabling seamless integration into workflows. This capability ensures that any errors or inconsistencies in the spray application are identified and corrected promptly, without requiring manual intervention or halting the spraying process.

[0214] In some implementations, the on-the-fly adjustments during spraying or during a spray job. Depth sensor 3002, in cooperation with spray measurements determination 1682 and calibration parameters determination 1686 of FIG. 16, can sense / monitor / measure one or more spray characteristics (e.g., edge of spray, fan width, and thickness distribution) and comparing them to expected values. Tool path parameters compensation 1688 can update tool path parameters on-the-fly to maintain consistent application quality. The tool path parameters can be updated while the spraying end effector is actively operating. The tool path parameters can be updated for the next spray during the spray job. By continuously comparing these real-time measurements to predefined expected values or specifications, the system can identify deviations or inconsistencies in the spray application. When deviations are detected, the calibration system can immediately update the tool path parameters of the spraying end effector without interrupting the spraying process. Tool path parameters may include adjustments to the position, orientation, spraying speed, or distance of the spraying end effector relative to the target surface. These updates ensure that the spray characteristics remain aligned with the intended specifications, maintaining consistent application quality across the surface. For example, if the edge of the spray begins to shift due to a yaw offset or if the fan width deviates from the expected value, the system can recalibrate the spraying end effector in real-time to correct these issues. On-the- fly adjustments allow the system to seamlessly integrate calibration into the spraying operation, eliminating the need for interruptions. On-the-fly adjustments not only enhance operational efficiency but also ensures that the material application remains uniform and accurate throughout the job. Moreover, on-the-fly adjustment capability reduces material waste and minimizes the need for rework, such as sanding or touch-ups, by ensuring that the sprayed material is applied precisely as intended.On-the-fly adjustment can also enable the system to adapt to changes in environmental conditions, material properties, or tip wear during the spraying process, further improving the reliability and consistency of the application. On-the-fly adjustments minimize disruptions to the spraying process, enabling seamless integration into workflows. This capability ensures that any errors or inconsistencies in the spray application are identified and corrected in real-time, without requiring manual intervention or halting the spraying process.

[0215] Exemplary pitch measurements and compensation of pitch offset

[0216] FIG. 31 A depicts a measurement associated with pitch offset and determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray burst or a calibrated spray burst. The solid line area represents an actual spray burst having a pitch offset. Based on 3D profile data, a length / distance mPitchcanbe measured based on two points: the origin or location of where the desired spray burst is expected and a center point of the actual spray burst. In some cases, the two points at which the length / distance is measured can vary, as long as the length / distance mpitchcan be deduced or derived from the measured length / distance. For example, a length / distance can be measured based on two points: the origin or location of where the desired spray burst is expected and an edge of the actual spray burst. The measured length / distance mpitchcan directly correspond to the calibration parameter for pitch offset, dpitch(as previously illustrated in FIG. 24). Once the calibration parameter for pitch offset, dpitchis determined, the spraying end effector pose (e.g., the pitch angle of the distal end of the robotic arm) may be adjusted using the pitch compensation value as described in FIG. 24.

[0217] FIG. 31 B depicts a measurement associated with pitch offset and determining a pitch compensation value based on the pitch offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray pass or a calibrated spray pass. The solid line area represents an actual spray pass having a pitch offset. Based on 3D profile data, a length / distance mPitch can be measured based on two points: the origin or location of where the desired spray pass is expected and a starting point of the actual spray pass. In some cases, the two points at which the length / distance is measured can vary, as long as the length / distance mpitchcan be deduced or derived from the measured length / distance. For example, a length / distance can be measured based on two points: the origin or location of where the desired spray pass is expected and a starting edge point of the actual spray pass. In another example, a length / distance can be measured based on two points: the origin or location of where the desired spray pass is expected and an end point of the actual spraypass. In another example, a length / distance can be measured based on two points: the origin or location of where the desired spray pass is expected and an ending edge point of the actual spray pass. The measured length / distance mpitchas depicted can directly correspond to the calibration parameter for pitch offset, dpitch(as previously illustrated in FIG. 24). Once the calibration parameter for pitch offset, dpitchis determined, the spraying end effector pose (e.g, the pitch angle of the distal end of the robotic arm) may be adjusted using the pitch compensation value as described in FIG. 24.

[0218] Exemplary yaw measurements and compensation of yaw offset

[0219] FIG. 32A depicts a measurement associated with yaw offset and determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray burst or a calibrated spray burst. The solid line area represents an actual spray burst having a yaw offset. Based on 3D profile data, a length / distance myawcan be measured based on two points: the origin or location of where the desired spray burst is expected and a center point of the actual spray burst. In some cases, the two points at which the length / distance is measured can vary, as long as the length / distance myawcan be deduced or derived from the measured length / distance. For example, a length / distance can be measured based on two points: the origin or location of where the desired spray burst is expected and an edge point of the actual spray burst. The measured length / distance myawcan directly correspond to the calibration parameter for yaw offset, dyaw(as previously illustrated in FIG. 27). Once the calibration parameter for yaw offset, dyawis determined, the spraying end effector pose (e.g, the yaw angle of the distal end of the robotic arm) may be adjusted using the yaw compensation value as described in FIG. 27.

[0220] FIG. 32B depicts a measurement associated with yaw offset and determining a yaw compensation value based on the yaw offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray pass or a calibrated spray pass. The solid line area represents an actual spray pass having a yaw offset. Based on 3D profile data, a length / distance myawcan be measured based on two points: the origin or location of where the desired spray pass is expected and a starting point of the actual spray pass. In some cases, the two points at which the length / distance is measured can vary, as long as the length / distance myawcan be deduced or derived from the measured length / distance. For example, a length / distance can be measured based on two lines: a center line of a desired spray pass and a centerline of the actual spray pass. In another example, a length / distance can be measured based on two points: the origin or location of where thedesired spray pass is expected and an edge point the actual spray pass. The measured length / distance myawas depicted can directly correspond to the calibration parameter for yaw offset, dyaw(as previously illustrated in FIG. 27). Once the calibration parameter for yaw offset, dyawis determined, the spraying end effector pose (e.g., the yaw angle of the distal end of the robotic arm) may be adjusted using the yaw compensation value as described in FIG. 27.

[0221] Exemplary roll measurements and compensation of roll offset

[0222] FIG. 33A depicts a measurement associated with roll offset and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray burst or a calibrated spray burst. The solid line area represents an actual spray burst having a roll offset.

[0223] Based on 3D profile data, a first length / distance mroa halong the height dimension or y-axis can be measured based on two points: the left edge of the spray burst and the right edge of the spray burst. In another example, the first length / distance mrouhalong the height dimension or y-axis can be measured based on two points: the origin or location of where the desired spray burst is expected and the left edge of the spray burst. In another example, the first length / distance mroa_halong the height dimension or y-axis can be measured based on two points: the origin or location of where the desired spray burst is expected and the right edge of the spray burst.

[0224] Based on 3D profile data, a second length / distance mroil walong the width dimension or x- axis can be measured based on two points: the left edge of the spray burst and the right edge of the spray burst. In another example, the second length / distance mrM walong the width dimension or x- axis can be measured based on two points: the origin or location of where the desired spray burst is expected and the left edge of the spray burst. In another example, the second length / distance mroll walong the width dimension or x-axis can be measured based on two points: the origin or location of where the desired spray burst is expected and the right edge of the spray burst.

[0225] Manner of tilt of the spray burst (e.g., left end higher than the right end, or right end higher than the left end) may determine the roll sign, e.g., rol l_sig n . RolLsign may be set to +1 if the left edge is higher than the right edge. RolLsign may be set to -1 if the right edge is higher than the left edge.

[0226] The spraying end effector pose (e.g., the roll angle of the distal end of the robotic arm) may be adjusted using the roll compensation value. The roll compensation value erMmay be determined according to the relationship as illustrated in FIG. 28. Once the calibration parameter for roll offset,9roUis determined, the spraying end effector pose (e.g., the roll angle of the distal end of the robotic arm) may be adjusted using the roll compensation value as described in FIG. 28.

[0227] FIG. 33B depicts a measurement associated with roll offset and determining a roll compensation value based on the roll offset, according to some aspects of the disclosed technology. The dotted line area represents a desired spray pass or a calibrated spray pass. The solid line area represents an actual spray pass having a roll offset.

[0228] Based on 3D profile data, a first length / distance mroll halong the height dimension or y-axis can be measured based on two points: the left lower corner of the spray pass and the right lower corner of the of the spray pass. In another example, the first length / distance mroll halong the height dimension or y-axis can be measured based on two points: the origin or location of where the desired spray pass is expected and the left lower corner of the spray pass. In another example, the first length / distance mroll_halong the height dimension or y-axis can be measured based on two points: the origin or location of where the desired spray pass is expected and the right lower corner of the spray pass. Other pairs of points can be used, as long as a height measurement of a right triangle (one leg of the right triangle) formed by the roll offset can be derived or deduced.

[0229] Based on 3D profile data, a second length / distance mrou _walong the width dimension or x- axis can be measured based on two points: the left lower corner of the spray pass and the right lower corner of the spray pass. In another example, the second length / distance mroll walong the width dimension or x-axis can be measured based on two points: the origin or location of where the desired spray pass is expected and the left lower corner of the spray pass. In another example, the second length / distance mrou_walong the width dimension or x-axis can be measured based on two points: the origin or location of where the desired spray pass is expected and the right lower corner of the spray pass. Other pairs of points can be used, as long as a width measurement of a right triangle (the other leg of the right triangle) formed by the roll offset can be derived or deduced.

[0230] Manner of tilt of the spray pass (e.g., left lower corner higher than the right lower corner, or right lower corner higher than the left lower corner) may determine the roll sign, e.g, roll_sign. Roll_sign may be set to +1 if the left lower corner is higher than the right lower corner. Roll_sign may be set to -1 if the right lower corner is higher than the left lower corner.

[0231] The spraying end effector pose (e.g, the roll angle of the distal end of the robotic arm) may be adjusted using the roll compensation value. The roll compensation value 0roamay be determined according to the relationship as illustrated in FIG. 28. Once the calibration parameter for roll offset,9roUis determined, the spraying end effector pose (e.g., the roll angle of the distal end of the robotic arm) may be adjusted using the roll compensation value as described in FIG. 28.

[0232] Exemplary fan width measurements, thickness measurements, and compensation of fan width or fan width deviation and thickness or thickness deviation

[0233] FIG. 34A depicts a measurement associated with fan width, according to some aspects of the disclosed technology. A fan width measurement may be mspray widthof a spray burst, which is a length / distance between the left edge of the spray burst and the right edge of the spray burst. It is not always necessary for the calibration system to establish an expected / intended / nominal spray width to implement the spray tip calibration procedure. The actual fan width can be measured, and the tool path parameters can be adjusted based on the measured actual fan width to achieve desired results of spraying. In some cases, fan width measurement may be twice the length / distance between the center point of the spray burst and the left edge of the spray burst. In some cases, fan width measurement may be twice the length / distance between the center point of the spray burst and the right edge of the spray burst. The fan width measurement mspray widtflmay be used to adjust one or more tool path parameters as described in FIG. 29.

[0234] FIG. 34B depicts a measurement associated with fan width, according to some aspects of the disclosed technology. A fan width measurement may be mspray widthof a spray pass, which is a length / distance between the left boundary of the spray pass and the right boundary of the spray pass. In some cases, fan width measurement may be twice the length / distance between the centerline of the spray pass and the left boundary of the spray pass. In some cases, fan width measurement may be twice the length / distance between the centerline of the spray pass and the right boundary of the spray pass. The fan width measurement mspray widthmay be used to adjust one or more tool path parameters as described in FIG. 29.

[0235] FIGS. 34C-D depict determining a fan width and a thickness from sensor data, according to some aspects of the disclosed technology. Sensor data, such as 3D profile data, can be generated using a depth sensor scanning of a spray burst as illustrated in FIG. 34A, or a spray pass as illustrated in FIG. 34B. It is not always necessary for the calibration system to establish an expected / intended / nominal spray width and / or an expected / intended / nominal spray thickness to implement the spray tip calibration procedure. The actual fan width and / or the actual spray thickness can be measured, and the tool path parameters can be adjusted based on the measured actual fan width and / or the actual spray thickness to achieve desired results of spraying.

[0236] In particular, FIG. 34C shows depth data before spray and after spray in the z-axis, across the x-axis, averaged over the y-axis for a certain distance or range on the y-axis (e.g., a width of a spray burst, or a length of a spray pass). FIG. 34D shows the differential depth data representing the depth data after spray subtracted by the depth data before spray.

[0237] The differential depth data can be used to extract a spray width measurement, mspray width, from the differential depth data. One or more features can be extracted from the depth data, such as the left edge / boundary of the spray, the center point or centerline of the spary, and the right edge / boundary of the spray. The one or more features can be used to derive a fan width measurement mspray widtfl, as discussed in FIGS. 34A-34B.

[0238] The differential depth data can be used to extract a spray thickness measurement, mthickness, from the differential depth data. One or more features can be extracted from the depth data, such as the left edge / boundary of the spray, and the right edge / boundary of the spray. In some cases, a transition zone along the x-axis from no spray to a spray with significant or substantial non-zero thickness may be identified, and the midpoint of the transition zone may be used as the edge / boundary of the spray. The features, such as the edges / boundaries of the spray, can be used to mask the differential depth data to form a data set of thickness measurements. The spray thickness measurement can be derived from the data set, such as a mean, median, or mode of the data set, or a mean, median, or mode of the data set with outliers or noise removed.

[0239] In some cases, the differential depth data may be processed by a template fitting algorithm and / or data smoothing algorithm to form a sanitized data set or a model of the spray. The features can be extracted from the sanitized data or the model to derive spray width and / or spray thickness.

[0240] The differential depth data can be used to extract smoothness / roughness or variance from the data set of thickness measurements. The differential depth data can be used to extract skewness from the data set of thickness measurements.

[0241] One or more of spray thickness measurement, smoothness / roughness or variance, skewness, can be used to adjust one or more tool path parameters, and / or spray patterns to be used to perform targeted spraying. One example of the one or more tool path parameters is speed. If the thickness measurement is too thick at the speed S, or is greater than the desired thickness, the speed of the robotic arm may be increased. If the thickness measurement is too thin, or is less than the desired thickness, the speed of the robotic arm may be decreased. Another example of one or more tool path parameters is pass offsets, e.g., how far apart two parallel spray passes are offset from each other.Another example of one or more tool path parameters is the number of spray passes. Another example of one or more tool path parameters is sprayer angle (e.g., roll angle). If the thickness measurement is too thick at the speed S, or is greater than the desired thickness, the roll angle of the robotic arm may be decreased. If the thickness measurement is too thin, or is less than the desired thickness, the roll angle of the robotic arm may be increased. Another example of one or more tool path parameters is sprayer distance (e.g., a distance from the target surface). If the thickness measurement is too thick at the speed S, or is greater than the desired thickness, the distance of the robotic arm may be decreased. If the thickness measurement is too thin, or is less than the desired thickness, the distance angle of the robotic arm may be increased.

[0242] Exemplary spray tip calibration techniques using a depth sensor

[0243] FIG. 35 is a flow diagram illustrating method 3500 executing calibration procedure or a verification procedure, according to some aspects of the disclosed technology. Method 3500 may be performed by calibration manager 1696 of FIG. 16. Calibration manager 1696 can orchestrate one or more components in robotic system 1600 to perform one or more operations illustrated in FIG. 35.

[0244] In some embodiments, the calibration manager may determine that one or more conditions for triggering calibration is met.

[0245] In some embodiments, the calibration manager may position the robotic system and / or the end effector to a designated location. The control system of the robotic system may transmit commands to a coarse positioning system and / or a fine positioning system to relocate or position the robotic system and / or the spraying end effector. Preferably, the designated location is the location that the robotic system and / or the spraying end effector is expected to be positioned for normal operation.

[0246] In some embodiments, the calibration manager may cause a control system of the robotic system to execute a calibration procedure (e.g., 3502, 3504, and 3506 of FIG. 35). In some embodiments, the calibration manager may initiate the calibration procedure in response to determining that the environment is safe to perform a calibration procedure. In some embodiments, the calibration manager may initiate the calibration procedure in response to receiving user input indicating that the calibration procedure may begin.

[0247] In 3502 of the calibration procedure, the calibration manager may trigger a sensor to perform a scan of the target surface and perception system to determine characteristics of the target surface prior to spraying. In some implementations, 3502 is optional.

[0248] In 3504 of the calibration procedure, the calibration manager may actuate a spraying end effector to perform a spray burst. The tool path parameters of the spray burst can include a fixed spraying duration (e.g. 0.25ms), 0° pitch degrees, 0° yaw degrees, 0° roll degrees, x1 -coordinate position, y1 -coordinate position, z1-coordinate position. {x1, y1, and z1 } may be set such that the spraying end effector is at a specific distance from the target surface (e.g., 0.8m), and at a specific height from the ground (e.g., 0.6m).

[0249] In 3508, the calibration manager may trigger a perception system (e.g., perception system 340, or more specifically, spray measurements determination 1682) to determine one or more characteristics of the spray burst based on the sensor data collected in 3502 and / or 3506. Characteristics may include lengths / distances of key features of the spray burst. Characteristics may include thickness of the spray burst.

[0250] In 3510, the calibration manager may trigger a planning system (e.g., planning 394, or more specifically, calibration parameters determination 1686) to determine calibration parameters based on the determined characteristics. The planning system may determine one or more calibration parameters, such as pitch offset, yaw offset, roll offset, actual fan width, and actual thickness based on the determined characteristics. Exemplary calculations are described and illustrated in FIGS. 31 A, 32A, 33A, and 34A.

[0251] In 3512, the calibration manager may trigger a planning system (e.g., planning 394, or more specifically, tool path parameters compensation 1688) to apply tool path parameters compensation based on the calibration parameters. The planning system may determine one or more calibration values based on the calibration parameters.

[0252] In 3514, the calibration manager may cause a control system of the robotic system to execute a verification procedure (e.g, performing a verification procedure according to 3502, 3504, and 3506 of method 3500) using calibrated tool path parameters. In some embodiments, the calibration manager may initiate the verification procedure in response to determining that the environment is safe to perform a verification procedure. In some embodiments, the calibration manager may initiate the verification procedure in response to receiving user input indicating that the verification procedure may begin. The result of the verification procedure, if calibrated tool path parameters are appropriately determined to minimize one or more of pitch offset, yaw offset, roll offset, optionally achieve a desired fan width and optionally achieve a desired thickness, the verification spray burst would achieveexpected characteristics. A perception system may determine from 3D profile data of the verification spray burst whether expected characteristics are achieved by performing a similar operation as 3508.

[0253] In some cases, 3514 may be optional. In some cases, 3514 may be repeated if one or more expected / desired characteristics are not achieved.

[0254] FIG. 36 is a flow diagram illustrating method 3600 executing calibration procedure or a verification procedure, according to some aspects of the disclosed technology. Method 3500 may be performed by calibration manager 1696 of FIG. 16. Calibration manager 1696 can orchestrate one or more components in robotic system 1600 to perform one or more operations illustrated in FIG. 36.

[0255] In some embodiments, the calibration manager may determine that one or more conditions for triggering calibration is met.

[0256] In some embodiments, the calibration manager may position the robotic system and / or the end effector to a designated location. The control system of the robotic system may transmit commands to a coarse positioning system and / or a fine positioning system to relocate or position the robotic system and / or the spraying end effector. Preferably, the designated location is the location that the robotic system and / or the spraying end effector is expected to be positioned for normal operation.

[0257] In some embodiments, the calibration manager may determine that operations of method 3500 have been performed to correct pitch offset and / or yaw offset, and trigger operations of method 3600 to begin.

[0258] In some embodiments, the calibration manager may cause a control system of the robotic system to execute a calibration procedure (e.g. , 3602, 3604, and 3606 of FIG. 35). In some embodiments, the calibration manager may initiate the calibration procedure in response to determining that the environment is safe to perform a calibration procedure. In some embodiments, the calibration manager may initiate the calibration procedure in response to receiving user input indicating that the calibration procedure may begin.

[0259] In 3602 of the calibration procedure, the calibration manager may trigger a sensor to perform a scan of the target surface and perception system to determine characteristics of the target surface prior to spraying. In some implementations, 3602 is optional.

[0260] In 3604 of the calibration procedure, the calibration manager may actuate a spraying end effector to perform a spray pass. The tool path parameters of the spray pass can include a fixed spraying duration (e.g. 200ms), a speed S, at 0° pitch degrees, 0° yaw degrees, 0° roll degrees, starting at x1 -coordinate position, y1 -coordinate position, z1 -coordinate position, and ending at x1 -coordinate position, y2-coordinate position (y2=y1 length of spray pass), and z1 -coordinate position. {x1 , y1, and z1} and {x1 , y2, and z1} may be set such that the spraying end effector is at a specific distance from the target surface (e.g., 0.8m), and at a specific height from the ground (e.g., 0.6m). In some cases, the tool path parameters of the spray pass can include a fixed spraying duration (e.g. 200ms), a speed S, at 0° pitch degrees, 0° yaw degrees, 0° roll degrees, starting at x1-coordinate position, y1-coordinate position, z1-coordinate position, and ending at x2-coordinate position (x2=x1 length of spray pass), y1-coordinate position, and z1-coordinate position. {x1 , y1 , and z1 } and {x2, y1, and z1} may be set such that the spraying end effector is at a specific distance from the target surface (e.g., 0.8m), and at a specific height from the ground (e.g., 0.6m). Speed S can be in the range of 0.1-0.2m / s. Length of spray pass can be in the range of 0.1 -0.5m.

[0261] In 3608, the calibration manager may trigger a perception system (e.g., perception system 340, or more specifically, spray measurements determination 1682) to determine one or more characteristics of the spray pass based on the sensor data collected in 3602 and / or 3606.Characteristics may include lengths / distances of key features of the spray burst. Characteristics may include thickness of the spray burst.

[0262] In 3610, the calibration manager may trigger a planning system (e.g., planning 394, or more specifically, calibration parameters determination 1686) to determine calibration parameters based on the determined characteristics. The planning system may determine one or more calibration parameters, such as pitch offset, yaw offset, roll offset, actual fan width, and actual thickness based on the determined characteristics. Exemplary calculations are described and illustrated in FIGS. 31 B, 32B, 33B, and 34B.

[0263] In 3612, the calibration manager may trigger a planning system (e.g., planning 394, or more specifically, tool path parameters compensation 1688) to apply tool path parameters compensation based on the calibration parameters. The planning system may determine one or more calibration values based on the calibration parameters.

[0264] In 3614, the calibration manager may cause a control system of the robotic system to execute a verification procedure (e.g., performing a verification procedure according to 3502, 3504, and 3506 of method 3500) using calibrated tool path parameters. In some embodiments, the calibration manager may initiate the verification procedure in response to determining that the environment is safe to perform a verification procedure. In some embodiments, the calibration manager may initiate the verification procedure in response to receiving user input indicating that the verification procedure maybegin. The result of the verification procedure, if calibrated tool path parameters are appropriately determined to minimize one or more of pitch offset, yaw offset, roll offset, optionally achieve a desired fan width and optionally achieve a desired thickness, the verification spray pass would achieve expected characteristics. A perception system may determine from 3D profile data of the verification spray pass whether expected characteristics are achieved by performing a similar operation as 3508.

[0265] In some cases, 3614 may be optional. In some cases, 3614 may be repeated if one or more expected / desired characteristics are not achieved.

[0266] Exemplary computing device

[0267] FIG. 37 is a block diagram of an apparatus or a system, e.g., an exemplary computing device 3700, according to some embodiments of the disclosure. One or more computing devices 3700 may be used to implement the functionalities described with the FIGS, and herein. A number of components are illustrated in FIG. 37. can be included in the computing device 3700, but any one or more of these components may be omitted or duplicated, as suitable for the application. In some embodiments, some or all of the components included in the computing device 3700 may be attached to one or more motherboards. In some embodiments, some or all of these components are fabricated onto a single system on a chip (SoC) die. Additionally, in various embodiments, the computing device 3700 may not include one or more of the components illustrated in FIG. 37, and the computing device 3700 may include interface circuitry for coupling to the one or more components. For example, the computing device 3700 may not include a display device 3706, and may include display device interface circuitry (e.g., a connector and driver circuitry) to which a display device 3706 may be coupled. In another set of examples, the computing device 3700 may not include an audio input device 3718 or an audio output device 3708 and may include audio input or output device interface circuitry (e.g., connectors and supporting circuitry) to which an audio input device 3718 or audio output device 3708 may be coupled.

[0268] The computing device 3700 may include a processing device 3702 (e.g., one or more processing devices, one or more of the same types of processing device, one or more of different types of processing device). The processing device 3702 may include electronic circuitry that process electronic data from data storage elements (e.g., registers, memory, resistors, capacitors, quantum bit cells) to transform that electronic data into other electronic data that may be stored in registers and / or memory. Examples of processing device 3702 may include a CPU, a GPU, a quantum processor, a machine learning processor, an artificial intelligence processor, a neural network processor, an artificial intelligence accelerator, an application specific integrated circuit (ASIC), an analog signal processor, ananalog computer, a microprocessor, a digital signal processor, a field programmable gate array (FPGA), a tensor processing unit (TPU), a data processing unit (DPU), etc.

[0269] The computing device 3700 may include a memory 3704, which may itself include one or more memory devices such as volatile memory (e.g., DRAM), nonvolatile memory (e.g., read-only memory (ROM)), high bandwidth memory (HBM), flash memory, solid state memory, and / or a hard drive.Memory 3704 includes one or more non-transitory computer-readable storage media. In some embodiments, memory 3704 may include memory that shares a die with the processing device 3702.

[0270] In some embodiments, memory 3704 includes one or more non-transitory computer-readable media storing instructions executable to perform operations described herein, such as the method 1700 of FIG. 17, method 1800 illustrated in FIG. 18, method 1900 illustrated in FIG. 19, method 3500 illustrated in FIG. 35, and method 3600 illustrated in FIG. 36.

[0271] Memory 3704 may store instructions that encode one or more exemplary parts. Exemplary parts, such as one or more parts of control system 334 may be encoded as instructions and stored in memory 3704 are depicted. The instructions stored in the one or more non-transitory computer- readable media may be executed by processing device 3702.

[0272] In some embodiments, memory 3704 may store data, e.g., data structures, binary data, bits, metadata, files, blobs, etc., as described with the FIGS, and herein. Exemplary data, such as sensor data, user input, tool path parameters, etc., may be stored in memory 3704.

[0273] In some embodiments, memory 3704 may store one or more machine learning models (and or parts thereof) that are used in at least one of control system 334 (e.g., in perception system 340).Memory 3704 may store training data for training the one or more machine learning models. Memory 3704 may store input data, output data, intermediate outputs, intermediate inputs of one or more machine learning models. Memory 3704 may store instructions to perform one or more operations of the machine learning model. Memory 3704 may store one or more parameters used by the machine learning model. Memory 3704 may store information that encodes how processing units of the machine learning model are connected with each other.

[0274] In some embodiments, the computing device 3700 may include a communication device 3712 (e.g., one or more communication devices). For example, the communication device 3712 may be configured for managing wired and / or wireless communications for the transfer of data to and from the computing device 3700. The term "wireless" and its derivatives may be used to describe circuits, devices, systems, methods, techniques, communications channels, etc., that may communicate datathrough the use of modulated electromagnetic radiation through a nonsolid medium. The term does not imply that the associated devices do not contain any wires, although in some embodiments they might not. The communication device 3712 may implement any of a number of wireless standards or protocols, including but not limited to Institute for Electrical and Electronic Engineers (IEEE) standards including Wi-Fi (IEEE 802.10 family), IEEE 802.16 standards (e.g., IEEE 802.16-2005 Amendment), Long-Term Evolution (LTE) project along with any amendments, updates, and / or revisions (e.g., advanced LTE project, ultramobile broadband (UMB) project (also referred to as "3GPP2"), etc.). IEEE 802.16 compatible Broadband Wireless Access (BWA) networks are generally referred to as WiMAX networks, an acronym that stands for worldwide interoperability for microwave access, which is a certification mark for products that pass conformity and interoperability tests for the IEEE 802.16 standards. The communication device 3712 may operate in accordance with a Global System for Mobile Communication (GSM), General Packet Radio Service (GPRS), Universal Mobile Telecommunications System (UMTS), High Speed Packet Access (HSPA), Evolved HSPA (E-HSPA), or LTE network. The communication device 3712 may operate in accordance with Enhanced Data for GSM Evolution (EDGE), GSM EDGE Radio Access Network (GERAN), Universal Terrestrial Radio Access Network (UTRAN), or Evolved UTRAN (E-UTRAN). The communication device 3712 may operate in accordance with Code-division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Digital Enhanced Cordless Telecommunications (DECT), Evolution-Data Optimized (EV-DO), and derivatives thereof, as well as any other wireless protocols that are designated as 3G, 4G, 5G, and beyond. The communication device 3712 may operate in accordance with other wireless protocols in other embodiments. The computing device 3700 may include an antenna 3722 to facilitate wireless communications and / or to receive other wireless communications (such as radio frequency transmissions). The computing device 3700 may include receiver circuits and / or transmitter circuits. In some embodiments, the communication device 3712 may manage wired communications, such as electrical, optical, or any other suitable communication protocols (e.g., the Ethernet). As noted above, communication device 3712 may include multiple communication chips. For instance, a first communication device 3712 may be dedicated to shorter-range wireless communications such as Wi-Fi or Bluetooth, and a second communication device 3712 may be dedicated to longer-range wireless communications such as global positioning system (GPS), EDGE, GPRS, CDMA, WiMAX, LTE, EV- DO, or others. In some embodiments, a first communication device 3712 may be dedicated to wirelesscommunications, and a second communication device 3712 may be dedicated to wired communications.

[0275] The computing device 3700 may include power source I power circuitry 3714. The power source I power circuitry 3714 may include one or more energy storage devices (e.g., batteries or capacitors) and / or circuitry for coupling components of the computing device 3700 to an energy source separate from the computing device 3700 (e.g., DC power, AC power, etc.).

[0276] The computing device 3700 may include a display device 3706 (or corresponding interface circuitry, as discussed above). Display device 3706 may include any visual indicators, such as a headsup display, a computer monitor, a projector, a touchscreen display, a liquid crystal display (LCD), a light-emitting diode display, or a flat panel display, for example.

[0277] The computing device 3700 may include an audio output device 3708 (or corresponding interface circuitry, as discussed above). The audio output device 3708 may include any device that generates an audible indicator, such as speakers, headsets, or earbuds, for example.

[0278] The computing device 3700 may include an audio input device 3718 (or corresponding interface circuitry, as discussed above). The audio input device 3718 may include any device that generates a signal representative of a sound, such as microphones, microphone arrays, or digital instruments (e.g., instruments having a musical instrument digital interface (MIDI) output).

[0279] The computing device 3700 may include a GPS device 3716 (or corresponding interface circuitry, as discussed above). The GPS device 3716 may be in communication with a satellite-based system and may receive a location of the computing device 3700, as known in the art.

[0280] The computing device 3700 may include a sensor 3730 (or one or more sensors). The computing device 3700 may include corresponding interface circuitry, as discussed above). Sensor 3730 may sense physical phenomenon and translate the physical phenomenon into electrical signals that can be processed by, e.g., processing device 3702. Examples of sensor 3730 may include: capacitive sensor, inductive sensor, resistive sensor, electromagnetic field sensor, light sensor, camera, imager, microphone, pressure sensor, temperature sensor, vibrational sensor, accelerometer, gyroscope, strain sensor, moisture sensor, humidity sensor, distance sensor, range sensor, time-of- flight sensor, pH sensor, particle sensor, air quality sensor, chemical sensor, gas sensor, biosensor, ultrasound sensor, a scanner, etc.

[0281] The computing device 3700 may include another output device 3710 (or corresponding interface circuitry, as discussed above). Examples of the other output device 3710 may include anaudio codec, a video codec, a printer, a wired or wireless transmitter for providing information to other devices, haptic output device, gas output device, vibrational output device, lighting output device, home automation controller, or an additional storage device.

[0282] The computing device 3700 may include another input device 3720 (or corresponding interface circuitry, as discussed above). Examples of the other input device 3720 may include an accelerometer, a gyroscope, a compass, an image capture device, a keyboard, a cursor control device such as a mouse, a stylus, a touchpad, a bar code reader, a Quick Response (QR) code reader, any sensor, or a radio frequency identification (RFID) reader.

[0283] The computing device 3700 may have any desired form factor, such as a handheld or mobile computer system (e.g. , a cell phone, a smart phone, a mobile Internet device, a music player, a tablet computer, a laptop computer, a netbook computer, a personal digital assistant (PDA), a personal computer, a remote control, wearable device, headgear, eyewear, footwear, electronic clothing, etc.), a desktop computer system, a server or other networked computing component, a printer, a scanner, a monitor, a set-top box, an entertainment control unit, a vehicle control unit, a digital camera, a digital video recorder, an I nternet-of-Things device, or a wearable computer system. In some embodiments, the computing device 3700 may be any other electronic device that processes data.

[0284] Select examples

[0285] Example 1 provides a method for calibrating a spraying end effector, including actuating a spraying end effector at a distal end of a robotic arm to perform a spray operation to generate a spray pattern on a target surface; receiving sensor data associated with the spray pattern captured using one or more sensors; determining one or more calibration parameters based on the captured sensor data; and adjusting one or more tool path parameters of the spraying end effector based on the one or more calibration parameters.

[0286] Example 2 provides the method of example 1 , where the one or more calibration parameters include a pitch offset.

[0287] Example 3 provides the method of example 1 or 2, where the one or more calibration parameters include a yaw offset.

[0288] Example 4 provides the method of any one of examples 1-3, where the one or more calibration parameters include a roll offset.

[0289] Example 5 provides the method of any one of examples 1-4, where the one or more calibration parameters include a fan width.

[0290] Example 6 provides the method of any one of examples 1-5, where the one or more calibration parameters include a spray thickness.

[0291] Example 7 provides the method of any one of examples 1-6, where one or more sensors include a camera.

[0292] Example 8 provides the method of any one of examples 1-7, where the one or more calibration parameters include a depth sensor, and the sensor data include three-dimensional profile data.

[0293] Example 9 provides the method of any one of examples 1-8, further including performing a verification spray procedure to confirm an accuracy of one or more adjusted tool path parameters.

[0294] Example 10 provides the method of any one of examples 1 -9, where the spray operation includes actuating the spraying end effector to perform two or more (e.g., two or three) spray bursts.

[0295] Example 11 provides the method of any one of examples 1 -9, where the spray operation includes actuating the spraying end effector to perform a spray burst.

[0296] Example 12 provides the method of any one of examples 1 -9, where the spray operation includes actuating the spraying end effector to perform a spray pass of a predetermined length at a predetermined speed.

[0297] Example 13 provides the method of any one of examples 1-12, where determining the one or more calibration parameters based on the captured sensor data includes determining a spray order of the spray pattern based on the sensor data.

[0298] Example 14 provides the method of any one of examples 1-13, where determining the one or more calibration parameters based on the captured sensor data includes determining one or more measurements of the spray pattern based on the sensor data; and deriving the one or more calibration parameters based on the one or more measurements.

[0299] Example 15 provides the method of any one of examples 1-14, further including detecting a condition for triggering a calibration procedure.

[0300] Example 16 provides the method of any one of examples 1-15, further including positioning the distal end of the robotic arm at a predetermined distance from the target surface.

[0301] Example 17 provides a method for calibrating a spraying end effector, including actuating a spraying end effector at a distal end of a robotic arm to perform a spray operation to generate a spray pattern on a target surface; receiving a user input associated with the spray pattern, the user input including one or more of: a spray order, and one or more distance measurements of the spray pattern;determining one or more calibration parameters based on the user input; and adjusting one or more tool path parameters of the spraying end effector based on the one or more calibration parameters.

[0302] Example 18 provides the method of example 17, where the one or more calibration parameters include one or more of: a pitch offset, a yaw offset, a roll offset, a fan width, and a thickness.

[0303] Example 19 provides the method of example 17 or 18, further including performing a verification spray procedure to confirm an accuracy of one or more adjusted tool path parameters.

[0304] Example 20 provides the method of any one of examples 17-19, where the spray operation includes actuating the spraying end effector to perform two or more (e.g., two or three) spray bursts.

[0305] Example 21 provides the method of any one of examples 17-20, further including detecting a condition for triggering a calibration procedure.

[0306] Example 22 provides the method of any one of examples 17-19, further including positioning the distal end of the robotic arm at a predetermined distance from the target surface.

[0307] Example 23 provides a robotic system for drywall finishing, including a robotic arm having a distal end; a spraying end effector at the distal end of the robotic arm; a positioning system to adjust a position and pose of the spraying end effector relative to a target surface; and a control system to perform any one of the methods according to any one of examples 1-22 to calibrate one or more tool path parameters controlling the spraying end effector and the robotic arm.

[0308] Example 24 provides the robotic system of example 23, further including one or more sensors at the distal end of the robotic arm.

[0309] Example 25 provides a non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic system, cause the robotic system to perform a method for calibrating a spraying end effector according to any one of examples 1-22.

[0310] Variations and other notes

[0311] The detailed description, such as the "Select examples" section, provide various examples of the embodiments disclosed herein.

[0312] In FIGS. 7-15, and 31-34, the pitch, yaw, and roll offsets, fan width errors, and thickness errors, are illustrated in isolation, but in practice, a combination of one or more offsets and fan width errors may be present for a given spraying end effector. The calibration techniques described and illustrated herein (e.g., FIGS. 16-36) can be applied to measure the offsets, fan width, thickness, and / or errors even if there are more than one type of offset and / or error present for the spraying end effector.

[0313] While some embodiments are described with respect to surface finishing, the techniques described herein may be applied to depositing insulation materials and / or fireproofing materials onto a surface. Some of the techniques are described with walls as an example. However, the techniques can also be applied to other types of building surfaces or structures, such as interior surfaces, exterior surfaces, building surfaces, ceilings, floors, etc. The techniques described herein may be applied to painting of a surface. The techniques described herein may be applied to fabrication and manufacturing.

[0314] The detailed description of illustrated implementations of the disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. While specific implementations of, and examples for, the disclosure are described herein for illustrative purposes, various equivalent modifications are possible within the scope of the disclosure, as those skilled in the relevant art will recognize. These modifications may be made to the disclosure in light of the detailed description.

[0315] For purposes of explanation, specific numbers, materials and configurations are set forth in order to provide a thorough understanding of the illustrative implementations. However, it will be apparent to one skilled in the art that the present disclosure may be practiced without the specific details and / or that the present disclosure may be practiced with only some of the described aspects. In other instances, well known features are omitted or simplified in order not to obscure the illustrative implementations.

[0316] Further, references are made to the accompanying drawings that form a part hereof, and in which are shown, by way of illustration, embodiments that may be practiced. It is to be understood that other embodiments may be utilized, and structural or logical changes may be made without departing from the scope of the present disclosure. Therefore, the detailed description is not to be taken in a limiting sense.

[0317] Various operations may be described as multiple discrete actions or operations in turn, in a manner that is most helpful in understanding the disclosed subject matter. However, the order of description should not be construed as to imply that these operations are necessarily order dependent. In particular, these operations may not be performed in the order of presentation. Operations described may be performed in a different order from the described embodiment. Various additional operations may be performed or described operations may be omitted in additional embodiments.

[0318] For the purposes of the present disclosure, the phrase “A or B” or the phrase "A and / or B'1means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase ‘A, B, or C” or the phrase "A, B, and / or C" means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). The term ''between,'1when used with reference to measurement ranges, is inclusive of the ends of the measurement ranges.

[0319] The description uses the phrases "in an embodiment" or "in embodiments," which may each refer to one or more of the same or different embodiments. The terms "comprising," "including," "having," and the like, as used with respect to embodiments of the present disclosure, are synonymous. The disclosure may use perspective-based descriptions such as "above," "below," "top," "bottom," and "side" to explain various features of the drawings, but these terms are simply for ease of discussion, and do not imply a desired or required orientation. The accompanying drawings are not necessarily drawn to scale. Unless otherwise specified, the use of the ordinal adjectives “first,” “second,” and “third,” etc., to describe a common object, merely indicates that different instances of like objects are being referred to and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking or in any other manner.

[0320] In the detailed description, various aspects of the illustrative implementations will be described using terms commonly employed by those skilled in the art to convey the substance of their work to others skilled in the art.

[0321] The terms “substantially,” “close,” “approximately,” “near,” and “about,” generally refer to being within + / - 20% of a target value as described herein or as known in the art. Similarly, terms indicating orientation of various elements, e.g., “coplanar,” “perpendicular,” “orthogonal,” “parallel,” or any other angle between the elements, generally refer to being within +1- 5-20% of a target value as described herein or as known in the art.

[0322] In addition, the terms “comprise,” “comprising,” “include,” “including,” “have,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a method, process, or device, that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such method, process, or device. Also, the term “or” refers to an inclusive “or” and not to an exclusive “or.”

[0323] The systems, methods and devices of this disclosure each have several innovative aspects, no single one of which is solely responsible for all desirable attributes disclosed herein. Details of one ormore implementations of the subject matter described in this specification are set forth in the description and the accompanying drawings.

Claims

1. CLAIMS1 . A method for calibrating a spraying end effector, comprising: actuating a spraying end effector at a distal end of a robotic arm to perform a spray operation to generate a spray pattern on a target surface; receiving sensor data associated with the spray pattern captured using one or more sensors; determining one or more calibration parameters based on the captured sensor data; and adjusting one or more tool path parameters of the spraying end effector based on the one or more calibration parameters.

2. The method of claim 1 , wherein the one or more calibration parameters comprise a pitch offset.

3. The method of claim 1 or 2, wherein the one or more calibration parameters comprise a yaw offset.

4. The method of any one of claims 1 -3, wherein the one or more calibration parameters comprise a roll offset.

5. The method of any one of claims 1 -4, wherein the one or more calibration parameters comprise a fan width.

6. The method of any one of claims 1 -5, wherein the one or more calibration parameters comprise a spray thickness.

7. The method of any one of claims 1 -6, wherein one or more sensors comprise a camera.

8. The method of any one of claims 1 -7, wherein the one or more calibration parameters comprise a depth sensor, and the sensor data comprise three-dimensional profile data.

9. The method of any one of claims 1 -8, further comprising: performing a verification spray procedure to confirm an accuracy of one or more adjusted tool path parameters.

10. The method of any one of claims 1 -9, wherein the spray operation comprises actuating the spraying end effector to perform two or more spray bursts.11 . The method of any one of claims 1 -9, wherein the spray operation comprises actuating the spraying end effector to perform a spray burst.

12. The method of any one of claims 1 -9, wherein the spray operation comprises actuating the spraying end effector to perform a spray pass of a predetermined length at a predetermined speed.

13. The method of any one of claims 1-12, wherein determining the one or more calibration parameters based on the captured sensor data comprises: determining a spray order of the spray pattern based on the sensor data.

14. The method of any one of claims 1-13, wherein determining the one or more calibration parameters based on the captured sensor data comprises: determining one or more measurements of the spray pattern based on the sensor data; and deriving the one or more calibration parameters based on the one or more measurements.

15. The method of any one of claims 1-14, further comprising: detecting a condition for triggering a calibration procedure.

16. The method of any one of claims 1-15, further comprising: positioning the distal end of the robotic arm at a predetermined distance from the target surface.

17. A method for calibrating a spraying end effector, comprising: actuating a spraying end effector at a distal end of a robotic arm to perform a spray operation to generate a spray pattern on a target surface; receiving a user input associated with the spray pattern, the user input including one or more of: a spray order, and one or more distance measurements of the spray pattern; determining one or more calibration parameters based on the user input; and adjusting one or more tool path parameters of the spraying end effector based on the one or more calibration parameters.

18. The method of claim 17, wherein the one or more calibration parameters comprise one or more of: a pitch offset, a yaw offset, a roll offset, a fan width, and a thickness.

19. The method of claim 17 or 18, further comprising: performing a verification spray procedure to confirm an accuracy of one or more adjusted tool path parameters.

20. The method of any one of claims 17-19, wherein the spray operation comprises actuating the spraying end effector to perform two or more spray bursts.21 . The method of any one of claims 17-20, further comprising: detecting a condition for triggering a calibration procedure.

22. The method of any one of claims 17-19, further comprising: positioning the distal end of the robotic arm at a predetermined distance from the target surface.

23. A robotic system for drywall finishing, comprising: a robotic arm having a distal end; a spraying end effector at the distal end of the robotic arm; a positioning system to adjust a position and pose of the spraying end effector relative to a target surface; and a control system to perform any one of the methods according to any one of claims 1-22 to calibrate one or more tool path parameters controlling the spraying end effector and the robotic arm.

24. The robotic system of claim 23, further comprising: one or more sensors at the distal end of the robotic arm.

25. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a robotic system, cause the robotic system to perform a method for calibrating a spraying end effector according to any one of claims 1-22.

Images