Systems and techniques for workpiece modification

Abstract

The present disclosure provides processes and systems for modifying a workpiece. A process includes capturing at least one image of the workpiece; processing the at least one image; obtaining a nominal tool path; measuring the surface of the workpiece to obtain a workpiece surface measurement result; generating an updated tool path; generating a planning path for the workpiece modification equipment; scanning the modification performed on the workpiece by the workpiece modification equipment; determining parameters of a modifier; and modifying the workpiece by the workpiece modification equipment according to the planned path and the modifier parameter. The modification may be to dispense material on the workpiece. A system includes image capture equipment, an on-line sensor, a robotic motion controller, an off-line registration unit, a nominal tool path, an on-line positioning unit, a path planner unit, a modifier including workpiece modification equipment, and a modification equipment controller.

Description

Background Art

[0001] Automation capabilities are widespread in the robotic modification of workpieces, for example, applying a dispensed adhesive. Summary of the invention

[0002] The present disclosure recognizes the general need for more accurate modification of workpieces. In a first aspect, a process for modifying a workpiece is provided. The process includes capturing an image of the workpiece using an image capture device; processing the image using an image processor to determine at least one of a position or an orientation of the workpiece; obtaining a nominal tool path; measuring a surface of the workpiece using an online sensor A to obtain a workpiece surface measurement result; and generating an updated tool path using a waypoint modifier based on at least the nominal tool path and the position and / or the orientation of the workpiece. The process also includes generating a planned path for a workpiece modification device using a path planner based on at least the updated tool path and the workpiece surface measurement result; scanning the modification performed on the workpiece by the workpiece modification device using an online sensor B; determining a modifier parameter based on at least one modifier characteristic; and modifying the workpiece by the workpiece modification device according to the planned path and the modifier parameter.

[0003] In a second aspect, another process for modifying a workpiece is provided. The process includes capturing an image of the workpiece using an image capture device; processing the image using an image processor to determine at least one of a position or an orientation of the workpiece; obtaining a nominal tool path; measuring a surface of the workpiece using an online sensor A to obtain a workpiece surface measurement result; and generating an updated tool path using a waypoint modifier based on at least the nominal tool path and the position and / or the orientation of the workpiece. The process also includes generating a planned path for a dispenser using a path planner based on at least the updated tool path and the workpiece surface measurement result; scanning beads of material dispensed by the dispenser on the workpiece using an online sensor B; determining a dispensing parameter based on a bead distribution; and communicating a control signal to a dispenser controller, the control signal causing the dispenser to dispense the material on the workpiece according to the planned path and the dispensing parameter.

[0004] In a third aspect, a system is provided. The system includes an image capture device configured to capture an image of the workpiece; an online sensor A configured to measure a surface of the workpiece; and an online sensor B configured to scan a modification of the workpiece. The system also includes a controller, the controller including: a robot motion controller configured to control the movement of a robotic arm; an offline registration unit configured to determine a waypoint list based at least on the image of the workpiece captured by the image capture device, a digital model of the workpiece, and a nominal tool path; an online positioning unit configured to determine an updated waypoint list based on a series of measurement results of the surface of the workpiece measured by the online sensor A and a series of joint postures of the robotic arm; a path planner unit configured to plan a path based on the waypoint list and the updated waypoint list; and a modification equipment controller configured to control a workpiece modification equipment according to the planned path and at least one modification parameter. The system also includes a modifier, the modifier including the workpiece modification equipment.

[0005] The above overview of the present disclosure is not intended to describe each disclosed embodiment or each specific implementation of the present disclosure. The following description illustrates exemplary embodiments in more detail. In several places throughout the present application, guidance is provided by example lists, which can be used in various combinations. In each case, the cited list is only used as a representative group and should not be interpreted as an exclusive list. Therefore, the scope of the present disclosure should not be limited to the specific exemplary structures described herein, but should at least be extended to the structures described by the language of the claims and the equivalents of these structures. Any elements cited as alternatives in this specification may be explicitly included in the claims or excluded from the claims in any combination as needed. Although various theories and possible mechanisms may have been discussed herein, such discussions should not be used to limit the subject matter that can be protected by the claims in any case. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 A system for modifying a workpiece is illustrated in which example embodiments may be implemented.

[0007] Figure 1A Offline registration is illustrated.

[0008] Figure 1B A path planner is instantiated.

[0009] Figure 2 Another system for modifying a workpiece in which example embodiments may be implemented is illustrated.

[0010] Figure 2AAnother offline registration is illustrated.

[0011] Figure 2B Another path planner is illustrated.

[0012] Figure 3 A schematic diagram illustrating a portion of a modification system in which example embodiments may be implemented is shown.

[0013] Figure 4 A schematic diagram illustrating a coordinate system of a portion of a modification system in which example embodiments may be implemented.

[0014] Figure 5 A method for modifying a workpiece is illustrated in which example embodiments may be implemented.

[0015] Figure 6 An example system for modifying a workpiece according to embodiments herein is illustrated.

[0016] 7A to 7E A schematic diagram illustrating feature correspondence between an image and a computer-aided design (CAD) file according to embodiments herein.

[0017] Figure 8 is a graph of average pixel error versus number of iterations according to an implementation described herein.

[0018] FIG. 9A to FIG. 9B Schematic diagrams of laser lines that may be used to implement example embodiments are illustrated.

[0019] Fig. 10A A schematic diagram illustrating the format of a tool path according to embodiments described herein.

[0020] Fig. 10B A schematic diagram illustrating adjusted waypoint positions based on a surface distribution scan according to embodiments described herein.

[0021] FIG. 11A to FIG. 11C The X-component, Y-component, and Z-component of a robot TCP trajectory according to embodiments described herein are illustrated.

[0022] Fig.11D Illustrated is a 3D plot of a robot TCP trajectory according to embodiments described herein.

[0023] Fig. 12A Schematic diagrams are illustrated of in-line orientations of designated nozzle axes in which example embodiments may be implemented.

[0024] Fig. 12B Schematic diagram illustrating in-line orientation specifying rotation about a nozzle axis in which example embodiments may be implemented.

[0025] Fig.13 A schematic diagram illustrating online surface normal estimation according to embodiments described herein is illustrated.

[0026] FIG. 14A to FIG. 14B Schematic diagrams illustrating orientation adjustments with non-perpendicularity that can be used to implement example embodiments.

[0027] Fig.15 Calculated surface normal vectors along a robotic tool path of embodiments described herein are illustrated.

[0028] FIG. 16A to FIG. 16B A schematic diagram illustrating the observability problem when the TCP follows a tangent direction of the tool path is illustrated.

[0029] Fig.17A Schematic diagram without in-plane rotation about the nozzle axis is illustrated.

[0030] Fig. 17B Schematic diagram illustrating in-plane rotation about the nozzle axis to achieve optimal field of view along the tool path.

[0031] Fig.18 A schematic diagram illustrating adjustment of waypoint locations based on the midpoint of a rib feature within a surface distribution scan according to embodiments described herein.

[0032] Fig.19 is a photograph of a test fixture having curved surfaces and parallel ribs according to embodiments described herein.

[0033] 20 includes a graph of robot trajectories recorded for rib feature tracking when rib feature tracking is applied according to embodiments described herein.

[0034] Fig.21 A schematic diagram is illustrated showing adjustment of waypoint positions based on edge points within a surface distribution scan according to embodiments described herein.

[0035] Fig. 22 is a photograph of a test fixture with curved edges according to embodiments described herein.

[0036] FIG. 23A to FIG. 23C Included are graphs of recorded robot trajectories for edge feature tracking of X-component, Y-component, and Z-component according to embodiments described herein.

[0037] Fig.23D Included is a graph of recorded robot trajectories for edge feature tracking according to embodiments described herein, the graph being a 3D plot of the TCP trajectory.

[0038] Fig.24A schematic diagram illustrating a method of modifying process parameters that can be used to implement example embodiments.

[0039] Fig.25 An artifact decoration system in an example network architecture is illustrated.

[0040] Figure 26 to Figure 28 An example computing device that may be used in embodiments herein is illustrated.

[0041] Fig.29 An example system of the present disclosure is illustrated.

[0042] Fig.30 A non-limiting example of a closed-loop robotic dispensing test station of the present disclosure is illustrated.

[0043] Fig.31 is a block diagram illustrating an example closed-loop robotic allocation in accordance with aspects of the present disclosure.

[0044] Fig.32 is a schematic diagram illustrating an example format of a tool path for the closed-loop robotic dispensing technique of the present disclosure.

[0045] FIG. 33 illustrates aspects of bead position detection according to aspects of the present disclosure.

[0046] FIG. 34 illustrates bead geometry estimation for two scan distribution samples.

[0047] Fig.35 is a graph illustrating a log of time consumption for each computation cycle during a closed-loop allocation test run.

[0048] Fig.36 is a schematic diagram showing an example of how changes in waypoint position and tool velocity can affect the temporal parameterization along the tool path.

[0049] FIG37 illustrates a schematic diagram showing the proposed time reparameterization strategy.

[0050] FIG. 38 illustrates the effectiveness aspects of utilizing the path streaming technique of the present disclosure with time reparameterization for variable speed closed loop allocation.

[0051] FIG. 39 illustrates the results of bead shape compensation using a simple control law.

[0052] FIG. 40 illustrates one or more potential problems with using a simple control law for bead shape compensation.

[0053] Figure 41 illustrates the transient behavior of bead width and tool speed along the dispensing path.

[0054] FIG42 illustrates a schematic diagram of a steady-state checker.

[0055] Fig.43 Illustrated are the tool speed distribution in the upper figure and the bead width distribution in the lower figure along a straight dispensing path in a test run with bead shape control.

[0056] FIG. 44 illustrates an example of a desired bead width of 10 mm ( Fig.44A )、8mm( Fig.44B ) and 5mm( Fig.44C )’s experimental results.

[0057] FIG. 45 illustrates a control gain of 0.2 ( Fig.45A )、0.5( Fig.45B ) and 0.8( Fig.45C )’s experimental results.

[0058] FIG. 46 illustrates an example of a waiting time of 2 seconds ( Fig.46A ) and 3 seconds ( Fig.46B )’s experimental results.

[0059] FIG. 47 shows that on a relatively flatter substrate ( Fig.47A ) and a relatively more curved base ( Fig.47B ) with online part shape compensation and bead shape compensation. DETAILED DESCRIPTION

[0060] The present disclosure relates to methods and systems for modifying workpieces, such as closed-loop feedback control of robot trajectories based on sensory feedback of surface geometry. In some cases, the methods and / or systems are used to achieve a higher level of process autonomy and higher dispensing quality when dispensing materials (e.g., adhesives, sealants, paints, thermally conductive materials, etc.) on parts with complex geometries. "Workpiece" is intended to include any object processed using a tool or machine, and the term is used interchangeably with "part" herein.

[0061] For certain tasks, in order to successfully modify the workpiece, a highly accurate registration of the position of the workpiece is required. It has been found that automatic part registration can be used to more accurately determine the position of the part relative to the robotic workpiece decorator. Another problem is that local changes in the geometry of the part may affect the ability to accurately modify the workpiece. It has also been found that the tool path of the robotic workpiece decorator can be adjusted based on part surface sensing and a closed-loop feedback system. Depending on how the workpiece is modified, adjustments may also need to be made based on changes in the modification. For example, if a material (e.g., adhesive, sealant, paint, thermally conductive material, etc.) is to be deposited on the part, the material properties (such as deposition rate, viscosity) may change. Additional sensors and information can be used and included in the closed-loop feedback system to adjust the modification parameters as needed.

[0062] Figure 1 1 illustrates a system for modifying a workpiece in which example embodiments may be implemented. The system 100 includes the main components of a controller 110, an inspection system 120, and a decorator 130. The system 100 is Figure 1 100 is illustrated as being in communication with a data repository 140. However, it is expressly contemplated that in some embodiments, the data repository 140 may be local to the system 100, or integrated into the system. Similarly, the system 100 is illustrated as being projected to a display 10. However, it is expressly contemplated that the system may be integrated into a processor of a device that includes the display 10. The system 100 may be implemented by one or more suitable computing devices that communicate with each of these major components.

[0063] The controller 110 includes an offline registration unit 111 that receives as input at least an image of a workpiece captured by an image capture device 121, a digital model of the workpiece (e.g., a workpiece model 142 from a data repository 140), a predefined modification tool path on the workpiece model (e.g., a nominal path 141), and a tool center point (TCP) pose from a robot motion controller 116 (e.g., controlling an articulated arm of the robot). As another example, the controller 110 also processes an image of a workpiece including a part of the robot (e.g., an end effector, such as a workpiece modification tool), and a digital model of at least a portion of the robot 153. The offline registration unit 111 outputs a spatially transformed tool path (in the format of a list of waypoints) based on the estimated position and orientation of the workpiece in the robot coordinate system. These waypoints may be transmitted to the path planner unit 113. In some cases, the waypoints may be transmitted to an optional prediction model 148 and / or an optional information database 149 for future use / reference.

[0064] For example, refer to Figure 1A , there are optionally several units within the offline registration unit 111. A nominal tool path retriever 111a retrieves a nominal tool path 141 (e.g., from a data repository 140). An image retriever 113b retrieves an image from the image capture device 121. An image processor 111c processes the image. The processing may include at least part registration (i.e., determining position and / or orientation) based on the captured image, feature detection in the image and on the surface model (e.g., feature detection of features such as edges, ribs, seams, corners, fiducial markers), and 3D pose estimation based on corresponding features. A waypoint decorator 111d generates an updated tool path based on the (e.g., processed) image of the workpiece and the nominal tool path, plus one of a set of joint positions of the robotic artwork (e.g., from which at least one TCP pose can be determined) or an image of the end effector of the workpiece decorating device 131.

[0065] The controller 110 also includes an online positioning unit 112, which receives at least the real-time streaming workpiece surface distribution from the online sensor A122 and the real-time streaming TCP posture from the motion controller of the articulated robot arm as input. The online positioning unit 112 outputs an updated waypoint position, which tracks the actual workpiece surface with a desired gap from the workpiece modification equipment 131.

[0066] The controller 110 also includes a path planner unit 113 that receives as input at least the list of waypoint positions from the online positioning unit 112 and outputs a smooth robot trajectory (e.g., with minimal jitter) for execution by a grooming equipment controller 114. "Jitter" is defined as the third-order derivative of the joint position trajectory, which is minimized throughout the path to achieve smoothness. The controller 110 also includes a grooming equipment controller 114 that applies grooming parameters 144 derived from a grooming model 145 and controls the on / off state of the workpiece grooming equipment 131. Reference Figure 1B , there are optionally several units within the path planner unit 113. The waypoint retriever 113a retrieves the current waypoint set (e.g., from the data repository 140). The path retriever 113b retrieves the actual path of the robot arm across the workpiece surface from the online sensor A 122. The path analyzer 113c compares the actual path with the waypoints. Based on the comparison, the waypoint decorator 113d calculates a new waypoint set. In some cases, the robot motion controller 116 may receive input from the path planner 113 and transmit instructions to the workpiece decoration equipment 131 to move the workpiece decoration equipment 131 along, for example, the updated tool path provided by the waypoint decorator 113d.

[0067] The controller 110 optionally also includes a graphical user interface (GUI) generator 115, which can be configured to transmit information to the display 10. The GUI generator 115 can generate a graphical user interface based on some or all of the information collected or generated by the controller 110 to be displayed on the display assembly 10. Suitable displays include, for example, but not limited to, a computer screen, a smart phone, or some other user device. Other units 117 may also be included in the controller 110. The controller 110 is described as having the function of transmitting communicable information to other devices and receiving communicable information from other devices. This can be accomplished through an application program interface, for example, so that the controller 110 can receive and communicate with any unit and / or model in each of the inspection system 120, the decorator 130, and the data repository 140.

[0068] The inspection system 120 includes an image capture device 121 that captures an image of the workpiece and outputs the image to the offline registration unit 111. Any suitable image capture device may be used, such as but not limited to a red, blue, and green (RGB) camera, a black and white (B&W) camera, or a three-dimensional (3D) image sensor.

[0069] The inspection system 120 further includes an online sensor A 122, which can be attached to a surface of the workpiece modification equipment 131 and performs online scanning of the workpiece surface before the workpiece modification equipment 131. The inspection system 120 further includes an online sensor B 123, which can be attached to a different (e.g., opposite) surface of the workpiece modification equipment 131 and performs online scanning of the modification performed on the workpiece by the workpiece modification equipment 131. Each of the online sensor A 122 and the online sensor B 123 senses in situ and obtains data under (e.g., close to) real-time conditions. Any suitable equipment for each of the line sensor A and the online sensor B can be used for scanning, such as but not limited to equipment independently selected from a laser profiler, an area snapshot sensor, a triangulation-based sensor, a time-of-flight sensor, a laser point sensor, an optical coherence tomography sensor, a confocal sensor, and a dynamic vision sensor. In some embodiments, a laser profiler is preferred as an online sensor because it has the advantage of being faster and simpler to scan than a point cloud, and a point cloud provides 2.5D data instead of 2-dimensional data of a laser profiler.

[0070] The inspection system 120 may further include other units 124 .

[0071] The decorator 130 includes workpiece decorating equipment 131. Any suitable equipment configured to decorate the workpiece may be used in the system, such as, but not limited to, dispensers, sanders, polishers, cutters, drills, sprayers, welders, etc. In some embodiments, a dispenser is used, which will be discussed in detail below. For example, the decorator 130 can be a robotic adhesive dispensing unit with a robotic arm having a dispenser (e.g., 131). The current state can be a TCP received from the robotic motion controller 116 of the controller 110.

[0072] The data repository 140 is configured to communicate with the controller 110, the inspection system 120, and the decorator 130. The data repository 140 may be local to the controller 110 or accessible through a cloud-based network. Figure 1 1 as being local to the system 100, it is expressly contemplated that the controller 110 may be remote from the system 100 and may use a wireless or cloud-based network to receive signals and transmit commands.

[0073] The data repository 140 includes a nominal path unit 141, which contains a predefined tool path for modifying a workpiece. The tool path is defined on a surface model 142, which can be a computer-aided design (CAD) model, a depth image, a point cloud, or other model, also included in the data repository 140 or capable of being retrieved in other ways.

[0074] The data repository 140 also includes a modification parameter unit 144 that contains information about measurable factors of the workpiece modification equipment (eg, speed, distance from the workpiece surface, angle with the workpiece surface, temperature, deposition / spraying rate, etc.).

[0075] The data repository 140 also includes a modifier property unit 143, which contains information about the properties of any material used to modify the workpiece that can be sensed by the online sensor B 123 (e.g., size, shape, location, etc.). Similarly, the data repository 140 includes a material property unit 146, which contains information about the physical properties of any material used to modify the workpiece (e.g., chemical composition, state of matter, temperature, viscosity, Mohs hardness, sharpness, adhesion, color, drill size, etc.). The data repository 140 also includes a workpiece property unit 147, which contains information about the properties of the workpiece (e.g., size, shape, material composition, etc.). The data repository 140 optionally also includes an information database 149, which contains any additional relevant information for access by any of the units in the data repository 140 or the controller 110.

[0076] The data repository 140 optionally also includes a prediction model 148 that receives as input at least historical information from previous groomings involving the same or similar workpieces and / or groomers. The model 148 outputs a grooming model to the grooming equipment controller 114 to help quickly respond to differences between the planned path / grooming and the actual path / grooming.

[0077] The data repository 140 optionally also includes a machine learning unit 151 configured to predict process parameters for modifying a workpiece, such as input to a modification model 145. The data repository 140 also includes a modification model 145 that receives as input at least the in-situ streamed modifier characteristics 143 from the online sensor B 123. Optionally, the modification model also receives as input a template / desired modifier characteristic defined by a user. The modification model 145 may also receive information from one or more of a material characteristic unit 146, a workpiece characteristic unit 147, a prediction model 148, an information database 149, or other units 154. The modification model 145 outputs the modification parameters 144 to the modification equipment controller 114 to provide closed-loop feedback for rapidly adjusting the modification distribution to correct errors between the actual modifier characteristics and the template modifier characteristics.

[0078] The data repository 140 also includes an image capture calibration parameters unit 152 that contains information used by the offline registration unit 111 to calibrate the image capture equipment 121 before capturing any images.

[0079] Optionally, the data repository 140 also includes a robot model unit 153, which contains a digital model of at least a portion of the robot, such as a robotic arm configured to modify a workpiece and / or an end effector of the robot (e.g., a workpiece modification device). In some cases, the robot model unit 153 contains a digital model of the entire robot.

[0080] The data repository 140 may also include other units 154 .

[0081] Figure 2 Another system for modifying a workpiece in which example embodiments may be implemented is illustrated. More specifically, Figure 2 An exemplary system is illustrated in which a bead of adhesive is deposited on a workpiece. The system includes the major components of a controller 210, an inspection system 220, a dispenser 230, and a data repository 240. The system may be implemented by one or more suitable computing devices in communication with each of these major components.

[0082] The controller 210 includes an offline registration unit 211 that receives as input at least an image of a workpiece captured by an image capture device 221, a digital model of the workpiece (e.g., a CAD model 242), a predefined modification tool path (e.g., a nominal path) on the workpiece model, and a TCP pose from a robot motion controller 216 (e.g., controlling an articulated arm of the robot). In another example, the controller 210 also processes an image of a workpiece including a part of the robot (e.g., a nozzle of a dispenser), and a digital model of at least a portion of the robot 253. The offline registration unit 211 outputs a spatially transformed tool path (in the format of a list of waypoints) based on the estimated position and orientation of the workpiece in the robot coordinate system. These waypoints may be transmitted to the path planner unit 213. In some cases, the waypoints may be transmitted to an optional prediction model 248 and / or an optional information database 249 for future use / reference.

[0083] For example, refer to Figure 2A , there are optionally several units within the offline registration unit 211. A nominal tool path retriever 211a retrieves a nominal tool path 241 (e.g., from a data repository 240). An image retriever 213b retrieves an image from an image capture device 221. An image processor 211c processes the image. The processing may include at least part registration (i.e., determining position and / or orientation) based on the captured image, feature detection in the image and on the surface model (e.g., feature detection of features such as edges, ribs, seams, corners, fiducial markers), and 3D pose estimation based on corresponding features. A waypoint decorator 211d generates an updated tool path based on the (e.g., processed) image of the workpiece and the nominal tool path, plus one of a set of joint positions of the robotic artwork (e.g., from which at least one TCP pose can be determined) or an image of the end effector (e.g., nozzle) of the dispenser 230.

[0084] The controller 210 further includes an online positioning unit 212 that receives as input at least the workpiece surface distribution streamed in real time from the online sensor A 222, the TCP pose streamed in real time from the motion controller of the articulated robot, and the waypoint list from the offline registration unit 211. The online positioning unit 212 outputs updated waypoint positions that track the actual workpiece surface at a desired gap from the workpiece modification equipment 231.

[0085] The controller 210 also includes a path planner unit 213 that receives as input at least the list of waypoint locations from the online positioning unit 212 and outputs a smooth robot trajectory (e.g., with minimal jitter) to a buffer for execution by the distributor controller 214. Figure 2B, there are optionally several units within the path planner unit 213. Waypoint retriever 213a retrieves the current waypoint set (e.g., from data repository 240). Path retriever 213b retrieves the actual path of the robot across the workpiece surface from online sensor A 222. Path analyzer 213c compares the actual path with the waypoints. Based on the comparison, waypoint decorator 213d calculates a new waypoint set.

[0086] The controller 210 also includes a dispenser controller 214 that applies a dispensing parameter 244 derived from a dispensing model 245 and controls an on / off state of the dispenser 230 .

[0087] The controller 210 optionally also includes a graphical user interface (GUI) generator 215, which can be configured to transmit information to the display 20. The GUI generator 215 can generate a graphical user interface based on some or all of the information collected or generated by the controller 210 to be displayed on the display 20. Suitable displays include, for example, but not limited to, a computer screen, a smart phone, or some other user device. Other units 217 may also be included in the controller 210. The controller 210 is described as having the function of transmitting communicable information to other devices and receiving communicable information from other devices. This can be done through an application program interface, for example, so that the controller 210 can receive and communicate with any unit and / or model in each of the inspection system 220, the dispenser 230, and the data repository 240.

[0088] The inspection system 220 includes an image capture device 221 that captures an image of the workpiece and outputs the image to the offline registration unit 211. Any suitable image capture device may be used, such as but not limited to an RGB camera, a B&W camera, or a 3D image sensor.

[0089] The inspection system 220 further includes an online sensor A 222 that can be attached to a surface of the workpiece modification equipment 231 and performs an online scan on the workpiece surface before the workpiece modification equipment 231. The inspection system 220 further includes an online sensor B 223 that can be attached to a different (e.g., opposite) surface of the dispenser 230 and performs an online scan on the adhesive beads dispensed on the workpiece by the dispenser 230. Each of the online sensor A 222 and the online sensor B 223 senses in situ and obtains data under (e.g., near) real-time conditions. Any suitable equipment for each of the line sensor A and the online sensor B can be used for scanning, such as, but not limited to, equipment independently selected from a laser profiler, an area snapshot sensor, a triangulation-based sensor, a time-of-flight sensor, a laser point sensor, an optical coherence tomography sensor, a confocal sensor, and a dynamic vision sensor. In some embodiments, a laser profilometer is preferred as the inline sensor because it has the advantage of being faster and simpler to scan than a point cloud, which provides 2.5D data instead of the 2-dimensional data of a laser profilometer or laser point sensor.

[0090] The inspection system 220 may further include other units 224 .

[0091] Dispenser 230 includes any suitable dispenser of adhesive material (e.g., including an extruder and a nozzle for depositing adhesive beads), such as at least Figure 3 , 12 and 14. For example, a suitable dispenser is described in detail in PCT Publication No. WO 2020 / 174394 (Napierala et al.), which is incorporated herein by reference in its entirety.

[0092] The data repository 240 is configured to communicate with the controller 210, the inspection system 220, and the dispenser 230. The data repository 240 may be local to the controller 210 or accessible through a cloud-based network. Figure 2 200, but it is expressly contemplated that the controller 210 may be remote from the system 200 and may use a wireless or cloud-based network to receive signals and transmit commands.

[0093] The data repository 240 includes a nominal path unit 241 containing a predefined tool path for modifying a workpiece. The tool path is defined on a CAD model 242, which is also included in the data repository 240 or can be retrieved in other ways.

[0094] The data repository 240 also includes a dispensing parameters unit 244 that contains information about measurable factors of the dispenser (eg, speed, distance from the workpiece surface, angle from the workpiece surface, temperature, deposition rate, etc.).

[0095] The data repository 240 also includes a bead distribution unit 243, which contains information about the properties of the adhesive beads (e.g., size, shape, location, etc.) that can be sensed by the online sensor B 223. Similarly, the data repository 240 includes a material property unit 246, which contains information about the physical properties of the adhesive material (e.g., chemical composition, state of matter, temperature, viscosity, adhesion, color, etc.). The data repository 240 also includes a workpiece property unit 247, which contains information about the properties of the workpiece (e.g., size, shape, material composition, etc.). The data repository 240 optionally also includes an information database 249, which contains any additional relevant information for access by any of the units in the data repository 240 or the controller 210.

[0096] The data repository 240 optionally also includes a predictive model 248 that receives as input historical information from previous modifications involving the same or similar workpieces and / or adhesives. The predictive model 248 outputs a modification model to the dispenser controller 214 to help quickly respond to differences between the planned path / modification and the actual path / modification.

[0097] The data repository 240 optionally also includes a machine learning unit 251 configured to predict process parameters for dispensing on a workpiece, such as input to a dispensing model 245. The data repository 240 also includes a dispensing model 245 that receives at least an in-situ streaming bead distribution 243 from an online sensor B 223 as input. Optionally, the modification model also receives a template / desired modifier characteristic defined by a user as input. The dispensing model 245 may also receive information from one or more of a material characteristic unit 246, a workpiece characteristic unit 247, a prediction model 248, an information database 249, or other units 255. The dispensing model 245 outputs a dispensing parameter 244 to the dispenser controller 214 to provide closed-loop feedback for rapidly adjusting the adhesive bead distribution to correct errors between the actual adhesive bead characteristic and the template adhesive bead characteristic.

[0098] The data repository 240 also includes an image capture calibration parameters unit 252 that contains information used by the offline registration unit 211 to calibrate the image capture equipment 121 before capturing any images.

[0099] Optionally, the data repository 240 also includes a robot model unit 253, which contains a digital model of at least a portion of the robot, such as a robotic arm configured to modify a workpiece and / or an end effector (e.g., a dispenser) of the robot. In some cases, the robot model unit 253 contains a digital model of the entire robot.

[0100] The data repository 240 may also include other units 254 .

[0101] Modification of distribution process parameters

[0102] For example, but not limitation, in one embodiment, the material modifier is an adhesive dispenser that dispenses adhesive onto a surface.

[0103] Adhesives have different properties based on environmental conditions, so it is difficult to know exactly how fast to move the dispenser (i.e., how fast the dispenser moves in space), the force to apply to the material to achieve a desired volumetric flow rate, the temperature to which to heat one or more components, etc. In scenarios where the adhesive is a 2 (or more) part mixture, each component has these issues. While some products may have a high tolerance to variation, other products, such as aircraft components, require precise application of the adhesive to ensure proper functionality.

[0104] Therefore, it is important to have a feedback system that is able to characterize the adhesive flow in situ and adjust the parameters to achieve the desired adhesive dispensing profile.

[0105] As described above Figure 2 system related, Fig.24 Schematic diagram illustrating a method of modifying process (eg, dispensing) parameters that can be used to implement example embodiments. Fig.24 Including information including at least one of equipment (e.g., robot arm) speed 2410, distance 2420 between dispenser and workpiece, or volume flow rate 2430 to change process parameters (2440). The method also includes detecting the effect (2450) on bead shape caused by changing process parameters (2440) and selecting new process parameters (2460). In some cases, the method performs at least one loop iteration between changing process parameters (2440) and detecting the effect (2450) on bead shape, for example, by using online sensor B. In certain embodiments of the present disclosure, the beads are beads of adhesive (e.g., pressure sensitive adhesive, structural adhesive, etc.), although it is clearly contemplated that other materials are dispensed on the workpiece in the form of beads. Machine learning can be used to help change process parameters (2440), select new process parameters (2460), or both. Machine learning will be described in more detail below.

[0106] Offline registration of part positions

[0107] refer to Figure 3 , provides a schematic diagram of a portion of a modification system in which an example embodiment can be implemented. More specifically, Figure 3 An illustration of a portion of a system in use is included. The system includes a dispenser 330, an online sensor A 322, an online sensor B 323, and an image capture device 321. The system is shown as a snapshot in time during a process of dispensing a bead of adhesive 334 onto a major surface 362 of a workpiece 360 ​​through a nozzle 332 of the dispenser 330. The arrow shows the direction D in which the dispenser 330 is traveling relative to the workpiece 360. The online sensor A 322 directs a signal 325 at the major surface 362 of the workpiece 360 ​​and then receives a return signal 326 that provides information about the actual distribution of the major surface 362 of the workpiece 360 ​​to allow the dispense characteristics to be adjusted as needed before the dispenser reaches the position that the online sensor A 322 has sensed. The online sensor B 323 directs a signal 327 to the bead of adhesive 334 that has been dispensed onto the major surface 362 of the workpiece 360 ​​and then receives a return signal 328 that provides information about the actual distribution of the bead of adhesive 334 for determining whether the dispense parameters need to be adjusted.

[0108] refer to Figure 4 , shows a schematic diagram of a robotic dispensing system consisting of a 6-axis robot arm, a dispensing tool mounted on the robot's tool flange, and a camera mounted on the dispensing tool. Let {A}, {B}, and {D} denote the coordinate systems of the robot base, the image capture device, and the dispensing tool, respectively. Let {E} denote the coordinate system fixed to the workpiece (user coordinate system). For the representation of coordinate transformations, the 3D translation and rotation from coordinate system 1 to coordinate system 2 are expressed in a compact form as the transformation matrix

[0109]

[0110] represents the rotation matrix from coordinate system 1 to coordinate system 2, and Represents the origin of coordinate system 1 expressed in coordinate system 2.

[0111] T AD and T BD This can be calibrated using the standard hand-eye calibration procedure. BE is the unknown quantity estimated by the vision-based positioning method, which will be described below. Then, the workpiece pose T expressed in the base coordinate system (required in the tool path planning algorithm) can be calculated using the chain rule AE :

[0112] T AE=T AD T DB T BE

[0113] It is assumed that before the workpiece positioning process, the image capture equipment parameters (intrinsic matrix, extrinsic matrix and distortion coefficients) and the 3D model of the workpiece are known. The image capture equipment parameters are obtained via a standard calibration procedure. The 3D model of the workpiece can be obtained from the original CAD design file or via a 3D scan of a sample workpiece.

[0114] In addition, one or more joint positions (e.g., poses) of a workpiece modification equipment (e.g., a robot) relative to a robot coordinate system may be streamed from a robot controller upon request. Figure 4 The robot 400 in FIG. 4 may include a main body portion 422 and a plurality of arm portions 424, 426, and 428, which are connected via joints 442, 444, and 446 (e.g., the main body portion 422 and the arm portion 424 are connected via the joint 442). The manipulation joint 442 forms an angle α between the main body portion 422 and the arm portion 424. 1 , so the angle α 1 The angle α changes with the specific relative position of the main body portion 422 and the arm portion 424. Similarly, the arm portion 424 and the arm portion 426 are connected by a joint 444. The manipulation joint 444 forms a variable angle α between the arm portion 424 and the arm portion 426. 2 Likewise, arm portion 426 and arm portion 428 are connected by joint 446. Joint 446 is manipulated to form a variable angle α between arm portion 426 and arm portion 428. 3 By obtaining the streamed joint positions from the robot controller, at least one joint angle between adjacent arm parts of the robot may be used to determine the TCP pose.

[0115] Figure 5 A process 50 for modifying a workpiece according to one or more aspects of the present disclosure is illustrated. Although it should be understood that a variety of systems may be configured to perform the process 50 according to the present disclosure, for ease of discussion, the method 50 is described as being comprised of Figure 1 and Figure 2 The system illustrated in is executed. Figure 5 An exemplary schematic diagram of how various system components may appropriately interact and / or be implemented to perform such methods is provided, although other interactions are also expressly contemplated. The process 50 may begin with the image capture device 121 or 221 capturing an image (500). The captured image may include one or more of an RGB image 501, a B&W image 502, a 3D image 503, or other image 504.

[0116] Then, the system 100 or 200 may process the captured image offline (510). For example, the system 100 or 200 may perform post-processing operations, including at least part registration (i.e., determining position and / or orientation) based on the captured image. The process optionally also includes feature detection in the image and on the surface model (e.g., feature detection of features such as edges, ribs, seams, corners, fiducial markers, etc.) and 3D pose estimation based on corresponding features. Additional post-processing operations such as smoothing, filtering, compression, fisheye correction, etc. may also be performed. The image processing optionally includes data from a surface model 511 (e.g., a CAD model) of the workpiece, a 3D rendering based on an image of the same part, and / or a previous laser scan of the same part. In some cases, the post-processing also includes searching for feature correspondences between the captured images (e.g., 501 to 504) and the surface model 511.

[0117] In addition, the path planner 113 or 213 generates a tool path (520). The tool path can be a spatially transformed tool path in a waypoint list format based on at least the joint positions from the robot 521 (e.g., which can be used to determine the TCP pose) or the digital model, the nominal path 522, the captured images 501 to 504, and / or the estimated position and orientation of the workpiece processed offline. Additionally, the online sensor A 122 or 222 measures the workpiece surface (530). The process 50 also generates an updated tool path (540) that includes at least the measurement results of the workpiece surface and the streamed TCP pose or joint angle 541.

[0118] In-line sensor B 123 or 223 provides an updated tool path, and optionally other information; the process 50 plans a path based at least on this data (550). When outputting the decorator parameters, the process 50 typically includes the desired decorator characteristics 561 (560). Additionally, (at least) a decorator equipment control and a robot controller 575 collaborate to decorate the workpiece (570). The decorator equipment control may include, for example, but not limited to, at least one of a dispenser control unit 571, a welding equipment control unit 572, a paint repair equipment control unit 573, or some other control unit 574. The actions involved in the process may be performed in various orders, and some actions are repeated multiple times to provide in-situ adjustment of the tool path, workpiece decoration, etc.

[0119] For example, Figure 6An example system for modifying a workpiece is illustrated, wherein some possible interactions between certain components are depicted. Workpiece 602 is depicted as interacting with each of image capture equipment 604, online sensor B 606, modification equipment control unit 608, robot control unit 610, and online sensor A 612. Image capture equipment 604 may interact with offline registration unit 614. Offline registration unit 614 may also interact with online positioning unit 616 and / or robot control unit 610. Online positioning unit 616 may also interact with robot control unit 610 and / or path planner unit 618. The path planner unit may also interact with robot control unit 610. Robot control unit 610 may interact with each of modification model 620, workpiece 602, modification equipment control unit 608, path planner 618, online positioning unit 616, and / or offline registration unit 614. The trimming model 620 may interact with each of the online sensor B 606, the machine learning unit 622, the trimming equipment control unit 608, and / or the robot control unit 610. When the system is used to trim the workpiece 602, each of the path planner unit 618 and the trimming equipment control unit 608 may provide inputs regarding specific tool paths and trimming characteristics to the robot control unit 610 based on the sensed positions of the surfaces of the workpiece 602 and the desired trimming parameters.

[0120] The machine learning unit 622 may be configured to predict process parameters for workpiece modification, such as input to the modification model 620. A non-exhaustive list of machine learning techniques that may be used for data obtained from the systems or methods of the present disclosure includes: support vector machines (SVM), logistic regression, Gaussian processes, decision trees, random forests, bagging, neural networks, deep neural networks (DNN), linear discriminants, Bayesian models, k-nearest neighbors (k-NN), and gradient boosting algorithms (GBA). However, it is expressly contemplated that other suitable machine learning techniques may be used.

[0121] refer to Fig. 7A To Figure E, given the image capture equipment parameters and the 2D image point set [x i y i ] T , i = 1, ..., n and a 3D point set [X i Y i Z i ] T, i=1,...,n, the 6D pose (3D translation and 3D rotation) of the target in the image capture device (e.g., camera) coordinate system can be estimated based on the perspective n-point (PnP) method. Unambiguously, the minimum number of 2D-3D correspondences for the PnP solution is four, but in practical applications, more correspondences are preferred so that the optimization-based PnP method can be performed for robust positioning when noise is present in the system. The method for calculating 2D-3D correspondences can be a major challenge in the process. Since all kinds of workpieces have a large number of edges, edge features (e.g., straight or curved) are selected instead of 2D texture or corner features that can be found on everyday objects (such as marked bottles and cardboard boxes) to identify 2D-3D correspondences. Other image features (e.g., Harris corners, ORB, SIFT, BRIEF, SURF, QR tags, etc.) may also or alternatively be used.

[0122] In 2D images, edge features are sharp changes in pixel intensity. Edge features can be extracted using, for example, the Canny edge detector, which is widely used as a default edge detector in image processing, or another suitable model. In order to achieve automatic edge detection and avoid manual adjustment of thresholds, a parameter-free Canny detector is implemented, which determines the lower and upper thresholds based on the median value of the image. Fig. 7A Edge features 710 of an object (eg, a part) obtained from an image capture device are illustrated.

[0123] In order to find 3D edge features in the CAD model that correspond to 2D Canny edges in the image (e.g., "3D model edges"), the edges from the CAD model need to be projected into a 2D image (e.g., "2D model edges") for subsequent search of correspondences in the image space. Model edges can be the result of two effects: (1) edge features based on curvature changes in the 3D model (represented here as "edges"), and (2) edge features based on depth changes in the 3D model (represented here as "contours"). Figure 7B illustrates edge features 720 of an object (eg, a part) obtained from a CAD model, and Figure 7C The outline of the object obtained from the CAD model is illustrated 730. A method of identifying these two types of edges in a 3D model is described as follows:

[0124] Edges: Edges of curvature are extracted by first calculating the angle between the normal directions of each adjacent mesh surface in the CAD model (e.g., in the form of a polygonal mesh), and then selecting element edges with angles above a predetermined threshold (e.g., 90°). Next, backface culling is implemented to remove invisible edges, given an initial guess of the camera pose, as they do not correspond to any 2D image edge features.

[0125] Contours: Contours are formed due to rapid depth changes of the model geometry measured in the image capture equipment coordinate system. The resulting contours in the image plane are shown as the boundary between the workpiece and the background. In order to reconstruct the contour edges based on the CAD model and an initial guess of the image capture equipment pose, a digital twin of the image capture equipment is implemented to render the CAD model. In the rendered image, the CAD model is masked in black and the background is masked in white, so that the Canny edge detector can be used to robustly extract the contour edges. The 3D coordinates of the extracted 2D contours can then be retrieved based on the depth data from the renderer.

[0126] Various software packages can be used for 3D edge extraction and contour detection, respectively. In some examples, these software packages may include application programming interfaces (APIs) and / or other combinations of modules / subroutines. In some non-limiting examples, such as when using PyVista and / or Pyrender, these software packages may include open source software libraries and / or free open source software libraries. CAD models in multiple formats are supported, including STL and OBJ. The overall method for workpiece positioning based on 2D-3D correspondence includes the following actions:

[0127] In the user interface, the user selects several 2D key points in the image that correspond to a predetermined set of 3D key points in the CAD model. The user is also presented with a schematic image of the CAD model with key point labels to guide the manual selection of 2D key points in the image.

[0128] The pose is estimated based on the 2D-3D correspondences specified by the user. This pose will be used as an initial guess of the pose of the image capture device in the subsequent workpiece localization step (ie, pose refinement based on the initial guess).

[0129] New 2D-3D correspondences are computed based on the current image capture device pose. The 2D model edges can be edges of curvature or edges of contours. To find the correspondence between the 3D model edges and the 2D Canny edges from the camera image, points are first sampled along the 2D model edges at a constant spacing and then projected to the image capture device image. Second, for each sampled point, the corresponding point on the Canny edge can be computed via a 1D search along a direction orthogonal to the 2D model edge. Fig.7DA schematic diagram illustrating the difference between edge features 720 and contour 730 of an object from a CAD model and edge features 710 of the object determined from an image capture device.

[0130] Update the pose estimate using the PnP-RANSAC method or another suitable method.

[0131] The calculation and update are repeated until the pose estimation result converges (eg, the error is below a predetermined threshold).

[0132] Fig. 7E Schematic diagram illustrating a 1D search method for finding 2D-3D correspondences in an image. The top angular line 740 represents the 2D model edge. Points 750 represent sampling points along the 2D model edge. The bottom curved line 760 represents the Canny edge.

[0133] The algorithm was tested using an iPhone 8 (rear) camera with a resolution of 12MP, an aperture of f / 1.8 and autofocus as the image capture device. The camera's intrinsic matrix and distortion parameters were calibrated using a printed chessboard.

[0134] The workpiece localization algorithm was first tested on a 3D printed block with a wavy top surface. Eight key points at the corners of the block were marked in the CAD model. For user initialization, a guidance image with marked key points was shown to the user to help manually select key points in the camera image. It should be noted that not all key points need to be identified by the user for successful initialization. The user can pick the key points with the best confidence and skip those that are invisible or difficult to identify (for example, the 8th key point was skipped in this example). In order to investigate the ability of the proposed method to compensate for large positioning errors in user initialization, a uniformly distributed artificial positioning error in the range (-2,2) mm was added to the workpiece position calculated based on user initialization.

[0135] The edge features of the CAD model are detected based on the edges of the contour and the edges of the curvature. Given an initial guess of the workpiece pose, the correspondence between the model edges and the Canny edges is computed via a bidirectional 1D search in the camera image. Ten pose refinement iterations are performed on the test image of the wavy block. Based on the workpiece pose from the initial guess and the final iteration, the 3D edges of the CAD model are projected to the camera image.

[0136] The localization error in image space is characterized by computing the Euclidean distance (L2 norm) from sampled points along the edge of the 2D model to their corresponding points along the Canny edge. Figure 8The average pixel error of all sampled points in each iteration is plotted in . For the wavy block, both edge-based and contour-based methods successfully correct the localization error from user initialization with similar convergence speed. The residual error may be caused by the geometric difference between the CAD model and the 3D printed part and the calibration error of the camera focal length.

[0137] Online tool path control based on part surface geometry

[0138] An advantage of methods according to at least some embodiments of the present disclosure is the ability to adjust a robotic tool path so that a workpiece modification device (e.g., a dispenser nozzle tip) accurately tracks a part surface geometry at a desired (e.g., dispensing) angle relative to the surface normal and a desired gap distance from the part surface. Given:

[0139] (1) A nominal tool path defined on the CAD model of the part and expressed in the robot coordinate system (based on the part registration process as described above).

[0140] (2) A two-dimensional surface distribution streamed from an online sensor A (e.g., a laser profilometer) mounted on the robot.

[0141] (3) The posture of the online sensor A relative to the TCP coordinate system fixed on the robot.

[0142] (4) The TCP pose of the workpiece modification equipment relative to the robot coordinate system, which can be streamed from the robot controller upon request. This combined with (2) and (3) can provide the position of the part surface distribution relative to the robot coordinate system.

[0143] The unknowns are target workpiece trimming equipment poses that maintain desired angles and gap distances relative to the part surface, as sensed in situ (e.g., in near real time) by online sensors A. Differences between these target workpiece trimming equipment poses and those from the nominal tool path require online tool path adjustments.

[0144] Fig.9A is a top view schematic of a laser line 910 from a laser profilometer that can be used in an embodiment of closed-loop part registration. The width d of the laser line can vary, for example, from 20 mm to 100 mm projected onto the part surface from a distance of 100 mm to 300 mm (although other widths and distances may also be suitable). A suitable laser sensor look-ahead distance L can be defined by the distance from the TCP to the laser projection plane (e.g., 20 mm to 100 mm). The laser sensor can be placed above the TCP tip so that the size of the laser sensor does not affect the reachable radius of the TCP tip. Reference Fig. 9B, provides a perspective schematic diagram of an assembly that includes a laser profilometer 920 attached to a robotic arm 930 that directs a laser line 910 to the front of a TCP of a workpiece modification equipment 940.

[0145] An example use of closed-loop part registration may also include allocation. Such an example is described herein. The tool path buffer stores a list of unfinished waypoints within the look-ahead distance. At each time step, waypoints are extracted from the tool path buffer and transmitted to the robot controller via a real-time data exchange (RTDE) interface. In the robot controller, each target waypoint in Cartesian space is transformed into a joint position via an inverse kinematics model and then fed to a proportional feedback controller for precise robot motion control. The actual joint position from the encoder is fed back to the RTDE interface after being transformed back to Cartesian space (i.e., TCP pose) via a forward kinematics model. At the same time, the laser profiler streams the part surface distribution to the scan data buffer. Once the current TCP pose becomes available from the RTDE interface, the latest surface distribution is extracted from the scan data buffer and transformed to the TCP coordinate system. This is done in an online positioning unit, which outputs the transformed surface distribution to a path planner unit. The path planner unit then calculates updated waypoints based on the nominal tool path and surface distribution (described in more detail below). The updated waypoints can be passed to the online path planner for path smoothing, or directly added to the toolpath buffer for future execution by the robot controller. The above robot controller architecture is implemented in Python 3.6 and runs on a Linux virtual machine on a 64-bit Windows 10 laptop (HP ZBook). The hardware specifications are as follows: Intel(R) Core(TM) i7-10850CPU@2.79GHz, 16GB memory.

[0146] The robot tool path is discretized into a series of waypoints. Fig. 10A , each waypoint w i is defined as a tuple consisting of TCP gesture p i and one of tool speed, tool speed and tool acceleration, a timestamp, or some combination of these. If a timestamp is specified, it takes precedence over tool speed and acceleration. In this example, a timestamp is specified.

[0147] The motion commands for each waypoint are transmitted to the robot via RTDE library functions that can take joint positions, timestamps, velocities, accelerations, look-ahead times, and gains as arguments. The inverse kinematics model can be used to generate the inverse kinematics from the Cartesian expression p iThe joint positions are calculated. lookahead is a variable that sets the lookahead time in order to smooth the trajectory. lookahead can be set to 0.03s, 0.04s, 0.05s, 0.06s, 0.07s, 0.08s, 0.09s, 0.10s, 0.11s, 0.12s, 0.13s, 0.14s, 0.15s, 0.16s, 0.17s, 0.18s, 0.19s, or 0.20s; for example, optionally in the range 0.03s to 0.2s. gain is the proportional gain for following the target position. If timestamps are specified, they take precedence over velocities and accelerations, otherwise velocities and accelerations are used to calculate the timestamps.

[0148] Fig. 10B An exemplary method for adjusting waypoint positions along a nominal path based on scanner data is shown in . First, both the nominal path 1010 defined in the robot base coordinate system and the scan profile 1020 defined in the scanner coordinates are transferred to the TCP coordinates. Second, the intersection of the nominal path 1010 with the laser projection plane 1030 is calculated, and the intersection becomes the adjusted waypoint. Then, a series of adjusted waypoints will form an actual tool path 1040 that conforms to the part surface (in addition, a dispenser nozzle gap offset value can be added to the final motion command).

[0149] For example, the TCP poses were recorded while the robot was traversing the tool path over a wrinkled paper surface (as a sample irregular part) with waypoint adjustments based on online scan data. FIG. 11A to FIG. 11C , respectively plotting the X component, Y component, and Z component of the robot TCP trajectory, including FIG. 11B to FIG. 11D The nominal path 1110 and the actual path 1120 in each of the two paths. It should be noted that the nominal path and the actual path are Fig.11A The overlap is so extensive that they are difficult to distinguish from each other. Fig.11D A 3D diagram of the robot TCP trajectory is shown. The robot follows a nominal path 1120 in the XY plane, as FIG. 11A to FIG. 11B The main adjustment occurs in the Z direction, where the Z component of the adjusted tool path follows the irregular shape of the wrinkled paper surface, as shown in Fig. 11C As a result, the executed tool path with closed-loop control deviates significantly from the nominal path in order to track the actual part surface.

[0150] In addition to adjusting the position of the end effector (e.g., the end-of-arm component that interacts with the surface, such as a dispenser or sander, etc.) based on the part surface distribution, it may also be necessary to adjust the end effector orientation (TCP pose), especially when the actual surface normal of the part deviates significantly from the surface normal of the nominal model (i.e., the CAD model). Here, the TCP coordinate system is defined so that the Y axis is aligned with the nozzle axis and the X axis is aligned with the direction of travel of the nozzle (e.g., the same direction as the look-ahead direction of a laser scanner). An axis-angle expression is used to express the 3D orientation adjustment: a rotation around the nozzle axis by an angle. Therefore, the online orientation adjustment can be divided into two steps. Reference Fig. 12A , first specify the direction of the Y axis based on the surface normal of the part (for example, the axial direction of the nozzle). Fig. 12B , and secondly, specify the rotation angle about the Y axis (e.g., rotation in the XZ plane). These principles also apply to any other type of workpiece modification equipment (e.g., sanders, polishers, cutters, drills, sprayers, welders, etc.).

[0151] Orientation adjustment based on surface normals

[0152] The surface normal of the part at a particular dispense location can be estimated based on multiple slices of laser scans obtained along the tool path before and after the dispense location. The surface distribution from the multiple laser scans forms a point cloud that approximates the part surface geometry around the nozzle location. A polynomial fit (e.g., a plane fit or a parabola fit) can be applied to the point cloud to analytically solve for the surface normal. Here, as an example, a plane is fitted to the point cloud, which can provide a good estimate of the local surface shape when the sampling area is small.

[0153] refer to Fig.13 , shows an exemplary schematic diagram of estimated surface normals of a curved part surface determined according to the above process. A local fitting plane 1300 is depicted, including three slices of a scan profile 1320 , three trajectory waypoints 1330 , and an axial direction of a nozzle 1340 .

[0154] Based on the calculated surface normal at each waypoint, the nozzle axis (Y axis) can be set to always align with the estimated normal direction of the part surface. Non-perpendicularity can also be applied to the orientation adjustment, for example, by first setting the nozzle axis to the surface normal direction, and then rotating the nozzle to maintain an arbitrary angle between the nozzle Y axis and the estimated normal direction of the workpiece surface (for example, rotating the nozzle around the travel / looking direction (X axis) by an angle α, such as Fig.14A ). This may be useful in scenarios such as avoiding collision of the nozzle 1440 with the part 1450 when dispensing the bead 1460 in a confined space, for example, as Fig. 14B14C . These principles are also applicable to any other type of workpiece finishing equipment (e.g., sanders, polishers, cutters, drills, sprayers, welders, etc.).

[0155] Surface normal estimation based on point clouds may be affected by errors caused by sensing noise from online sensors (e.g., laser profilometers). Additionally, small features (such as small step changes on the part surface) may cause non-smooth transitions of surface normals along the robot tool path. Given such problems with surface normal estimation, the robot may exhibit jerky movements when following surface normals, resulting in reduced quality of modification (e.g., dispensing). Two methods are proposed to address these issues. First, two subsequent surface normal vectors along the tool path (e.g., at t n-1 and t n If the angle is greater than a predetermined value, it will not be updated to timestamp t n Estimated surface normal. t n-1 The surface normal at will be assigned to t n . Second, outliers in the point cloud can be automatically identified and rejected before applying plane fitting to the surface normal estimate. Random Sampling Consensus (RANSAC) is a suitable example method for iteratively determining a set of outlier-free points in a point cloud (see, e.g., Fischler, Martin A., and Robert C. Bolles., "Random Sampling Consensus: A Model Fitting Paradigm with Applications to Image Analysis and Automated Cartography." Communications of the ACM, Vol. 24, No. 6 (1981): pp. 381-395). For example, RANSAC can be performed by the following actions:

[0156] (1) Randomly select three points from the point cloud to form a hypothetical set of inliers.

[0157] (2) Fit the plane (Ax+By+Cy+Z=0) to the assumed set of interior points.

[0158] (3) Calculate the distance from all other points in the point cloud to the fitted plane, which represents the plane fitting error.

[0159] (4) Points whose distance error is larger than a predetermined threshold are classified as part of the consistent set.

[0160] (5) If the size of the consistent set is large enough (e.g., greater than 90% of the total number of points in the point cloud), perform a refined plane fit (linear least squares estimation) on the consistent set. Otherwise, go to (1) until the maximum number of iterations set by the user is reached.

[0161] Although RANSAC is based on a random process, it can achieve (e.g., close to) real-time performance using a small number of iterations. It has been shown that the probability p of finding a set of outliers free of points can be calculated based on the following equation:

[0162] p=1-(1-(1-e) s ) N

[0163] Where e represents the probability that a point is an outlier, s represents the minimum number of points to fit the model (form a hypothesized inlier set), and N represents the total number of iterations. Given that for the plane fitting scenario, s=3, and assuming that 20% of the points in the point cloud are outliers (e=0.2), the number of iterations N required to have a probability of p=0.99 leading to a set without outliers can be solved:

[0164]

[0165] This means that, with 99% probability, only seven iterations at most are needed to produce an estimate of the surface normal that is not perturbed by outliers.

[0166] Surface normal tracking combined with online Z-tracking was tested on crumpled paper, and a nominal path was defined in the XY plane of the robot coordinate system. The time parameterization of the nominal path was set based on a tool speed of 20 mm / s. The robot orientation along the nominal path was set to be constant, with the nozzle axis aligned with the Z axis of the robot coordinate system and the look-ahead direction at each waypoint was set to be tangential to the tool path. The parameters of the surface normal estimation algorithm were set as follows: Subsampling of the laser points in each scan slice was applied to reduce the size of the final point cloud used for surface normal estimation. Specifically, according to the Z-tracking algorithm, one out of every four points was sampled around the adjusted waypoint. Reference Fig.15 , provides a graph of the calculated surface normal vectors plotted along the robot tool path.

[0167] Orientation adjustment for optimal viewing

[0168] After specifying the orientation of the nozzle axis, any rotation angle about this axis can be assigned to the nozzle without affecting the dispensing quality, since the nozzle has a centrosymmetric geometry. However, when considering a laser scanner assembly that projects a laser line in the look-ahead direction, such as Fig.16AAs depicted in , the rotation angle around the nozzle axis may affect the coverage of the scan area. Typically, the rotation angle is specified so that the look-ahead direction is aligned with the tangent direction of the tool path. Fig. 16B , this default setting may cause observability problems when the robot navigates a curved toolpath with a small turning radius. Let L denote the distance from the laser projection plane to the TCP, and d denote half the width of the projected laser line. The critical turning radius R c is defined as:

[0169]

[0170] When the turning radius R is less than the critical radius R c When R is greater than or equal to R, the tool path does not fall within the scanning area of ​​the laser line, so the laser sensor cannot observe the tool path. Therefore, it is not possible to adjust waypoints online along the tool path (e.g., for Z tracking) based on laser sensor feedback. c The laser can observe the tool path to enable potential closed-loop adjustments. For example, in the dispensing system tested, R c = 68 mm (L = 66 mm, d = 50 mm when the scanning height is 240 mm). However, in scenarios where dispensing around tight corners is required, this limitation of the turning radius of closed-loop adaptive dispensing may result in unpredictable bead quality at corner areas that cannot be observed by the line sensor.

[0171] In order to ensure that the waypoints to be updated are always observable in front of the look-ahead horizon, the rotation angle around the nozzle axis can be set so that the online sensor always points to the newly updated waypoints from the previous time step, for example FIG. 17A to FIG. 17B 17. The schematic top view illustration of the in-plane rotation about the nozzle axis along the tool path is depicted in . Without the in-plane rotation, when the robot follows the tangential direction of the tool path, the waypoints 1710 to be updated within the look-ahead horizon 1720 are not visible to the laser sensor. FIG. 17A to FIG. 17B The next waypoint 1730 to be followed and the front laser line 1740 are also indicated. Fig. 17B The most recently updated waypoint 1750 is also indicated. In the case of in-plane rotation, the robot orientation is adjusted so that the waypoint to be updated always falls within the field of view of the laser sensor. Specifically, when a new added slice of the surface distribution at the front laser line is added, the adjusted waypoint of the TCP in that scan slice becomes the most recently updated waypoint p j . Click p for the next route i After calculating the adjusted nozzle axis (e.g., based on surface normal estimation), the forward direction is adjusted from p to p by a so-called in-plane rotation (rotation in a plane perpendicular to the nozzle axis so that the forward direction is adjusted from p to p i Point to pj ) to calculate the new look-ahead direction.

[0172] Tracking Rib Features

[0173] On various parts (e.g., automotive parts such as spoiler assemblies), rib features of small height on the surface of the part are present as markers that indicate locations for adhesive bead deposition or ultrasonic welding. The ability to track these rib features allows adhesive to be accurately and repeatably deposited at the desired location to achieve optimal bonding performance.

[0174] Fig.18 A suitable method for adjusting the position of waypoints along the nominal path based on the rib position is depicted in FIG. First, the nominal path 1810 defined in the robot base coordinate system and the scan segment 1820 defined in the scanner coordinate system are both transferred to the TCP coordinate system. Second, the intersection of the nominal path 1810 with the laser projection plane 1830 is calculated by finding the first waypoint on the nominal tool path that is behind the laser projection plane, and the intersection is given by p j This will be the waypoint to be updated based on the actual rib position. Third, the rib peak is detected and the midpoint of the peak becomes the position of the adjusted waypoint, given by If there is no rib feature, a Z tracking strategy can be performed. Then, the j The timestamp assigned to A series of adjusted waypoints An actual tool path 1840 is formed that conforms to the part surface and also follows the midpoints of the parallel ribs.

[0175] If two rib peaks are detected from the distribution, the midpoint on the surface distribution between these two peaks can be calculated, which will be the location of the adjusted waypoint. The detection method was tested on a curved surface with rib features.

[0176] The rib feature tracking function combined with the orientation adjustment method described above was tested on a fixture with parallel ribs on a curved surface. The 3D nominal path was defined as a piecewise linear trajectory that passed through the spaces between the parallel ribs but did not follow the centerline exactly. The time parameterization of the nominal path was set based on a tool speed of 20 mm / s. The robot orientation along the nominal path was set to be constant, with the nozzle axis aligned with the Z axis of the robot coordinate system. Fig.19 A photograph of a test fixture 1910 having a curved surface 1920 and parallel ribs 1930 is shown, with the directions of the X-axis, Y-axis, and Z-axis of the robot base coordinate system marked.

[0177] like FIG. 20A to FIG. 20EAs shown in Figure 1, the TCP poses are recorded as the robot traverses the nominal tool path with rib feature tracking, including FIG. 20B to FIG. 20D Both the nominal path 2010 and the actual path 2020 in each of . It should be noted that the nominal path and the actual path are Fig. 20A The tool paths are adjusted in each of the X, Y, and Z directions so that the resulting trajectory accurately follows the centerline of the parallel ribs.

[0178] Tracking edge features

[0179] Another example of surface feature tracking is edge detection. The goal is to detect the edge of a part and track waypoints that have a desired offset from the detected edge. For example, a potential application in the automotive industry is to apply structural adhesive along the edge of a metal sheet for subsequent bonding to another panel to form a door panel assembly. Due to manufacturing tolerances, metal sheets can have variable edge locations. Without a closed-loop dispensing system that can perform in-line tracking of edge features, the dispensed adhesive bead cannot accurately follow the edge line on multiple panels with variable shapes, resulting in inconsistent bonding quality.

[0180] Fig.21 Such a method for adjusting the position of waypoints along a nominal path based on edge positions is depicted in . First, the nominal path 2110 defined in the robot base coordinate system and the scan profile 2120 defined in the scanner coordinates are transferred to the TCP coordinates. Second, the intersection of the nominal path with the laser projection plane 2130 is calculated by finding the first waypoint on the nominal tool path 2110 that is behind the laser projection plane, given by p j This will be the waypoint to be updated based on the scan distribution. Third, detect edge points in the scan distribution and calculate the adjusted waypoints so that it maintains a predetermined offset from the edge. Then, j The timestamp assigned to A series of adjusted waypoints An actual tool path 2140 is formed that conforms to the part surface and also follows the contour of the curved edges.

[0181] The edge feature tracking functionality combined with the orientation adjustment method described above was tested on a fixture with a 3D curved edge. The 3D nominal path was defined as a piecewise linear trajectory on one side of the curved edge. The goal was to adjust the nominal tool path to maintain a constant offset of 10 mm from the actual edge. The time parameterization of the nominal path was set based on a tool speed of 20 mm / s. The robot orientation along the nominal path was set to be constant, with the nozzle axis aligned with the Z axis of the robot coordinate system, as Fig. 22 , which is a photograph of a test fixture 2210 having a curved edge 2220, with the directions of the X-axis, Y-axis, and Z-axis of the robot base coordinate system marked.

[0182] The TCP poses were recorded as the robot traversed the toolpath with waypoint adjustments based on the online scan data. FIG. 23A to FIG. 23C As shown, the robot follows the actual edge contour in the X, Y and Z directions, including FIG. 23B to FIG. 23D Both the nominal path 2310 and the actual path 2320 in each of the two paths. It should be noted that the nominal path and the actual path are in Fig.23A The edges of the curved edge are so widely overlapped that they are not easily distinguishable from each other. Since the general trend of the edge line follows the X direction, the significant adjustments caused by online feature tracking are most noticeable in the Y and Z directions. Therefore, the adjusted tool path based on edge tracking deviates significantly from the nominal path in order to maintain a 10 mm offset from the curved edge, as shown in Figure 2. Fig.23D Drawn in.

[0183] Fig.25 An artifact decoration system architecture is illustrated. Architecture 2500 illustrates one embodiment of a specific implementation of artifact decoration system 2510. For example, architecture 2500 can provide computing, software, data access, and storage services without requiring the end user to know the physical location or configuration of the system delivering these services. In various embodiments, remote servers can deliver these services over a wide area network (such as the Internet) using appropriate protocols. For example, remote servers can deliver applications over a wide area network, and they can be accessed through a web browser or any other computing component. Figures 1 to 24 The software or components shown or described and the corresponding data may be stored on a server at a remote location. The computing resources in a remote server environment may be consolidated at a remote data center location, or they may be dispersed. The remote server infrastructure may deliver services through a shared data center, even though they appear as a single access point for users. Therefore, the components and functions described herein may be provided from a remote server at a remote location using a remote server architecture. Alternatively, they may be provided by a conventional server, installed directly on a client device, or provided in other ways.

[0184] exist Fig.25 In the example shown, some items are similar to those shown in previous figures. Fig.25Specifically shown is that the controller 2510 can be located at a remote server location 2502. Thus, the computing device 2520 accesses the controller 2510 through the remote server location 2502. The user 2550 can also use the computing device 2520 to access the user interface 2522. For example, the user 2550 can be a user who wants to check the progress of the workpiece modification while sitting in a parking lot and interacting with an application on the user interface 2522 of their smart phone 2520 or laptop 2520 or other computing device 2520 (e.g., an augmented reality (AR) device such as AR glasses).

[0185] Fig.25 It is shown that some elements of the system described herein are also expected to be arranged at the remote server location 2502, while other elements are not arranged at the remote server location. By way of example, data repository 2530, inspection system 2560 and decorator 2570 can be arranged at a location separated from location 2502, and accessed by the remote server at location 2502. No matter where it is located, data repository 2530 can be directly accessed by computing device 2520 through a network (wide area network or local area network), hosted at a remote site by a service, provided as a service, or accessed by a connection service residing in a remote location. Moreover, the data can be stored in substantially any location, and can be intermittently accessed by interested parties or forwarded to interested parties. For example, a physical carrier can be used to replace an electromagnetic wave carrier, or in addition to an electromagnetic wave carrier, a physical carrier can also be used. This can allow users 2550 to interact with controller 2510 through their computing device 2520.

[0186] It should also be noted that the elements of the system described herein or their parts can be arranged on a variety of different devices. Some of these devices include servers, desktop computers, laptop computers, embedded computers, industrial controllers, tablet computers or other mobile devices, such as handheld computers, cellular phones, smart phones, multimedia players, personal digital assistants, etc.

[0187] Figure 26 to Figure 28 Example apparatus that may be used in conjunction with the embodiments shown in the previous figures is illustrated. Fig.26 An example mobile device that may be used in the embodiments shown in the previous figures is illustrated. Fig.26 is a simplified block diagram of an illustrative example of a handheld or mobile computing device that can be used, for example, as a worker device or supervisor / safety officer device, in which the present system (or a portion thereof) can be deployed. For example, the mobile device can be deployed in an operator compartment of a computing device for generating, processing, or displaying data.

[0188] Fig.26A general block diagram of the components of a mobile cellular device 2616 that can run some of the components shown and described herein is provided. The mobile cellular device 2616 interacts with these components, or runs some components and interacts with them. In the device 2616, a communication link 2613 is provided, which allows the handheld device to communicate with other computing devices, and in some embodiments provides a channel for automatically receiving information (such as by scanning). Examples of communication links 2613 include allowing communication through one or more communication protocols, such as wireless services for providing cellular access to a network, and protocols for providing local wireless connections to a network.

[0189] In other examples, the application may be received on a removable secure digital (SD) card connected to the interface 2615. The interface 2615 and the communication link 2613 communicate with the processor 2617 (which may also be embodied as a processor) along a bus 2619, which is also connected to the memory 2621 and the input / output (I / O) components 2623, as well as the clock 2625 and the positioning system 2627.

[0190] In one embodiment, I / O components 2623 are provided to facilitate input and output operations, and the device 2616 may include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors, and output components such as display devices, speakers, and / or printer ports. Other I / O components 2623 may also be used.

[0191] The clock 2625 illustratively includes a real-time clock component that outputs the time and date. The clock can also provide a timing function for the processor 2617.

[0192] Illustratively, positioning system 2627 includes components that output the current geographic location of device 2616. The positioning system may include, for example, a global positioning system (GPS) receiver, a LORAN system, a dead reckoning system, a cellular triangulation system, or other positioning system. It may also include, for example, mapping software or navigation software that generates desired maps, navigation routes, and other geographic functions.

[0193] Memory 2621 stores operating system 2629, network settings 2631, applications 2633, application configuration settings 2635, data repository 2637, communication driver 2639, and communication configuration settings 2641. Memory 2621 may include all types of tangible volatile and non-volatile computer-readable memory devices. It may also include computer storage media (described below). Memory 2621 stores computer-readable instructions that, when executed by processor 2617, cause the processor to perform computer-implemented steps or functions according to these instructions. Processor 2617 may also be activated by other components to facilitate its functions. It is clearly contemplated that, although physical memory repository 2621 is illustrated as part of the device, cloud computing options may be used, in which some data and / or processing is completed using remote services. Although shown as a single unit for ease of illustration, it should be understood that in various examples according to various aspects of the present disclosure, the processor 2617 may include one or more processors, including one or more microprocessors, CPUs, GPUs, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), processing circuits (e.g., fixed function circuits, programmable circuits, or any combination of fixed function circuits and programmable circuits), or any other equivalent integrated or discrete logic circuits, and any combination of such components. The term "processor" or "processing circuit" may generally refer to any of the above logic circuits, alone or in combination with other logic circuits, or any other equivalent circuit. A control unit including hardware may also perform one or more of the techniques of the present disclosure.

[0194] Fig. 27 The device is shown to be a smart phone 2771. Smart phone 2771 has a touch-sensitive display 2773 that displays icons or tiles or other user input mechanisms 2775. The user can use the mechanisms 2775 to run applications, make phone calls, perform data transfer operations, etc. In general, smart phones 2771 are built on mobile operating systems and provide more advanced computing capabilities and connectivity than non-smart phones. It should be noted that other forms of devices are possible. However, although Fig. 27 An embodiment is illustrated in which the device 2700 is a smart phone 2771, but it is expressly contemplated that the display may be presented on another computing device.

[0195] Fig.28 is an example of a computing environment in which elements or (for example) portions of the systems and methods described herein may be deployed. Fig.28, an example system for implementing some embodiments includes a general purpose computing device in the form of a computer 2810. Components of the computer 2810 may include, but are not limited to, a processing unit 2820 (which may include a processor), a system memory 2830, and a system bus 2821 that couples various system components including the system memory to the processing unit 2820. The system bus 2821 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The memories and programs described with respect to the systems and methods described herein may be deployed in Fig.28 in the corresponding part of .

[0196] Computer 2810 typically includes various computer-readable media. Computer-readable media can be any available media that can be accessed by computer 2810, and includes both volatile / non-volatile media and removable / non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media. Computer storage media is different from, and does not include, modulated data signals or carrier waves. Computer storage media include hardware storage media that include volatile / non-volatile and removable / non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disc (DVD) or other optical disk storage, cassettes, tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by computer 2810. Communication media can embody computer-readable instructions, data structures, program modules, or other data in a transport mechanism, and include any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0197] The system memory 2830 includes computer storage media in the form of volatile and / or nonvolatile memory, such as read-only memory (ROM) 2831 and random access memory (RAM) 2832. A basic input / output system 2833 (BIOS), containing the basic routines that help to transfer information between elements within the computer 2810 (such as during startup), is typically stored in ROM 2831. RAM 2832 typically contains data and / or program modules that are immediately accessible to and / or currently being operated on by the processing unit 2820. By way of example and not limitation, Fig.28Operating system 2834 , application programs 2835 , other program modules 2836 , and program data 2837 are illustrated.

[0198] The computer 2810 may also include other removable / non-removable and volatile / non-volatile computer storage media. By way of example only, Fig.28 Illustrated are a hard disk drive 2841 that reads or writes to a non-removable, non-volatile magnetic medium, a non-volatile magnetic disk 2852, an optical drive 2855, and a non-volatile optical disk 2856. The hard disk drive 2841 is typically connected to the system bus 2821 through a non-removable memory interface, such as interface 2840, and the optical drive 2855 is typically connected to the system bus 2821 through a removable memory interface, such as interface 2850.

[0199] Alternatively or in addition, the functions described herein can be performed at least in part by one or more hardware logic components. For example, but not limited to, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (e.g., ASICs), application specific standard products (e.g., ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), etc.

[0200] Discussed above and in Fig.28 The drives and their associated computer storage media illustrated in FIG. 28 provide storage means for computer readable instructions, data structures, program modules and other data for the computer 2810. Fig.28 , for example, hard disk drive 2841 is illustrated as storing operating system 2844, application programs 2845, other program modules 2846, and program data 2847. It should be noted that these components may be the same as or different from operating system 2834, application programs 2835, other program modules 2836, and program data 2837.

[0201] A user may enter commands and information into the computer 2810 through input devices such as a keyboard 2862, a microphone 2863, and a pointing device 2861 (such as a mouse, trackball, or touch pad). Other input devices (not shown) may include a joystick, a game pad, a satellite dish, a scanner, or the like. These and other input devices are typically connected to the processing unit 2820 through a user input interface 2860 coupled to the system bus, but may also be connected through other interface and bus structures. A visual display 2891 or other type of display device is also connected to the system bus 2821 via an interface such as a video interface 2890. In addition to a monitor, the computer may also include other peripheral output devices such as speakers 2897 and a printer 2896, which may be connected through an output peripheral interface 2895.

[0202] The computer 2810 operates in a networked environment with one or more remote computers, such as the remote computer 2880 , using logical connections, such as a local area network (LAN) or a wide area network (WAN).

[0203] When used in a LAN networking environment, the computer 2810 is connected to the LAN 2871 through a network interface or adapter 2870. When used in a WAN networking environment, the computer 2810 typically includes a modem 2872 or other components for establishing communications over the WAN 2873 (such as the Internet). In a networking environment, program modules may be stored in a remote memory storage device. For example, Fig.28 It is illustrated that remote application programs 2885 can reside on remote computer 2880.

[0204] Velocity-based closed-loop control of bead shape

[0205] Aspects of the present disclosure relate to closed-loop control based on speed of bead shape.Automation capabilities are widely used in the application of liquid adhesives and heat-bonded thermoplastics.The development of adhesives has produced dispensable pressure-sensitive adhesives (PSAs) and structural adhesives.The adhesive (RDA) platform of automatic dispensing described above better solves the user's demand for complex bonding applications by providing an automatic dispensing solution.In this process, it is observed that the material flow rate is inconsistent, which is caused by factors such as batch-to-batch variation of adhesive materials and back pressure at the beginning of the dispensing process.These material flow rate inconsistencies may cause inconsistent bead geometries, which tend to have a negative impact on output and bonding performance.

[0206] The disclosed system solves and mitigates several potential problems. For example, the disclosed system can address the need for online bead sensing for cycle-to-cycle dispensing process control. During cycle-to-cycle dispensing control, bead shape data collected from past process cycles is used to generate control commands for the current cycle. This helps compensate for system and environmental uncertainties / disturbances that continuously or slowly change over multiple process cycles, such as batch-to-batch variations in adhesive materials and humidity / temperature fluctuations on the production line.

[0207] Bead shape data for cycle-to-cycle process control can be obtained using either an offline or online sensing system. Offline sensing requires two separate dispensing and scanning operations in one cycle using a single robotic arm (dispense-then-scan process), or two robotic arms working in parallel for dispensing and bead sensing. Alternatively, online bead sensing with coordinated robotic motion requires only one run per cycle using a single robotic arm (dispense-while-scan), with the potential benefit of reduced cycle time and system complexity compared to offline sensing.

[0208] Another example of a potential problem solved by the system of the present disclosure is the need for online bead sensing and the ability to operate a speed-based controller to provide intra-cycle process control to improve bead quality and yield. During cycle process control, the system of the present disclosure can collect bead shape data in real-time to generate control commands for the current cycle. Intra-cycle control provides a faster response to detected bead defects than inter-cycle process control. Intra-cycle control not only compensates for slowly changing disturbances from the system and environment, but also compensates for occasional disturbances that cause inter-cycle variations in material flow rate. Working together with the inter-cycle process controller, the intra-cycle controller of the present disclosure can further improve the bead quality on each sample and can reduce the number of defective samples, thereby resulting in higher yields for the end user.

[0209] The online bead sensing system for intra-cycle control can capture bead parameters with low latency and high time resolution, and combine with advanced detection algorithms to robustly extract bead parameters, such as bead shape and position. In view of the detected bead shape error, a control strategy is to adjust the material flow rate by changing the distribution variable, which is such as extruder motor speed (for example, for screw drive dispenser), pump pressure (for example, for pressure drive dispenser) or temperature (for example, for hot melt adhesive). There are several disadvantages in this method for intra-cycle control, such as the complexity of the time-varying multi-physics model representing the distribution system is very high, which makes it difficult to calculate the correct control input based on the expected material flow rate, or the control time window may be missed due to the low responsiveness of the distribution variable (for example, for hot melt adhesive distribution, after commanding a new extruder motor speed or heater temperature set point, due to factors such as the viscoelasticity of the adhesive material fed through a long path (for example, a hose) and the long heating / cooling time required to reach the target material temperature, there is usually a significant time delay before the nozzle tip reflects the expected flow rate change).

[0210] As an alternative control strategy, the speed-based bead shape controller of the present disclosure can provide faster compensation for bead shape errors without directly controlling material flow rate. Robot tool speed can be adjusted more quickly in response to over-extrusion or under-extrusion detected by the bead sensing system than dispensing variables such as pressure and temperature. These controllers of the present disclosure are also designed and constructed to handle the inevitable sensor noise and delay from the online sensing system to deliver smooth and stable robot motion at variable tool speeds.

[0211] The disclosed systems and techniques relate to material dispensing with in-line bead sensing and closed-loop feedback control of tool speed to improve the accuracy of the dispensing process. Fig.29 An example system of the present disclosure is illustrated. An example of the present disclosure includes one or more of the following:

[0212] 1. A system having a dispensing robot equipped with a bead sensor (an example of which is described below);

[0213] 2. Estimation process of bead / seam parameters based on laser profilometer data (an example of which is described below)

[0214] a. Bead / seam position estimation process based on rising / falling edge detection (an example of which is described below);

[0215] b. Bead geometric parameter (e.g., width, thickness, cross-sectional area) estimation techniques based on bead distribution segmentation (examples of which are described below);

[0216] 3. Robotic tool velocity control techniques for compensating for detected bead shape errors (an example of which is described below)

[0217] a. A path streaming technique with variable tool speed and consistent time step duration for time-based reparameterization (an example of which is described below); and

[0218] b. A technique based on determining tool speed setpoints by means of noise filtering and condition checking of time series data of bead parameters and tool speed (an example of which is described below).

[0219] Various aspects of the present disclosure use feedback data from a bead sensor to adjust the dispensing speed in order to compensate for bead shape errors. For example, various techniques of the present disclosure employ tool speed as an independent variable to correct detected bead shape errors, using a robust speed controller that takes into account online bead parameter estimation, noise handling, and steady-state checks.

[0220] Robotic dispensing techniques, such as those associated with advanced robotics and / or RDAA platforms, are described in International Patent Applications Publication Nos. WO2020 / 174394, WO2020 / 174397, WO2021 / 074744, and WO2021 / 124081, the entire disclosure of each of which is incorporated herein by reference. The velocity-based bead control aspects of the present disclosure represent an improvement in the field of closed-loop robotic dispensing devices and techniques for compensating for part shape and bead shape variations.

[0221] Fig.30 A non-limiting example of a closed-loop robotic dispensing test station of the present disclosure is illustrated. Fig.30The system includes a six-degree-of-freedom (6-DoF) robotic arm (such as the model UR 10 available from Universal Robots), a hot melt adhesive dispenser mounted on the robotic arm, a first laser profilometer (part scanner) mounted on one side of the dispenser for part surface scanning, and a second laser profilometer (bead scanner) mounted on the other side of the dispenser for bead shape sensing. Fig.30 In the particular example of FIG. 5 , both scanners are laser profilometers, but it should be understood that other types of scanners are also compatible with the system of the present disclosure.

[0222] Fig.31 is a block diagram illustrating an example closed-loop robotic allocation in accordance with aspects of the present disclosure. Fig.31 The "bead scanner" box and the "speed adjustment" box of the present disclosure relate to the speed-based bead shape control aspect of the present disclosure. In one example, Fig.31 The controller program is implemented in Python 3, which can be used on a system equipped with a or based on In this example, the PC-based controller uses The cable connects to the URCB3 controller via a TCP / IP connection. In this example, the various actions for streaming robot motion commands and reading the current robot pose communicate via a Real-Time Data Exchange (RTDE) interface at a frequency of 125Hz (maximum). The PC-based controller also uses The cables are connected to the part scanner and bead scanner via UDP. The 2D laser profiles are streamed from the scanner to the scan data buffer on the PC-based controller at up to 4kHz using the scanCONTROL Software Development Kit (SDK) from Micro-Epsilon. Fig.31 In the example of , within the UR robot controller, each target waypoint in Cartesian space is transformed into a joint position via the robot's inverse kinematics model and then fed to a proportional feedback controller for motion control. The actual joint positions from the encoders are fed back to the RTDE interface after being transformed into the tool center pose (TCP) via the robot's forward kinematics model.

[0223] exist Fig.31In the example of , a nominal path that defines the projected adhesive bead locations on the workpiece / part is first generated within a PC-based controller using an offline path planner, which includes part registration methods and a CAD-to-path process (where "CAD" stands for computer-aided design). The user-defined nominal path is then discretized into a series of waypoints for the robot to execute. At each calculation step during the closed-loop dispensing process (e.g., a cycle time of 10ms to 20ms), the current robot TCP is first read from the robot controller. Next, the most recent laser profiles from the scan data buffers of both scanners are transformed to the robot TCP coordinate system for determining adjustments to the waypoint poses and velocities. Next, the updated waypoints are added to the tool path buffer for future execution by the UR motion controller.

[0224] Fig.32 is a schematic diagram showing an example format of a tool path for the closed-loop robotic dispensing technique of the present disclosure. Each waypoint w along the robot tool path i is defined as a tuple consisting of a time step duration Δt i and TCP attitude p i In some examples of experiments conducted with the disclosed technique, the tool velocity was not explicitly defined in the waypoint expression (direct velocity control was not provided by the UR CB3 controller), but could be controlled by varying the time step duration Δt i The value of is implicitly controlled. Fig.32 In the example, the motion command for each waypoint is sent to the robot via the RTDE command servoJ(q,t,lookahead,gain). q is the vector of joint positions, which can be obtained from the Cartesian expression p using the inverse kinematics model. i Calculated. Δt represents the blocking time to move to the next posture, which is related to Δt i Lookahead is a variable used to set the lookahead time to smooth the trajectory (the range is 0.03s to 0.2s). Gain is the proportional gain for tracking the target position (the range is 100 to 2000).

[0225] According to various aspects of the bead sensing technology disclosed herein, the 2D surface distribution captured by the bead scanner can be expressed as a point in the XZ plane. In this example, the Z axis defines the direction of the laser projection. The bead scanner used for the specific experiment of the present disclosure has a measurement range of [190,290] (mm) along the Z axis and a measurement range of [-72,72] (mm) along the X axis. In each calculation cycle, the bead sensing algorithm inputs a 2D distribution consisting of 640 points as raw data at run time, and extracts useful bead parameters via bead position detection (described in more detail below) and bead geometry estimation (described in more detail below). These parameters are saved and used as sensor inputs for a velocity-based bead shape controller, which will be described in more detail below.

[0226] FIG. 33 illustrates aspects of bead position detection according to aspects of the present disclosure. Fig.33A A schematic diagram showing the detection of rising and falling edges within a bead or seam distribution F(x) (indicated by arrows) via detection of zero crossing positions in the second derivative F"(x) and thresholding in the first derivative F'(x). Fig.33B An example graphical user interface (GUI) for configuring a search window for bead position detection is shown (shown in box). The distribution of beads on the substrate can be characterized as a rising edge followed by a falling edge in the positive x-direction. The shape of the seam is the opposite of the beads, with a falling edge followed by a rising edge in the positive x-direction ( Fig.33A ). Given this similarity, the bead position detection algorithm based on rising / falling edge detection can also be used for seam detection in applications such as seam tracking during robotic welding. An example bead detection technique of the present disclosure is described below.

[0227] According to this particular example, first, the system of the present disclosure may receive user input that defines a refined search window in the distribution where beads are most likely to appear. For example, the system of the present disclosure may implement a GUI that enables a user to provide input that defines the center of the search window using a reference index of a selected center point in the distribution (e.g., near the nozzle location). The search range (e.g., Fig.33B Refining the search window can alleviate or even potentially eliminate ambiguity in bead detection when multiple rising and falling edges are present on the part surface near the bead.

[0228] Next, first-order and second-order Gaussian filters are applied to the surface distribution within the user-defined search window, resulting in Fig.33AThe degree of smoothing used to remove noise is controlled by the standard deviation parameter of the Gaussian filter, which can be received as user input via the GUI (e.g., as Fig.33B The pseudo code for the bead position detection function is presented below:

[0229]

[0230] Examples of the disclosed bead geometry estimation techniques are described below. Figure 34 illustrates bead geometry estimation for two scan profile samples. Fig.34A is the scanning distribution of a round bead on an inclined and curved substrate. Fig.34B exemplifies a scan profile of a flat bead on a flat substrate, where the point data at the bead boundary (e.g., points below the substrate plane) are discontinuous and erroneous. Fig.34A and Fig.34B In both, the user-defined search window is indicated using a box with a dashed border. The center of the search window is indicated using a circle with a solid border, the bead position is indicated using a cross, the rising and falling edges are indicated using adjacent circles, and the fitted base plane is indicated using a marker line.

[0231] The goal of the bead geometry estimation process is to extract bead shape characteristics that define the quality of the deposited bead, such as width, maximum and / or average thickness, and cross-sectional area. The rising and falling edges from the previous bead position detection step are estimates of the bead boundaries. For example, the detected bead boundaries (rising / falling edges) are zero curvature locations that may differ from the true bead boundaries by several millimeters (e.g., 0.0005 mm). Fig.34B As another example, the portion of the scan profile near the bead boundary is sometimes affected by sparse and noisy data points caused by high discontinuities and reflections between the bead and substrate surfaces (e.g. Fig.34B ). Therefore, these edge locations cannot be used to estimate bead shape properties (such as width) with reliable high accuracy and consistency.

[0232] The bead geometry estimation technology disclosed in the present invention refines the position of the bead boundary by segmenting the base distribution from the bead distribution, providing a technical solution to the above-mentioned technical problems. Assuming that the bead is located on a portion of a relatively "flat" base with low curvature (which may be the case for many or perhaps most applications), the bead geometry estimation technology disclosed in the present invention can use random sampling consensus (RANSAC) to fit a straight line to the "flat" portion of the base distribution to remove noise and outliers, followed by linear regression to find the best line fit. All points above the fitted base line distribution can be considered as part of the bead and can be used to calculate the bead width, thickness and / or cross-sectional area. The pseudo code for the example implementation of the technology is presented as follows:

[0233]

[0234]

[0235] Various aspects of the bead shape control techniques of the present disclosure are described below. Some bead shape control techniques of the present disclosure involve path streaming with speed control. In order to achieve the purpose of online bead shape control, the path streaming method needs to be compatible with adjustable tool speed. In order to improve stability and motion smoothness, robot manufacturers often recommend streaming path commands at a higher frequency. For example, for path streaming to a UR 10 robot via an RTDE interface, it is currently recommended to stream waypoint commands at a constant rate of 125Hz. Each waypoint can be represented by a time interval Δt (travel time from the previous waypoint to the current waypoint), a position p = [x, y, z], and an orientation o = [α, β, γ] (Euler angles). By setting Δt to a constant value (e.g., 8ms), a smooth path can be achieved using the following path streaming method for online part shape compensation (e.g., in Fig.31 In the example):

[0236]

[0237]

[0238] The executed path has a different velocity profile than the nominal path. For example, the nominal tool velocity at the jth waypoint can be estimated by v j =‖p j -p j-1 ‖ / Δt j , and the tool speed performed can be estimated by When the adjusted waypoint Approaching the nominal waypoint p jHowever, for applications where the bead quality is sensitive to such deviations of the tool speed from the nominal value, the time interval Δt can be adjusted online to maintain the nominal tool speed:

[0239]

[0240] As the waypoint locations used for part shape compensation change, the time intervals become inconsistent along the adjusted toolpath as a trade-off with maintaining nominal tool speed:

[0241]

[0242] Such uncontrolled update rate changes present the following potential problems:

[0243] Too small a time interval may reach the lower limit of the time interval for message exchange with the robot controller. For example, the streaming of waypoints to the UR 10 robot via the RTDE interface cannot exceed 125Hz (minimum time interval is 8ms);

[0244] Too small an interval may reach the lower limit of the processing time required to complete each calculation cycle. Fig.35 , which is a graph illustrating a time consumption log of each computing cycle during a closed-loop allocation test run. Fig.35 As shown), the cycle time allocated for processing sensor data and calculating control commands should not be less than 14ms.

[0245] Too large a time interval may cause the robot to move in a jagged manner due to the coarse temporal discretization of the pose commands.

[0246] Fig.36 is a schematic diagram showing an example of how changes in waypoint position ||p|| (the highest horizontal axis in each of the three illustrated boxes) and tool speed v (the lowest horizontal axis in each of the three illustrated boxes) can affect the time parameterization t along the tool path (the middle horizontal axis in each of the three illustrated boxes). In the upper box, the time parameterization along the nominal path is configured to be evenly spaced with a constant update time interval of 20ms. In the middle box, when the position of each waypoint is adjusted for part shape compensation, changes in the time intervals are introduced along the resulting adaptive path. In the lower box, when tool speed adjustments are introduced into the process, the level of change in the time intervals is further increased, with a maximum time interval of 75ms and a minimum time interval of 7ms.

[0247] The problem of inconsistent time intervals becomes more severe when adaptive tool speed control is applied using feedback data from the bead scanner (e.g. Fig.36In addition to changing the waypoint position for part shape compensation, the tool speed Δv for bead shape compensation is adjusted. j (relative to nominal tool speed) also contributes to the change in time interval, which is related as follows:

[0248]

[0249] FIG37 illustrates a schematic diagram showing the proposed time reparameterization strategy. Fig.37A In the example of , new waypoints (unshaded dots) are interpolated at 20ms intervals based on the fact that the current time interval (40ms) is greater than the expected time interval (20ms). Fig.37B In the example of , the current waypoint is skipped based on that the current time interval (10 ms) is less than the expected time interval (20 ms).

[0250] To address potential issues related to inconsistent update rates along the adjusted toolpath, the disclosed technique provides time reparameterization for each command execution cycle. c (with an allowable range [Δt min ,Δt max ])'s time reparameterization strategy is as follows:

[0251] (1) If Δt k Less than Δt max and is greater than Δt min , then execute the current waypoint command [p k ,o k ,Δt k ];

[0252] (2) If Δt k Greater than Δt max , then at a constant time interval Δt c Interpolate new waypoints at previously commanded waypoints [p k-1 ,o k-1 ,Δt k-1 ] and the current waypoint [p k ,o k ,Δt k ]between( Fig.37A );and

[0253] (3) If Δt k Less than Δt min , then skip the current waypoint [p k ,o k ,Δt k ] for execution ( Fig.37B ).

[0254] The modified path streaming method with time reparameterization is as follows:

[0255]

[0256]

[0257] FIG38 illustrates the effectiveness aspects of utilizing the disclosed path streaming technique with temporal reparameterization for variable speed closed loop distribution. An additional benefit of temporal reparameterization is that the part surface can be sampled at a higher temporal resolution than without temporal reparameterization. Conformal tool paths planned based on this surface mesh with higher fidelity can produce better bead quality with more precise gap height control. Fig.38A Experimental results of closed-loop allocation using the path streaming technique without temporal reparameterization are shown. Fig.38B Experimental results of closed-loop distribution using the path streaming technique with time reparameterization are shown. The first row of plots shows the velocity profiles for the nominal path ("Nominal") and the adjusted path ("Planned") for bead shape compensation. The second row of plots shows the time parameterization along the adjusted tool path, where the desired time interval is 20 ms. The third row of plots shows the sampled point cloud from the part scanner. The adaptive tool path for part shape compensation is indicated by the trajectory from left to right (and ascending).

[0258] FIG. 39 illustrates the results of bead shape compensation using a simple control law. Fig.39A The bead width distributions of three runs with good control quality (top curve, bead width converges to the target width), poor control quality (middle curve, bead width deviates from the target width), and medium control quality (bottom curve, bead width oscillates around the target width) are shown. Fig.39B Tool velocity profiles are shown for the nominal path (“Nominal”), the adjusted / commanded path (“Planned”), and the executed path (“Executed”, smoothed using a moving average filter with a window size of 20).

[0259] Given that tool speed is the only independent variable for bead shape control, one of the bead geometry parameters estimated by the bead sensing algorithm can be selected as the dependent variable to be controlled along the tool path. Since bead thickness can be effectively controlled via gap height tracking using a part scanner, bead width can be considered the dependent variable in this example. Without loss of generality, this control law can be applied to other bead parameters, such as cross-sectional area.

[0260] The control law of the present disclosure is based on the following constant volume assumption: when Δt approaches zero, the volume of deposited material in a constant time period Δt can be approximated as a constant by two adjacent time steps. This assumption is valid when the material flow rate varies smoothly or remains constant over time. Let w and w d Denote the bead width measured from the current time step and the desired bead width for the next time step, respectively. Let v and v d denote the tool velocity measured from the current time step and the desired tool velocity for the next time step, respectively. The constant volume assumption gives the following:

[0261] w·v·Δt=w d ·v d ·Δt

[0262] The expected tool speed for the next time can then be calculated using the following formula:

[0263]

[0264] A simple approach for bead shape control is to reduce the difference between the current tool speed and the desired tool speed. Let g denote the adjustable gain for tool speed regulation. The tool speed command v for the next time step can then be calculated using c :

[0265] v c =v+g·(v d -v)

[0266] exist Fig.39A The example of is shown in Figure 1. The distribution of bead widths along three dispensing paths using the simple control law described above. In this particular example, the controller does not provide repeatable bead shape control quality, with the bead width sometimes deviating from the desired width or oscillating around the desired width. The tool velocity distribution from the test run reveals problems with the controller (e.g. Fig.39B In the example of t of , that is, the adjusted tool velocity command calculated by the control law is excessively oscillatory and cannot effectively correct the bead shape error. Additionally, even if the oscillating velocity command can compensate the bead shape error to some extent, the actual velocity profile of the executed tool path may be different from the command value due to the dynamic limitations of the robot arm. There are three reasons for the oscillatory behavior of the calculated velocity command, namely:

[0267] (1) Bead width measurement delay w. To avoid the light path being blocked by the extruder body, the laser line from the bead scanner is projected at an offset distance from the nozzle (e.g. Fig.40B The control commands calculated based on this delayed measurement may lead to instability issues (oscillatory motion) and bead width response overshoot (e.g. Fig.40A shown).

[0268] (2) Bead width measurement result w (such as Fig.39A ) and tool speed measurement results v (as Fig.40C These noise values ​​are transferred to the controller without any noise compensation, resulting in an oscillating speed command v c .

[0269] (3) The transient duration of bead width w and tool speed v is long (as shown in Figure 41). After the tool speed changes or when the flow rate fluctuates in the extrusion system, the bead width is in a transient state (e.g. Fig.41A After the tool speed set point is commanded and before the set point is reached, the tool speed is in a transient state (as shown in Fig.41B ). The duration of this transient depends on the gain settings of the robot motion controller and the acceleration / deceleration limits of the robot. When using these transient variables to calculate control commands, the constant volume assumption may not be valid, resulting in bead width tracking errors.

[0270] FIG. 40 illustrates one or more potential problems with using a simple control law for bead shape compensation. Fig.40A is a schematic diagram (top view) of a nozzle (circle with solid line border) through a straight path. The tracking error of the bead width is caused by delayed sensing of the bead scanner (vertical line indicates laser line). In each of the graphs shown in FIG40 , the deposited bead is indicated by the filled (or shaded) area and the desired bead width is indicated by the two dashed lines. The laser line is indicated by the vertical line intersecting the deposited bead and the desired bead. Fig.40B The delayed sensing distance of the bead scanner is shown. Fig.40C The tool velocity profile read from the robot controller is shown.

[0271] Figure 41 illustrates the transient behavior of bead width and tool speed along the dispensing path. Fig.41A is a graph of the bead width distribution along the dispensing path, where the transient state is indicated by a box with a dashed border line (and labeled “Transient”) and the steady state is indicated by a box with a dashed border line (and labeled “Steady State”). Fig.41B is a graph of tool velocity profiles along a dispense path, with commanded velocity indicated by the "Planned" curve and measured / executed velocity indicated by the "Executed_Original" curve. Transients in the executed velocity are indicated by boxes with dashed border lines (labeled "Transient").

[0272] Various aspects of the control law with signal filtering and steady-state checks are described below. To address the various potential problems described herein, the techniques of the present disclosure include an advanced bead shape control method that performs signal filtering and steady-state checks on the measured bead width w and tool speed v. According to these techniques, a sliding average is used for signal filtering of the bead width and tool speed measurements. As new measurements are received from / arrived at the sensor during each calculation cycle, an average of a series of measurements from a past time window is calculated as the filter output. A specific implementation of the algorithm can be described using the following pseudo code:

[0273]

[0274] The algorithm sliding average (shown in the pseudo code block above) acts as a low pass filter to remove high frequency noise present in the measured bead width and tool speed. Incrementally update the sliding average and sliding variance σ n 2 This method improves computational efficiency and achieves optimal real-time performance.

[0275] In each calculation cycle, the mean and standard deviation of bead width and tool speed, which are the output of the algorithm sliding average, are fed to two steady-state checkers for bead width and tool speed, respectively. The binary values ​​returned from the two steady-state checkers indicate the preparation of the bead width based on the filtered and tool speed Calculate the new tool speed command v c .

[0276] FIG42 illustrates a schematic diagram of a steady-state checker. Fig.42A Illustrated is a steady state check for bead width measurements. The "Return False" and "Return True" areas indicate the time steps at which false and true values ​​are returned by the algorithm, respectively. Fig.42B Steady-state checks for tool speed measurements are illustrated. The "Return False" area indicates the time steps at which the steady-state distance d is zero (returns false), and also indicates the time steps at which the steady-state distance d increases (returns false). The "Return True" area indicates the time steps at which the steady-state distance d is greater than a threshold value d_thres (returns true). Fig.42C is a schematic diagram showing that the minimum steady-state distance d_thres is equal to the delayed sensing distance of the bead scanner.

[0277] The algorithm used to check bead homeostasis can be described using the following pseudo code (with Fig.42A Related):

[0278]

[0279] The conditions for tool velocity steady state (with Fig.42B Related):

[0280]

[0281]

[0282] In the algorithmic speed steady-state checker presented above, the minimum steady-state distance d thres Set to be equal to or greater than the delay sensing distance of the bead scanner (with Fig.42C The overall control law with signal filtering and steady-state checking can be described by the following pseudo code:

[0283]

[0284]

[0285] Fig.43 The tool speed distribution in the upper figure and the bead width distribution in the lower figure along a straight dispensing path in a test run with bead shape control are illustrated. In the upper figure of tool speed versus time, the "nominal" curve indicates the nominal speed, the "planned" curve indicates the commanded / adjusted speed, the "execution_raw" curve indicates the raw speed measurement, and the "execution_smoothed" curve indicates the filtered speed measurement. The "steady state" area indicates the time step in which the steady state distance d increases. In the lower figure of bead width versus time, the "expected width" line indicates the expected width, while the "sliding average of bead width" curve above the "expected width" line indicates the smoothed bead width measurement, and the "sliding standard value of bead width" curve below the "expected width" line indicates the standard deviation ("standard") of the bead width measurement. The "dead zone" area indicates the dead zone of the bead width tracking error, and the "oscillation threshold" area indicates the range of allowed standard deviations, which indicates the steady state of the bead width. The thick vertical line indicates the time step for the bead width condition check when the steady state distance d reaches the target threshold.

[0286] Fig.43 The example shows tool speed and bead width data recorded from a test run with bead shape control. In this example, the filter window sizes for the sliding averages of bead width and tool speed were set to 50 samples and 20 samples, respectively. The control gain g was set to 0.8. From t = 4 s to t = ~7 s, the standard deviation σ of the bead width increased due to the rapid change from 12 mm to 8 mm. w Greater than the threshold σ w_thres0.2mm), so the steady-state condition of the bead width is not met. When t=6s, the steady-state condition of the tool speed is met, and thus the steady-state distance d begins to accumulate. When t=7s, the steady-state conditions of both the bead width and the tool speed are met, and the steady-state distance d is already greater than the threshold d thres (20 mm). This indicates readiness to accept a new tool speed command, which is then calculated (~30 mm / s) based on the bead shape controller, resulting in a step change in the command speed profile. The ramp-up phase of the robot tool speed immediately interrupts the steady state of the tool speed until steady state conditions are met again at t = ~8 s to restart the accumulation of d. This adjustment of the tool speed is repeated until the sliding average of the bead width is close enough to the desired bead width (with an error less than ε deadzone =0.5mm).

[0287] FIG. 44 illustrates an example of a desired bead width of 10 mm ( Fig.44A )、8mm( Fig.44B ) and 5mm( Fig.44C ). The control gain was 0.8. The waiting time at the start of the dispensing process was 2 seconds (compensating for the delayed extrusion of the hot melt adhesive). Thus, FIG. 44 illustrates experimental results demonstrating the effectiveness of the bead shape controller to achieve different desired bead widths.

[0288] FIG. 45 illustrates a control gain of 0.2 ( Fig.45A )、0.5( Fig.45B ) and 0.8( Fig.45C ). The desired bead width is 5 mm. The waiting time at the start of the dispensing process is 2 seconds (compensating for delayed extrusion of the hot melt adhesive). Thus, FIG. 45 illustrates experimental results demonstrating how control gain affects the tool speed command and the resulting bead shape.

[0289] FIG. 46 illustrates an example of a waiting time of 2 seconds ( Fig.46A ) and 3 seconds ( Fig.46B ). The desired bead width is 5 mm. The control gain is 0.8. The dispensing experiments that produced the results shown in FIG46 were performed at different wait time settings at the beginning of the tool path to demonstrate the effectiveness of the bead shape controller under different initial conditions and dispensing experiments.

[0290] FIG. 47 shows that on a relatively flatter substrate ( Fig.47A ) and a relatively more curved base ( Fig.47B) on a flat substrate, where online part shape compensation and bead shape compensation were used. The control gain was 0.8. The desired bead width was 5 mm. The waiting time at the start of the dispensing process was 2 seconds (compensating for delayed extrusion of the hot melt adhesive). The dispensing experiments that produced the results shown in Figure 47 were performed on flat substrates and curved substrates to demonstrate simultaneous part shape compensation and bead shape compensation.

[0291] In the detailed description of the exemplary embodiments of the present invention, reference is made to the accompanying drawings, which illustrate specific embodiments in which the present invention can be put into practice. The illustrated embodiments are not intended to be exhaustive of all embodiments according to the present invention. It should be understood that other embodiments may be utilized and structural or logical changes may be made without departing from the scope of the present invention. Therefore, the following detailed description should not be considered to have a limiting meaning, and the scope of the present invention is limited by the appended claims.

[0292] Unless otherwise indicated, all numbers expressing feature sizes, amounts, and physical properties used in the specification and claims are to be understood in all instances as modified by the terms "about," "approximately," or "substantially." Accordingly, unless indicated to the contrary, the numerical parameters set forth in the foregoing specification and the appended claims are approximations that may vary depending upon the desired properties sought to be obtained by those skilled in the art utilizing the teachings disclosed herein.

[0293] Unless the content clearly dictates otherwise, as used in this specification and the appended claims, the singular forms "a / kind", "the" and "the" encompass embodiments having plural referents. Unless the content clearly dictates otherwise, as used in this specification and the appended claims, the term "or" is generally employed in its sense including "and / or".

[0294] It should be appreciated that, according to this example, certain actions or events of any of the methods described herein may be performed in a different order, may be added, combined, or omitted together (e.g., not all described actions or events are necessary for the practice of the method). In addition, in some examples, actions or events may be performed simultaneously rather than sequentially, such as by multithreading, interrupt processing, or multiple processors.

[0295] The techniques described in this disclosure may be implemented at least in part in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented in one or more processors, including one or more microprocessors, CPUs, GPUs, DSPs, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), processing circuits (e.g., fixed function circuits, programmable circuits, or any combination of fixed function circuits and programmable circuits), or any other equivalent integrated or discrete logic circuits, and any combination of such components. The term "processor" or "processing circuit" may generally refer to any of the above logic circuits, alone or in combination with other logic circuits, or any other equivalent circuit. A control unit including hardware may also perform one or more of the techniques of the present disclosure.

[0296] Such hardware, software, and firmware may be implemented within the same device or within different devices to support the various operations and functions described in this disclosure. In addition, any described units, modules, or components may be implemented together or separately as discrete but cooperating logic devices. Depicting different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be implemented by separate hardware or software components. On the contrary, the functions associated with one or more modules or units may be performed by separate hardware or software components or integrated in common or separate hardware or software components.

[0297] The techniques described in the present disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. For example, when the instructions are executed, the instructions embedded or encoded in the computer-readable storage medium may cause a programmable processor or other processor to perform the method. The computer-readable storage medium may include: random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk, CD-ROM, floppy disk, cassette, magnetic media, optical media, or other computer system readable media.

Claims

1. A process for modifying a workpiece, the process comprising: capturing at least one image of the workpiece using an image capture device; processing the at least one image using an image processor to determine at least one of a position or an orientation of the workpiece; Obtaining a nominal tool path; Using an online sensor A to measure the surface of the workpiece to obtain a workpiece surface measurement result; generating an updated toolpath using a waypoint decorator based at least on the nominal toolpath and the position and / or the orientation of the workpiece; generating a planned path for a workpiece modification device using a path planner based at least on the updated tool path and the workpiece surface measurement result; Scanning modification performed on the workpiece by workpiece modification equipment using an online sensor B; determining a decorator parameter based on at least one decorator characteristic; and The workpiece is modified by the workpiece modifying equipment according to the planned path and the modifying parameters. 2 . The process of claim 1 , wherein modifying the workpiece by the workpiece modifying equipment comprises communicating a control signal to a modifying equipment controller, the control signal causing the workpiece modifying equipment to modify the workpiece.

3. The process of claim 1 or claim 2, wherein the at least one image of the workpiece further comprises an image of at least one other part of an end effector or robotic arm of the workpiece modification equipment.

4. A process for dispensing material on a workpiece, the process comprising: capturing at least one image of the workpiece using an image capture device; processing the at least one image using an image processor to determine at least one of a position or an orientation of the workpiece; Obtaining the nominal tool path; Using an online sensor A to measure the surface of the workpiece to obtain a workpiece surface measurement result; generating an updated toolpath using a waypoint decorator based at least on the nominal toolpath and the position and / or the orientation of the workpiece; generating a planned path for a dispenser using a path planner based at least on the updated tool path and the workpiece surface measurement result; scanning a bead of material dispensed by the dispenser onto the workpiece using an inline sensor B; determining a partitioning parameter based on the bead distribution; as well as A control signal is communicated to a dispenser controller, the control signal causing the dispenser to dispense the material onto the workpiece according to the planned path and the dispensing parameters.

5. The process of claim 4, wherein the at least one image of the workpiece further comprises an image of an end effector of the workpiece modification equipment or at least one other part of the robotic arm.

6. The process according to claim 5, further comprising changing at least one processing parameter, detecting at least one effect on the shape of the material bead by an online sensor B, and selecting at least one new processing parameter.

7. The process of claim 6, wherein varying at least one processing parameter comprises varying a movement speed of the dispenser, a distance of the dispenser from the workpiece, and / or a volumetric flow rate of the material from the dispenser.

8. A process according to any one of claims 4 to 7, wherein the material comprises an adhesive, a sealant, a paint or a thermally conductive material.

9. The process of any one of claims 4 to 8, wherein the dispenser comprises a nozzle having a Y-axis, and planning a path for the dispenser comprises aligning the nozzle Y-axis with an estimated normal direction to the workpiece surface.

10. The process of claim 9, further comprising making the nozzle non-vertical by rotating the nozzle to maintain an arbitrary angle between the nozzle Y-axis and the estimated normal direction of the workpiece surface.

11. A process according to claim 9 or claim 10, wherein the nozzle is rotated so that the inline sensor A points to an updated waypoint from the most recent time step.

12. The process of any one of claims 1 to 11, wherein at least one of the inline sensor A or the inline sensor B comprises a laser profilometer.

13. The process of any one of claims 1 to 12, wherein the workpiece surface includes at least one feature selected from the group consisting of an edge, a rib, a seam, a corner point, and a fiducial mark, and the updated tool path includes detection of the at least one feature on the workpiece surface.

14. A process according to any one of claims 1 to 13, wherein the nominal tool path comprises a predefined tool path for modifying the workpiece, the predefined tool path being defined on a surface model of the workpiece.

15. A system for modifying a workpiece, the system comprising: an image capture device configured to capture at least one image of the workpiece; An online sensor A, wherein the online sensor A is configured to measure a surface of the workpiece; and an online sensor B configured to scan modifications to the workpiece; A controller, the controller comprising: a robot motion controller configured to control movement of the robot arm; an offline registration unit configured to determine a list of waypoints based on at least one of the following data sources: the at least one image of the workpiece captured by the image capture equipment, a digital model of the workpiece, and a nominal tool path; an online positioning unit configured to determine an updated waypoint list based on a series of measurement results of the surface of the workpiece measured by the online sensor A and a series of joint positions of the robot arm; a path planner unit configured to plan a path based on the waypoint list and the updated waypoint list; a finishing equipment controller configured to control a workpiece finishing equipment according to the planned path and at least one finishing parameter; and A decorator, the decorator comprising the workpiece decorating equipment.

16. The system of claim 15, wherein the at least one image of the workpiece further comprises an image of an end effector of the workpiece modification equipment or at least one other part of the robotic arm.

17. The system of claim 15, wherein the list of waypoints is further determined based on at least one of a set of joint positions of the robotic arm, or a digital model of the workpiece modification equipment or robot.

18. The system of any one of claims 15 to 17, further comprising a data repository communicatively coupled to at least each of the controller and the decorator, the data repository comprising: Nominal path unit; Surface model unit; modifier feature unit; Modify parameter unit; a modification model unit configured to determine modification parameters and output the modification parameters to the modification equipment controller; Material properties unit; Workpiece characteristic unit; Image capture equipment calibration parameter unit; and Robot model unit.

19. The system of claim 18, wherein the data repository is communicatively coupled to at least one of the offline registration unit, the online positioning unit, the path planner unit, the grooming equipment controller, the robotic motion controller unit, the image capture equipment unit, the online sensor A, the online sensor B, or the grooming model.

20. A system according to claim 18 or claim 19, wherein the data repository further comprises an information database.

21. The system of claim 20, wherein the data repository further comprises a predictive model configured to determine a modification model based at least on data from the information database.

22. The system of any one of claims 15 to 21, wherein the controller further comprises a graphical user interface (GUI) generator configured to generate a graphical user interface for a display.

23. The system according to any one of claims 15 to 22, wherein the offline registration unit comprises: a nominal tool path retriever configured to retrieve a nominal tool path; an image retriever configured to retrieve the image of the workpiece; an image processor configured to process the image of the workpiece; and A waypoint decorator is configured to generate an updated tool path based on the processed image of the workpiece, the nominal tool path, and the set of joint positions of the workpiece decorating equipment or an image of an end effector of the workpiece decorating equipment.

24. The system according to any one of claims 15 to 23, wherein the path planner unit comprises: a waypoint retriever configured to retrieve a set of waypoints; a path finder configured to retrieve an actual path of the robot arm; a path analyzer configured to compare the actual path with the set of waypoints; and A waypoint decorator is configured to determine a new set of waypoints based on the comparison of the path analyzer.

25. The system of any one of claims 15 to 24, wherein the data repository further comprises a machine learning unit configured to determine process parameters for modifying the workpiece.

26. The system of any one of claims 15 to 25, wherein the workpiece grooming equipment comprises a dispenser and the grooming equipment controller is a dispenser controller.

27. The system of claim 26, wherein the modifier property unit comprises a bead distribution unit, the modifier parameter unit comprises a distribution parameter unit, and the modifier model unit comprises a distribution model unit.

Images