Robotic surface modification system and method

Abstract

An image acquisition system for a reflective surface is disclosed, the image acquisition system including a first light source mounted on a robotic arm and a second light source mounted on the robotic arm different from the first light source, and an image acquisition device positioned to acquire an image of the reflective surface. The first light source, the second light source, and the image acquisition device are mounted on the robotic arm. The first light source is positioned relative to the image acquisition device such that a field of view of the image acquisition device captures a specular reflection from the first light source.

Description

[Background technology]

[0001] Surface modification on specular surfaces presents challenges in terms of image acquisition, surface trajectory design, and pre- and post-modification evaluation. Summary of the Invention

[0002] An image capture system for a reflective surface is disclosed, the system including a first light source mounted on a robotic arm and a second light source different from the first light source. The system also includes an image capture device positioned to capture an image of the reflective surface. The first light source, the second light source, and the image capture device are mounted on the robotic arm. The first light source is positioned relative to the image capture device such that a field of view of the image capture device captures a specular reflection from the first light source. [Brief explanation of the drawings]

[0003] The drawings, which are not necessarily drawn to scale and in which the same numbers may refer to similar elements in different figures, illustrate generally, by way of example, and not by way of limitation, various embodiments discussed in the present document.

[0004] [Figure 1] FIG. 1 is a schematic diagram of a robotic surface modification system in which embodiments of the present invention are useful.

[0005] [Figure 2] FIG. 2 is a diagram illustrating a surface modification method according to an embodiment of the present disclosure.

[0006] [Figure 3A] FIG. 3A is a diagram illustrating an image acquired by an image acquisition system described in embodiments herein. [Figure 3B] FIG. 3B is a diagram illustrating an image acquired by an image acquisition system described in embodiments herein. [Figure 3C]FIG. 3C is a diagram illustrating an image acquired by an image acquisition system described in embodiments herein. [Figure 3D] FIG. 3D is a diagram illustrating an image acquired by an image acquisition system described in an embodiment herein. [Figure 3E] FIG. 3E is a diagram illustrating an image captured by an image capture system described in embodiments herein.

[0007] [Figure 4A] FIG. 4A illustrates an embodiment of a robotic repair unit with an image acquisition system for surface modification. [Figure 4B] FIG. 4B illustrates an embodiment of a robotic repair unit with an image acquisition system for surface modification.

[0008] [Figure 5A] FIG. 5A shows a schematic diagram of an illumination arrangement that may be useful for different evaluations of surfaces. [Figure 5B] FIG. 5B shows a schematic diagram of an illumination arrangement that may be useful for different evaluations of surfaces.

[0009] [Figure 6A] FIG. 6A illustrates different configurations of an image acquisition system according to embodiments herein. [Figure 6B] FIG. 6B illustrates a different configuration of an image acquisition system according to embodiments herein. [Figure 6C] FIG. 6C illustrates a different configuration of an image acquisition system according to embodiments herein. [Figure 6D] FIG. 6D illustrates a different configuration of an image acquisition system according to embodiments herein.

[0010] [Figure 7A] FIG. 7A illustrates one embodiment of an image acquisition system that may be attached to a robotic surface modification unit in accordance with embodiments herein. [Figure 7B] FIG. 7B illustrates one embodiment of an image acquisition system that can be attached to a robotic surface modification unit in accordance with embodiments herein.

[0011] [Figure 8] FIG. 8 illustrates different structured light patterns that can be placed on a backlight to provide structured illumination for, for example, an image acquisition system.

[0012] [Figure 9A] FIG. 9A shows the results of an image acquired using the system herein and processed with the systems and methods described herein. [Figure 9B] FIG. 9B shows the results of an image acquired using the system herein and processed with the systems and methods described herein. [Figure 9C] FIG. 9C shows the results of an image acquired using the system herein and processed with the systems and methods described herein. [Figure 9D] FIG. 9D shows the results of an image acquired using the system herein and processed with the systems and methods described herein. [Figure 10A] FIG. 10A shows the results of an image acquired using the system herein and processed with the systems and methods described herein. [Figure 10B] FIG. 10B shows the results of an image acquired using the system herein and processed with the systems and methods described herein.

[0013] [Figure 11A-1] FIG. 11A-1 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11A-2] FIG. 11A-2 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11A-3] FIG. 11A-3 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11B-1] FIG. 11B-1 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11B-2] FIG. 11B-2 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11B-3] FIG. 11B-3 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11C-1] FIG. 11C-1 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11C-2] FIG. 11C-2 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11C-3] FIG. 11C-3 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11D-1] FIG. 11D-1 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11D-2] FIG. 11D-2 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11E-1] FIG. 11E-1 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11E-2] FIG. 11E-2 shows a reflection image of a grid of light on a surface generated using the systems and methods herein. [Figure 11F] FIG. 11F shows a reflection image of a grid of light on a surface generated using the systems and methods herein.

[0014] [Figure 12A] FIG. 12A is a diagram illustrating a static structured light set and an illuminated specular surface, according to embodiments herein. [Figure 12B] FIG. 12B is a diagram illustrating a static structured light set and an illuminated mirror surface, according to embodiments herein.

[0015] [Figure 13A] FIG. 13A shows a surface image acquired using static structured light in a grid pattern. [Figure 13B] FIG. 13B shows a surface image acquired using static structured light in a grid pattern. [Figure 13C] FIG. 13C shows a surface image acquired using static structured light in a grid pattern. [Figure 13D] FIG. 13D shows a surface image acquired using static structured light in a grid pattern. [Figure 13E] FIG. 13E shows a surface image acquired using static structured light in a grid pattern. [Figure 13F] FIG. 13F shows a surface image acquired using static structured light in a grid pattern. [Figure 13G] FIG. 13G shows a surface image acquired using static structured light in a grid pattern.

[0016] [Figure 14] FIG. 14 illustrates a diffuser stack that can be used in some embodiments described herein.

[0017] [Figure 15A] FIG. 15A shows a schematic diagram of a surface image acquisition system according to embodiments herein. [Figure 15B] FIG. 15B illustrates a schematic diagram of a surface image acquisition system according to embodiments herein.

[0018] [Figure 16A]FIG. 16A illustrates a schematic diagram of an outward-facing surface image acquisition system according to embodiments herein. [Figure 16B] FIG. 16B illustrates a schematic diagram of an outward-facing surface image acquisition system according to embodiments herein.

[0019] [Figure 17A] FIG. 17A illustrates a schematic diagram of a binocular front-facing surface image acquisition system according to embodiments herein. [Figure 17B] FIG. 17B illustrates a schematic diagram of a binocular front-facing surface image acquisition system according to embodiments herein.

[0020] [Figure 18] FIG. 18 illustrates a schematic diagram of a flat dome optical surface image acquisition system according to embodiments herein.

[0021] [Figure 19] FIG. 19 is a diagram showing a schematic diagram of a surface image acquisition system.

[0022] [Figure 20] FIG. 20 is a diagram illustrating the repair strategy generation system architecture.

[0023] [Figure 21] FIG. 21 shows an example of a mobile device that can be used in the embodiment shown in the previous figure. [Figure 22] FIG. 22 shows an example of a mobile device that can be used in the embodiment shown in the previous figure.

[0024] [Figure 23] FIG. 23 illustrates a block diagram of a computing environment that can be used in the embodiments illustrated in the previous figures. DETAILED DESCRIPTION OF THE INVENTION

[0025] Recent advances in image acquisition technology and computing systems have made clearcoat inspection processes feasible at production speeds. In particular, stereodeflectometry has recently been shown to be capable of providing images and locations of paint and clearcoat defects at suitable resolution, along with spatial information that provides coordinate location information and defect classification, thereby enabling subsequent accurate location reconstruction and automated spot repair.

[0026] However, especially in the automotive industry, many surfaces containing defects are not flat. Also, the vehicle may move from the initial inspection location to the repair location. Therefore, it is important to verify the exact location of the vehicle in order to pinpoint the exact defect location. It may also be important to re-evaluate the characteristics of the defect or confirm the initial evaluation just before repair is performed.

[0027] The systems and methods described herein enable image acquisition of surfaces using a compact end-of-arm system in a robotic surface modification unit. In some embodiments, the image acquisition system is mounted near the surface modification tool on the robotic unit. The ability to mount the system on the end of the same robotic arm as the surface modification tool (as opposed to a separate robotic unit) provides important advantages, such as in-situ measurements during the surface modification process and reduced error in the motion transition between the tool and the image acquisition system. However, the image acquisition system must be sufficiently compact to allow the robotic arm to move the tool into position for the surface modification process. Images may be processed in-situ to generate surface characterization, surface modification trajectories, etc. The same (or a different) image acquisition system may be used to re-evaluate the surface after the surface modification process to understand whether the surface modification was sufficient. For example, a vehicle may have a clear coat defect in an area of ​​the surface that, after repair (e.g., sanding and polishing), has noticeable haze that is cosmetically unacceptable.

[0028] However, while the systems and methods described herein contemplate an end-of-arm image acquisition system, it is expressly contemplated that in some embodiments the image acquisition system may be mounted on a separate robotic unit, hi some embodiments, the end-of-arm image acquisition system may be mounted on the same robotic arm at a different mounting location than the surface modification tool.

[0029] As used herein, the term "vehicle" is intended to broadly encompass any mobile structure that receives at least one layer of paint and / or clear coat during manufacture. While many of the examples herein relate to automobiles, it is expressly contemplated that the methods and systems described herein are also applicable to trucks, trains, boats (with or without motors), airplanes, helicopters, motorcycles, and the like.

[0030] As used herein, the term "paint" is intended to broadly encompass various layers of paint on a vehicle, such as electrocoat, filler, primer, paint, and clearcoat. Furthermore, "paint repair" includes detecting and repairing visual defects on or within these paint layers. In some embodiments, the systems and methods described herein use clearcoat as the target paint repair layer. However, the systems and methods described herein are applicable to specific paint layers (e.g., electrocoat, filler, primer, paint, and clearcoat) with little or no modification.

[0031] As used herein, the term "defect" refers to an area on a work surface that detracts from its visual aesthetic. For example, many vehicles may have a specular, or highly reflective, surface that appears shiny or metallic after painting is complete. A "defect" may include foreign matter trapped in one or more layers of paint on the work surface. Defects also include stains during painting, overapplication of paint (smears or drips), and dents. As used herein, "defect" encompasses both aesthetic disturbances that occur during the painting process or the repair process. For example, a surface may have haze that is exacerbated by a defect repair operation. Alternatively, an area containing a defect (e.g., a trapped foreign matter, a scratch) may not have significant haze before the repair operation, but may have an aesthetically unacceptable level of haze after the repair.

[0032] FIG. 1 is a schematic diagram of a robotic paint repair system in which embodiments of the present invention are useful. The system 100 generally includes two units: a vision inspection system 110 and a defect repair system 120. Both systems may be controlled by motion controllers 112, 122, respectively, which may be configured to receive commands from one or more application controllers 150. The application controllers may receive inputs or provide outputs from a user interface 160. The repair unit 120 includes a force control unit 124 alignable with an end effector 126. Note that, as shown in FIG. 1, the end effector 126 includes two tools 128, which are further detailed in co-pending U.S. Provisional Application No. 62 / 940,950 (filed November 27, 2019). The end effector 126 also includes an image acquisition system 127, which is positioned such that rotation or translation of the end effector 126 allows for switching between one of the tools 128 and the image acquisition system 127. However, other configurations are expressly contemplated. For example, while FIG. 1 shows the repair unit 120 operating simultaneously with the image acquisition system 110, configurations in which the repair unit 120 operates behind the image acquisition system 110 are explicitly contemplated, in which at least part of the operation of the repair unit 120 is controlled based on data collected from the image acquisition system 110.

[0033] One of the two main challenges, the inspection of the vehicle 130 by the inspection unit 110, is challenging due to the inherent nature of the technology. Surfaces of interest are typically much larger than defects, often by orders of magnitude. As a result, sensor and lens selection require a tradeoff between field of view and resolution, which is critical to achieving the required viewing angle. Furthermore, each paint layer in the finishing process (e.g., electrocoat, primer, paint, clear coat, etc.) has a different visual appearance, particularly specularity. Highly specular (i.e., highly glossy or reflective) surfaces present unique image acquisition challenges. These challenges compound to make inspection difficult. Recent increases in computing resources have led to progress in this area, with several commercially available solutions becoming available. A sufficiently sophisticated inspection system 110 is crucial to identifying defects for repair by the repair unit 120.

[0034] The current state of the art in vehicle paint restoration involves manually sanding / polishing defects using fine abrasives and polishes, with or without power tools, while maintaining the desired finish (e.g., mirror-like consistency of a clear coat). When skilled workers perform such repairs, they utilize highly trained skills while simultaneously using their senses to monitor the progress of the repair and make corrections accordingly. Such sophisticated behaviors are difficult to replicate with robotic systems with limited sensing capabilities.

[0035] While modifications to vehicle surfaces to remove paint-related defects are provided as examples throughout this specification, other surface modifications are expressly contemplated, such as other abrasive processes (sanding, grinding), other additive processes (e.g., additive manufacturing, applying adhesives, etc.), or subtractive processes (material removal, cutting, etc.).

[0036] 2 illustrates a surface modification method in accordance with one embodiment of the present invention. While method 200 is described in the context of repairing surface defects in vehicles, it is expressly contemplated that other applications may also benefit from the systems and methods described herein.

[0037] In block 210, an initial scan of the surface to be modified is performed, which may be performed at an inspection station, for example, in the context of vehicle repair.

[0038] As shown in FIG. 2 , the steps of imaging 220, surface characterization 230, surface modification 240, and post-modification evaluation are repeated for multiple defects on the surface. In the context of a vehicle, there may be multiple discrete defects on the surface that require repair. However, some defects detected during the initial scan of block 210 may not require repair or may not be repairable by an on-site repair unit. For a number of defects that can be repaired by a robotic repair unit, steps 220, 230, 240, and 260 are repeated until all defects are repaired to an acceptable level or as production / time constraints permit. The acceptable level may be determined by other criteria, such as industry-accepted size, manufacturer quality tolerances, or human visibility.

[0039] In block 270, a second scan of the entire surface may be performed, for example, using the same image acquisition system as in block 210, the image acquisition system used in steps 220-260, or a different image acquisition system.

[0040] In the context of paint defect repair, the scan performed in block 210 may not necessarily be aimed at detailed characterization of the defects or selection of a surface modification sequence to address the detected defects. The initial scan in block 210 may be used, for example, to identify detected defects that require repair and can be repaired by an on-site robotic surface modification unit.

[0041] Local surface imaging is performed in block 220. In the context of vehicle defect repair, a dedicated image acquisition system 222 may acquire surface information at the location of the detected defect. The dedicated image acquisition system 222 may be an image acquisition system separate from the robotic surface modification unit or may be part of the end-of-arm system 224 of the robotic surface modification unit.

[0042] In block 230, a characterization of the imaged surface is performed. The characterization may include identifying the exact location 232 on the surface where modification is required. For example, the defect location may be identified with high accuracy in three-dimensional space. Furthermore, a modification sequence 234 may be generated based on the type of modification required. For example, a scratch may be repaired by a robotic repair unit differently than a bulge caused by a trapped foreign object, and a crater (depression) may be repaired differently. The modification sequence 234 may be selected based on the severity 236 of the detected defect. For example, a large trapped foreign object may require more pressure, longer contact time, or a different abrasive article than a small foreign object. Other surface characterization factors 238, such as expected vehicle usage and the condition of other paint layers, may also be important. For example, an initial orange peel characterization may be performed on the surface surrounding the detected defect, so that the selected surface modification sequence maintains or blends in the orange peel.

[0043] In block 240, a surface modification process is performed. The surface modification 240 may be performed based on the modification sequence selected in block 230 or based on other factors. The surface modification 240 may be either an additive or subtractive modification, depending on the needs of the work surface being modified. The surface modification 240 includes a trajectory having a path including multiple waypoints, between which the surface modification tool moves with controlled speed, angle, and applied force.

[0044] In block 260, after the modification sequence is complete, a post-modification evaluation of the surface may be performed. It is expressly contemplated that post-modification evaluation 260 may be performed using the image acquisition system of block 220, the image acquisition system of block 210, or a separate image acquisition system. However, as described herein, the flexibility of end-of-arm system 224 allows the same image acquisition system to be used in both blocks 220 and 260, thereby increasing efficiency and accuracy in addressing multiple repairable defects. Post-modification evaluation 260 may include evaluation and measurement of the surface for, for example, the presence or absence of haze 262, orange peel 264 disturbances, or other characteristics 266 (e.g., newly developed scratches, etc.).

[0045] At block 250, the defective area is inspected to determine whether the repair is sufficient. If additional repair is required, method 200 may receive a new command (indicated by arrow 260) and the method may be repeated. Inspection of the defect repair may include capturing a post-repair image 252, which may be presented to a repair operator or stored as needed. Inspection may also include verification of the repair, as shown in block 254. This may include comparing the before and after images, detecting whether the defect is visible or recognizable to the human eye, or other suitable verification techniques. In some embodiments, the captured images may be analyzed by operational systems, quality assurance, tracking, and process management.

[0046] 3A-3E illustrate images captured by an image capture system in accordance with embodiments described herein. As noted above, the image capture system may be mounted to the end effector of the robotic repair unit. However, in other embodiments, other mounting locations are possible.

[0047] Figure 3A shows a structured light image 310 of a surface containing a defect. The defect shown in image 310 is a nib. Figure 3B shows a structured light image 330 of a surface containing a defect. The defect shown in image 330 is a trapped fiber. Figure 3C shows a structured light image 320 of a surface containing a defect. The defect shown in image 320 is a crater.

[0048] 3D and 3E show raw and processed images 340 and 350 of a surface after modification. As shown in image 350, the systems and methods described herein can be used to identify the boundaries of the surface modification and assess the amount of haze introduced by the modification (darker areas indicate greater haze). Haze is caused by non-specular reflections from micro-flaws in the surface, which scatter incident light rather than reflect it resonantly. Note that these images are inverted. In embodiments, the images are inverted to improve human visibility, even though more light is received from the micro-flaws. It is expressly contemplated that analysis may be performed on either the acquired image or the inverted image.

[0049] The systems and methods described herein enable automated defect detection on mirrored surfaces in a single image without moving the object. Current inspection systems for mirrored surfaces are complex and often require multiple cameras, light sources, or moving objects. The systems and methods described herein enable coordination of machine vision devices, image acquisition under different lighting conditions, and feature and defect identification using machine vision algorithms. The systems and methods described herein address submillimeter defects on mirrored surfaces, but can also address challenges in other surface conditions.

[0050] 4A and 4B illustrate one embodiment of a robotic repair unit equipped with an image acquisition system for surface modification. The repair robot 400 may include one or more rotation points 402 that allow the robot arm to approach the surface to be modified. A revolute joint 404 allows the end effector equipped with the image acquisition system described herein to navigate curved or irregular surfaces to acquire the images necessary for constructing and evaluating surface modification sequences. The robotic arm 400 shown in FIG. 4A has multiple degrees of freedom that allow for surface approach and surface modification.

[0051] Although FIG. 4A illustrates a flat surface 420, it is expressly contemplated that the systems and methods described herein are also useful on curved or inclined surfaces that would be difficult for conventional large, complex systems to maneuver into position to obtain the required images.

[0052] The end effector of the robot 400 includes one or more tools 412 and an image acquisition system 410. The tools 412 are rotationally interchangeable with the image acquisition system 410, and after the surface modification process, the image acquisition system 410 may be moved into position to acquire images for evaluation of the surface modification.

[0053] Figure 4B shows a close-up view 450 of the end-of-arm system. Figure 4B shows tool 452 being rotated into position and retracting image acquisition system 454. Similarly, image acquisition system 454 may be rotated into position as needed, for example, after tool 452 has completed a surface modification process.

[0054] As will be described in more detail with respect to subsequent figures, the robot controller responsible for moving the tool 452 and image acquisition system 454 into and out of position is controlled by a controller that selectively triggers the scattered light system, the image acquisition system, and the general operation of the robot system. In some embodiments, the system may include multiple controllers, such as a controller that physically moves the robot into position, a repair controller that executes a selected repair strategy, and an image acquisition controller that moves parts into position, turns light sources on and off, and acquires images.

[0055] The systems and methods described herein use different lighting configurations to acquire different images of a surface prior to surface modification. Figure 5A is a schematic diagram of different lighting operations that may be useful for different evaluations of a surface. Incident light 512 is projected onto surface 510 at an angle of incidence. The light is then reflected from surface 510 as diffuse reflection 514 or specular reflection 516. The diffuse reflection of incident light 512 is reflected from the surface at various directions and angles, as shown in Figure 5A. The specular reflection 516 is reflected from surface 510 at an angle opposite to that of the incident light.

[0056] 5B illustrates a chart 550 showing the angles of reflection of different types of light that may be used to characterize a surface. Chart 550 also illustrates applications for which each measurement technique may be particularly useful. Note that chart 550 is not intended to be an exhaustive list. While the systems and methods described herein are described as being configured to measure specular gloss and / or haze, by adjusting the relative positioning of the camera and light source, it is also possible to measure scene, gloss appearance, gloss definition, or surface uniformity.

[0057] 6A and 6B show different configurations of a single image acquisition system and how different types of images can be acquired using a single image acquisition system. Setup 600 shows a backlight 610 positioned at an angle relative to the surface, projecting light as light projection 612. Image acquisition device 630 is also positioned at an angle relative to the surface and has a field of view 632.

[0058] Setup 650 shows a configuration in which the surface backlight is turned off and instead high brightness light 620 is projected onto the surface. Setup 610 is useful for characterizing specular reflections, i.e., defects. Setup 650 is useful for acquiring images using diffuse illumination, especially when acquiring haze.

[0059] Figures 6C and 6D show schematic diagrams of how the image capture systems of Figures 6A and 6B, respectively, can be used to characterize a surface. In Figure 6C, light source 652 is positioned at an angle relative to surface 660 and image capture device 654, so that a light beam is projected toward surface 660 and reflected from surface 660 to camera 654, where the specular reflection is captured by the image capture device. In Figure 6D, light source 672 is positioned at an angle relative to surface 670 and image capture device 674, so that a light beam is reflected from surface 670 in multiple directions as diffuse light, providing the diffuse reflection to image capture device 674.

[0060] 7A and 7B illustrate one embodiment of an image acquisition system 700 that can be attached to an end effector of a robotic surface modification unit. The image acquisition system 700 includes an image acquisition device 708, such as a camera. The image acquisition device may include one or more lenses, e.g., a lens stack, and may be highly vibration-resistant. Vibration resistance is particularly important in clearcoat defect repair applications, where the robotic repair unit may move laterally along the vehicle or the image acquisition system 708 may move into position to acquire the necessary images. The system 700 may also include multiple lighting options, such as a surface backlight 702 and / or a high-intensity line light source 704. As described above, the surface backlight 702 may be used with one or more structured light patterns. The surface backlight 702 may project light at multiple different intensities depending on the application. In some embodiments, the surface backlight 702 may have a modified spectral profile, for example, by adding or removing color elements.

[0061] Mount 710 is designed to mount image capture system 700 to an end effector, for example, directly or with a mounting plate. System 700 may also include a transparent cover 706, which protects backlight 702, high-intensity light 704, and image capture device 708 from debris, flying objects, etc. The cover may be made of glass or plastic, for example. While high-intensity light 704 is illustrated as a line light, it is expressly contemplated that a high-intensity spotlight or projector may be used in some embodiments.

[0062] Figure 7B shows the system 750 in an operational configuration, with some mounting structures removed for clarity. The surface backlight 752 projects diffuse light 762 that overlaps with the field of view 768 projected by the image capture device 758. While Figure 7B shows an embodiment in which the surface backlight 752 and the high-intensity line light 754 projecting the high-intensity strip of light 764 are operating simultaneously, it is expressly contemplated that in many applications, only one of them will be used.

[0063] 7B further illustrates that the surface backlight 752, the high brightness line light 754, and the image capture device 758 are positioned at an angle relative to one another. The angles of any of the components 752, 754, 758 may be fixed or, in some embodiments, adjustable.

[0064] The systems and methods described herein use novel illumination techniques to detect clearcoat defects using directional illumination or surface defects using scattered illumination. The operation of each component is controlled by a robotic controller (not shown in FIGS. 7A-7B). For example, system 750 is mounted on a robotic surface modification unit at an appropriate location relative to the surface. Surface defects caused by micro-scratches, such as haze, can be made more visible using a backscattered illumination system. This can be achieved using an angled line light, as described in FIG. 5A. On-axis specular reflection and scattered ring observation can also be used.

[0065] In the systems and methods described herein, it is important to maintain alignment between the field of view and the surface being imaged. On a specular surface, the chief ray is reflected off the surface, with the angle of reflection equal to the angle of incidence. Due to specular reflection, the incident and reflected light are in the same plane. Therefore, the principal axis of the camera / lens must be positioned and oriented correctly to accurately capture these chief rays. The field of view must coincide with the normal vector from the surface of the area of ​​interest (e.g., the area containing the defect). It is also important to maintain the stability of the image acquisition system to maintain alignment.

[0066] FIG. 8 illustrates various structured light patterns that can be used to provide structured illumination to an image acquisition system. In some embodiments, one or more of the grid patterns shown in FIG. 8 may be placed as a mask on the backlight and maintained throughout the surface modification process, for example, for all defect repairs for a particular vehicle. However, in some embodiments, the grid pattern may be removable from the backlight, and different grid patterns may be used for repairing different detected defects. Also, in some embodiments, the patterns may be built into the backlight and not easily removed or replaced between operations.

[0067] The use of a fixed pattern allows for faster image acquisition. Using only a single grating pattern improves the efficiency of the overall process and reduces the cycle time between modification steps, as the analysis process for a single pattern is significantly simplified compared to traditional complex structured light patterns.

[0068] For example, some prior art systems require a 3-foot by 4-foot high-brightness display screen on which multiple patterns are displayed in sequence. As each pattern is displayed, an image must be captured, analyzed, and combined to generate a single surface map. The systems and methods described herein are able to capture the surface information required to perform a surface modification sequence from a sequence of images captured of a single grid pattern. In some embodiments, only a single image is captured. However, it may be beneficial to capture multiple images without significantly increasing cycle time.

[0069] The systems described herein can be mounted on an end effector and can be more easily moved over a surface to acquire surface topography information. In embodiments where a single grid pattern is used to acquire a single image, resolution in the Z direction (e.g., defect depth or protrusion) is sacrificed. However, in some embodiments, it is sufficient to locate the defect, determine whether it is above or below the clear coat layer, and estimate its height. Additionally, in some embodiments, multiple cameras can be used to improve depth detection by positioning additional cameras at different angles / positions.

[0070] The various structured light patterns illustrated in FIG. 8 are exemplary only and are not limiting. For example, while pattern 802 depicts vertical lines, it is expressly contemplated that horizontal or diagonal lines may be used. Furthermore, while pattern 804 depicts a grid pattern of alternating sizes, it is expressly contemplated that a single-size grid pattern may be used. Images 806 and 808 depict different patterns including circular apertures. Patterns 802-808 are provided by a patterned illumination device. However, it is expressly contemplated that other grid aperture shapes and sizes may also be used. Furthermore, dynamically changing light patterns (e.g., deflectometry, single-shot deflectometry, etc.) may also be used.

[0071] In a camera system, there are multiple variables that can be adjusted to obtain different information about the surface—gain, aperture and exposure time, acceptance angle (angle of incidence / vector that can be mapped to pixels), field of view angle, and many other settings. For example, increasing the gain can increase the signal-to-noise ratio of haze on a surface. In some embodiments described herein, multiple images can be acquired while changing the gain multiple times. Comparing images acquired at different gains can enhance the measurement of haze. Haze can be a combination of both signal and noise, so variability in images acquired at different gains can help identify and quantify haze. Gain and / or light intensity can also be adjusted based on the color of the basecoat paint.

[0072] While other suitable light sources can be used, some systems and methods described herein utilize LED illumination systems with varying intensities. Image acquisition conditions may include high frequencies and small apertures in the camera lens, necessitating LEDs for greater depth of field. High-brightness LEDs are particularly useful for specular surfaces, which do not diffuse light and reflect most light in one direction. By increasing the amount of incident light, high-brightness LEDs also increase the amount of captured light for a given exposure time, improving the probability of capturing defect information. Generally, a smaller aperture requires a longer exposure time, but compensating with high-brightness light can reduce the time required to acquire each image. The systems and methods described herein use static structured light to identify defects in clearcoats and scattered light to identify optical haze in clearcoats.

[0073] Figures 9-10 show the results of images acquired using the system described herein and processed by the systems and methods further described. Figures 9A-9C show the results of processing the images to detect haze on the surface. Figures 9A-9C show the curved surface of a motorcycle fuel tank within specular reflection patches. These processed images provide an objective and quantitative indication of surface issues and defects.

[0074] Haze can be quantified by calculating the density of flaws within a particular region. Using the systems and methods described herein, the system can automatically identify haze regions, determine the density of flaws within the haze regions, and calculate a haze percentage that may be perceived by the human eye. This calculation may be performed using a segmentation algorithm, a machine learning algorithm, or other suitable algorithm. The images in FIGS. 9A-9C are post-modification inspection images acquired after the surface modification process is complete. In some embodiments, images such as those in FIGS. 9A-9C are presented to a user via a user interface. In other embodiments, a haze percentage output may be presented. In yet another embodiment, a satisfaction indicator, such as "repair successful," "repair unsuccessful," or "rework required," may be presented.

[0075] FIG. 9D shows a processed image of the surface after modification. The surface exhibits haze, outlined by outline 952. The density of micro-scratches varies across the surface, with high-density areas indicated by arrows 954 and low-density areas indicated by arrows 956. Shadowing 958 is also visible in the processed image, partially due to specular light artifacts. In contrast, the area outlined by outline 952 is a backscattered area. Micro-scratch defects are most visible when using a backscattered lighting system. A high-intensity line light is used at a different angle of incidence than a diffused light panel. The angle used for backscattered lighting is commonly referred to as the "backscatter angle."

[0076] FIGS. 10A and 10B illustrate the results of image acquisition for defect detection. A structured light pattern of vertical lines, such as pattern 802 in FIG. 8, was used to image the surface. As shown, in each image, defects are clearly visible using static structured light processing. As described above with respect to FIG. 3, after defects are detected in the images, they can be characterized, including their type and severity. The images shown in FIGS. 10A and 10B are of a surface requiring surface modification and in which defects were detected. Clearcoat defects are visible due to the degree of Z deformation visible above and below the defect. Based on the shadowing effect and defect feature filtering shown in FIGS. 10A-10B, an image segmentation algorithm can be used to detect whether the defect is above or below a depth point in the clearcoat and to characterize the type and severity of the defect. However, while some embodiments use an image segmentation algorithm, other embodiments contemplate using unsupervised or trained machine learning algorithms.

[0077] Defect detection is best achieved using a collimated light illumination system. Because defects can have three-dimensional characteristics (e.g., defect size, shape, and / or location in the Z-axis of the clear coat layer), it is important to note the shadowing effect of defects in the clear coat layer. A specular illumination configuration is configured such that the light source and camera are tilted so that reflected light is incident on the camera.

[0078] In some embodiments, a preliminary scan of the surface may provide initial location information of surface anomalies that may be used to generate a surface modification trajectory, however, in other embodiments, the results of specular image acquisition in the vicinity of where defect repair will occur may be used to generate a surface modification strategy or to modify a previously selected surface modification strategy.

[0079] It is expressly contemplated herein that the illumination configuration, including the selected angle of the line or area light, as well as the structured light pattern, is selected based at least in part on a preliminary scan of the surface. While it is possible to perform both specular and backscattered image acquisition in a single image acquisition step, it is also contemplated that in some embodiments, these are performed sequentially as separate image acquisition steps. In such embodiments, information obtained from the first image acquisition step may be used to better adjust the settings for the second image acquisition step.

[0080] 11A-11F show images of a grid of light reflected from a surface, generated using the systems and methods described herein. Figures 7A-7F show images of fringes on a surface. The images in Figures 7A-7F were acquired using planar light, but the light bars are expected to shrink due to the shape of the light and the shape of the surface being imaged. This effect can be corrected for because the curvature of the light is known. The projection is based on the assumption that only the surface being imaged is changing. A convex light source is expected to cause shrinkage, and a concave light source is expected to cause expansion.

[0081] Given a constant curvature, a particular fringe (interference pattern) is produced, and if there is an area of ​​high curvature change on the surface, reflections from multiple fringes, or all fringes in other embodiments, may be obtained.

[0082] The systems and methods described herein are also useful for verifying that parts are correctly positioned. For example, in vehicle surface repair, the vehicle may move between the initial image capture and the time of repair. Also, in door repair, there may be a standoff between the door and the vehicle, and its position may change. Such changes may result in repairs being made to the wrong surface area or a collision between the repair equipment and the vehicle.

[0083] The systems and methods described herein can also be used to re-identify or reposition a defect using an end-of-arm system before repair begins. Verifying that a defect is in the expected location and identifying how a robotic repair system should adjust based on its new location can provide upstream information for an assembly, repair, or manufacturing system. The systems and methods described herein can provide feedback regarding system tolerances. For example, if the defect's movement is consistently within a certain threshold range, the tolerance is stable, and repair can continue with the repair recipe and repair area size selected during the previous repair process. On the other hand, if the tolerance is tighter than expected (i.e., the defect is within a narrower range of the predicted location), the repair area may be reduced, potentially leading to a shorter repair time. Conversely, if the tolerance deteriorates (i.e., the defect is outside the threshold range), additional action is required to ensure the planned repair is completed and avoid re-repair. Such action may include expanding the repair area, repositioning the repair robot, or selecting a new repair strategy.

[0084] Particularly in areas of the surface where smaller repair areas are desired, localization using the systems and methods described herein can reduce the need for re-repair and shorten repair time by localizing in-situ.

[0085] Using the grid light, images 1100A1-1100F of the reflective surface are acquired, as shown in Figures 11A-11F. The images are processed to calculate a boundary 1100, which is the smallest area that can contain the entire light bar. This image is processed to generate two outputs: (1) the rotation angle, and (2) the area of ​​the reflected light. The boundary 1100 may be defined by a center point 1120.

[0086] The processed image can be used to verify that the field of view within boundary 1100 is as expected, based on previous image capture or a CAD model of the vehicle (or other surface) being repaired. Surface curvature has a predictable effect on light reflection.

[0087] If the rotation angle is 0°, 90°, 180°, 270°, etc., the boundary 1100 will be an upright rectangle. A square boundary 1100 will have C4 symmetry (2π / 4, i.e., rotational symmetry by 90 degrees) and may result in an indistinguishable shape. A rectangular boundary will have C2 symmetry (2π / 2, i.e., rotational symmetry by 180 degrees), and the boundary sides may be aligned with the "vertical" and "horizontal" axes.

[0088] Figures 11A-1 to 11A-3 show light reflection from a flat panel, i.e., a surface with no curvature. Whether the part's orientation is correct for the vision system can be confirmed by checking whether the rotation angle belongs to one of the rectangular symmetry groups (C2). Specifically, this can be confirmed by checking whether the rotation angle is within a ±threshold range, such as 0°, 90°, 180°, or 270°.

[0089] Additionally, the location of boundary 1100 may be determined based on the area in the field of view relative to the expected area. For example, in image 1100A1, some of the light is outside the panel, and the entire field of view (FOV) is not utilized. Such information allows for feedback between the image acquisition system and position adjustments.

[0090] The end-of-arm vision system allows the system to capture images while it is moving relative to the surface. Images 1100B1 and 1100B2 are images captured while the end-of-arm system is moving. The light angle and area within boundary 1120 change as the system moves along the curved surface. If the expected surface is known (e.g., from a CAD file, 3D scan, or previous image capture), the captured angle / light area can be compared to the expected area to detect any deviations.

[0091] Generally, as the image acquisition system approaches the normal direction, the area of ​​the boundary 1100 increases, and the angle of the region of interest similarly approaches 0° / 90° / 180° / 270°. In some embodiments, prior to image acquisition and repair of the defect, the repair system is ensured to be aligned with the normal direction.

[0092] The position can be further confirmed using the area of ​​the light grating reflection in the actual field of view relative to the expected area. For example, in the first image, some of the light is off the panel, so the entire possible field of view (FOV) is not being utilized. This area information allows feedback b / w between the image acquisition system and position adjustments.

[0093] 11B-1 and 11B-2 show images of light reflection on a curved surface acquired using a grid of light. From images 1100B1 and 1100B2, the curvature of the surface can be detected. The center point 1120 of each calculated boundary 1110 is shown. Image 1100B1 was acquired at a rotation angle of 9.77°, resulting in a boundary area of ​​2,521,694.0 pixels.

[0094] The rotation angle is calculated as shown in Figure 11B-3. After the rectangle boundary is identified, the four corner points are ordered clockwise, starting with the point with the highest y coordinate. If two points have the same highest y value, the point on the right becomes the origin. The points are numbered 0, 1, 2, and 3 (0 is the start point and 3 is the end point), and the angle between the line connecting the start and end points and the horizontal line is shown in Figure 11B-3.

[0095] The rotation angle is calculated as shown in Figure 11B-3, and the bounding area of ​​the rectangle is measured. If the camera position and surface curvature information are known, it can be determined whether the system is aligned to within a given tolerance for the curved region. If the alignment is not acceptable, a transformation is performed from image 11B-1 to 11B-2 to obtain a better reflective region within the camera's field of view.

[0096] Image 1100B2 was acquired at a rotation angle of 10.28° relative to the surface, resulting in a boundary area of ​​3,896,456.0 pixels. Images 1100B1 and 1100B2 may be two images acquired at different times during the image acquisition sequence. The observed boundary area change and reflected light pattern are compared to that predicted for a known surface. If the observed change does not match that predicted, the image acquisition system is not in the expected position. For example, a smaller-than-expected area may indicate a more convex curvature than expected, or if the grating is only partially reflected and off-screen, it may indicate a misaligned system.

[0097] Similarly, by observing the changes in interface area and reflected light patterns, the curvature or topography of the imaged surface can be determined, allowing the observed locations to be compared to, for example, a CAD model of the entire surface to identify locations on the surface being imaged.

[0098] Figures 11C-1 through 11C-3 show examples of depressions in a surface. The optical cone (also known as the field of view cone in other fields) extends downward from the sensor and includes all light rays received by the sensor (e.g., a camera). As the optical axis extends toward the surface of the object being imaged, the optical cone expands, and the origin of these reflected rays becomes more diffuse when observed. For a flat surface, when the camera and light source are positioned at a specular angle, a uniform square reflection area is expected. As the curvature of the surface increases, more or fewer reflected rays are received by the camera sensor, changing the shape of the reflection area. This information allows us to estimate the roughness (convexity / concaveness) of the observation area. If the surface is convex, the optical cone expands more rapidly and the reflection area is smaller. If the surface is concave, the optical cone expands more slowly and the reflection area is larger.

[0099] Concave shapes diffuse light, creating various angles. A concave surface is indicated when the area of ​​light in the region of interest is greater than the maximum area on a flat surface. Figure 11C-1 shows an image acquired at a rotation angle of 27.8°, with an observed boundary area of ​​15,701,842.0 pixels. Figure 11C-2 shows an image of the same surface acquired at a rotation angle of 90.0°, with an observed boundary area of ​​7,414,352.0 pixels. Figure 11C-3 shows an image of the same surface acquired at a rotation angle of 47.57°, with an observed boundary area of ​​9,584,036.0 pixels. As shown in Figures 11C-1 to 11C-3, even though these are images of the same region, slight displacements (such as rotation or translation) can result in different image results for the same surface, and the center of gravity of the boundary rectangles can also differ. This is due to changes in the light reflected back to the camera.

[0100] Figures 11D-1 and 11D-2 show examples of surfaces with convex curves. Convex curves cause light to converge, resulting in a smaller area of ​​light in the region of interest than the maximum area on a flat surface. Also, the reflection angle is typically not close to any of 0°, 90°, 180°, or 270°. Figure 11D-1 shows an image acquired at a rotation angle of 21.4°, with an observed boundary area of ​​1,122,413.0 pixels. Figure 11D-2 shows an image acquired at a rotation angle of 14.9°, with an observed boundary area of ​​1,066,667.0 pixels.

[0101] Figures 11E-1 and 11E-2 show surface images with the image acquisition system generally aligned normal. For flat or substantially flat features, rotation angles near 0° / 90° / 180° / 270° result in observed areas below the maximum light area. Figure 11E-1 shows a surface image acquired with rotation angles near 0° / 90° / 180° / 270°.

[0102] Whether the part orientation is correct with respect to the image acquisition system can be verified by checking whether the rotation angle is a member of the rectangular symmetry group (C2). In the example of Figures 11A-11E, this is indicated by being within ± thresholds such as 0°, 90°, 180°, and 270°.

[0103] Figure 11F illustrates a scenario in which the shape of the imaged surface results in multiple light regions appearing within a single image. In this illustration, angular information can be used to determine relative curvature within the two-dimensional image of Figure 11F. Light region 1100F-1 has a rotation angle of 0°, is aligned nearly normal to the image acquisition system, and can be considered a flat surface. Light regions 1100F-2 and 1100F-3 have rotation angles not close to 0° / 90° / 180° / 270° and exhibit greater relative curvature. These images can be compared to 3D renderings, CAD models, or other topographical information to verify the position of the image acquisition system.

[0104] FIG. 12 illustrates a static structured light setup according to an embodiment of the present disclosure. As shown in FIG. 12A, an area light is mounted with a diffuser and a diffuser grid pattern. The diffuser grid pattern can be a simple grid. FIG. 12B illustrates an illuminated specular surface, with the grid consisting of 8 mm squares. Uniform illumination is provided across the entire area of ​​the light. The light, grid spacing, and / or pattern can be modified depending on the specific application. The grid breaks up the diffuse light source and directs the light toward defects, allowing them to cast shadows and characterize the defects. Therefore, it is important to balance the width of the grid lines with the width of the grid spacing. If the grid is too wide, defects may be lost in the spaces between the grid lines; if the grid lines are too narrow, insufficient shadows may be cast.

[0105] 13A-13G show surface images acquired using a structured light pattern with a grid structure. FIG. 13A shows a surface image 1200 acquired without a structured light pattern, in which defect 1202 is small and difficult to detect using diffuse illumination alone. FIG. 13B shows image 1240 containing defect 1242 aligned with one line of the grid pattern, creating a visible shadow around it. This shadow can potentially be used by machine learning algorithms to identify the type, depth, severity, etc. of the defect. Similarly, defect 1232 in image 1230 of FIG. 13C, defect 1222 in image 1220 of FIG. 13D, and defect 1212 in image 1210 of FIG. 13E can also be characterized using their shadows.

[0106] Comparing Figure 13A with Figures 13B-13E shows that the systems and methods described herein provide quantitative characterization information about a surface. Using machine learning algorithms, it is possible to correlate the shadows in the images with, for example, the size and depth of a foreign object. The images in Figures 13B-13E are sequential images of a single defect taken at different illumination angles, allowing for detailed characterization of the defect. Using an image acquisition system mounted at the end of a robotic arm allows for precise positional control and allows for image acquisition from multiple angles.

[0107] Figure 13F shows an image 1250 containing a surface defect 1262, and Figure 13G shows a close-up view. In some embodiments, a machine learning algorithm is configured to identify the grid frame 1252 of a structured light pattern used on a background light source, allowing for more accurate characterization of the defect 1262. Identifying the grid square or squares containing the defect 1262 in image 1254 allows for more efficient surface characterization. Using an edge detection algorithm to locate the defect within the grid reduces the image processing footprint, allowing for faster cycle times and reduced noise.

[0108] FIG. 14 illustrates a diffuser plate stack that may be used in one embodiment of the present disclosure. FIG. 14 shows a magnified image 1300 of an image acquisition assembly. An image acquisition device 1302 is positioned at an angle relative to a high-intensity line light 1354. A diffuser stack 1310 is positioned in front of the line light 1354. As shown in FIG. 14, the stack 1310 includes three diffuser plates 1302 with a spacer 1304 between them. The diffuser plates 1302 diffuse the line light, allowing for the acquisition of an intensity profile along an axis perpendicular to the line light. This allows for the conversion of line light into an area light in some embodiments. The diffuser plates 1302 may also be used to create a gradient of light that illuminates the defect as if the light were irradiated from a single direction or at a shallower angle. The spacer 1304 may be the same or different distances between adjacent plates 1302.

[0109] 14 shows an embodiment in which the diffuser stack 1310 includes three diffuser plates 1302, it is expressly contemplated that other embodiments may use more or fewer plates 1302. For example, placing only one diffuser plate 1302 away from the line of light 1354 may be sufficient for some applications. Configurations using two plates 1302 are also useful. Configurations using four, five, six, or more plates 1302 are also possible depending on the application.

[0110] In addition to some of the embodiments of the diffuser plate stacks shown herein, it is expressly contemplated that various numbers and configurations of diffuser plates are possible, including diffuser plates with varying specifications for haze transmission, transparency, and thickness, allowing for a variety of configurations suitable for embodiments herein.

[0111] In some embodiments, one or more diffuser plates 1302 are movable within the system 1300, allowing one or more plates 1302 to be moved in and out of the path of the light line. The spacing 1304 between the plates 1302 and their distance from the light line can also be changed. This can be achieved, for example, by using a sliding mechanism, where a robotic controller controls the sliding to move the plate 1302 to the desired position.

[0112] While it is contemplated that the light sources herein will be turned on and off as needed by the robotic controller, heat may be generated during the process and this heat will need to be dissipated, so one or more thermal management options may be implemented, such as fans, conductive materials, insulation, coolants, or other suitable thermal management means.

[0113] The use of a diffuser plate stack allows for the creation of a diffuse lighting environment without incorporating a dedicated diffuse light source or specialized optical elements into an area backlight. This allows for the conversion of a conventional non-diffused lighting environment into a uniform, diffuse lighting environment as needed. It is also useful for expanding the illumination range within the field of view of an image capture device. The greater the spacing between diffuser plates, the more widely the plates closer to the light source diffuse the light, and the further each subsequent plate diffuses the light. Diffuser plates receive collimated light from the light source and spread the light across the surface by diffusing the angle of incidence. The spacing between plates provides a means to adjust the degree of diffusion that occurs. In particular, a larger spacing between the first and second plates results in a stronger diffusion effect, as the initially diffused light reaches the second plate with a more diffused profile.

[0114] Image capture, especially of reflective surfaces, often requires a dedicated light source that exceeds the intensity and characteristics of ambient light. For example, when using light sources such as LEDs, a diffuser plate is useful to spread the light evenly across the field of view of the image capture device. The combination of a structured light bar and a diffuser plate can create a diffuse lighting environment without the need for specialized diffuser plates or other optical elements. Additionally, some embodiments enable a compact image capture system that can be mounted on the end of a robotic arm, as shown in Figures 7A and 7B, while also improving the mobility of the surface modification tool mounted thereon.

[0115] 6-7 illustrate an embodiment in which the light source is positioned offset from the camera; however, other configurations are expressly contemplated. Some current systems may rely on projection systems requiring expensive high-lumen screens (e.g., LCD / LED) to project tunable reflections onto a surface. These systems also require multiple cameras, making processing the reflected images computationally intensive (e.g., image stitching). Embodiments herein enable similar analysis with smaller light panels and fewer cameras. The end-effector-mounted systems described herein are highly mobile, allowing the associated field of view to be kept small, thereby reducing the computational complexity of image processing.

[0116] The systems shown in Figures 15-18 are configured such that the image capture camera captures images through an optical panel, thereby reducing the mechanical "footprint" required for the system at the end of the robot arm. These embodiments also allow the camera to be positioned at a steeper angle relative to the surface normal, thereby reducing the overall length of the system. The systems and methods disclosed herein also contribute to reducing the overall length, width, and height of the vision system at the end of the robot arm by utilizing specialized lenses, such as folded optics and thin lens stacks. Each of the embodiments shown in Figures 15-18 has the advantage of reducing the space required at the end of the robot arm.

[0117] 15A and 15B show schematic diagrams of a surface image acquisition system 1500 according to one embodiment of the present disclosure. The image acquisition system 1500 includes at least two cameras 1510, which image a specularly reflective surface 1520 through a light panel 1530, which may be a grid light panel or other suitable illumination system. Each camera is positioned at an angle 1512 or 1514 relative to the surface 1520, which may be the same or different. Each camera images the surface 1520 through a region of the light panel 1530, e.g., regions 1522 and 1524. These regions may include openings that extend partially or completely through the light source 1530. FIG. 15A shows a side view of the system 1500, and FIG. 15B shows a perspective view 1550 illustrating the relative placement of the cameras 1510. The cameras are spaced apart along the length 1570 and width 1560 of the light source 1530. In some embodiments, the cameras 1510 are positioned diagonally across from the light panel 1530. The system 1500 is designed to minimize holes in the grating reflection image so that no areas are unilluminated by the light source 1530.

[0118] The ability to reduce the volume occupied by system 1500 is limited by the size of light source 1530. Some applications require a larger light source, while others require a smaller one. System 1500 extends the width of the scanned image while reducing the effects of ambient light. Here, "length" is defined as the dimension in the principal plane of the reflected image, and "width" is the dimension perpendicular to it. For example, while the current industry standard uses light sources on the order of one square meter, the system of the present invention can utilize light sources as small as a few centimeters. This miniaturization allows for use in end-of-arm robotic systems, reducing the risk of collision with target surfaces or other robot components.

[0119] 16A and 16B show schematic diagrams of an outward-facing surface image acquisition system according to an embodiment of the present disclosure. The system 1600 includes two or more cameras 1610 that image a specularly reflective surface 1620 through a light panel 1630. The cameras 1610 are positioned to face in opposite directions and have a field of view through an area 1640 of the light panel. The area 1640 may include or be defined by an opening in the light source 1630. The cameras 1610 are positioned such that a first field of view 1622 (camera angle 1562) does not overlap with a second field of view 1624 (also angle 1562). In addition to the two cameras shown, a four-camera configuration is also contemplated, with adjacent cameras spaced approximately 90 degrees apart.

[0120] In system 1600, the physical size of the reflected image increases longitudinally, making full use of the length of the light source. Because cameras 1610 are not imaging the same area, the overall field of view is expanded.

[0121] However, because camera 1610 is positioned at an angle to widen the field of view, the area directly below the field of view may not be directly illuminated. As shown in Figure 16B, defect 1660 may be illuminated by diffuse light in image 1650.

[0122] In some embodiments, the field of view may be reduced so that fields of view 1622 and 1624 overlap or do not overlap, eliminating or nearly eliminating the gap. In some embodiments, light source 1630 may be fully or nearly transparent, allowing for viewing directly through surface 1620. Furthermore, the light source may provide uniform illumination downward toward surface 1620, allowing for an image without shadows or unevenness.

[0123] 17A and 17B show schematic diagrams of a binocular front-view surface image acquisition system according to an embodiment of the present disclosure. Similar to human vision, the system 1700 includes two cameras 1710, each capturing an image of a surface 1720 through a light source 1730. Knowing the relative positions of the cameras 1710 allows for depth information to be obtained from differences in the image 1750, as shown in FIG. 17B, for example. Furthermore, binocular vision provides a highly visually accurate image, as seen by a consumer. The camera 1710 can capture an image through an aperture (partial or full) in the light source 1730.

[0124] In some embodiments, the camera 1710 may be positioned on the optical axis and point directly downward, or one camera 1710 may view the surface 1720 through an opening in the center of the light source 1730.

[0125] FIG. 17B shows an example of a stereo image, where images acquired from two different positions are computationally combined, which is useful for obtaining 3D shape information.

[0126] In system 1700, there may be areas between fields of view 1712 and 1714 that are not completely imaged, depending on the distance between cameras 1710 and the distance between each camera 1710 and surface 1720. However, some embodiments contemplate a configuration in which three cameras 1710 are arranged in a triangular configuration.

[0127] FIG. 18 shows a schematic diagram of a scattered-light surface image acquisition system 1800 according to an embodiment of the present disclosure. The system 1800 uses directional lighting techniques to illuminate a surface 1820. The light source 1830 is a panel-like light source that projects multiple lights through a panel, projecting the light downward toward the surface 1820. The light source 1830 may include multiple light sources (e.g., a flat dome light) positioned around the periphery, which transmit light through the panel. Additionally, the panel may have a concave-convex shape to project the light toward the surface. For example, a light source such as the "LFX3-PT series" manufactured by CCS Corporation may be used in an embodiment of the present disclosure. The light source 1830 uses a light guide plate with a feature configured to project light downward toward the surface and not toward the camera, thereby simultaneously providing a diffused effect like a dome light and an on-axis lighting effect like a causal light.

[0128] 18 shows a configuration in which two cameras 1840 are positioned at a distance from each other, it is expressly contemplated that an additional camera 1810 may be placed. This configuration allows the camera 1840 to have a field of view normal to the surface 1820. Also, dark field illumination techniques can be used to provide supplemental illumination to areas of the surface that are not illuminated by specular reflection. Additionally, camera 1810 does not necessarily need to be positioned normal to the defect for image acquisition.

[0129] System 1800 provides the flexibility of easily adding or removing additional cameras (e.g., camera 1810) without interfering with the reflected image captured by existing camera 1840, because no permanent opening in the light source is required.

[0130] However, while system 1800 is shown based on a configuration similar to that of FIG. 17, it is expressly contemplated that transparent, scattered light source 1830 may be incorporated as an alternative to light sources 1530, 1630, 1730 in any of systems 1500, 1600, or 1700.

[0131] Systems 1500-1800 may reduce the size of a mirror inspection system by stacking the camera above the light source. This allows the camera to image through the light source. Using the systems described herein, the size of the reflected image of the light source can be maintained or increased. While some systems may have diffusely illuminated areas or holes (unilluminated areas) in the reflective grating, as previously mentioned, precise camera placement can reduce these interruptions. The present system also allows for the reduction of the overall size of the image acquisition system by rearranging components. For example, the need to place the light source on the same plane as one or more cameras can be eliminated. Instead, the light source can be placed on a different plane (non-coplanar) than the camera, thereby reducing the system's footprint (e.g., planar footprint). Stacking the devices reduces the overall system volume, allowing for a wider effective field of view, fewer images for highly curved surfaces, and ultimately shorter inspection cycle times. While the present system is useful for detecting defects in the clear coat layer of, for example, mirror-finished surfaces on automobiles, it is expressly contemplated that it can also be used for image acquisition of other mirror-finished surfaces.

[0132] 19 shows a schematic of a surface image acquisition system 1900. The system 1900 may be designed to be attached to a robotic surface modification unit 1970 via a mounting portion 1930. For example, the robotic surface modification unit 1970 may include an end effector 1972 that receives the mounting portion 1930. The end effector 1972 may be provided at the end of a robotic arm 1975.

[0133] Surface image acquisition system 1900 includes image acquisition system 1910. Image acquisition system 1910 may include one or more image acquisition devices 1911 (e.g., a camera, video camera, or other suitable image acquisition device). Image acquisition system 1910 may also include one or more light sources 1914 (e.g., an area light source for diffuse illumination, a line light source for specular image acquisition, a panel light, a flat dome light, etc.). It is expressly contemplated that in some embodiments, a single light source 1914 may be sufficient.

[0134] In some embodiments, the light source 1914 is at least partially coplanar with the one or more image capture devices 1910. However, in other embodiments, the light source 1914 is not coplanar with the one or more image capture devices 1910, but may be configured to be located between the image capture device 1911 and the surface 1990. In some embodiments, the image capture device 1911 images the surface through the light source 1914. The light source 1914 may have an aperture to allow the image capture device 1911 to view. However, in some embodiments, the light source 1914 is sufficiently transparent to allow the image capture device 1911 to capture an image through the light source 1914 without significant distortion.

[0135] The image acquisition system 1910 is shown including a movement mechanism 1916. The movement mechanism 1916 may be responsible for adjusting the angle of the image acquisition device 1911 or the light source 1914 to change the relative angle between the image acquisition device 1910 and the light source 1914. Alternatively, the movement mechanism 1916 may be configured to change the relative positions of the image acquisition devices 1911 relative to one another or relative to the light source 1914. For example, the system 1900 may be reconfigured within the same device or between surface image acquisition operations (e.g., when moving from a first defect location to a second defect location). The system 1900 may be able to adjust the position and / or orientation of the image acquisition devices 1910 relative to one another or relative to the light source 1914.

[0136] The moving mechanism 1916 may also be capable of adding, removing, or modifying the diffusing mechanism 1912. The diffusing mechanism 1912 may include a pattern 1904 (to provide structured illumination) positioned relative to the light source 1914. Alternatively, the diffusing mechanism 1912 may include a diffuser plate stack 1906. The stack 1906 may include one plate or two or more spaced apart plates. The stack 1906 may include three, four, five, six, or more plates. Other components 1918 may be included in the image acquisition system 1910.

[0137] Although the surface image acquisition system 1900 is shown including a controller, the controller 1960 may be located elsewhere within the image acquisition system 1900 or the robotic surface modification unit 1970, or may be remote from either. The controller 1960 includes a light source selector 1962 that can select whether the first or second light source 1914 is turned on, or whether both are turned on or off. For each light source, a light intensity selector 1464 can adjust the intensity of the output light.

[0138] The controller 1960 may also include an image capture device position selector 1967. From the position selector 1967, the movement mechanism 1916 receives instructions regarding the physical location and / or orientation of the image capture device 1911.

[0139] The controller 1960 may include a diffuser selector 1966. In some embodiments, the selector 1966 selects one of a plurality of structured patterns 1904 to place in front of the backlight. In some embodiments, a single pattern 1904 is used throughout the repair process, eliminating the need for switching. The diffuser selector 1966 may select a pattern 1904 to place between the light source 1914 and the work surface 1990, or may adjust the placement of a diffuser in the diffuser stack 1906. For example, the controller 1960 may instruct the movement mechanism 1916 to increase or decrease the spacing between the diffusers or the spacing between the diffuser stack 1906 and the light source 1914. Based on feedback from the surface analyzer 1950, the controller 1960 may use a repair strategy generator 1982 to generate a repair strategy to address the detected defects. However, if a repair strategy has already been generated based on a pre-scan of the work surface 1990, the repair strategy modifier 1984 can be used to modify the repair strategy based on information obtained from the surface analyzer 1950.

[0140] The surface analysis unit 1950 can acquire the acquired images using an image receiving unit 1952. The surface analysis unit 1950 can detect defects on the work surface 1990 using a defect identification unit 1954, for example, utilizing statistical image processing and feature detection algorithms trained by the algorithm learning unit 1922. The defect analysis unit 1956 can use the acquired images to determine information about the detected defects, such as the type of defect, its size, its location on the work surface 1990, its location in the clear coat layer of the surface 1990, and the severity of the defect. If the image acquisition system 1910 acquires images of the repaired surface 1990, a haze assessment unit 1980 can assess the amount of haze that has occurred on the surface 1990. The surface analysis unit 1950 can also include other functional units 1957.

[0141] The surface analysis unit 1950 may further include a position verification unit 1955, which can verify the position of the image acquisition system 1910 relative to the work surface 1990. Images may be acquired by the image receiving unit 1452. From the acquired images, the topography calculation unit 1953 may calculate the curvature of the imaged area. The position verification unit 1955 may then compare the curvature at the current position with the surface characteristic data 1924 to verify whether the image acquisition system 1910 and / or the surface modification unit 1970 are correctly positioned for the surface modification process. The calculated topography and / or position verification information may be stored in the data store 1920. The surface analysis unit 1950 may monitor drift over time, i.e., whether the image acquisition system 1910 and / or the surface modification unit 1970 are always in the correct position, approaching the correct position, or moving away from the correct position during a series of surface modification processes. Based on the trends, the repair strategy generator 1982 may adjust the repair strategy to reflect the need to adjust the starting location of the repair operation.

[0142] 19 includes a data store 1920, the data store 1920 may be removed from the surface image acquisition system 1900 and accessed via the communication component 1902. The data store 1920 may include an algorithm learner 1922 for modifying machine learning algorithms to improve the characterization of defects by the defect analyzer 1956 or the quantification of haze by, for example, the haze evaluator 1958. One or more algorithm learners 1922 may be stored in the data store 1924 for generation of repair strategies by the repair strategy generator 1982 or modification by the repair strategy modifier 1984. However, while supervised algorithmic techniques are possible, it is expressly contemplated that unsupervised algorithmic techniques may be used; for example, an image segmentation algorithm may be used in some embodiments.

[0143] Surface characteristic data 1924 may also be stored in data store 1920 and may provide information regarding the characterization of detected defects and detected surface haze. Data store 1920 may also include one or more light source options 1926 obtained by controller 1960. For example, light source options 1926 may include possible angles relative to image capture device 1911 or angles between first and second light sources 1914. Data store 1920 may also include a repair strategy component 1928, which may include previously generated repair strategies and surface conditions associated with those repair strategies. Repair strategy data 1928 may be used to inform machine learning algorithms driving repair strategy generator 1982 or repair strategy modifier 1984.

[0144] In some embodiments, the surface image acquisition system 1900 may output data to a display 1940 using a communication component 1902. The communication component 1902 may be in communication with a graphical user interface generator 1944, which is shown in FIG. 19 as part of the display 1940, but may also be part of the controller 1960, a remote controller, or other suitable computing device. The generated GUI is displayed on the display 1940 via a user interface 1942.

[0145] A user can interface with system 1900, for example, using user interface 1942. User interface 1942 may provide access to applications used, for example, for workflow control by controller 1960. User interface 1942 may also be used to display acquired images, results of image processing, metadata associated with acquired images, defect characterization information, etc.

[0146] In some embodiments, the work surface 1990 may be a mirrored surface having reflective properties. The work surface 1990 may be moved during the surface modification process and may be moved using a movement mechanism 1994, such as when a vehicle moves from a first location to a second location on an assembly line. In embodiments in which the work surface 1990 is movable, a stabilizer 1990 or stabilization system may be used to maintain the relative position of the work surface 1990 and the image acquisition system 1910.

[0147] Described herein are surface inspection systems that include an image acquisition device, such as a camera, one or more light sources, a distance sensor, etc. The systems and methods herein provide components for managing and executing the image acquisition and for processing the acquired images to obtain defect characterization information and surface characterization information. Additionally, systems and methods are described that can store and retrieve acquired images, image metadata, and results of defect detection and characterization, and manipulate such information to generate or improve repair strategies. It is expressly contemplated that the systems herein will be interoperable with, and may be controlled by, the controller of the robotic arm to which they are attached. It is also contemplated that these systems will be interoperable with other components of the robotic system.

[0148] The systems and methods herein enable the coordination of machine vision devices, efficient and highly portable image acquisition under lighting conditions, and identification of surface characteristics and defects on specular surfaces.

[0149] However, it is expressly contemplated that the systems and methods herein are useful in other industries. For example, while the vehicles and use cases described herein are intended for repair work at the initial manufacturing site, automotive aftermarket use cases are also applicable. They may also be applied to periodic or continuous assessment of internal or external processes, such as part repair, evaluation of metal or paint finishes against other product families, and even high spatial resolution mapping of environments using mobile robots.

[0150] Additionally, it is envisioned that the surface image acquisition system herein may be useful for other specular surfaces, such as capturing images of a surface before and after adhesive application.

[0151] FIG. 20 is a block diagram of a repair strategy generation architecture. The remote server architecture 2000 illustrates one embodiment of the repair strategy generator 2010. For example, the remote server architecture 2000 can provide computation, software, data access, and storage services, without requiring end users to know their physical location or configuration. In each embodiment, the remote server may provide services using an appropriate protocol over a wide area network, such as the Internet. For example, the remote server may provide applications over the wide area network, which users can access through a web browser or other computing component. The software or components shown and described in FIGS. 1-19 and their corresponding data may be stored on a remote server. Computing resources in a remote server environment may be centralized or distributed in a remote data center. The remote server infrastructure may be configured to provide services through a shared data center and appear to users as a single point of access. Thus, the components and functionality described herein may be provided remotely using the remote server architecture, installed directly on traditional servers or client devices, or otherwise provided.

[0152] In the example shown in Figure 20, several items are similar to those shown in the previous figures. Figure 20 particularly shows that repair strategy generation systems may be located at a remote server location 2002. Thus, computing device 2020 accesses these systems through remote server location 2002. Operator 2050 can also access user interface 2022 using computing device 2020.

[0153] FIG. 20 also illustrates another example of a remote server architecture. FIG. 20 illustrates that configurations are contemplated in which some elements of the systems described herein are located at a remote server location 2002 while others are not. By way of example, storage devices 2030, 2040, or 2060, or repair system 2070, may be located at a location different from location 2002 and accessed via remote server 2002. Regardless of their location, they may be accessed directly from computing device 2020, through a network (wide area network or local network), hosted or provided as a service, or through a remotely located connectivity service. Additionally, data may be stored in virtually any location and intermittently accessed or transferred by interested parties as needed. For example, physical carrier media may be used in place of or in conjunction with electromagnetic carrier media.

[0154] Additionally, components of the systems described herein, or portions thereof, may be located on various types of devices, including servers, desktop computers, laptop computers, embedded computers, industrial controllers, tablet computers, or other mobile devices (such as palmtop computers, mobile phones, smartphones, multimedia players, personal digital assistants, etc.).

[0155] 21 and 22 show examples of mobile devices that may be used in the embodiments shown in the figures.

[0156] 21 is a simplified block diagram illustrating an example of a portable or mobile computing device that may be used as a user or client portable device 2121 (e.g., computing device 2020 in FIG. 20). The system (or portions thereof) may be deployed on such a portable device. For example, the portable device may be located in an operator compartment of computing device 920 and used to generate, process, or display data. FIG. 17 is another example of a portable or mobile device.

[0157] 21 provides a general block diagram of components of a client device 2116, which executes and cooperates with some of the components shown and described herein, or executes some of the components and cooperates with others. A communications link 2113 is provided in the device 2116, which allows the mobile device to communicate with other computing devices and, in some embodiments, provides a channel for automatically receiving information (e.g., by scanning). Examples of communications link 2113 include wireless services that provide access to cellular networks, protocols that provide wireless connections to local networks, etc.

[0158] As another example, the application may be received by a removable Secure Digital (SD) card connected to the interface 2115. The interface 2115 and communication link 2113 communicate via a bus 2119 with a processor 2117 (which also has processor functionality), along with memory 2121, input / output (I / O) components 2123, a clock 2125, and a location system 2127.

[0159] I / O components 2123 may be provided to facilitate input and output operations, and the device 2116 may include input components such as buttons, touch sensors, optical sensors, microphones, touch screens, proximity sensors, accelerometers, orientation sensors, etc., and output components such as a display device, speakers, and / or a printer port. Other I / O components 2123 may also be used.

[0160] The clock 2125 may comprise a real-time clock component that outputs the time and date, and may also provide timing functions for the processor 2117.

[0161] As an example, location system 2127 includes components that output the current geographic location of device 2116. This may include, for example, a Global Positioning System (GPS) receiver, a Loran system, a dead-reckoning system, a cell tower triangulation system, or other location determination systems. It may also include mapping or navigation software that generates destination maps, navigation routes, and other geographic features.

[0162] Memory 2121 stores operating system 2129, network settings 2131, applications 2133, application configuration settings 2135, data store 2137, communication drivers 2139, and communication configuration settings 2141. Memory 2121 may include all types of tangible, volatile and non-volatile computer-readable memory devices. It may also include computer storage media (described below). Memory 2121 stores computer-readable instructions that, when executed by processor 2117, cause computer-implemented steps or functions to be performed in accordance with the instructions. Processor 2117 may be activated by other components to support their functions.

[0163] 22 shows an example in which the device is a smartphone 2271. The smartphone 2271 has a touch-sensitive display 2273 that displays icons, tiles, or other user interface features 2275. Features 2275 can be used by a user to run applications, make calls, and perform data transfer operations. Generally, smartphones 2271 are built on mobile operating systems and offer more advanced computing capabilities and connectivity than feature phones.

[0164] It should be noted that other configurations of device 2216 are also contemplated.

[0165] FIG. 23 is a block diagram of a computing environment that can be used with the embodiments shown in the previous figures.

[0166] FIG. 23 is an example of a computing environment in which the systems and methods described herein, or portions thereof (for example), may be deployed. With reference to FIG. 23, an exemplary system for executing some embodiments is in the form of a general-purpose computing device referred to as a computer 2310. Components of the computer 2310 may include a processing unit 2320 (which may include a processor), a system memory 2330, and a system bus 2321 that couples various system components connecting the system memory to the processing unit 2320. The system bus 2321 may be any type of bus structure, such as a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. The memory and programs described in connection with this specification may be deployed in the corresponding portions of FIG. 23.

[0167] The computer 2310 typically includes a variety of computer-readable media. Computer-readable media are any media accessible by the computer 2310, including volatile and nonvolatile media and removable and non-removable media. For example, computer-readable media include computer storage media and communication media. Computer storage media do not include modulated data signals or carrier waves; they are volatile and non-volatile, removable and non-removable hardware storage media implemented in any method or technology for storage of information (such as computer-readable instructions, data structures, program modules, or other data). For example, RAM, ROM, EEPROM, flash memory, CD-ROM, DVD or other optical disk storage media, magnetic cassettes, magnetic tape, magnetic disk storage media, or any other media accessible by the computer 2310. Communication media may embodi a means for carrying computer-readable instructions, data structures, program modules, or other data and include any information transmission media. A "modulated data signal" means a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal.

[0168] The system memory 2330 includes computer storage media in the form of volatile and / or nonvolatile memory such as read-only memory (ROM) 2331 and random access memory (RAM) 2332. A basic input / output system (BIOS) 2333 is stored in ROM 2331 and contains the basic routines that transfer information between elements within the computer 2310, such as during start-up. RAM 2332 typically contains data and / or program modules that are immediately accessed by or presently operated on by the processing unit 2320. For example, FIG. 23 illustrates operating system 2334, application programs 2335, other program modules 2336, and program data 2337.

[0169] The computer 2310 may also include other removable / non-removable and volatile / non-volatile computer storage media. For example, Figure 23 shows a hard disk drive 2341, a non-volatile magnetic disk 2352, an optical disk drive 2355, and a non-volatile optical disk 2356 that read from and write to non-removable, non-volatile magnetic media. The hard disk drive 2341 is typically connected to the system bus 2321 through a non-removable memory interface, such as interface 2340. The optical disk drive 2355 is also connected to the system bus 2321 through a removable memory interface, such as interface 2350.

[0170] Alternatively or additionally, the functionality described herein may be performed in part by one or more hardware logic components, such as, without limitation, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SoCs), complex programmable logic devices (CPLDs), etc.

[0171] The drives and their associated computer storage media discussed above and illustrated in Figure 23 store computer-readable instructions, data structures, program modules, and other data for computer 2310. For example, Figure 23 illustrates hard disk drive 2341 as storing operating system 2344, application programs 2345, other program modules 2346, and program data 2347. These components can either be the same as or different from operating system 2334, application programs 2335, other program modules 2336, and program data 2337.

[0172] A user may enter commands and information into the computer 2310 through input devices such as a keyboard 2362, a microphone 2363, and a pointing device 2361, such as a mouse, trackball, or touch pad. Other input devices (not shown) may include a joystick, game pad, satellite receiver, scanner, or the like. These and other input devices are typically connected to the processing unit 2320 through a user interface 2360, which is connected to the system bus, although they may be connected by other interface and bus structures. A visual display, such as a display device 2391, is also connected to the system bus 2321 through an interface, such as a video interface 2390. In addition to the monitor, other peripheral output devices such as speakers 2397 and printer 2396 may also be connected through an output peripheral interface 2395.

[0173] The computer 2310 operates in a networked environment where it is connected to remote computers, such as a remote computer 2380, using logical connections such as a local area network (LAN) or a wide area network (WAN).

[0174] When used in a LAN networking environment, the computer 2310 is connected to the LAN 2371 through a network interface or adapter 2370. When used in a WAN networking environment, the computer 2310 typically includes means, such as a modem 2372, for establishing communications with the WAN 2373, such as the Internet. In a networked environment, program modules may be stored in remote memory storage devices. For example, Figure 23 illustrates that remote application programs 2385 may reside on the remote computer 2380.

[0175] An image capture system for a reflective surface is disclosed, the system including a first light source mounted on a robotic arm, a second light source mounted on the robotic arm different from the first light source, and an image capture device positioned to capture an image of the reflective surface, wherein the first light source, the second light source, and the image capture device are mounted on the robotic arm, and the first light source is positioned such that the field of view of the image capture device captures a specular reflection from the first light source.

[0176] In the image acquisition system, the second light source may be disposed relative to the image acquisition device so that scattered light from the second light source falls within the field of view.

[0177] The image capture system may include a second light source disposed between the image capture device and the first light source.

[0178] The image acquisition system may be such that the first light source, the second light source, and the image acquisition device are at least partially housed within a housing, and the housing is attached to a robot arm.

[0179] The image acquisition system may be such that the first light source comprises a static structured light pattern.

[0180] The static structured light pattern may include a repeating pattern.

[0181] The repeating pattern may include a line, a polygon, or a circle.

[0182] The image acquisition system may be attached to an end effector of a robotic arm.

[0183] The end effector may have a surface modification tool attached to it.

[0184] The end effector may be configured to move from a first position (where the tool contacts the surface) to a second position (where the image acquisition system acquires images).

[0185] The image acquisition system may include a controller that generates a trajectory for the tool based on the acquired images.

[0186] The end effector may also be configured to move to a third position where a second tool contacts the surface.

[0187] The end effector may also be configured to move to a fourth position where a third tool contacts the surface.

[0188] The image capture device may be a camera.

[0189] An image capture system for a reflective surface is disclosed, the system including a mount configured to be connected to a robotic arm, a light source connected to the mount, and an image capture device positioned to capture an image based on light irradiated onto the reflective surface.

[0190] In the image acquisition system, the light source may be a high-intensity line light, and the light source may be positioned so that specular reflection falls within the field of view.

[0191] The image acquisition system may further include a diffuser disposed in front of the light source.

[0192] The diffuser may be a first diffuser, and light from the light source may pass through the first diffuser and then through the second diffuser. A gap may be provided between the first and second diffusers.

[0193] The image acquisition system may further include a third diffuser, the third diffuser being arranged in a stacked structure, wherein light from the light source passes through the second diffuser and then through the third diffuser, and the spacing between the second and third diffusers may be the same.

[0194] The diffuser plate may be removable.

[0195] The image capture system may include a controller that controls the robot arm so that the image capture device is aligned with respect to a normal vector of the reflecting surface.

[0196] The image acquisition system may further include a second light source, which may be an area light.

[0197] The image acquisition system may include a structured light pattern disposed between the second light source and the reflective surface.

[0198] In the image acquisition system, the light source may be arranged on the same plane (coplanar) as the image acquisition device.

[0199] The light source may be disposed between the image capture device and the reflective surface.

[0200] The image capture device may be configured to capture an image of the surface through an opening in the light source.

[0201] The image capture device may be positioned normal to the light source.

[0202] The image capture device may be disposed at an angle to the light source.

[0203] The image capture device may be a first image capture device and the system may further include a second image capture device, the second image capture device being positioned at a different angle relative to the light source.

[0204] The first and second image capture devices may be arranged facing each other (cross-facing configuration).

[0205] The first image capture device may be offset from the second image capture device along the length of the light source.

[0206] The first image capture device may be offset from the second image capture device along the width of the light source.

[0207] A first image capture device may image the surface through a first portion of the light source, and a second image capture device may image the surface through a second portion of the light source.

[0208] The first and second portions may overlap.

[0209] The first portion and the second portion may be spaced apart.

[0210] The first portion may include an opening through which the light source passes.

[0211] The image acquisition system may further include a movement mechanism.

[0212] The image capture system may be configured such that the movement mechanism changes the position or orientation of the image capture device.

[0213] A robotic surface modification system is disclosed that includes a robotic arm, a force control unit connected to the robotic arm, an end effector connected to the force control unit, and an image acquisition system attached to the end effector, the image acquisition system configured to acquire an image of a surface.

[0214] The robotic system may include a housing in which the image acquisition system is attached to the end effector, the housing at least partially enclosing the image acquisition device and the light source.

[0215] The light source may be positioned such that specular reflection from the area light source is captured by the image capture device.

[0216] The light source may include a structured light pattern.

[0217] The light source may be a first light source and may further include a second light source, the second light source being different from the first light source.

[0218] The second light source may be a line light and the image capture device may be positioned to receive scattered light.

[0219] The robotic system may further include a surface modification tool configured to modify the surface when aligned with the force control unit, where the surface modification includes adding or removing material.

[0220] The robotic system may include a second surface modification tool, which may be attached to the end effector.

[0221] The system may be such that the end effector is rotationally movable from a first position (where the image acquisition system is aligned with the normal vector of the surface) to a second position (where the surface modifying tool is aligned with the force control unit).

[0222] The system may include a second tool, and the end effector may be rotationally movable to a third position where the second surface modifying tool is aligned with the force control unit.

[0223] The system may include a fourth tool, the end effector being rotationally movable to a fourth position in which a third surface modifying tool is aligned with the force control unit.

[0224] The system may include a robotic control unit configured to send movement commands to the robotic arm.

[0225] The system may include a robotic control unit configured to send movement commands to the rotation mechanism.

[0226] The system may include an image analyzer that processes the acquired images.

[0227] The system may include a controller that selects a trajectory for the modifying motion of the surface modifying tool based on the analyzed image.

[0228] The system may be configured such that the image analyzer quantifies the amount of haze on the surface.

[0229] The processing may include detecting the curvature of the surface based on the acquired images.

[0230] The system may further include a position verification system, which may include: a topography acquisition unit that acquires a known topography; and a position verification unit that compares the detected curvature with the known topography.

[0231] If the detected topography is detected to be different from the known topography, the trajectory of the surface modification may be updated.

[0232] Updating the trajectory of the surface modification may include changing the starting point, expanding or shrinking the modification area, or selecting a different trajectory.

[0233] The system may include a light source in the image acquisition system.

[0234] The light source may be located in the same plane as the image capture device.

[0235] A light source may be disposed between the image capture device and the surface.

[0236] The image capture device may be configured to image the surface through a light source.

[0237] The image capture device may be configured to image the surface through an opening in the light source.

[0238] A method for modifying a surface is disclosed, the method including the steps of: imaging a surface a first time using an image acquisition system attached to a surface modification system; characterizing the surface based on the image acquired by the image acquisition system; switching the relative positions of the image acquisition system and a tool of the surface modification system based on the characterization; performing a surface modification process using the tool; imaging the surface a second time using the image acquisition system; and evaluating the surface modification process based on the second image acquired.

[0239] The method may be implemented such that the image acquisition system includes an image acquisition device, a first light source, and a second light source, the first light source being used in the first imaging step and the second light source being used in the second imaging step, and the first and second light sources being different.

[0240] The method may be performed such that, in a first imaging step, the first light source is tilted relative to the image capture device, and the image capture device is configured to capture specular reflection.

[0241] The method may be performed such that, in the second imaging step, the second light source is tilted relative to the image acquisition device, and the image acquisition device is configured to acquire scattered light.

[0242] The method may be implemented such that the first light source may include a structured light pattern.

[0243] The method may be performed such that a diffuser may be disposed between the second light source and the surface.

[0244] The method may be practiced to include a second diffuser spaced apart from the diffuser.

[0245] The method may include a third diffuser plate positioned further away from the second diffuser plate, the third diffuser plate being located on the opposite side of the first diffuser plate, and may be performed such that the spacing between the first and second diffuser plates is approximately equal to the spacing between the second and third diffuser plates.

[0246] The method may be performed such that the characterizing step may include locating surface defects.

[0247] The method may be performed such that the characterizing step may include identifying a type of surface defect.

[0248] The method may be performed such that the characterizing step may include quantifying the amount of haze on the surface.

[0249] A method for positioning an image acquisition system over a surface is disclosed, the method comprising the steps of acquiring an image of the surface using an image acquisition device, generating a surface topography of the surface using a topography generator based on the acquired image, acquiring a known surface topography of the surface, comparing the generated surface topography with the known surface topography, and generating a misalignment indicator based on differences detected by the comparison.

[0250] The method may further include obtaining a surface correction trajectory template for the surface correction system, the surface correction trajectory template including a starting point, a path, and a region, and generating a new surface correction trajectory that changes at least one of the starting point, the path, or the region based on the deviation indicator.

[0251] The method may be performed such that generating a surface topography includes fitting a circumscribing rectangular boundary to a portion of the acquired image, the rectangular boundary including an area within which reflections of a light source on the surface can be acquired.

[0252] The method may be implemented such that generating the surface topography may include calculating a rotation angle based on the rectangular boundary.

[0253] The method may be implemented such that generating the surface topography may include calculating an area of ​​the rectangular boundary.

[0254] The method may be performed such that the comparing step may include determining that an image capture system including the image capture device is aligned with the surface.

[0255] The method may be implemented such that the comparing step includes determining that an image capture system including the image capture device is not aligned with the surface, and may further include the step of transforming the image based on the determination.

[0256] The method may be performed such that generating a surface topography may include classifying the surface as concave if the area is greater than the maximum area of ​​a flat surface.

[0257] The method may be implemented such that generating a surface topography may include classifying the surface as convex if the area is smaller than the maximum area of ​​a flat surface.

[0258] The method may be implemented such that it may further comprise the step of storing the deviation indicator.

[0259] The method may be implemented such that, if the deviation index is one of a plurality of deviation indexes, the method may further include the step of obtaining the plurality of deviation indexes and detecting a deviation trend.

[0260] The method may be implemented such that it may further include modifying the surface-corrected trajectory template if the deviation trend is one that tends to increase over time.

[0261] The method may be performed such that the image capture device includes a grid light source.

[0262] The method may be implemented such that generating the surface topography may include calculating a boundary area within which a reflection of the gridded light source can be obtained.

[0263] The method may be implemented such that generating the surface topography may include calculating a rotation angle of the grid light relative to the boundary region.

[0264] A surface image acquisition system is disclosed, the system including a first image acquisition device, a light source configured to be positioned between the first image acquisition device and a surface to be imaged, and a mount for coupling the surface image acquisition system to a robotic arm, wherein the first image acquisition device acquires an image of the surface through the light source.

[0265] The system may further include a second image capture device.

[0266] The system may be configured such that the second image capture device is coplanar with the first image capture device.

[0267] The system may be configured such that a first image capture device captures an image of the surface through a first portion of the light source, and a second image capture device captures an image of the surface through a second portion of the light source.

[0268] The system may be configured such that the first and second portions may overlap.

[0269] The system may be configured such that the first and second parts may be separate.

[0270] The system may be configured such that the first and second portions may be spaced apart along both the width and length of the light source.

[0271] The system may be configured such that the first portion may include an opening through which the light source passes.

[0272] The system may be configured such that the light source comprises a panel.

[0273] The system may be configured so that the light source is transparent.

[0274] The system may be configured such that the light source scatters light towards the surface.

[0275] The system may be configured to prevent scattering of light from the light source towards the first image capture device.

[0276] The system may be configured such that the panel comprises a transparent panel and the light source comprises a light emitter, the light emitter being arranged to emit light into the transparent panel.

[0277] The system may be configured such that the panel may include a plurality of surface features configured to scatter light toward the surface.

[0278] The system may be configured such that the first image capture device is tilted at a first angle relative to the light source and the second image capture device is tilted at a second angle relative to the light source.

[0279] The system may be configured such that the first image capture device is positioned along an axis normal to the light source.

[0280] The system may further include a third image capture device.

Claims

1. 1. An image acquisition system for a reflective surface, comprising: a first light source attached to the robot arm; a second light source attached to the robot arm, different from the first light source; an image capture device positioned to capture an image of the reflective surface; Equipped with the first light source, the second light source, and the image capture device are mounted on the robot arm, and the first light source is positioned relative to the image capture device such that a field of view of the image capture device captures specular reflection from the first light source.

2. The image acquisition system of claim 1 , wherein the second light source is positioned relative to the image acquisition device such that a field of view of the image acquisition device acquires scattered illumination from the second light source.

3. The image acquisition system of claim 1 , wherein the second light source is disposed between the image acquisition device and the first light source.

4. The image acquisition system of any one of claims 1 to 3, wherein the first light source, the second light source, and the image acquisition device are at least partially housed in a housing, and the housing is attached to the robot arm.

5. The image acquisition system of any one of claims 1 to 4, wherein the first light source comprises a static structured light pattern.

6. The image acquisition system of any one of claims 1 to 5, wherein the image acquisition system is attached to an end effector of a robotic arm.

7. the end effector is configured to move from a first position to a second position; The image acquisition system of claim 6 , wherein the first position is configured for a tool to contact the surface and the second position is configured for the image acquisition system to acquire an image.

8. The image acquisition system of claim 7 , further comprising a controller that generates a trajectory for the tool based on the acquired images.

9. 1. An image acquisition system for a reflective surface, comprising: a mount configured to couple the image acquisition system to a robotic arm; a light source that illuminates the reflective surface and is connected to the mounting portion; an image capture device arranged to capture an image of the illuminated reflective surface; An image acquisition system comprising:

10. 10. The image acquisition system of claim 9, wherein the light source is a high intensity line light source and the light source is positioned such that a specular reflection is received in the field of view.

11. The image acquisition system of claim 10 , wherein a diffuser is positioned in front of the light source.

12. The image acquisition system of claim 10 , wherein the diffuser is removable.

13. The image acquisition system according to any one of claims 9 to 12, wherein a controller is configured to control the robot arm so that the image acquisition device is positioned at a position that coincides with a normal vector from the reflecting surface.

14. The image acquisition system according to any one of claims 9 to 13, further comprising a second light source, the second light source being an area light source.

15. The image acquisition system of claim 14 , wherein a structured light pattern is disposed between the second light source and the reflective surface.

16. The image acquisition system of claim 9 , wherein the image acquisition device images the surface through an opening in the light source.

17. 1. A robotic surface modification system comprising: A robotic arm, a force control unit connected to the robot arm; an end effector connected to the force control unit; an image acquisition system configured to acquire an image of the surface, the image acquisition system being attached to the end effector; A robotic surface modification system comprising:

18. 20. The robotic system of claim 17, wherein the image acquisition system includes a housing attached to the end effector, the housing at least partially housing an image acquisition device and a light source.

19. The robotic system of claim 18 , wherein the light source is positioned such that specular reflection from an area light source is received by the image capture device.

20. The robotic system of claim 19 , wherein the light source comprises a structured light pattern.

21. 20. The robotic system of claim 18, wherein the light source is a first light source and further comprising a second light source different from the first light source.

22. 22. The robotic system of claim 21, wherein the second light source is a line light source and the second light source is positioned to receive diffuse illumination by the image capture device.

23. 23. The robotic system of claim 17, further comprising a surface modification tool configured to modify a surface, the modification comprising adding or removing material from the surface, and configured to modify the surface when the surface modification tool is aligned with the force control unit.

24. The system of any one of claims 17 to 23, further comprising an image analyser for processing the acquired images.

25. The system of claim 24 , wherein a controller selects a trajectory for the surface modification tool relative to a surface modification command on the surface based on the analyzed image.

26. 25. The system of claim 24, wherein the image analyzer is configured to quantify the amount of haze on the surface.

27. The system of claim 24 , wherein the processing includes detecting a curvature of the surface based on the acquired image.

28. 1. A location verification system comprising: a topographical information acquisition unit that acquires known topographical information of the surface; a position verification unit that compares the detected curvature with the known topographical information; 28. The system of claim 27, further comprising a location verification system comprising:

29. 1. A method for modifying a surface, comprising: imaging the surface a first time using an image acquisition system attached to a surface modification system; characterizing the surface based on images acquired by the image acquisition system; switching the relative positions of the image acquisition system and a tool of the surface modification system; performing a surface modification process using the tool based on the characterization; and imaging the surface a second time using the image acquisition system; and evaluating the surface modification based on the second acquired image; and A method for modifying a surface, comprising:

30. 30. The method of claim 29, wherein the image capture system includes an image capture device, a first light source, and a second light source, the first light source being used in a first imaging step and the second light source being used in a second imaging step, and the first and second light sources being different light sources.

31. 31. The method of claim 30, wherein in the first imaging step, the first light source is positioned at an angle relative to the image capture device, and the image capture device is configured to capture specular reflections.

32. 31. The method of claim 30, wherein in the second imaging step, the second light source is positioned at an angle relative to the image capture device, and the image capture device is configured to capture diffuse illumination.

33. The method of any one of claims 29 to 32, wherein said characterizing comprises quantifying the haze of the surface.

34. 1. A method for positioning an image acquisition system over a surface, comprising: capturing an image of the surface using an image capture device; generating topographical information of the surface using a surface topography generator based on the acquired images; obtaining known topographical information of the surface; comparing the generated terrain information with the known terrain information; generating a deviation indicator based on a difference between the generated terrain information and the known terrain information; A method for positioning an image acquisition system, comprising:

35. obtaining a surface modification trajectory template for the surface modification system; the surface modification trajectory template includes a starting point, a path, and a region; 35. The method of claim 34, further comprising generating a new surface modification trajectory that changes at least one of the starting point, path, or region based on the deviation metric.

36. 36. The method of claim 34 or 35, wherein generating the surface topography information comprises fitting a bounding rectangle to a portion of the acquired image, the bounding rectangle comprising an area from which reflections of a light source on the surface can be acquired.

37. 37. The method of claim 36, wherein generating the surface topography information includes generating a rotation angle based on the bounding rectangle.

38. 37. The method of claim 36, wherein generating the surface topography information comprises generating an area of ​​the bounding rectangle.

Images